Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

Nothing happens silently, parts 2 and 3: degraded results, and the remaining fallbacks, skips and warnings (#350)

Shogo Kajiyama206e4d324fed
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4ce6bf6..349930a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,26 @@
## Unreleased
-(nothing yet)
+### Added
+
+- **The remaining silent fallbacks, skips and warnings are recorded.** Each still prints its stderr line and is now also a `diagnostics` entry, and `contract --json` lists every code with its bucket under `json_output.diagnostic_codes`: an odd-sized frame scaled to even before encoding (`odd_dimensions_scaled_even`), subtitle/data streams dropped on a retry with the first attempt's error (`non_av_streams_dropped`, beside `dropped_non_av_streams`), a `--dry-run` measurement skipped for an input an earlier stage would write (`measurement_input_pending`), `ffmpeg -encoders`/`-bsfs` that could not be read (`ffmpeg_list_unreadable`), an invalid or negative `FFMPEG_SKILL_TIMEOUT` (`timeout_env_invalid`), the libaom fallback for a missing SVT-AV1 (`av1_encoder_libaom`), a CRF capped at the encoder's maximum (`crf_clamped`), a timeline export that does not read back (`timeline_parse_failed`, with the reason), and every tool's former stderr-only `warning:` line (emoji fallbacks, silent, short or duplicate inputs, memory-heavy jobs, `--strip-dovi` without Dolby Vision, a numbering gap in `sequence.py`, `render.py`'s frame-from-preset and default `--normalize`).
+- **`font_used` on the tools that draw text.** `overlay.py --text`, `graphics.py`, `caption.py` (burn) and `grid.py --label auto` report `{asked, file, family, how, substituted}`; a family that is not installed and was drawn in another (fontconfig's answer, or Arial on Windows) is a `font_substituted` fallback.
+- **Counts for what was skipped.** `cropdetect.py` `windows` / `windows_failed`; `sync.py --fix-drift` `drift_skipped_reason` (`too_short` / `low_confidence`); `caption.py --ass` `caption.ass_shifted` / `ass_unshifted` / `ass_malformed`; `silence.py --filler` `filler.matched_untimed` / `matched_too_long`.
+- **`--codec av1`/`prores` refusals say why.** `error.reason: encoder_missing` (`kind: missing_tool`, as before) when the build's encoder list lacks the encoder; `encoder_list_unreadable` (`kind: ffmpeg`) when the list could not be read, instead of claiming the build lacks it. `--hw`'s CPU note says the same.
+- **The MCP server answers a line that is not JSON** with a JSON-RPC parse error (`-32700`, `id: null`) instead of dropping it.
+
+### Fixed
+
+- **A degraded or partial result is no longer reported as the asked-for one** (nothing happens silently, part 2). Additive fields, plus the `batch.py` row `error` noted below:
+ - `render.py`'s `verified` is the conjunction of every stage's. A stage's own verification steps (beyond its intermediate's probe) are in the result's `verification`, tagged with `stage`, and its notes in `notes`, prefixed with its tool: an export that missed its platform loudness used to leave the render `verified: true`. An executed plan and a pack carry their children's the same way, and a `--cache` hit replays what the cached stage verified.
+ - `render.py`'s `"check": {"content": true}` without a platform runs `check.py --content` (it was skipped without a word; a plan's check step already ran it). A check not run under `--dry-run` is a `check_dry_run` skip, and check.py output that does not parse is a failure document (`error.reason: check_output_unparsed`) instead of a bare string.
+ - `render.py --cache` says why each stage missed (`cache.miss_reasons`: `not_cached`, `sidecar_missing`, `artifact_missing`, `sidecar_corrupt`, `artifact_truncated`, `no_output_path`); a damaged entry is a `cache_entry_damaged` warning and an audit line in the cache directory; a failed store, an input that cannot be fingerprinted and an unreadable ffprobe banner are recorded.
+ - `export.py` reports `trimmed: {from, to}` (and a note) when a platform maximum cut the clip. A loudness measurement that cannot be read is `loudness.measured: false` with a failing `{step: loudness, ok: false, reason: not_measured}` and a `loudness_not_measured` warning, instead of no loudness block and `verified: true`; after `--normalize`, the measurement from before the pass is no longer reused.
+ - `multicam.py`'s `cuts` name the camera actually rendered (camera 0 where the asked-for camera had not started), with a `camera_not_started` fallback naming the one asked for.
+ - `report.py` lists each section it built as a verification step (`loudness`, `compliance` with `passed`, `contact_sheet`); one it could not build is `ok: false` with the child's reason and a `report_section_missing` warning, so the report is no longer `verified: true` with a section missing.
+ - `batch.py`: a failed row's `error` is the failing step's own `{step, tool, kind, message, code, reason, details, hint, exit_code}` (it was absent, or a string such as `"1"` for a worker that exited); rows add `verified` and `unverified` (a completed but unverified step stays `ok`, as design-decisions decides), and the batch's `verification` has one `item` step per row. Ctrl-C reports a row for every input (`interrupted: true`, or `skipped: "interrupted"`) and a `total` that counts them all. A cache hit is served only while its output's size and mtime are the ones the batch wrote (`cache_miss` on the row, a warning), so an entry written by 2.10.11 or earlier runs once more; a cache file that does not parse is a `batch_cache_reset` warning and an audit line. A recipe error that is the same for every item is refused before any item runs.
+ - `verify.py` runs each step with `--json` and keeps the tool's own `kind`, `reason`, `message` and `exit_code` instead of the last stderr line or a fixed string; steps that do not apply are listed under each file's `skipped` with a reason.
+ - `export.py`'s and `multicam.py`'s last stderr-only `warning:` lines are records too (`colour_not_converted`, `output_audio_silent`, `loudness_outside_spec`, `sync_low_confidence`), so no tool prints a warning that `--json` does not carry.
## 2.11.0
diff --git a/README.md b/README.md
index 2c403f3..2e050b7 100644
--- a/README.md
+++ b/README.md
@@ -389,7 +389,7 @@ The short list for humans. The agent-facing version, with the reasoning, is the
- **Silence detection finds nothing?** The default threshold is −35 dBFS. The tool prints a hint with the track's measured level; raise the threshold (`silence.py --threshold -25`) or shorten `--min-silence`.
- **Sync results carry a confidence.** Below 0.3, or an offset near the edge of the analysis window, is probably wrong: enlarge `--analyze-seconds` or find a clap. Recordings over ten minutes from separate devices need `sync.py --fix-drift`.
- **Outputs are never overwritten silently.** An existing output path is refused (`kind: input`, `error.reason: output_exists`, before anything runs); pass `--overwrite` where a replacement is intended.
-- **Nothing happens silently.** A refusal names its `error.reason` and `error.details`; a stale output lock taken over, a run without a lock and a plan input nobody could fingerprint are recorded in `diagnostics` and on stderr. A refused plan and a lock takeover also append a line to an audit log (`<plan>.audit.jsonl`, or `render.py --audit-log FILE`).
+- **Nothing happens silently.** A refusal names its `error.reason` and `error.details`; a stale output lock taken over, a run without a lock, a plan input nobody could fingerprint, a fallback (an odd-sized frame scaled to even, a substituted font, libaom for SVT-AV1), a skipped step and every warning are recorded in `diagnostics` and on stderr (codes: `json_output.diagnostic_codes`); tools that draw text name the font they drew with (`font_used`). A refused plan and a lock takeover also append a line to an audit log (`<plan>.audit.jsonl`, or `render.py --audit-log FILE`). A degraded or partial result says so: `render.py`'s `verified` is every stage's, an unmeasured loudness or a report section that could not be built is a failing verification step, `export.py` reports `trimmed`, `multicam.py`'s `cuts` name the camera rendered, and `batch.py` rows carry the failing step's `error`, `verified`, and a row for every input after Ctrl-C.
- **Long chains belong in a plan.** Three hand-chained re-encodes lose quality and are hard to change; `render.py` runs the whole edit from one JSON file, and `--dry-run` shows every ffmpeg command before anything is written.
## FAQ
diff --git a/docs/contract.md b/docs/contract.md
index e4f20a8..7816b29 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -376,7 +376,9 @@ limit (`check.py` reports it instead). `_platforms.resolve()` is the one alias m
New in the same release, all additive: `export.py --preset tiktok|shorts|linkedin|facebook`
(real presets, not aliases of `reels`/`youtube`), `--preset youtube-hdr` (HEVC Main10 keeping
the source's HDR tags; `kind: input` on an SDR source) and `--preset youtube-av1`
-(`kind: missing_tool` when the build has neither SVT-AV1 nor libaom); `caption.py --platform`
+(`kind: missing_tool`, `error.reason: encoder_missing`, when the build has neither SVT-AV1 nor
+libaom; `kind: ffmpeg`, `encoder_list_unreadable`, when its encoder list could not be read at all,
+which says nothing about the build); `caption.py --platform`
and `graphics.py --platform` / `--margin` and `overlay.py --platform` (margins from the safe
zone, an explicit `--margin`/`--position` wins); `look.py --safe NAME`; `fit.py --fit blur`; `report.py --pack`;
`graphics.py --template sticker|hook|meme`; and `render.py --template NAME INPUT`
@@ -546,7 +548,7 @@ the script, `missing` means fontconfig knows none, `unknown` means there is no w
ask (no `fc-list` on PATH, or it failed). `caption.py`, `graphics.py` and `overlay.py --text` resolve
a font by script automatically and fail with `kind: input` rather than render boxes **only for
`missing`**: `unknown` is not `missing` here any more than anywhere else in this document — the job
-runs with the font as given and one info line says the coverage could not be verified. So a
+runs with the font as given and a `font_coverage_unverified` warning says the coverage could not be verified. So a
`missing` script here is a job that will not run until a font is installed — but, like the default
font, it never affects `ok` or any tool's
`usable` (the tool works, this machine just has no glyphs for that language). The plain-text
@@ -634,7 +636,7 @@ Per-tool keys added in 1.17, all additive:
| `filler`, `removed_seconds_total` | `silence.py --filler` | `{lang, source, engine, words, removed, removed_count, removed_seconds, removed_words, word_timings, list, warnings}`. The existing `removed_seconds` is unchanged in name and meaning — the seconds of *silence* removed, which is what it has always held — and `removed_seconds_total` is the additive sibling covering silence plus filler |
| `jobs`, `jobs_requested`, `wall_seconds`, `item_seconds_total`, `timed_out` | `batch.py` | the parallelism actually applied and the number asked for, the batch's wall clock, the sum of the per-item times (so the speed-up can be quoted), and whether the shared timeout budget ran out. A timed-out item carries `"skipped": "timeout"` in its result row |
| `timeline` | `render.py --export-timeline FILE` | `{format, rate, duration, frames, clips, transition, transition_frames, music, markers, notes, not_exported}`: the written timeline's own numbers (`rate` is exact, `"30000/1001"` for 29.97) and what the project asked for that an editor timeline cannot carry. Nothing is encoded (`commands: []`); `verified` means the file was written and reads back as its format (`verification` step `parse`) -- not that an editor opened it |
-| `cache` | `render.py --cache` | `{dir, ffmpeg, hits, misses, saved_seconds, entries}`, plus `would_hit` under `--dry-run`. The ffmpeg build banner, the skill version, the contract version, the forwarded flags (`--fast`, `--codec`, …), the GPU setting (`FFMPEG_SKILL_HW` and whether `--hw`/`--no-hw` made it explicit) and the output's extension are all part of every key, so a cache is never reused across any of them — a `--fast` draft is never served to a run that did not ask for one |
+| `cache` | `render.py --cache` | `{dir, ffmpeg, hits, misses, miss_reasons, saved_seconds, entries}`, plus `would_hit` under `--dry-run`. `miss_reasons` (unreleased, additive): `[{stage, reason, ...}]`, `reason` one of `not_cached`, `sidecar_missing`, `artifact_missing`, `sidecar_corrupt`, `artifact_truncated`, `no_output_path`. The ffmpeg build banner, the skill version, the contract version, the forwarded flags (`--fast`, `--codec`, …), the GPU setting (`FFMPEG_SKILL_HW` and whether `--hw`/`--no-hw` made it explicit) and the output's extension are all part of every key, so a cache is never reused across any of them — a `--fast` draft is never served to a run that did not ask for one |
Per-tool keys added in 1.17.1, all additive:
@@ -676,6 +678,16 @@ Per-tool keys added after 2.2.2, all additive:
| `vfr_check` | `cut.py` (a video input that could still stream-copy) | the sampled frame timing, `{heuristic: true, measured, method: "sampled", guard, windows, deltas}`; `measured` is `sampled_cfr`, `vfr` or `inconclusive`. `--vfr-guard average` (the default) keeps the 2.x guard: a source whose nominal and average frame rates differ (`variable_frame_rate_suspected`) re-encodes as `--accurate` (reason `vfr`, stderr says "variable-frame-rate"), and only then is the timing sampled and reported, with a `notes` line naming `--vfr-guard sampled` when it measured `sampled_cfr`; the key is absent otherwise. `--vfr-guard sampled` samples before every copy and re-encodes on `vfr` (reason `vfr`) or `inconclusive` (fewer than 20 intervals; `vfr_inconclusive`); `--vfr-guard off` samples and keeps the copy, with a `notes` line. Absent for audio-only input or output, `--accurate`, `--codec` and a pending `--dry-run` input |
| `join_check`, `segment_end_snap_seconds` | `cut.py --segments` | `join_check`: a video stream-copy join measured by demuxing, `{packets, expected_packets, max_step_seconds, audio_offset_ms, audio_parts_checked, ok}`. `expected_packets` counts the source's video packets from each part's start keyframe to its end (or the end of the video); `max_step_seconds` is the largest presentation step, `null` when `vfr_check` measured anything but `sampled_cfr` (a `--vfr-guard off` join of variable timing: count only), and each step at a join must be the step its parts predict within 1 ms. `audio_offset_ms`: per part, its sound's shift minus its picture's in ms (+ = sound late), from audio packets matched by CRC32 to the source stream shaped like the joined audio (FFmpeg's default selection may copy one other than `a:0`; an MPEG-TS source's AAC is hashed without its ADTS headers); `null` for a part with no unique match (digital silence, a periodic tone), and as a whole when the source has no audio or the video check failed. `audio_parts_checked` counts the measured parts. `ok` is the conjunction (count, steps, every measured part within 5 ms); `ok: false` means the join was re-cut (`concat_fallback`) and this is the failed measurement. `null` for a single segment, audio, `--dry-run`, any re-encode forced for every segment, a segment that re-encoded, or a join re-cut before a copy was tried (open GOPs, a later B-frame segment starting between keyframes, parts that differ, a concat error). `segment_end_snap_seconds`: per segment, the signed seconds its end moved to the keyframe the copied part stops at, the first at or after the requested end (`null` for a part that reaches the end of the video, a part of a source with no B-frames, cut to its requested end as in 2.5.1, or a Matroska/MPEG-TS part, which is not snapped); `null` as a whole unless the output is those copied parts |
+Per-tool keys added after 2.10.11 ("nothing happens silently"), all additive:
+
+| key | tool | what it holds |
+|---|---|---|
+| `font_used` | `overlay.py --text`, `graphics.py`, `caption.py` burn mode, `grid.py --label auto` | the font the text was drawn with: `{asked, file, family, how, substituted, via?, fonts_dir?}`. `how`: `font_file` (the caller's file), `script` (picked to cover the text's script), `resolved` (the asked family resolved to a file by `fc-match` or the Windows Fonts folder; for libass, the file fontconfig picks for the family), `by_name` (no file resolved: ffmpeg or libass look the family up themselves, `file` null), `fonts_dir` (`caption.py`: the family is looked up in `--fonts-dir` first), `ass_styles` (`caption.py --ass`: the file's own styles name the fonts). `substituted: true` -- the file is not the asked family (a Windows machine without it gets a CJK system font or Arial; fontconfig answers every name) -- is also a `font_substituted` fallback |
+| `windows`, `windows_failed` | `cropdetect.py` | the sampled windows and how many ffmpeg could not decode (skipped; a `cropdetect_windows_failed` warning when above 0). Every window failing stays a `kind: ffmpeg` failure |
+| `drift_skipped_reason` | `sync.py --fix-drift` (top level for one SOURCE, and in each `sources` entry) | why the drift correction was not applied: `too_short` (no tail window far enough from the head) or `low_confidence` (the tail did not correlate); `null` when it was. Also a `drift_correction_skipped` entry under `diagnostics.skipped` |
+| `ass_shifted`, `ass_unshifted`, `ass_malformed` | `caption.py --ass` (inside `caption`) | with `--offset`: the Dialogue lines moved, and those whose timestamps could not be read (copied as they were); every run: Dialogue lines that could not be read (too few fields, an unreadable start/end) and are not counted as cues. A count above 0 is a warning |
+| `matched_untimed`, `matched_too_long` | `silence.py --filler` (inside `filler`) | listed words that were not cut: no usable timing in the transcript (also a `filler_words_untimed` warning), or longer than a filler (`FILLER_MAX_WORD`, a held vowel) |
+
`caption.py --transcribe` and `silence.py --filler --transcribe`: when a local speech engine ran
and produced no cue, the refusal is `"<engine> found no speech in <input>"` with `kind: input`,
`reason: "no_speech"` and `engine` in the failure document — distinct from the "no local
@@ -741,8 +753,9 @@ here claims otherwise until real exit-code/stderr sniffing exists to back that u
A refusal also carries `error.reason`, a stable snake_case code for *why*
(`contract --json` → `json_output.error_reasons` lists them: `output_exists`, `output_is_input`,
-`output_is_directory`, `output_dir_missing`, `output_dir_not_writable`, `output_locked`, and
-`render.py`'s `plan_*` reasons), and `error.details`, the facts the message is written from, so a
+`output_is_directory`, `output_dir_missing`, `output_dir_not_writable`, `output_locked`,
+`render.py`'s `plan_*` reasons, and `--codec av1`/`prores`'s `encoder_missing` and
+`encoder_list_unreadable`), and `error.details`, the facts the message is written from, so a
caller never parses the prose:
```
@@ -759,15 +772,53 @@ A tool that runs another (`render.py` stages and plans) forwards the child's `re
skipped -- is recorded as it happens (a stderr line too) and lands in the document as
`diagnostics: {warnings, fallbacks, skipped}`, each entry `{code, message, ...facts}`; it is
present on success and failure documents alike, only when one list is not empty, and an entry a
-`render.py` stage reported carries `stage`. Codes today: `stale_lock_taken_over` (fallbacks),
-`output_unlocked`, `lock_deadline_unset`, `audit_log_unwritable` and `plan_unbound_inputs`
-(warnings).
+`render.py` stage reported carries `stage`. `contract --json` → `json_output.diagnostic_codes`
+lists every code with its bucket and meaning (a test reads every `warn()`/`record_fallback()`/
+`record_skip()` call against it). Among them: the runner's `odd_dimensions_scaled_even` (an encode
+re-run with an even scale; the frame loses a row or column), `non_av_streams_dropped` (with the
+first attempt's error beside `dropped_non_av_streams`), `measurement_input_pending` (skipped: a
+dry-run measurement of an earlier stage's output), `ffmpeg_list_unreadable` (`ffmpeg -encoders` or
+`-bsfs` could not be read; what needed it names the read failure, not a missing encoder) and
+`timeout_env_invalid` (an `FFMPEG_SKILL_TIMEOUT` that is not a number, or is negative);
+`av1_encoder_libaom` and `crf_clamped`; `font_substituted` (with `font_used`); `drift_correction_skipped`
+(with `drift_skipped_reason`); `cropdetect_windows_failed` (with `windows_failed`);
+`ass_lines_unshifted` / `ass_lines_malformed`; `timeline_parse_failed`; and every `warning:` line a
+tool used to print to stderr only (emoji fallbacks, silent or short inputs, memory-heavy jobs,
+`--strip-dovi` without Dolby Vision, `render.py`'s frame-from-preset and `--normalize` defaults).
+`render.py`'s plan, check and cache records (`check_dry_run`, `check_output_unparsed`, `cache_entry_damaged`, ...),
+`export.py`'s `loudness_not_measured`, `loudness_outside_spec` and `colour_not_converted`, `multicam.py`'s
+`camera_not_started`, `report.py`'s `report_section_missing` and `batch.py`'s cache records
+(`batch_cache_reset`, `batch_cache_output_changed`, ...) are listed there too. Every one of them
+still prints its stderr line; the record is additive.
+
+**A result that is not the one asked for says so.** `verified` is the conjunction of every
+`verification` step, and a tool that runs others counts theirs: `render.py` lists each stage's own
+steps beyond its intermediate's probe, tagged with `stage` (a cached stage's as the cache recorded
+them), and its `notes` carry each stage's notes prefixed with the stage's tool, so an export that
+missed its platform loudness leaves the render `verified: false`. A step that could not be
+measured is a failing step with a `reason`, never an absent one: `export.py`'s unreadable loudness
+is `{step: loudness, ok: false, reason: not_measured}` with `loudness.measured: false`, and
+`report.py` lists each section it built (`loudness`, `compliance` with `passed`,
+`contact_sheet`), one it could not with `ok: false` and the child's `reason`/`message`. A
+deliberate change of result is a field: `export.py`'s `trimmed: {from, to}` when a platform
+maximum cut the clip, `multicam.py`'s `cuts` naming the camera rendered (camera 0 where the one
+asked for had not started, with a `camera_not_started` fallback). `batch.py` keeps `ok` as "every
+step completed" (a spec miss is `completed` + `verified: false`, not a failure) and adds `verified`
+and `unverified: [{step, tool, failed}]` to each row, one `item` verification step per row, a
+failed row's `error: {step, tool, kind, message, code, reason, details, hint, exit_code}` from the
+failing step's own document, and `cache_miss` when a cache entry was not served because its output
+changed or could not be checked; an interrupted batch has a row for every input
+(`interrupted: true`, or `skipped: "interrupted"` when it had not started) and `total` counts them
+all. `verify.py`'s steps keep the tool's `kind`, `reason`, `message` and `exit_code`, and the steps
+that do not apply are listed under each file's `skipped` with a reason.
**Audit log.** Events a later reader must be able to reconstruct are appended as JSON lines,
`{ts, tool, event, pid, plan_sha256, details}` (`plan_sha256` on plan events): `plan_rejected`,
-`lock_takeover`, `output_unlocked`, and `output_is_input_refused` when a log was named. The file
+`lock_takeover`, `output_unlocked`, `output_is_input_refused` when a log was named,
+`cache_entry_damaged` (`render.py --cache`) and `batch_cache_reset`. The file
is `render.py --audit-log FILE` when given; otherwise a refused plan's `<plan>.audit.jsonl` next
-to the plan, and a lock event's hidden `.<output>.ffskill-audit.jsonl` beside the output. Nothing
+to the plan, a lock event's hidden `.<output>.ffskill-audit.jsonl` beside the output, and a cache
+event's `.ffskill-audit.jsonl` in the cache directory or the batch's `output_dir`. Nothing
is appended under `--dry-run`. A record that cannot be written is a `diagnostics.warnings` entry
(`audit_log_unwritable`) and never fails or blocks the run; the refusal's `details.audit_log`
says `{path, written}` either way.
@@ -844,7 +895,12 @@ returns one user-role text message naming the actual command lines to run and wh
afterward -- the same facts SKILL.md's own request table states, addressed by workflow instead of
by request phrasing. A missing required argument is a JSON-RPC error (`prompts/get` never guesses
a path or a platform); an unknown prompt name is a JSON-RPC error too, distinct from "method not
-found" since `prompts/get` itself is a real method. These are recipes, not new tool calls: every
+found" since `prompts/get` itself is a real method.
+
+A line on stdin that is not JSON is answered with a JSON-RPC parse error (`-32700`, `id: null`)
+rather than dropped, so a client whose request was mangled does not wait for an answer that
+never comes; a JSON value that is not an object, and a notification, still get no response.
+An unknown method is `-32601`, any other handler error `-32000`. These are recipes, not new tool calls: every
command line a prompt names is one `tools/call` (or the CLI) can already run.
## Consuming the contract from an agent
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index 594769f..a80f133 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -34,8 +34,11 @@ exists. When a decision changes, edit the entry in the same PR.
`_contract.DRY_RUN_ANALYSIS`. Tests: `test_dry_run_never_runs_ffmpeg_and_writes_nothing`,
`test_dry_run_plans_rest_on_real_measurements`.
- **When a measured input is an intermediate an earlier dry-run stage would have written**
- (render/batch plans), the measurement is skipped with a note rather than failing the plan.
- Code: `_common.dry_run_input_pending()`.
+ (render/batch plans), the measurement is skipped rather than failing the plan, and the skip is
+ recorded: a `diagnostics.skipped` entry (`measurement_input_pending`, once per path) beside the
+ stderr line, which until the "nothing happens silently" work was the only "note" there was.
+ Code: `_common.dry_run_input_pending()`. Test:
+ `test_a_measurement_skipped_for_a_pending_input_reaches_the_document`.
- **`join.py --dry-run` plans on an input that does not exist yet, whatever `--on-missing`
says.** The file is taken for an earlier step's output (render's cut clips, a TTS line): it
stays in the planned command and is named under `pending` and in `notes`, never under
@@ -1037,7 +1040,11 @@ not a new file format this tool would have to maintain.
left to add); an explicit `--no-hw` (or `render.py`/`batch.py --no-hw`) is the caller's choice of
the CPU and is not overridden, ffmpeg then refuses it as before; `FFMPEG_SKILL_HW=0` is no flag
and does not stop it; AV1 and ProRes are out of scope (their own missing-encoder refusals stand).
- An unreadable encoder list (empty) proves nothing is missing and changes nothing. No CPU line is
+ An unreadable encoder list (empty) proves nothing is missing, so here it moves no encode onto
+ VideoToolbox. (Through 2.10.11 this sentence said it "changes nothing", which held only for this
+ function: the same empty set made `--codec av1`/`prores` refuse with "this build has neither" and
+ `--hw`'s note say "this ffmpeg build has no h264_videotoolbox". The read failure is now kept and
+ named -- see "nothing happens silently", the encoder list.) No CPU line is
recorded for it, so a job VideoToolbox refuses is a plain `kind: ffmpeg` failure naming
`h264_videotoolbox`, not a retry that could not run. `doctor` agrees: a missing
`encoder:libx264`/`libx265` whose VideoToolbox encoder is in the build, on a machine that passes the
@@ -1821,3 +1828,194 @@ under it; the rest of the silent paths are converted onto the same infrastructur
`test_with_no_time_limit_a_live_holders_lock_is_never_taken_over`,
`test_a_lock_an_older_version_wrote_is_taken_over_after_an_hour_as_before`,
`test_running_without_a_lock_is_a_recorded_warning`.
+
+### Part 2: a degraded or partial result is never reported as the asked-for one
+
+- **A stage's verdict is the parent's.** `absorb_stage()` now also carries a stage's `notes`
+ (prefixed with its tool: `export: loudness ... is outside x's ...`) and, for a stage that
+ completed in a real run, its verification steps tagged with `stage`; `emit()` adds them to the
+ parent's own, so `render.py`'s `verified` is the conjunction of every stage's. Through 2.10 an
+ export that missed its loudness (`completed`, `verified: false`) made a render that said
+ `verified: true`. Carried: every step except a passing `probe`/`exists` of the stage's
+ intermediate (a completed stage passed them by definition, and they would only lengthen the
+ list); a stage that said `verified: false` without a failing step adds `{step: verified, stage,
+ ok: false}`. The same carry applies to an executed plan (its tool's steps used to be copied by
+ hand, untagged) and to a pack's renders. The entry "`verified` is what the tool measured itself"
+ above still holds: render's stages *are* what it ran. Code: `_common.emit.absorb_stage()`,
+ `emit()`. Test: `test_render_verified_is_every_stages_and_carries_their_notes`.
+- **A cache hit replays what the stage verified.** A `render.py --cache` sidecar keeps the stage's
+ `verified`, `verification`, `notes` and `diagnostics`; a hit feeds them through
+ `absorb_stage()` as a live stage would, so a cached unverified export stays unverified.
+ `references/scripts.md` said "Any mismatch is a silent miss" (and `cache_lookup()` returned
+ False "silently, because a miss is not an error"); a miss is still not an error, but every
+ miss is now in `cache.miss_reasons` (`not_cached`, `sidecar_missing`, `artifact_missing`,
+ `sidecar_corrupt`, `artifact_truncated`, `no_output_path`); the two that mean something damaged
+ a file this tool wrote are also a `cache_entry_damaged` warning and an audit line in the cache
+ directory's `.ffskill-audit.jsonl` (the cache directory is the place the event names). A store
+ that fails is a `cache_store_failed` warning, an input that cannot be fingerprinted a
+ `cache_input_unreadable` warning (it keys as `"missing"`), and an unreadable ffprobe banner a
+ `cache_ffmpeg_banner_unreadable` fallback: the sentence "Unreadable falls back to the parsed
+ pair" in "The ffmpeg version is inside the render cache key" still describes the key, and the
+ fallback is now said. Code: `render.cache_lookup()`, `cache_store()`. Tests:
+ `test_render_cache_says_why_a_stage_missed_and_replays_what_a_hit_verified`,
+ `test_render_cache_store_failure_banner_fallback_and_unreadable_input_are_recorded`.
+- **`"check": {"content": true}` without a platform runs.** A plan's check step already honoured
+ `content` alone (`render.execute_plan()`), and `check.py --content` without `--platform` is a
+ valid call (youtube assumed, judgement rows WARN), so the project stage now follows the same rule
+ rather than refusing a key the schema accepts; a failure is reported as a platform check's is
+ (`kind: verification`, "the content check failed"). A check stage the run does not perform is
+ recorded: `check_dry_run` under `--dry-run` (a plan's check steps too), `check_nothing_to_check`
+ for a stage with neither key. check.py output that does not parse is a failure document
+ (`status: failed`, `error.reason: check_output_unparsed`, `exit_code`, `stderr_tail`) where a
+ bare string stood. Code: `render.run_check()`, `render.main()`. Tests:
+ `test_render_runs_a_content_only_check_and_records_a_dry_run_skip`,
+ `test_check_output_that_does_not_parse_is_a_structured_failure`.
+- **`export.py` says what it cut and what it could not measure.** A trim to the platform maximum
+ is `trimmed: {from, to}` and a `notes` line (it is the preset's promise, so not a warning). A
+ loudness measurement that did not parse is `loudness: {lufs: null, tp: null, ok: null,
+ measured: false, reason: not_measured, target_*}`, a failing `{step: loudness, ok: false,
+ reason: not_measured}` and a `loudness_not_measured` warning -- check.py's WARN row for the same
+ case is the model; leaving the block out read as "nothing to check" and kept `verified: true`.
+ After `--normalize` a re-measurement that fails is the same record: the earlier measurement no
+ longer describes the file, and reusing it (2.10) reported the old level as the new one. A
+ platform preset on a source without audio records a `loudness_no_audio` skip. Code:
+ `export._loudness_unmeasured()`. Tests: `test_export_says_it_trimmed_to_the_platform_maximum`,
+ `test_export_an_unmeasured_loudness_is_a_failing_step_not_an_absence`.
+- **`multicam.py`'s `cuts` name the camera on screen.** A range whose camera has not started plays
+ camera 0 (unchanged); `cuts` now says 0 for it, as `--write-project` already did, and a
+ `camera_not_started` fallback names the camera asked for. Code: `multicam.main()`. Test:
+ `test_multicam_names_the_camera_it_rendered_not_the_one_asked_for`.
+- **`report.py` accounts for every section.** Each section it tries to build -- loudness of each
+ file with audio, the `--platform` compliance table, each contact sheet -- is a verification step;
+ one that could not be built is `ok: false` with the child's `reason`/`kind`/`message` and a
+ `report_section_missing` warning, so `verified` is false and `check: null` no longer sits beside
+ `verified: true`. The compliance step's `ok` means "the table is in the report"; what the table
+ says about the file is `passed`. Under `--dry-run` a contact sheet is a `report_section_dry_run`
+ skip (look.py draws nothing in a dry run). `--no-sheets` is the caller's request and is not
+ recorded. Test: `test_report_records_a_section_it_could_not_build`.
+- **`batch.py`: `ok` stays "every step completed"; `verified` is new beside it.** The entry
+ "`verified` is what the tool measured itself" decides that a spec miss is `completed` +
+ `verified: false`, not a failure, so a step that completed unverified is not counted as failed;
+ the row says `verified: false` and `unverified: [{step, tool, failed}]`, the batch's own
+ `verification` has one `item` step per row (its `verified` was always false before: it had no
+ steps), and a note names the items. A failed row's `error` is the failing step's own
+ `{step, tool, kind, message, code, reason, details, hint, exit_code}`; the old string `error`
+ (only ever a worker's exception text, or `"1"` -- `str(SystemExit)`) is replaced by that object,
+ with `exception` for a worker that raised. A recipe error that is the same for every item (a step
+ script outside `scripts/`, no steps) is refused once before any item runs. Code:
+ `batch.process()`, `step_error()`. Tests:
+ `test_batch_a_failed_item_says_which_step_failed_and_why`,
+ `test_batch_counts_a_completed_but_unverified_step_as_ok_and_says_it_is_unverified`,
+ `test_batch_a_worker_that_raised_names_its_error_not_its_exit_code`.
+- **An interrupted batch reports every item.** The shared signal handler exited with "nothing was
+ written" and, inside an item, surfaced as that item's failure while the batch went on. batch.py
+ installs its own SIGINT/SIGTERM handler: it stops the running steps (`stop_children()`, the
+ shared handler's first half) and raises KeyboardInterrupt, and the table then has one row per
+ input -- finished, `interrupted: true` (stopped while running), or `skipped: "interrupted"` (not
+ started) -- so `total` is the folder's count; exit 130 (143 for SIGTERM), `kind: interrupted`. A
+ `--watch` loop stops after an interrupted pass. Test:
+ `test_batch_interrupted_reports_every_item_and_the_right_total`.
+- **A batch cache hit is served only while its output is the file the batch wrote.** An entry now
+ keeps the output's `{size, mtime_ns}` (a stat, no read: a multi-GB delivery is not re-hashed per
+ run); a hit whose output changed is a miss (`cache_miss.reason: output_changed`, a
+ `batch_cache_output_changed` warning). An entry written by 2.10.11 or earlier has no identity and is not
+ served either (`entry_unverifiable`): one re-run per item is the cost of not serving a replaced
+ file as "cached ok". The GPU entry's "older caches still hit" is about the key and still holds; it
+ is this identity check that re-runs them once. A cache file that does not parse is a
+ `batch_cache_reset` warning plus an audit line in the output folder's `.ffskill-audit.jsonl`,
+ and an unreadable `recipe.project` (its content is in the key) a `recipe_project_unreadable`
+ warning. Code: `batch.cached_output_changed()`, `recipe_key()`. Tests:
+ `test_batch_serves_a_cached_output_only_while_it_is_unchanged_and_records_a_reset_cache`,
+ `test_batch_recipe_key_records_an_unreadable_project`.
+- **`verify.py` keeps each tool's own failure.** Steps run with `--json`; a failed step keeps the
+ tool's `kind`, `reason`, `message`, `details` and `exit_code` (`step_timeout`,
+ `step_output_unparsed` where the tool printed none), and the probe of an unreadable file keeps
+ probe's own message instead of "ffprobe failed". `error` stays the one-line table cell. A step
+ that does not apply is listed under the file's `skipped` (`quick`, `no_video`, `no_audio`,
+ `not_hdr`, `not_multichannel`, `input_unreadable`) and as a SKIP row in the report; `total` and
+ `failed` count only steps that ran, as before. Test:
+ `test_verify_keeps_the_tools_own_failure_and_lists_the_steps_it_skipped`.
+
+Part 3 converts the remaining fallbacks, skips and stderr-only warnings onto that infrastructure.
+What was decided there:
+
+- **A record is kept once per run; its stderr line is printed each time.** `_record()` does not
+ append an entry identical to one already there, so a decision taken for every encode of a job
+ (the libaom fallback, a CRF clamp, an odd-dimension retry) is one record. Code:
+ `_common.emit._record()`. Test: `test_the_libaom_fallback_and_a_clamped_crf_are_recorded`.
+- **Every code is a contract enum, and no tool prints a `warning:` line with `info()`.**
+ `json_output.diagnostic_codes` lists each code with its bucket (warnings / fallbacks / skipped)
+ and meaning; a source test reads every `warn()`/`record_fallback()`/`record_skip()` call against
+ it in both directions, and another refuses `info("warning ...")` (or `print`/`stderr.write`) under
+ `scripts/`, with no exception: part 2 converted `export.py`'s four (`colour_not_converted`,
+ `output_audio_silent`, `loudness_outside_spec` twice) and `multicam.py`'s low-confidence line
+ (`sync_low_confidence`, as `sync.py` records it), and the allowance the test held for them is gone. Code: `_contract.DIAGNOSTIC_CODES`. Tests:
+ `test_every_diagnostic_code_is_in_the_contract_under_its_bucket`,
+ `test_no_stderr_only_warning_is_left_in_the_sources`.
+- **Which bucket.** A fallback is another route the run took and went on with, whose result
+ differs from the asked one: an odd-sized frame scaled to even (`odd_dimensions_scaled_even`; the
+ output is a row or column smaller), subtitle/data streams dropped on a retry
+ (`non_av_streams_dropped`, with the first attempt's last stderr lines; `dropped_non_av_streams`
+ stays the boolean it was), libaom for a missing SVT-AV1 (`av1_encoder_libaom`), a substituted
+ font (`font_substituted`). A skip is a step that would normally run and did not: a dry-run
+ measurement of a pending input, a `--fix-drift` correction (`drift_correction_skipped`). The rest
+ are warnings, including `render.py`'s two defaults (`frame_from_export_preset`,
+ `normalize_on_by_default`): they are choices made for the caller, and the stderr wording is
+ kept (tests read `frame: 1920x1080 from the youtube export preset`). Tests:
+ `test_an_odd_sized_source_scaled_to_even_is_a_recorded_fallback`,
+ `test_run_keeping_subtitles_records_the_first_attempts_error`,
+ `test_stderr_only_warnings_reach_the_document`.
+- **An encoder list that could not be read is named, not taken for a missing encoder.**
+ `ffmpeg_encoders()`/`ffmpeg_bsfs()` still return an empty set (every caller's membership test
+ keeps working), but keep why (`ffmpeg_list_error()`: did not run, timed out, exited non-zero,
+ listed nothing) and record it once (`ffmpeg_list_unreadable`). `--codec av1`/`prores` then refuse
+ under their own reason, `kind: ffmpeg`, `error.reason: encoder_list_unreadable` -- not
+ `missing_tool`, which a build that was never read cannot be said to be -- while a list that was
+ read and lacks the encoder is `missing_tool`/`encoder_missing` as before. They still refuse
+ rather than guess an encoder: the encode would run on the binary that could not even list its
+ encoders. `--hw`'s CPU note says the VideoToolbox encoder "could not be confirmed". Code:
+ `_common.runner._read_ff_list()`, `_common.decision._refuse_missing_encoder()`, `_vt_args()`.
+ Test: `test_an_unreadable_encoder_list_is_named_not_taken_for_a_missing_encoder`.
+- **The CRF clamp is reported where it bites, which is the HEVC HDR line.** It asks x265 for
+ CRF+2, so `--quality 50`/`51` is encoded at 51 (`crf_clamped`, with requested, asked and used);
+ the AV1 cap at 63 is checked the same way but `apply_common()` already refuses a higher
+ `--quality`. Code: `_common.decision._clamped_crf()`.
+- **`FFMPEG_SKILL_TIMEOUT` keeps its meaning and gains a record.** Not a number (or NaN): the
+ default 1800 s, as before; negative: 0, i.e. no limit, as before (`max(0, value)`); each is a
+ `timeout_env_invalid` warning, recorded only when no `--timeout` overrides the variable. Blank
+ is unset. Changing a negative value to the default was considered and left: it would change what
+ a running deployment's setting does in a patch. Code: `_common.runner._read_env_timeout()`.
+ Test: `test_an_invalid_ffmpeg_skill_timeout_is_a_recorded_warning`.
+- **`font_used` says which font file drew the text, as far as this process can know it.**
+ `how` is `font_file`, `script`, `resolved` (fc-match, or the Windows Fonts folder), `by_name` (no
+ file: ffmpeg or libass resolve the name themselves, so the file is not claimed), `fonts_dir` or
+ `ass_styles` (`caption.py`). A resolved file is a substitute when none of the families
+ fontconfig lists for it is the asked one, compared without case, spaces or punctuation and as a
+ prefix either way ("DejaVu Sans Bold" is DejaVu Sans); a generic alias (`sans-serif`, `mono`,
+ ...) is never a substitute. On Windows a family that is not in the Fonts folder is a substitute
+ (Arial, or a CJK system font for a CJK-looking request). For libass on Windows nothing is
+ resolved (`by_name`): libass may find the family itself, and claiming Arial there would be the
+ same unverified guess this work removes. The script-coverage lines in `script_font_for_text()`
+ became records too (`font_lacks_script`, `font_coverage_unverified`, `fonts_dir_lacks_script`).
+ Code: `_common.fonts.resolve_font()`, `font_choice()`, `record_font_choice()`. Test:
+ `test_the_font_drawn_with_is_reported_and_a_substitute_recorded`.
+- **Counted skips get a field as well as a record.** `cropdetect.py` `windows`/`windows_failed`;
+ `sync.py --fix-drift` `drift_skipped_reason` (`too_short` / `low_confidence` / null, also in each
+ `sources` entry); `caption.py --ass` `caption.ass_shifted`/`ass_unshifted` (with `--offset`) and
+ `ass_malformed`; `silence.py --filler` `filler.matched_untimed` and `matched_too_long`. The
+ filler reader used to drop an untimed word before `filler_spans()` saw it, so it now hands them
+ over to be counted; `word_timings` still counts only timed words. `sync.py` printed the
+ single-source low-confidence warning twice (once per source, once overall); it is one
+ `sync_low_confidence` record. Tests: `test_cropdetect_counts_the_windows_it_could_not_decode`,
+ `test_a_skipped_drift_correction_says_why`, `test_caption_counts_ass_lines_it_could_not_shift_or_read`,
+ `test_filler_words_with_no_timing_are_counted`.
+- **A timeline that does not read back keeps its reason in a warning, not in the step.** The
+ `parse` verification step stays `{step, ok}` (`render.py` is not touched here);
+ `timeline.parses()` records `timeline_parse_failed` with the exception or what was read instead.
+ A JSON value that is not an object is now a reason, not an `AttributeError`. Code:
+ `_common.timeline.parses()`, `parse_error()`. Test: `test_a_timeline_that_does_not_parse_keeps_the_reason`.
+- **The MCP server answers a line that is not JSON with `-32700`, id null.** A JSON value that is
+ not an object and a notification still get no response (`test_mcp_server_survives_a_non_object_json_line`
+ pins that): JSON-RPC answers an invalid request with `-32600`, but a scalar on stdin is no request
+ this server can name, and replying to one was not asked for. Code: `mcp/server.py` `main()`. Test:
+ `test_mcp_server_error_paths_raw_argv_and_call_helper`.
diff --git a/mcp/server.py b/mcp/server.py
index d01fddd..456ffe3 100755
--- a/mcp/server.py
+++ b/mcp/server.py
@@ -247,7 +247,13 @@ def main() -> int:
continue
try:
req = json.loads(line)
- except ValueError:
+ except ValueError as exc:
+ # JSON-RPC 2.0: a line that is not JSON is answered with -32700 and id null, not
+ # dropped -- a client whose request was mangled in transit otherwise waits for an
+ # answer that never comes, with nothing to say why
+ resp = {"jsonrpc": "2.0", "id": None, "error": {"code": -32700, "message": f"parse error: {exc}"}}
+ stdout.write((json.dumps(resp) + "\n").encode("utf-8"))
+ stdout.flush()
continue
# json.loads accepts any valid JSON value, not just an object -- a bare `42`, `null`,
# `true` or `[1,2]` line parses fine but isn't a JSON-RPC request. Without this guard,
diff --git a/references/scripts.md b/references/scripts.md
index 51683c5..6435c99 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -1,6 +1,6 @@
# Script reference
-Every script prints the same information with `--help`; this file exists so the agent can read several at once. All scripts accept `--dry-run`, `--json`, `--json-brief` (since 1.11.0: the same JSON result trimmed to `status`, `output`, `dry_run`, `verified`, a compact `summary` of the output probe -- duration_s, width, height, fps, vcodec, acodec, channels, and lufs when measured -- the tool's own keys, and the count of commands instead of the command lines; it implies `--json`, leaves `--json`'s own output untouched, and failures print the usual full failure document), `--fast`, `--progress`, `--timeout SECONDS`, `--overwrite`, `--plan FILE` (the dry run written as a plan document that `render.py FILE` executes later; see render.py), `-o OUT`; every re-encoding tool and `export.py` also take `--hw`/`--no-hw` (h264/hevc/prores on Apple VideoToolbox, Apple Silicon only, CRF mapped to `-q:v`; AV1 stays on the CPU; a job VideoToolbox itself refuses -- the encoder cannot open or its compression session fails -- is re-encoded on the CPU and reported in `hw.fallback`/`hw.notes`, any other failure is not retried; a GPU encode adds a `notes` line that its quality is not CRF-equivalent; `FFMPEG_SKILL_HW=1` makes it the default for every tool except `export.py`, whose delivery presets need `--hw` itself; `render.py`/`batch.py --hw`/`--no-hw` apply to every stage, export included; results name the `encoder` that ran); every editing tool that re-encodes (not `export.py`, whose preset decides the codec) also takes `--codec h264|hevc|av1|prores` (the encoder for the re-encode; default x264 for SDR, x265 Main10 for HDR, unchanged) and `--quality N` (CRF scale, default 18 -- 30 for `proxy.py`; up to 63 for av1; ignored by prores). 2.0 removed `--crf` from these tools (it was `--quality`'s alias); `export.py` keeps its own `--crf`, which is not an alias. Every tool refuses (`kind: input`, `error.reason: output_exists`, before anything runs) to replace an existing output unless `--overwrite` is given. Refusals carry `error.reason` and `error.details` (the facts behind the message; `contract --json` → `json_output.error_reasons`), and what a run decided on its own -- a stale output lock taken over, no lock possible, a plan input nobody fingerprinted -- is in `diagnostics: {warnings, fallbacks, skipped}` as well as on stderr. `--codec hevc` on SDR writes 8-bit BT.709 HEVC (`hvc1`), `av1` uses SVT-AV1 (libaom fallback), `prores` is 422 HQ and needs an explicit `-o NAME.mov` (or `.mkv`), `h264` refuses an HDR source (`kind: input`, run `color.py --to-sdr` first). `export.py` keeps choosing the codec from its preset and has neither `--codec` nor `--quality`; a `render.py` project cannot choose a codec either -- but `--dry-run` only guarantees nothing is written for writing tools: `probe` (read-only, `--dry-run` changes nothing) still runs ffprobe, `check`/`sync`/`multicam`/`scenes`/`cropdetect`/`report`/`silence`/`loudness`/`stabilize`/`join` still run their ffmpeg/ffprobe measurements (a dry-run plan rests on real numbers; they just don't write the final artifact), `overlay` measures a multi-line `--text`'s lines on FFmpeg 8.0, 8.0.1 and 8.1.0 (one drawtext on a generated frame), and `verify` accepts the flag but ignores it entirely. Exact per-tool semantics: `contract --json`'s `dry_run` field (or `docs/contract.md`).
+Every script prints the same information with `--help`; this file exists so the agent can read several at once. All scripts accept `--dry-run`, `--json`, `--json-brief` (since 1.11.0: the same JSON result trimmed to `status`, `output`, `dry_run`, `verified`, a compact `summary` of the output probe -- duration_s, width, height, fps, vcodec, acodec, channels, and lufs when measured -- the tool's own keys, and the count of commands instead of the command lines; it implies `--json`, leaves `--json`'s own output untouched, and failures print the usual full failure document), `--fast`, `--progress`, `--timeout SECONDS`, `--overwrite`, `--plan FILE` (the dry run written as a plan document that `render.py FILE` executes later; see render.py), `-o OUT`; every re-encoding tool and `export.py` also take `--hw`/`--no-hw` (h264/hevc/prores on Apple VideoToolbox, Apple Silicon only, CRF mapped to `-q:v`; AV1 stays on the CPU; a job VideoToolbox itself refuses -- the encoder cannot open or its compression session fails -- is re-encoded on the CPU and reported in `hw.fallback`/`hw.notes`, any other failure is not retried; a GPU encode adds a `notes` line that its quality is not CRF-equivalent; `FFMPEG_SKILL_HW=1` makes it the default for every tool except `export.py`, whose delivery presets need `--hw` itself; `render.py`/`batch.py --hw`/`--no-hw` apply to every stage, export included; results name the `encoder` that ran); every editing tool that re-encodes (not `export.py`, whose preset decides the codec) also takes `--codec h264|hevc|av1|prores` (the encoder for the re-encode; default x264 for SDR, x265 Main10 for HDR, unchanged) and `--quality N` (CRF scale, default 18 -- 30 for `proxy.py`; up to 63 for av1; ignored by prores). 2.0 removed `--crf` from these tools (it was `--quality`'s alias); `export.py` keeps its own `--crf`, which is not an alias. Every tool refuses (`kind: input`, `error.reason: output_exists`, before anything runs) to replace an existing output unless `--overwrite` is given. Refusals carry `error.reason` and `error.details` (the facts behind the message; `contract --json` → `json_output.error_reasons`), and what a run decided on its own -- a stale output lock taken over, no lock possible, a plan input nobody fingerprinted, an odd-sized frame scaled to even, subtitle streams dropped on a retry (with the first attempt's error), a font substituted, a measurement skipped under `--dry-run`, an encoder list that could not be read, an invalid `FFMPEG_SKILL_TIMEOUT`, and every `warning:` line -- is in `diagnostics: {warnings, fallbacks, skipped}` as well as on stderr (codes: `contract --json` → `json_output.diagnostic_codes`). `--codec av1`/`prores` with no encoder for it refuses `kind: missing_tool`, `error.reason: encoder_missing`; when the encoder list could not be read at all, `kind: ffmpeg`, `encoder_list_unreadable`. Tools that draw text (`overlay.py --text`, `graphics.py`, `caption.py`, `grid.py --label auto`) report `font_used: {asked, file, family, how, substituted}`. `--codec hevc` on SDR writes 8-bit BT.709 HEVC (`hvc1`), `av1` uses SVT-AV1 (libaom fallback), `prores` is 422 HQ and needs an explicit `-o NAME.mov` (or `.mkv`), `h264` refuses an HDR source (`kind: input`, run `color.py --to-sdr` first). `export.py` keeps choosing the codec from its preset and has neither `--codec` nor `--quality`; a `render.py` project cannot choose a codec either -- but `--dry-run` only guarantees nothing is written for writing tools: `probe` (read-only, `--dry-run` changes nothing) still runs ffprobe, `check`/`sync`/`multicam`/`scenes`/`cropdetect`/`report`/`silence`/`loudness`/`stabilize`/`join` still run their ffmpeg/ffprobe measurements (a dry-run plan rests on real numbers; they just don't write the final artifact), `overlay` measures a multi-line `--text`'s lines on FFmpeg 8.0, 8.0.1 and 8.1.0 (one drawtext on a generated frame), and `verify` accepts the flag but ignores it entirely. Exact per-tool semantics: `contract --json`'s `dry_run` field (or `docs/contract.md`).
## Time grammar (every time-taking flag, 1.9)
@@ -251,7 +251,10 @@ itself with no black-bar measurement involved. A rectangle that matches the
full source frame means no bars were found. Does not decide whether
removing detected bars is wanted -- genuine letterboxed content (a
scope-ratio film in a 16:9 frame) "detects" the same way as accidental
-bars; look at the frame before cropping it away.
+bars; look at the frame before cropping it away. `windows` / `windows_failed`
+say how many windows were sampled and how many ffmpeg could not decode (those
+are skipped, with a `cropdetect_windows_failed` warning; all of them failing is
+a `kind: ffmpeg` failure).
`--motion-centre` (1.18) reports the motion centroid once per second, sampled
across the same windows: `motion_centre: [{time, x, y, x_frac, y_frac, motion}]`,
@@ -595,6 +598,9 @@ per-word `start`/`end`) or `--transcribe` (the same local engine `caption.py`
uses, never required). There is no heuristic fallback — finding an "um" by
looking for short quiet blips would cut real speech, so the tool refuses
instead. `--filler-list` reports what would go and writes nothing.
+`filler.matched_untimed` counts listed words the transcript gives no usable
+timing for (not removed, and a `filler_words_untimed` warning), and
+`matched_too_long` those longer than a filler (a held vowel, kept).
Built-in lists, one per language:
@@ -748,6 +754,15 @@ step for the black/frozen/silence rows), and reports `plan`, `plan_sha256`, `too
`tool_result`, `check` and `unbound_inputs` (each also a `diagnostics.warnings`
entry: nothing checked those files). Show the plan to the user, get the yes, execute:
one round trip instead of re-deriving the command.
+Render's `verified` is every stage's: each stage's own verification steps (beyond
+its intermediate's probe) are in the result's `verification`, tagged with `stage`,
+and each stage's notes are in `notes`, prefixed with its tool -- an export that
+misses its platform loudness leaves the render `completed` with `verified: false`
+and the note naming the fix. `"check": {"platform": ...}`, `{"content": true}` or
+both run `check.py` after the last stage (content alone: the black/frozen/silence
+rows, youtube's spec assumed, as `check.py --content` without `--platform`); under
+`--dry-run` the check is a `check_dry_run` entry in `diagnostics.skipped`.
+
`"export": {"preset": "reels", "normalize": true}` forwards `export.py --normalize`
so the rendered file meets the platform's loudness without a separate pass. Since
1.9.0 it is on by default when the preset is a platform (`youtube|youtube4k|reels|x`)
@@ -808,7 +823,14 @@ artifact the old one produced. Each entry is `DIR/<key><ext>` plus a
`DIR/<key>.json` sidecar (stage, versions, creation time, size, seconds). A hit
hardlinks the artifact into the work directory, or copies it where the
filesystem will not link — never moves it, since the cache has to outlive the
-run's own cleanup. Any mismatch is a silent miss.
+run's own cleanup. A miss is work to do, not an error, but never a silent one:
+`cache.miss_reasons` says why each stage missed (`not_cached`, `sidecar_missing`,
+`artifact_missing`, `sidecar_corrupt`, `artifact_truncated`, `no_output_path`), and a
+damaged entry (the last two of those that name a file this tool wrote) is also a
+`cache_entry_damaged` warning and a line in `DIR/.ffskill-audit.jsonl`. A sidecar keeps
+the stage's `verified`, `verification`, `notes` and `diagnostics`, and a hit replays
+them, so a cached stage counts toward the render's `verified` as a live one does. A
+store that fails is a `cache_store_failed` warning (the run is unaffected).
A stage whose flags changed misses the cache, so editing the captions block (for
example to `"fit_size": "off"`) re-runs captions and everything after it and
@@ -817,7 +839,7 @@ serves the earlier stages from the cache.
`stages_done` is unchanged: a cached stage is still a stage that happened.
Nothing is written under `--dry-run`, which instead reports `cache.would_hit`.
Result: `cache.dir`, `cache.ffmpeg`, `cache.hits`, `cache.misses`,
-`cache.saved_seconds`, `cache.entries`. `--from STAGE` starts at that stage and
+`cache.miss_reasons`, `cache.saved_seconds`, `cache.entries`. `--from STAGE` starts at that stage and
takes every earlier one from the cache; without `--cache`, or when an earlier
stage is not there, it refuses (`kind: input`) rather than quietly re-encoding
what it promised to skip. An unwritable cache directory is `kind: output`.
@@ -1047,10 +1069,28 @@ above the cap is clamped with a note, not refused, and both `jobs` and
the table in file order when the cache is only partially warm — and each item's
log lines are buffered and flushed in file order, so the per-item table and the
log read exactly as a serial run's whatever order the encodes finish in.
-- **A failed item is a row, not a dead run**: a worker that raises becomes
- `ok: false` with its reason, and the summary and the table are still printed.
- Ctrl-C cancels what has not started, keeps what finished, and exits 130
- `kind: interrupted` with the partial table.
+- **A failed item is a row, not a dead run**: a step that fails makes its row
+ `ok: false` with `error: {step, tool, kind, message, code, reason, details,
+ hint, exit_code}` taken from that step's own JSON document (a worker that raised:
+ `exception` and its message), and the summary and the table are still printed.
+ A recipe error that is the same for every item (a step script outside
+ `scripts/`, no steps) is refused before any item runs.
+- **`ok` is "every step completed"; `verified` is beside it**: a step that
+ completed with `verified: false` (an export's loudness miss) leaves the row
+ `ok: true`, `verified: false`, with `unverified: [{step, tool, failed}]`; the
+ batch's own `verification` has one `item` step per row and a note names the
+ unverified items. A row also carries its steps' `diagnostics` (tagged with `step`).
+- **Ctrl-C** (or SIGTERM) stops the running steps and exits 130 (143)
+ `kind: interrupted` with a row for every input: finished, `interrupted: true`
+ (stopped while running) or `"skipped": "interrupted"` (not started), so `total`
+ is the folder's count.
+- **The cache serves an output only while it is the file the batch wrote**: an
+ entry keeps the output's size and mtime, and an output replaced or edited since
+ is processed again (`cache_miss: {reason: output_changed, expected, actual}` on
+ the row, a `batch_cache_output_changed` warning; an entry from 2.10.11 or
+ earlier has no such record and is processed again once, `entry_unverifiable`).
+ A `.ffskill_cache.json` that does not parse is started empty with a
+ `batch_cache_reset` warning and a line in `output_dir/.ffskill-audit.jsonl`.
- With `--jobs > 1` each item gets its own work subdirectory
`<workdir>/<index>-<stem>/`, because step file names are stem-derived and two
sources sharing a stem would otherwise write over each other. `--jobs 1` (the
@@ -1182,7 +1222,12 @@ report.py --after FINAL [--before SOURCE] [--platform youtube] [--commands cmds.
```
One self-contained HTML: before/after facts and contact sheets, loudness,
compliance table with fixes, commands. Produce it for any multi-step job and
-hand the path to the user together with the numbers.
+hand the path to the user together with the numbers. Each section it builds is a
+`verification` step (`loudness` per file with audio, `compliance` for
+`--platform` with `passed` saying what the table found, `contact_sheet` per
+file); a section that could not be built is `ok: false` with the child's
+`reason` and `message` and a `report_section_missing` warning, so the report is
+`verified: false` rather than silently shorter.
### multicam.py — align several cameras and switch between them
```
@@ -1195,7 +1240,10 @@ All inputs are aligned to the first one by audio (same engine as `sync.py`,
each range of the reference timeline (gaps fall back to camera 0), `--auto N`
simply alternates every N seconds. Audio comes from the reference unless
`--audio` picks another input, e.g. an external recorder that has no video.
-`--offsets-only` reports offsets and confidence without rendering.
+`--offsets-only` reports offsets and confidence without rendering. `cuts` names
+the camera actually rendered: a range whose camera has not started yet plays
+camera 0, and a `camera_not_started` entry in `diagnostics.fallbacks` names the
+camera that was asked for.
**`--switch energy` (1.18)** auto-switches to whichever camera (of those with
a video stream) measures the loudest audio at each 0.25 s window on the
@@ -1222,6 +1270,11 @@ Runs the toolchain on the user's own files (phone HDR, GoPro, OBS, Log, Zoom)
and prints a PASS/FAIL table per step (probe, copy cut, accurate cut, fit,
caption, overlay, look, export, loudness, silence, plus `color --to-sdr` for
HDR and `audio --downmix` for >2 channels). Exit code 1 if anything fails.
+Each step runs with `--json`, and a failed one keeps its tool's own `kind`,
+`reason`, `message` and `exit_code` (`error` is the one-line table cell); a
+completed one carries the tool's `verified`. Steps that do not apply are listed
+under the file's `skipped` (`quick`, `no_video`, `no_audio`, `not_hdr`,
+`not_multichannel`, `input_unreadable`) and as SKIP rows in the report.
Run this first when a user hands over footage from a device you have not
seen before, and fix or report what fails.
@@ -1384,7 +1437,10 @@ and says so. A cue that would need more than `--max-lines` (default 2) is split
into consecutive cues sharing its time; a cue shorter than `--min-duration`
(default 1.0 s) is held longer, never past the next cue's start; `--offset
TIME` shifts every cue (seconds, `mm:ss`, `hh:mm:ss.ms` or `hh:mm:ss:ff`, a
-leading `-` for earlier) for `--text`, `--srt` and `--ass`. One `cues:` info line reports what changed. A file you passed in is
+leading `-` for earlier) for `--text`, `--srt` and `--ass`. One `cues:` info line reports what changed;
+with `--ass`, `caption.ass_shifted` / `ass_unshifted` count the Dialogue lines moved and those whose
+timestamps could not be read (copied as they were), and `ass_malformed` the lines that could not be
+read at all and are not counted as cues (each above 0 is a warning). A file you passed in is
never edited: the adjusted copy is written next to the output
(`<out>_adjusted.srt`, `<out>_offset.ass`) and burned instead — under `--dry-run`
/`--plan` the planned command names that same copy and the plan says where it
@@ -1525,7 +1581,11 @@ padded with silence, never allowed to cut the picture.
`--fix-drift` measures the offset again near the end of the overlap, reports
the clock difference in ppm, and resamples the source file so a 60-minute
take stays in sync (typical consumer devices drift 20-500 ppm = up to 1.8 s/h).
-Use it whenever the recording is longer than ~10 minutes. Check `confidence`
+Use it whenever the recording is longer than ~10 minutes. A correction that
+could not be measured is skipped and says why: `drift_skipped_reason` is
+`too_short` (no tail window far enough from the head) or `low_confidence`
+(top level for one source, and in each `sources` entry; null when applied).
+Check `confidence`
(0–1, normalised correlation with a runner-up penalty); below 0.3 the match is
doubtful. Benchmark on real dialogue/music (±30 s offsets, gain, noise, EQ):
with the default 120 s window 40/40 within 10 ms (max 1.1 ms); with a 60 s
@@ -1676,13 +1736,18 @@ export.py INPUT --preset youtube|youtube4k|reels|tiktok|shorts|linkedin|facebook
export.py --list
```
Scales into the preset frame (pad by default), tags BT.709 (a BT.601-matrix or RGB source is converted to it, as by every re-encode), sets `+faststart`,
-trims to platform maximums (Reels 90 s, X 140 s) unless `--allow-long`. It
+trims to platform maximums (Reels 90 s, X 140 s) unless `--allow-long`, and says
+so: `trimmed: {from, to}` and a `notes` line. It
does not touch levels: for youtube / youtube4k / reels / x the written file is
measured and the result's `loudness` (and a `notes` line) says when it is
outside the platform's LUFS / true-peak spec, naming the `loudness.py` call
that fixes it -- or pass `--normalize`, which runs that call on the written
file itself (audio re-encoded, video copied; `loudness.normalized: true`) so a
-platform export is one command instead of export, loudness, export again.
+platform export is one command instead of export, loudness, export again. A
+measurement that cannot be read is `loudness.measured: false` (values null) with
+a failing `{step: loudness, ok: false, reason: not_measured}` and a
+`loudness_not_measured` warning, so the export is `verified: false` rather than
+unchecked.
Since 1.14 each social destination is its own preset rather than an alias: `tiktok`
(1080x1920, max 600 s), `shorts` (1080x1920, max 180 s), `reels` (1080x1920, max 90 s),
diff --git a/scripts/_common/__init__.py b/scripts/_common/__init__.py
index 9f0a3bd..492a087 100644
--- a/scripts/_common/__init__.py
+++ b/scripts/_common/__init__.py
@@ -56,7 +56,7 @@ from _common.runner import (
child_limit, _CHILDREN, _cleanup_partial_output, _cmdline, CODECS, Context, DEFAULT_TIMEOUT,
_DRAWTEXT_PENDING, _DRAWTEXT_TMPDIR, _drawtext_tmpdir, dry_run_input_pending, _ENCODERS, _env_timeout,
ERROR_CODE, ERROR_RETRYABLE, EVEN_SCALE, _execute, _fail, ffmpeg_base, ffmpeg_encoders, ffmpeg_exe, _FFMPEG_RELEASE, _FFMPEG_VERSION,
- ffmpeg_bsfs, ffmpeg_release, ffmpeg_version, flush_drawtext_textfiles, INSTALL_HINTS, install_signal_handlers, _is_ffmpeg, _limit_for,
+ ffmpeg_bsfs, ffmpeg_list_error, ffmpeg_release, ffmpeg_version, flush_drawtext_textfiles, INSTALL_HINTS, install_signal_handlers, _is_ffmpeg, _limit_for,
_odd_dimension_retry, _on_signal, _OutputLock, _pid_dead, place_output, PROBE_TIMEOUT, _progress_line,
read_text_or_die, refuse_output_is_input, _remember_output, require_tool, run, run_analysis, _run_captured,
run_keeping_subtitles, run_tool, _run_with_progress, shell_quote, sibling_temp, _SIGNALS_INSTALLED, _stage_existing_output,
@@ -198,7 +198,7 @@ __all__ = [
"_emoji_name_candidates", "EMOJI_RANGES", "_EMOJI_REGIONAL", "emoji_support", "_EMOJI_SUPPORT_CACHE",
"_EMOJI_TAIL", "encoder_args", "_ENCODERS", "_env_timeout", "ERROR_CODE", "ERROR_RETRYABLE", "escape_drawtext",
"escape_filter_path", "EVEN_SCALE", "_execute", "_fail", "_family_rank", "FC_LANG", "_fc_list_fonts",
- "FC_UNKNOWN", "ffmpeg_base", "ffmpeg_bsfs", "ffmpeg_encoders", "ffmpeg_exe", "_FFMPEG_RELEASE", "_FFMPEG_VERSION", "ffmpeg_release", "ffmpeg_version", "fingerprint",
+ "FC_UNKNOWN", "ffmpeg_base", "ffmpeg_bsfs", "ffmpeg_encoders", "ffmpeg_list_error", "ffmpeg_exe", "_FFMPEG_RELEASE", "_FFMPEG_VERSION", "ffmpeg_release", "ffmpeg_version", "fingerprint",
"flush_drawtext_textfiles", "fold_bt709_matrix", "fmt_secs", "fmt_smpte_time", "fmt_srt_time", "font_covers_script",
"font_family_for_script", "font_family_of_file", "FONT_FLAG_HINT", "font_for_script", "FONT_INSTALL_HINT",
"fonts_dir_covers_script", "_fraction", "has_emoji", "info", "INSTALL_HINTS", "install_signal_handlers",
diff --git a/scripts/_common/asr.py b/scripts/_common/asr.py
index 067c592..7e8f4ad 100644
--- a/scripts/_common/asr.py
+++ b/scripts/_common/asr.py
@@ -16,7 +16,7 @@ from pathlib import Path
from typing import Any, Dict, List, NamedTuple, Optional, Tuple
from _common.decision import fmt_srt_time, parse_time
-from _common.emit import die, info
+from _common.emit import die, info, warn
from _common.runner import read_text_or_die
# The engines this skill knows how to drive, and how to install each -- one string, so every
@@ -296,7 +296,7 @@ def _parakeet_attempt(engines: "List[str]", facts: Dict[str, Any], wanted: str,
else:
run.notes.append(assumed_english_note(language_flag, whisper_failed=facts.get("routing") == ROUTING_LAST_RESORT,
parakeet_model=facts.get("routing") == ROUTING_PARAKEET_MODEL))
- info("warning: " + run.notes[-1])
+ warn(run.notes[-1], "asr_language_assumed")
return run, os.path.basename(chosen)
return None
diff --git a/scripts/_common/decision.py b/scripts/_common/decision.py
index e16ec9e..784a05b 100644
--- a/scripts/_common/decision.py
+++ b/scripts/_common/decision.py
@@ -16,8 +16,8 @@ from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Tuple
from _common.color import (bt709_label_filter, bt709_matrix_args, bt709_matrix_filter, bt709_tag_args, drawn_stream_format,
is_neutral_color, rgb_island, rgb_to_bt709_filter, _sdr_bt709)
-from _common.emit import die
-from _common.runner import CODECS, STATE, ffmpeg_encoders, ffmpeg_version, hw_platform_reason
+from _common.emit import die, record_fallback, refuse, warn
+from _common.runner import CODECS, STATE, ffmpeg_encoders, ffmpeg_list_error, ffmpeg_version, hw_platform_reason
def pad_filters(out_w: int, out_h: int, fill: str, color: str, blur: int, darken: float = 0.0,
@@ -321,7 +321,11 @@ def _vt_args(codec: str, crf: int, meta: Optional[Dict[str, Any]], keep_bt709: b
reason = None if name else f"no VideoToolbox encoder for {codec}"
reason = reason or hw_platform_reason()
if not reason and name not in ffmpeg_encoders():
- reason = f"this ffmpeg build has no {name}"
+ unread = ffmpeg_list_error("encoders")
+ # an encoder list that could not be read says nothing about the build: name the read
+ # failure, not a missing encoder (the read itself is a ffmpeg_list_unreadable warning)
+ reason = (f"{name} could not be confirmed (this ffmpeg build's encoder list could not be read: {unread})" if unread
+ else f"this ffmpeg build has no {name}")
if reason:
note = f"{reason}; encoded on the CPU"
if note not in STATE.hw_notes:
@@ -471,7 +475,7 @@ def _cpu_encoder_args(codec: str, crf: int, preset: str, meta: Optional[Dict[str
if codec == "hevc":
if hdr:
x265 = f"log-level=error:colorprim={prim}:transfer={trc}:colormatrix={cs}:range=limited:hdr10-opt=1" if trc == "smpte2084" else f"log-level=error:colorprim={prim}:transfer={trc}:colormatrix={cs}"
- return ["-c:v", "libx265", "-preset", preset, "-crf", str(min(51, crf + 2)), "-pix_fmt", "yuv420p10le", "-tag:v", "hvc1",
+ return ["-c:v", "libx265", "-preset", preset, "-crf", str(_clamped_crf("libx265", crf + 2, 51, crf)), "-pix_fmt", "yuv420p10le", "-tag:v", "hvc1",
"-x265-params", x265] + hdr_tags + ["-movflags", "+faststart"]
params, extra = _sdr_bt709("libx265") if keep_bt709 else ("", [])
return ["-c:v", "libx265", "-preset", preset, "-crf", str(crf), "-pix_fmt", "yuv420p", "-tag:v", "hvc1",
@@ -479,27 +483,56 @@ def _cpu_encoder_args(codec: str, crf: int, preset: str, meta: Optional[Dict[str
if codec == "av1":
pix = "yuv420p10le" if hdr else "yuv420p"
if "libsvtav1" in ffmpeg_encoders():
- args = ["-c:v", "libsvtav1", "-preset", str(SVT_PRESET.get(preset, 6)), "-crf", str(min(63, crf)), "-pix_fmt", pix]
+ args = ["-c:v", "libsvtav1", "-preset", str(SVT_PRESET.get(preset, 6)), "-crf", str(_clamped_crf("libsvtav1", crf, 63, crf)), "-pix_fmt", pix]
if hdr:
args += hdr_tags
elif keep_bt709:
params, extra = _sdr_bt709("libsvtav1")
args += (["-svtav1-params", params] if params else []) + extra
elif "libaom-av1" in ffmpeg_encoders():
- args = ["-c:v", "libaom-av1", "-crf", str(min(63, crf)), "-b:v", "0", "-cpu-used", "6", "-row-mt", "1", "-pix_fmt", pix]
+ # recorded: libaom is the slower AV1 encoder with its own CRF scale and speed setting,
+ # so the same --quality does not give the file SVT-AV1 would have
+ record_fallback("--codec av1: this ffmpeg build has no libsvtav1 (SVT-AV1), so libaom-av1 encodes instead "
+ "(slower; -cpu-used 6, its own CRF scale)", "av1_encoder_libaom", wanted="libsvtav1", used="libaom-av1")
+ args = ["-c:v", "libaom-av1", "-crf", str(_clamped_crf("libaom-av1", crf, 63, crf)), "-b:v", "0", "-cpu-used", "6", "-row-mt", "1", "-pix_fmt", pix]
args += hdr_tags if hdr else (_sdr_bt709("libaom-av1")[1] if keep_bt709 else [])
else:
- die("--codec av1 needs an AV1 encoder (libsvtav1 or libaom-av1) and this ffmpeg build has neither", kind="missing_tool",
- hint="install an ffmpeg built with SVT-AV1 (most distribution builds are), or use --codec hevc")
+ _refuse_missing_encoder("av1", ["libsvtav1", "libaom-av1"], "an AV1 encoder (libsvtav1 or libaom-av1)",
+ "install an ffmpeg built with SVT-AV1 (most distribution builds are), or use --codec hevc")
return args + ["-movflags", "+faststart"]
if codec == "prores":
if "prores_ks" not in ffmpeg_encoders():
- die("--codec prores needs the prores_ks encoder and this ffmpeg build lacks it", kind="missing_tool")
+ _refuse_missing_encoder("prores", ["prores_ks"], "the prores_ks encoder", None)
return ["-c:v", "prores_ks", "-profile:v", "3", "-vendor", "apl0", "-pix_fmt", "yuv422p10le"] + (hdr_tags if hdr else [])
die(f"unknown --codec {codec!r} (one of {', '.join(CODECS)})")
return []
+def _clamped_crf(encoder: str, value: int, top: int, requested: int) -> int:
+ """`value` capped at the encoder's maximum CRF, the cap recorded (`crf_clamped`) when it
+ bites: the HEVC HDR line asks x265 for CRF+2, so --quality 50 or 51 there is encoded at 51."""
+ if value <= top:
+ return value
+ warn(f"{encoder}: CRF {value} (from {requested}) is above the encoder's maximum {top}; encoded at CRF {top}",
+ "crf_clamped", encoder=encoder, requested=requested, asked=value, used=top)
+ return top
+
+
+def _refuse_missing_encoder(codec: str, encoders: List[str], what: str, hint: Optional[str]) -> None:
+ """--codec av1/prores with no encoder for it -- or with an encoder list that could not be read,
+ which proves nothing about the build and is refused under its own reason rather than as a
+ missing encoder (ffmpeg_list_error)."""
+ unread = ffmpeg_list_error("encoders")
+ if unread:
+ refuse("ffmpeg", "encoder_list_unreadable",
+ f"--codec {codec} needs {what}, and whether this ffmpeg build has one is unknown: its encoder list "
+ f"could not be read ({unread})",
+ hint="check that ffmpeg runs (python3 scripts/_contract.py doctor), then retry",
+ codec=codec, encoders=encoders, error=unread)
+ refuse("missing_tool", "encoder_missing", f"--codec {codec} needs {what} and this ffmpeg build has "
+ + ("neither" if len(encoders) > 1 else "none"), hint=hint, codec=codec, encoders=encoders)
+
+
def _x264_raw(crf: int, preset: str, keep_bt709: bool = True) -> List[str]:
args = ["-c:v", "libx264", "-preset", preset, "-crf", str(crf), "-pix_fmt", "yuv420p", "-movflags", "+faststart"]
if keep_bt709:
@@ -1159,17 +1192,22 @@ def normalise_filler_token(word: str) -> str:
def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = FILLER_MIN_GAP,
- max_word: float = FILLER_MAX_WORD) -> "List[Dict[str, Any]]":
+ max_word: float = FILLER_MAX_WORD, stats: "Optional[Dict[str, int]]" = None) -> "List[Dict[str, Any]]":
"""Time spans to remove, from measured word timings. Pure: no subprocess, no I/O.
`words` is [{"word", "start", "end"}] as whisper emits. A word is removed only when its
normalised form is in `wordlist` AND it carries a real start < end pair AND its length is
<= max_word. Adjacent spans closer than min_gap merge. Returns [{"start", "end", "word"}]
sorted and non-overlapping.
+
+ `stats`, when given, receives the listed words that were kept anyway, so the caller can say
+ so: `matched_untimed` (no usable start < end: nothing to cut on) and `matched_too_long`
+ (longer than max_word: a held vowel someone meant).
"""
listed = {normalise_filler_token(w) for w in (wordlist or set())}
listed.discard("")
hits = []
+ untimed = too_long = 0
for entry in words or []:
if not isinstance(entry, dict):
continue
@@ -1179,8 +1217,13 @@ def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = F
try:
start, end = float(entry["start"]), float(entry["end"])
except (KeyError, TypeError, ValueError):
+ untimed += 1
continue # no measured timing: nothing to cut
- if not (end > start) or (end - start) > max_word:
+ if not (end > start):
+ untimed += 1
+ continue
+ if (end - start) > max_word:
+ too_long += 1
continue
hits.append({"start": max(0.0, start - pad), "end": end + pad, "word": token})
hits.sort(key=lambda h: (h["start"], h["end"]))
@@ -1194,6 +1237,9 @@ def filler_spans(words, wordlist, *, pad: float = FILLER_PAD, min_gap: float = F
for m in merged:
m["start"] = round(m["start"], 4)
m["end"] = round(m["end"], 4)
+ if stats is not None:
+ stats["matched_untimed"] = untimed
+ stats["matched_too_long"] = too_long
return merged
diff --git a/scripts/_common/emit.py b/scripts/_common/emit.py
index d8d4d59..61a3d9c 100644
--- a/scripts/_common/emit.py
+++ b/scripts/_common/emit.py
@@ -74,7 +74,10 @@ def _record(bucket: str, msg: str, code: str, ctx: "Optional[Context]", data: Di
ctx = ctx or STATE
entry: Dict[str, Any] = {"code": code, "message": msg}
entry.update(data)
- getattr(ctx, bucket).append(entry)
+ # once per run: a decision taken for every encode of a job (an AV1 encoder fallback, a CRF
+ # clamp) is one record, though its stderr line is printed each time it is taken
+ if entry not in getattr(ctx, bucket):
+ getattr(ctx, bucket).append(entry)
sys.stderr.write(f"{prefix}{msg}\n")
return entry
@@ -82,7 +85,8 @@ def _record(bucket: str, msg: str, code: str, ctx: "Optional[Context]", data: Di
def warn(msg: str, code: str, *, ctx: "Optional[Context]" = None, **data: Any) -> Dict[str, Any]:
"""Say something went other than asked, on stderr AND in the document
(`diagnostics.warnings`: {code, message, ...data}). Use it where a bare info("warning: ...")
- would leave the caller of --json nothing to read."""
+ would leave the caller of --json nothing to read (a source test refuses that spelling).
+ Every code is listed in the contract (`json_output.diagnostic_codes`)."""
return _record("warnings", msg, code, ctx, data, "warning: ")
@@ -291,10 +295,32 @@ def absorb_stage_hw(stage: str, doc: Any, ctx: "Optional[Context]" = None) -> No
"used": hw.get("used"), "fallback": bool(hw.get("fallback")), "quality_note": quality})
+def _stage_hw_notes(doc: Dict[str, Any]) -> List[str]:
+ """The notes lines a stage's document carries only because of its GPU encode (its `hw.notes`
+ and the VideoToolbox quality line): absorb_stage_hw() already reports those, once."""
+ hw = doc.get("hw") if isinstance(doc.get("hw"), dict) else {}
+ out = [str(n) for n in hw.get("notes") or []]
+ enc, line = _video_encoders([str(c) for c in (doc.get("commands") or [])])
+ for no_cpu in (False, True):
+ q = hw_quality_note(enc, line, no_cpu)
+ if q:
+ out.append(q)
+ return out
+
+
+# A stage's own artifact checks: a stage that completed passed them (verify_output() fails the
+# stage otherwise), and its artifact is an intermediate, so carrying them would only lengthen the list.
+_STAGE_ARTIFACT_STEPS = ("probe", "exists")
+
+
def absorb_stage(stage: str, doc: Any, ctx: "Optional[Context]" = None) -> Dict[str, Any]:
- """A child tool's --json document into this result: its `hw` facts (absorb_stage_hw) and its
- `diagnostics` records, each tagged with the stage it came from. Returns the fields a re-raise
- of the stage's failure carries in its own `error`: the child's `reason` and `details`."""
+ """A child tool's --json document into this result: its `hw` facts (absorb_stage_hw), its
+ `diagnostics` records and its `notes` (each tagged with the stage it came from), and -- for a
+ stage that completed in a real run -- its verification steps beyond the artifact probe, tagged
+ with `stage`, which emit() adds to this result's own: the parent's `verified` is then the
+ conjunction of every stage's (a stage that said `verified: false` without a failing step adds
+ one `{step: verified, stage, ok: false}`). Returns the fields a re-raise of the stage's failure
+ carries in its own `error`: the child's `reason` and `details`."""
ctx = ctx or STATE
absorb_stage_hw(stage, doc, ctx)
if not isinstance(doc, dict):
@@ -305,6 +331,22 @@ def absorb_stage(stage: str, doc: Any, ctx: "Optional[Context]" = None) -> Dict[
tagged = dict(entry, stage=entry.get("stage") or stage)
if tagged not in getattr(ctx, bucket):
getattr(ctx, bucket).append(tagged)
+ hw_only = _stage_hw_notes(doc)
+ for n in doc.get("notes") or []:
+ if not isinstance(n, str) or n in hw_only or n.endswith(ENV_HW_NOTE) or any(n.endswith(": " + h) for h in hw_only):
+ continue
+ # a nested stage's note (render.py inside a pack) already names its stage
+ tagged_note = n if n.startswith(f"{stage}: ") else f"{stage}: {n}"
+ if tagged_note not in ctx.notes:
+ ctx.notes.append(tagged_note)
+ if doc.get("status") == "completed" and not doc.get("dry_run") and not ctx.dry_run:
+ carried = [dict(s, stage=s.get("stage") or stage) for s in doc.get("verification") or []
+ if isinstance(s, dict) and (s.get("step") not in _STAGE_ARTIFACT_STEPS or not s.get("ok"))]
+ if doc.get("verified") is False and all(s.get("ok") for s in carried):
+ carried.append({"step": "verified", "stage": stage, "ok": False})
+ for s in carried:
+ if s not in ctx.stage_verification:
+ ctx.stage_verification.append(s)
err = doc.get("error") if isinstance(doc.get("error"), dict) else {}
return {k: err[k] for k in ("reason", "details") if err.get(k) is not None}
@@ -351,6 +393,8 @@ def emit(output: Optional[str], *, ctx: "Optional[Context]" = None, **extra: Any
# verified nothing. Spec failures the tool cannot fix on its own (export's loudness gap)
# keep status completed and say verified: false, so a caller keys on one field.
steps: List[Dict[str, Any]] = ([{"step": "probe", "ok": True}] if meta else []) + list(extra.pop("verification", None) or [])
+ # a tool that ran others (render.py) verified what its stages verified too (absorb_stage())
+ steps += [s for s in ctx.stage_verification if s not in steps]
if output and not ctx.dry_run and os.path.splitext(output)[1].lower() not in MEDIA_EXT:
steps.insert(0, {"step": "exists", "ok": True})
doc["verified"] = not ctx.dry_run and bool(steps) and all(s.get("ok") for s in steps)
diff --git a/scripts/_common/fonts.py b/scripts/_common/fonts.py
index bd00416..8ab0bc7 100644
--- a/scripts/_common/fonts.py
+++ b/scripts/_common/fonts.py
@@ -11,20 +11,21 @@ import shutil
import subprocess
from pathlib import Path
from typing import Dict, List, Optional, Sequence, Tuple
-from _common.emit import die, info
+from _common.emit import die, info, record_fallback, warn
from _common.emoji import _is_emoji_char
def default_font_file(font_name: str) -> Optional[str]:
"""Resolve `font_name` to a concrete on-disk font file, so a caller can tell drawtext
- `fontfile=<path>` instead of `font=<name>`, when possible.
+ `fontfile=<path>` instead of `font=<name>`, when possible (resolve_font()'s `file`).
On some real Windows ffmpeg builds (winget's gyan.dev 9.x), drawtext's own fontconfig
resolution crashes with an access violation whenever it has to resolve a font by family name
-- with or without a valid fonts.conf on FONTCONFIG_FILE. `fontfile=` is the only form
- confirmed not to crash (#100), since it never touches fontconfig at all. `font_name` itself is
- ignored on Windows for that reason: a fixed, near-universally-present system font is used
- instead of trying to resolve the requested family (which would crash the same way).
+ confirmed not to crash (#100), since it never touches fontconfig at all. On Windows the
+ requested family is looked for by file name in the Fonts folder; a family that is not there
+ falls back to a CJK system font (for a CJK-looking request) or Arial, a near-universally-present
+ system font, instead of trying to resolve the requested family (which would crash the same way).
On Linux/macOS this is best-effort and uses the real requested family: `fc-match` reports the
same file fontconfig would resolve `font_name` to anyway, so a caller gets the identical font,
@@ -35,6 +36,23 @@ def default_font_file(font_name: str) -> Optional[str]:
Returns None when nothing could be resolved (fc-match missing/unavailable, or no well-known
Windows font file present); the caller falls back to font=<font_name>, the prior behaviour.
"""
+ return resolve_font(font_name)["file"]
+
+
+# generic fontconfig aliases: resolving one to a concrete family is what they are for, not a substitution
+_GENERIC_FAMILIES = {"sans", "sansserif", "serif", "mono", "monospace", "cursive", "fantasy", "systemui", "emoji"}
+
+
+def _family_key(name: "Optional[str]") -> str:
+ return re.sub(r"[^a-z0-9]", "", (name or "").lower())
+
+
+def resolve_font(font_name: str) -> "Dict[str, object]":
+ """default_font_file() with what it found: {file, family, substituted, via}. `substituted` is
+ True when the file is not the requested family -- a Windows machine without it (a CJK system
+ font or Arial instead), or fc-match answering with another family (fontconfig always answers
+ something). `via` is windows_fonts or fc-match; file and family are None when nothing resolved."""
+ none: "Dict[str, object]" = {"file": None, "family": None, "substituted": False, "via": None}
if platform.system() == "Windows":
windir = os.environ.get("WINDIR", "C:\\Windows")
fonts = Path(windir) / "Fonts"
@@ -42,33 +60,78 @@ def default_font_file(font_name: str) -> Optional[str]:
# removed (Noto Sans CJK JP -> NotoSansCJKjp-Regular.otf, Meiryo -> meiryo.ttc), then the
# common CJK system fonts when the request looks CJK (so Japanese text does not render as
# boxes in Arial), and Arial only as the last resort.
- wanted = re.sub(r"[^a-z0-9]", "", (font_name or "").lower())
+ wanted = _family_key(font_name)
try:
files = sorted(fonts.iterdir()) if fonts.is_dir() else []
except OSError:
files = []
if wanted:
for f in files:
- stem = re.sub(r"[^a-z0-9]", "", f.stem.lower())
+ stem = _family_key(f.stem)
if f.suffix.lower() in (".ttf", ".otf", ".ttc") and stem.startswith(wanted):
- return str(f)
+ return {"file": str(f), "family": font_name, "substituted": False, "via": "windows_fonts"}
if any(k in wanted for k in ("cjk", "gothic", "mincho", "meiryo", "yugoth", "msgothic", "malgun", "simhei", "simsun", "jp", "kr", "sc", "tc")):
for name in ("NotoSansCJKjp-Regular.otf", "NotoSansCJK-Regular.ttc", "meiryo.ttc", "YuGothM.ttc", "msgothic.ttc", "malgun.ttf", "msyh.ttc"):
if (fonts / name).exists():
- return str(fonts / name)
+ return {"file": str(fonts / name), "family": Path(name).stem, "substituted": bool(wanted), "via": "windows_fonts"}
candidate = fonts / "arial.ttf"
- return str(candidate) if candidate.exists() else None
+ if candidate.exists():
+ return {"file": str(candidate), "family": "Arial",
+ "substituted": bool(wanted) and wanted != "arial" and wanted not in _GENERIC_FAMILIES, "via": "windows_fonts"}
+ return none
exe = shutil.which("fc-match")
if not exe:
- return None
+ return none
try:
- proc = subprocess.run([exe, "--format=%{file}\n", font_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", timeout=5)
+ proc = subprocess.run([exe, "--format=%{file}\n%{family}\n", font_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", timeout=5)
except (subprocess.TimeoutExpired, OSError):
- return None
+ return none
if proc.returncode != 0:
- return None
- path = proc.stdout.splitlines()[0].strip() if proc.stdout.strip() else ""
- return path if path and os.path.exists(path) else None
+ return none
+ lines = proc.stdout.splitlines()
+ path = lines[0].strip() if lines else ""
+ if not (path and os.path.exists(path)):
+ return none
+ # fontconfig lists a face's family names comma-separated (localised names included); the
+ # request is met when one of them is the requested family, or the request names a style of it
+ families = [f.strip() for f in (lines[1] if len(lines) > 1 else "").split(",") if f.strip()]
+ wanted = _family_key(font_name)
+ met = (not wanted or wanted in _GENERIC_FAMILIES or not families
+ or any(_family_key(f) and (wanted.startswith(_family_key(f)) or _family_key(f).startswith(wanted)) for f in families))
+ return {"file": path, "family": families[0] if families else None, "substituted": not met, "via": "fc-match"}
+
+
+def font_choice(asked: "Optional[str]", *, font_file: "Optional[str]" = None, script_file: "Optional[str]" = None,
+ script_family: "Optional[str]" = None, renderer: str = "drawtext") -> "Dict[str, object]":
+ """The font a text-drawing tool draws with, as its result's `font_used`: {asked, file, family,
+ how, substituted}. `how`: font_file (the caller's own file), script (a font picked to cover the
+ text's script, 1.12), resolved (the asked family resolved to a file: fc-match, or the Windows
+ Fonts folder), by_name (no file resolved; ffmpeg/libass look the family up themselves, so the
+ file is not known here). For libass (`renderer="libass"`) a resolved file is the one fontconfig
+ picks for the family, which is what libass loads through fontconfig; on Windows libass is
+ given the family by name, so nothing is resolved for it."""
+ if font_file:
+ return {"asked": font_file, "file": font_file, "family": None, "how": "font_file", "substituted": False}
+ if script_file:
+ return {"asked": asked, "file": script_file, "family": script_family, "how": "script", "substituted": False}
+ if renderer == "libass" and platform.system() == "Windows":
+ return {"asked": asked, "file": None, "family": asked, "how": "by_name", "substituted": False}
+ found = resolve_font(asked or "")
+ if not found["file"]:
+ return {"asked": asked, "file": None, "family": asked, "how": "by_name", "substituted": False}
+ return {"asked": asked, "file": found["file"], "family": found["family"], "how": "resolved",
+ "substituted": bool(found["substituted"]), "via": found["via"]}
+
+
+def record_font_choice(choice: "Dict[str, object]") -> "Dict[str, object]":
+ """Record a substituted font (`diagnostics.fallbacks`, `font_substituted`) and return `choice`
+ for the result's `font_used`: fontconfig answers every request with some font, and drawtext and
+ libass draw with it and exit 0, so without this the only trace was the picture."""
+ if choice.get("substituted"):
+ record_fallback(f"font: '{choice['asked']}' is not installed here; drawing with {choice.get('family') or '?'} "
+ f"({choice['file']}) instead", "font_substituted",
+ asked=choice["asked"], file=choice["file"], family=choice.get("family"), via=choice.get("via"))
+ return choice
# --------------------------------------------------------------------------- script detection
@@ -399,14 +462,14 @@ def script_font_for_text(text: str, *, lang: "Optional[str]" = None, font: "Opti
Returns (script, None, None) when the caller's own choice stands: Latin text, an explicit
--font-file, an explicit --font (which is kept even when fontconfig says it does not cover
- the script -- with one info line saying so, because overriding a user's stated font silently
- is worse than a warning), or a --fonts-dir that does carry the script. Otherwise the resolved
+ the script -- with one recorded warning saying so, `font_lacks_script`, because overriding a
+ user's stated font silently is worse than a warning), or a --fonts-dir that does carry the script. Otherwise the resolved
file is returned with ONE info line naming it.
A script fontconfig says nothing covers is a failed job (tofu is not a delivery). A machine
with no working fontconfig at all answers "unknown", not "missing": the job continues with
- the caller's font -- libass and drawtext still have their own font backends -- and one info
- line says the coverage could not be verified.
+ the caller's font -- libass and drawtext still have their own font backends -- and one recorded
+ warning (`font_coverage_unverified`) says the coverage could not be verified.
"""
script = detect_script(text or "", lang)
if script == "latin":
@@ -415,24 +478,26 @@ def script_font_for_text(text: str, *, lang: "Optional[str]" = None, font: "Opti
return script, None, None
if font_explicit and font:
if not font_covers_script(font, script):
- info(f"font: '{font}' does not cover {LANGUAGE_NAMES[script]} text on this machine; keeping it as asked "
- f"(drop --font, or {FONT_FLAG_HINT}, to pick one by script automatically)")
+ warn(f"font: '{font}' does not cover {LANGUAGE_NAMES[script]} text on this machine; keeping it as asked "
+ f"(drop --font, or {FONT_FLAG_HINT}, to pick one by script automatically)", "font_lacks_script",
+ font=font, script=script)
return script, None, None
if fonts_dir:
covered = fonts_dir_covers_script(fonts_dir, script)
if covered:
return script, None, None
if covered is None:
- info(f"font: could not verify that {fonts_dir} covers {LANGUAGE_NAMES[script]} text "
- "(no fc-scan on this machine); using it as given")
+ warn(f"font: could not verify that {fonts_dir} covers {LANGUAGE_NAMES[script]} text "
+ "(no fc-scan on this machine); using it as given", "font_coverage_unverified", fonts_dir=fonts_dir, script=script)
return script, None, None
- info(f"font: no face in {fonts_dir} covers {LANGUAGE_NAMES[script]} text; "
- "picking one by script instead (the directory is still searched first)")
+ record_fallback(f"font: no face in {fonts_dir} covers {LANGUAGE_NAMES[script]} text; "
+ "picking one by script instead (the directory is still searched first)", "fonts_dir_lacks_script",
+ fonts_dir=fonts_dir, script=script)
entry = _script_font_entry(script)
if entry is FC_UNKNOWN:
- info(f"font: could not verify that this machine can render {LANGUAGE_NAMES[script]} text "
+ warn(f"font: could not verify that this machine can render {LANGUAGE_NAMES[script]} text "
"(no working fontconfig); rendering with the font as given -- "
- "doctor --json .fonts.scripts reports what is known")
+ "doctor --json .fonts.scripts reports what is known", "font_coverage_unverified", script=script)
return script, None, None
if not entry:
die(f"no installed font covers {LANGUAGE_NAMES[script]} text on this machine — {FONT_INSTALL_HINT}", kind="input")
diff --git a/scripts/_common/runner.py b/scripts/_common/runner.py
index 21e7367..28910cd 100644
--- a/scripts/_common/runner.py
+++ b/scripts/_common/runner.py
@@ -71,10 +71,26 @@ PROBE_TIMEOUT = 120.0
def _env_timeout() -> float:
+ return _read_env_timeout()[0]
+
+
+def _read_env_timeout() -> "Tuple[float, Optional[str]]":
+ """(the per-call limit FFMPEG_SKILL_TIMEOUT sets, why its value was not taken as written or
+ None). Unset or blank is the default with nothing to say. A value that is not a number (or is
+ NaN) is the default, and a negative one is 0 (no limit) as it always was; both used to pass
+ without a word, and apply_common() now records them (`timeout_env_invalid`)."""
+ raw = os.environ.get("FFMPEG_SKILL_TIMEOUT")
+ if raw is None or not raw.strip():
+ return DEFAULT_TIMEOUT, None
try:
- return max(0.0, float(os.environ.get("FFMPEG_SKILL_TIMEOUT", DEFAULT_TIMEOUT)))
+ value = float(raw)
+ if value != value:
+ raise ValueError("NaN")
except ValueError:
- return DEFAULT_TIMEOUT
+ return DEFAULT_TIMEOUT, f"FFMPEG_SKILL_TIMEOUT={raw!r} is not a number of seconds; using the default {DEFAULT_TIMEOUT:g} s"
+ if value < 0:
+ return 0.0, f"FFMPEG_SKILL_TIMEOUT={raw!r} is negative; read as 0, which means no time limit"
+ return value, None
# None of the four kinds above are retryable in practice: an "input"/"missing_tool" failure is
@@ -218,7 +234,7 @@ class Context:
__slots__ = ("dry_run", "json", "json_brief", "progress", "fast", "duration_hint", "commands", "timeout", "overwrite", "written", "preexisting", "plan", "plan_written", "plan_inputs", "codec",
"hw", "hw_source", "hw_notes", "hw_swaps", "hw_fallback", "hw_env_ignored", "hw_stages", "hw_auto",
- "notes", "warnings", "fallbacks", "skipped", "audit_log")
+ "notes", "warnings", "fallbacks", "skipped", "audit_log", "stage_verification")
def __init__(self) -> None:
self.reset()
@@ -256,6 +272,10 @@ class Context:
self.fallbacks: List[Dict[str, Any]] = []
self.skipped: List[Dict[str, Any]] = []
self.audit_log: Optional[str] = None # render.py --audit-log FILE: where audit() appends; None = the event's own default
+ # render.py (stages, an executed plan, a pack's renders): the verification steps each child
+ # tool reported, tagged with `stage`, which emit() adds to this result's own so that
+ # `verified` is the conjunction of every stage's (absorb_stage())
+ self.stage_verification: List[Dict[str, Any]] = []
STATE = Context()
@@ -330,6 +350,10 @@ def apply_common(args: "argparse.Namespace") -> None:
STATE.overwrite = bool(getattr(args, "overwrite", False))
if getattr(args, "timeout", None) is not None:
STATE.timeout = max(0.0, float(args.timeout))
+ else:
+ env_limit, env_problem = _read_env_timeout()
+ if env_problem:
+ warn(env_problem, "timeout_env_invalid", value=os.environ.get("FFMPEG_SKILL_TIMEOUT"), timeout=env_limit)
if STATE.fast and getattr(args, "preset", None) in X264_PRESETS:
args.preset = "veryfast"
STATE.codec = getattr(args, "codec", None) or None
@@ -387,9 +411,10 @@ _CHILDREN: List[Tuple[subprocess.Popen, Sequence[str]]] = []
_SIGNALS_INSTALLED = False
-def _on_signal(signum: int, frame: Any) -> None:
- import signal as _signal
- name = {getattr(_signal, "SIGINT", None): "SIGINT", getattr(_signal, "SIGTERM", None): "SIGTERM"}.get(signum, str(signum))
+def stop_children() -> None:
+ """Stop every child this tool is waiting on and remove each one's partial output: what the
+ signal handler does before it exits, for a tool that reports more than "interrupted" after a
+ signal (batch.py's partial table)."""
for proc, cmd in list(_CHILDREN):
try:
proc.terminate() # ffmpeg exits promptly on SIGTERM; a sibling script runs this same handler
@@ -403,6 +428,12 @@ def _on_signal(signum: int, frame: Any) -> None:
if cmd:
_cleanup_partial_output(cmd)
_CHILDREN.clear()
+
+
+def _on_signal(signum: int, frame: Any) -> None:
+ import signal as _signal
+ name = {getattr(_signal, "SIGINT", None): "SIGINT", getattr(_signal, "SIGTERM", None): "SIGTERM"}.get(signum, str(signum))
+ stop_children()
die(f"interrupted by {name}: the running command was stopped and its partial output removed; nothing was written",
code=128 + signum, kind="interrupted")
@@ -1031,7 +1062,10 @@ def run(cmd: Sequence[str], *, quiet: bool = False, check: bool = True, ctx: "Op
return proc, ran
retry = _odd_dimension_retry(ran, proc.stderr or "")
if retry is not None:
- info("source has odd dimensions; scaling to even before encoding (yuv420p needs it)")
+ # recorded, not only said: the output's frame is a pixel narrower or shorter than
+ # the source's, which a caller comparing sizes has to be able to read
+ record_fallback("source has odd dimensions; scaling to even before encoding (yuv420p needs it)",
+ "odd_dimensions_scaled_even", ctx=ctx, output=cmd[-1], scale=EVEN_SCALE)
ctx.commands[-1] = _cmdline(retry[:-1] + [cmd[-1]])
return _execute(retry), retry
if "not divisible by 2" in (proc.stderr or ""):
@@ -1130,10 +1164,12 @@ def run_analysis(cmd: Sequence[str], *, check: bool = True, text: bool = True, r
def dry_run_input_pending(path: str) -> bool:
"""True when a measurement cannot run because its input does not exist yet under --dry-run:
in a render.py/batch.py plan each stage's input is the previous stage's output, which a dry
- run never wrote. The measurement is then skipped (with a note) rather than failing the plan;
- on a real file the measurement runs even under --dry-run."""
+ run never wrote. The measurement is then skipped rather than failing the plan, and the skip is
+ recorded (`diagnostics.skipped`, `measurement_input_pending`, once per path) beside its stderr
+ line; on a real file the measurement runs even under --dry-run."""
if STATE.dry_run and not os.path.exists(path):
- info(f"[dry-run] {path} does not exist yet (an earlier dry-run stage would write it); measurement skipped")
+ record_skip(f"[dry-run] {path} does not exist yet (an earlier dry-run stage would write it); measurement skipped",
+ "measurement_input_pending", path=path)
return True
return False
@@ -1196,9 +1232,19 @@ def run_keeping_subtitles(cmd: List[str], output: str) -> bool:
(e.g. a container change) makes that first attempt fail; retry the same command without the
extra maps rather than let a tool that never touched subtitles start hard-failing because of
them. `cmd` is the full argv *without* the output path. Returns True only when the
- retry-without-subtitles path was actually needed (i.e. subtitle/data streams were dropped)."""
- if run(cmd + ["-map", "0:s?", "-map", "0:d?", "-c:s", "copy", "-c:d", "copy", output], check=False).returncode == 0:
+ retry-without-subtitles path was actually needed (i.e. subtitle/data streams were dropped).
+
+ The retry is recorded with the first attempt's error (`diagnostics.fallbacks`,
+ `non_av_streams_dropped`): the caller's `dropped_non_av_streams` says that streams went, the
+ record says why, which is the only way to tell a container that cannot hold them from a first
+ attempt that failed for another reason and happened to succeed the second time."""
+ first = run(cmd + ["-map", "0:s?", "-map", "0:d?", "-c:s", "copy", "-c:d", "copy", output], check=False)
+ if first.returncode == 0:
return False
+ error = " | ".join([ln.strip() for ln in (first.stderr or "").splitlines() if ln.strip()][-3:])[-500:]
+ record_fallback(f"the subtitle/data streams could not be copied into {os.path.basename(output)} "
+ f"({error or 'ffmpeg exit ' + str(first.returncode)}); encoding again without them",
+ "non_av_streams_dropped", output=output, returncode=first.returncode, error=error)
run(cmd + [output])
return True
@@ -1459,41 +1505,70 @@ def ffmpeg_encoders() -> set:
ffmpeg_version(). That is the one case for ffprobe. On a mixed install (a Homebrew ffprobe next
to a static ffmpeg) its list is another build's, so a real run never uses it: the AV1/ProRes
refusals and the VideoToolbox check must describe the binary that encodes, and a plan that
- read ffprobe's list is re-decided against ffmpeg's when it runs."""
+ read ffprobe's list is re-decided against ffmpeg's when it runs.
+
+ An empty list is not "this build has none": when the list could not be read (the binary did
+ not run, timed out, exited non-zero, or printed nothing a list is made of), the reason is kept
+ (ffmpeg_list_error("encoders")) and recorded once as a `ffmpeg_list_unreadable` warning, so a
+ refusal or a CPU note built on the empty set names the read failure, not a missing encoder."""
global _ENCODERS
if _ENCODERS is None:
- _ENCODERS = set()
- tool = "ffprobe" if STATE.dry_run else "ffmpeg"
- # --timeout bounds this read too: on Apple Silicon every CPU re-encode asks for the list
- # (#314), so a deadlocked ffmpeg must not outlive the limit the caller set before the encode
- limit = min(PROBE_TIMEOUT, STATE.timeout) if STATE.timeout else PROBE_TIMEOUT
- try:
- out = subprocess.run([ffmpeg_exe(tool)[0] or tool, "-hide_banner", "-encoders"], stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL, text=True, encoding="utf-8", errors="replace", timeout=limit).stdout
- _ENCODERS = set(re.findall(r"^\s*[VAS][.\w]{5}\s+(\S+)", out, re.M))
- except (OSError, subprocess.SubprocessError):
- pass
+ _ENCODERS = _read_ff_list("-encoders", r"^\s*[VAS][.\w]{5}\s+(\S+)")
return _ENCODERS
def ffmpeg_bsfs() -> set:
- """Names from `ffmpeg -bsfs`, read once, empty when it cannot be read; from ffprobe under
- --dry-run, for the reason ffmpeg_encoders() gives. color.py --retag rewrites an H.264/HEVC
- stream's colour description with h264_metadata/hevc_metadata only when the build has it."""
+ """Names from `ffmpeg -bsfs`, read once, empty when it cannot be read (the reason is kept and
+ recorded as ffmpeg_encoders() keeps it); from ffprobe under --dry-run, for the reason
+ ffmpeg_encoders() gives. color.py --retag rewrites an H.264/HEVC stream's colour description
+ with h264_metadata/hevc_metadata only when the build has it."""
global _BSFS
if _BSFS is None:
- _BSFS = set()
- tool = "ffprobe" if STATE.dry_run else "ffmpeg"
- limit = min(PROBE_TIMEOUT, STATE.timeout) if STATE.timeout else PROBE_TIMEOUT
- try:
- out = subprocess.run([ffmpeg_exe(tool)[0] or tool, "-hide_banner", "-bsfs"], stdout=subprocess.PIPE,
- stderr=subprocess.DEVNULL, text=True, encoding="utf-8", errors="replace", timeout=limit).stdout
- _BSFS = set(re.findall(r"^\s*([a-z0-9_]+)\s*$", out, re.M))
- except (OSError, subprocess.SubprocessError):
- pass
+ _BSFS = _read_ff_list("-bsfs", r"^\s*([a-z0-9_]+)\s*$")
return _BSFS
+# why the last read of a capability list came back empty: "-encoders"/"-bsfs" -> reason
+_LIST_ERRORS: Dict[str, str] = {}
+
+
+def ffmpeg_list_error(flag: str) -> Optional[str]:
+ """Why `ffmpeg -encoders` / `-bsfs` (flag "encoders" or "bsfs") could not be read, or None when
+ it was read (or not yet asked for). Only while the cached list is empty: a list a test or a
+ later read filled in has nothing to explain."""
+ flag = "-" + flag.lstrip("-")
+ cache = _ENCODERS if flag == "-encoders" else _BSFS if flag == "-bsfs" else None
+ return None if cache else _LIST_ERRORS.get(flag)
+
+
+def _read_ff_list(flag: str, pattern: str) -> set:
+ tool = "ffprobe" if STATE.dry_run else "ffmpeg"
+ # --timeout bounds this read too: on Apple Silicon every CPU re-encode asks for the list
+ # (#314), so a deadlocked ffmpeg must not outlive the limit the caller set before the encode
+ limit = min(PROBE_TIMEOUT, STATE.timeout) if STATE.timeout else PROBE_TIMEOUT
+ _LIST_ERRORS.pop(flag, None)
+ names: set = set()
+ why = None
+ try:
+ proc = subprocess.run([ffmpeg_exe(tool)[0] or tool, "-hide_banner", flag], stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", timeout=limit)
+ names = set(re.findall(pattern, proc.stdout or "", re.M))
+ if not names:
+ tail = [ln.strip() for ln in (proc.stderr or "").splitlines() if ln.strip()][-1:]
+ why = (f"{tool} {flag} exited {proc.returncode}" if proc.returncode else f"{tool} {flag} listed nothing") + \
+ (f": {tail[0][:200]}" if tail else "")
+ except subprocess.TimeoutExpired:
+ why = f"{tool} {flag} did not answer within {limit:g} s"
+ except (OSError, subprocess.SubprocessError) as exc:
+ why = f"{tool} {flag} could not run: {exc}"
+ if why:
+ _LIST_ERRORS[flag] = why
+ what = "encoder" if flag == "-encoders" else "bitstream filter"
+ warn(f"could not read this ffmpeg build's {what} list ({why}); nothing that needs it can be confirmed, which is not the same as the build lacking it",
+ "ffmpeg_list_unreadable", list=flag.lstrip("-"), tool=tool, error=why)
+ return names
+
+
def read_text_or_die(path: str, flag: str) -> str:
"""Read a caller-supplied UTF-8 text file (a cue list, chapters, notes) or fail as kind input
with the flag named, instead of a FileNotFoundError / UnicodeDecodeError traceback."""
@@ -1518,4 +1593,4 @@ def read_text_or_die(path: str, flag: str) -> str:
# emit needs runner's Context/STATE/ERROR_CODE, so the two form a cycle that has to be cut
# somewhere: by the time this line runs every name emit reads from runner above is defined, and
# every name runner reads from emit is only ever read inside a function body, never at import.
-from _common.emit import _plan_at_exit, audit, die, hw_report, info, record_fallback, refuse, warn # noqa: E402
+from _common.emit import _plan_at_exit, audit, die, hw_report, info, record_fallback, record_skip, refuse, warn # noqa: E402
diff --git a/scripts/_common/timeline.py b/scripts/_common/timeline.py
index 219fe9f..d58e354 100644
--- a/scripts/_common/timeline.py
+++ b/scripts/_common/timeline.py
@@ -29,6 +29,7 @@ from typing import Any, Dict, List, Optional, Tuple
from xml.sax.saxutils import escape as _xml_escape
from _common.decision import MissingFpsError, aspect_ratio, frame_size, parse_time
+from _common.emit import warn
FORMATS = {".edl": "edl", ".fcpxml": "fcpxml", ".otio": "otio"}
# Common NTSC rates are stored as the exact ratio an editor expects, not as 29.97.
@@ -482,17 +483,32 @@ WRITERS = {"edl": to_edl, "fcpxml": to_fcpxml, "otio": to_otio}
def parses(path: str, fmt: str) -> bool:
- """The verification step: the written file reads back as what it claims to be."""
+ """The verification step: the written file reads back as what it claims to be. A file that
+ does not is recorded with the reason (`diagnostics.warnings`, `timeline_parse_failed`: the
+ exception, or what was read instead), beside the step's `ok: false`."""
+ why = parse_error(path, fmt)
+ if why:
+ warn(f"{path} does not read back as {fmt}: {why}", "timeline_parse_failed", path=path, format=fmt, error=why)
+ return why is None
+
+
+def parse_error(path: str, fmt: str) -> Optional[str]:
+ """Why `path` does not read back as `fmt`, or None when it does (parses() without the record)."""
try:
text = Path(path).read_text(encoding="utf-8")
if fmt == "fcpxml":
import xml.etree.ElementTree as ET
- return ET.fromstring(text.encode("utf-8")).tag == "fcpxml"
+ tag = ET.fromstring(text.encode("utf-8")).tag
+ return None if tag == "fcpxml" else f"the root element is <{tag}>, not <fcpxml>"
if fmt == "otio":
- return json.loads(text).get("OTIO_SCHEMA") == "Timeline.1"
- return text.startswith("TITLE:") and any(line[:3].isdigit() for line in text.splitlines())
- except (OSError, ValueError, SyntaxError):
- return False
+ doc = json.loads(text)
+ schema = doc.get("OTIO_SCHEMA") if isinstance(doc, dict) else None
+ return None if schema == "Timeline.1" else f"OTIO_SCHEMA is {schema!r}, not 'Timeline.1'"
+ if not text.startswith("TITLE:"):
+ return "the first line is not TITLE:"
+ return None if any(line[:3].isdigit() for line in text.splitlines()) else "no numbered event line"
+ except (OSError, ValueError, SyntaxError) as exc:
+ return f"{type(exc).__name__}: {exc}"
def summary(tl: Dict[str, Any], fmt: str) -> Dict[str, Any]:
diff --git a/scripts/_contract.py b/scripts/_contract.py
index 19c7304..c262442 100644
--- a/scripts/_contract.py
+++ b/scripts/_contract.py
@@ -425,6 +425,98 @@ ERROR_REASONS = {
"plan_unknown_tool": "render.py PLAN: the plan names no tool render.py can run; details: {plan, plan_sha256, checked_at, tool, audit_log}",
"plan_cwd_missing": "render.py PLAN: the directory the plan was made in is gone; details: {plan, plan_sha256, checked_at, cwd, audit_log}",
"plan_argv_invalid": "render.py PLAN: argv is not a list; details: {plan, plan_sha256, checked_at, argv_type, audit_log}",
+ "encoder_missing": "--codec av1/prores: the build's encoder list was read and has no encoder for the codec (kind missing_tool); details: {codec, encoders}",
+ "encoder_list_unreadable": "--codec av1/prores: the build's encoder list could not be read, so whether it has the encoder is unknown (kind ffmpeg); details: {codec, encoders, error}",
+}
+
+# The codes of the collector's records (docs/design-decisions.md, "Nothing happens silently"):
+# code -> (bucket, what it records). Every warn()/record_fallback()/record_skip() literal under
+# scripts/ is listed here under its bucket (a test reads the sources), so a caller can switch on
+# a code the contract publishes.
+DIAGNOSTIC_CODES = {
+ # infrastructure (part 1)
+ "stale_lock_taken_over": ("fallbacks", "a dead or past-deadline writer's output lock was taken over; {lock, output, pid, pid_alive, deadline, why}"),
+ "output_unlocked": ("warnings", "no output lock could be created and the run went on without one; {lock, output, error}"),
+ "lock_deadline_unset": ("warnings", "the lock's deadline (its mtime) could not be set; {lock, error}"),
+ "audit_log_unwritable": ("warnings", "an audit record could not be appended; never fails the run; {path, event, error}"),
+ "plan_unbound_inputs": ("warnings", "render.py PLAN: inputs that did not exist when the plan was made, so were not checked; {paths}"),
+ # the runner
+ "odd_dimensions_scaled_even": ("fallbacks", "an encode that failed on odd dimensions was run again scaled to even ones (the frame loses a row or column); {output, scale}"),
+ "measurement_input_pending": ("skipped", "--dry-run: a measurement skipped because its input is an earlier stage's output that does not exist yet; {path}"),
+ "non_av_streams_dropped": ("fallbacks", "the subtitle/data streams could not be copied, so the encode ran again without them (dropped_non_av_streams: true); {output, returncode, error: the first attempt's last stderr lines}"),
+ "ffmpeg_list_unreadable": ("warnings", "`ffmpeg -encoders` / `-bsfs` could not be read; what needs the list says so instead of 'the build lacks it'; {list, tool, error}"),
+ "timeout_env_invalid": ("warnings", "FFMPEG_SKILL_TIMEOUT was not a number (the default was used) or was negative (read as 0, no limit); {value, timeout}"),
+ # encoder decisions
+ "av1_encoder_libaom": ("fallbacks", "--codec av1 on a build without libsvtav1: libaom-av1 encoded instead; {wanted, used}"),
+ "crf_clamped": ("warnings", "the CRF handed to the encoder was above its maximum and was capped (the HEVC HDR line adds 2); {encoder, requested, asked, used}"),
+ # fonts and text
+ "font_substituted": ("fallbacks", "the asked font family is not installed: fontconfig or the Windows Fonts folder gave another (font_used names it); {asked, file, family, via}"),
+ "font_lacks_script": ("warnings", "an explicit --font does not cover the text's script and is kept as asked; {font, script}"),
+ "font_coverage_unverified": ("warnings", "whether a font covers the text's script could not be checked (no fontconfig / fc-scan); {script, fonts_dir?}"),
+ "fonts_dir_lacks_script": ("fallbacks", "no face in --fonts-dir covers the text's script, so one was picked by script; {fonts_dir, script}"),
+ "emoji_monochrome": ("warnings", "emoji drawn by the text font (monochrome at best): no colour path here"),
+ "emoji_stripped": ("warnings", "overlay.py --emoji none: emoji removed from the drawn text; {mode}"),
+ "emoji_assets_missing": ("warnings", "--emoji-assets has no PNG for these clusters; the text font draws them; {missing}"),
+ "drawtext_newline_glyph": ("warnings", "graphics.py --text-render drawtext on FFmpeg 8.0-8.0.1/8.1.0: a box is drawn at each line break of a multi-line label; {text_render}"),
+ "timeline_parse_failed": ("warnings", "render.py --export-timeline: the written file does not read back as its format (the parse step is ok: false); {path, format, error}"),
+ # captions
+ "ass_lines_unshifted": ("warnings", "caption.py --ass --offset: Dialogue lines whose timestamps could not be read were copied unshifted; {path, count, offset}"),
+ "ass_lines_malformed": ("warnings", "caption.py --ass: Dialogue lines that could not be read (fields or start/end) are not counted as cues; {path, count}"),
+ "cues_outside_video": ("warnings", "caption.py: cues wholly outside the video are not drawn; {count, duration}"),
+ "asr_language_assumed": ("warnings", "--transcribe: the language was assumed (English) and the transcript is wrong if it is not"),
+ # analysis
+ "drift_correction_skipped": ("skipped", "sync.py --fix-drift not applied for a source (drift_skipped_reason); {path, reason: too_short | low_confidence, ...}"),
+ "sync_low_confidence": ("warnings", "sync.py / multicam.py: a source's offset has a correlation confidence below 0.1; {path, confidence}"),
+ "cropdetect_windows_failed": ("warnings", "cropdetect.py: sampled windows ffmpeg could not decode (windows_failed); {failed, windows, starts}"),
+ "filler_words_untimed": ("warnings", "silence.py --filler: listed words in the transcript with no usable timing were not removed (filler.matched_untimed); {count}"),
+ "filler_removal_caution": ("warnings", "silence.py --filler: a removed word that is often an ordinary word, or a discourse marker added with --filler-extra (also in filler.warnings)"),
+ # media and defaults
+ "silent_track_mixed": ("warnings", "audio.py --on-silent warn: a silent track was mixed anyway; {tracks}"),
+ "silent_inputs": ("warnings", "join.py --on-silent warn: silent inputs joined anyway (silent); {count}"),
+ "silent_input": ("warnings", "waveform.py: the input audio is silent and the visualisation is a flat line; {path, peak_db}"),
+ "short_inputs": ("warnings", "join.py: inputs shorter than two frames joined anyway (short_segments); {count}"),
+ "duplicate_inputs": ("warnings", "join.py: an input listed more than once is joined each time (duplicates); {count}"),
+ "sequence_gap": ("warnings", "sequence.py: a frame is missing from the numbering and the frames after it are not in the video; {missing, frames_left_out, first_left_out}"),
+ "memory_heavy": ("warnings", "reverse.py / loop.py --boomerang hold every decoded frame in memory and this input is large"),
+ "dovi_not_detected": ("warnings", "color.py --strip-dovi on a stream with no Dolby Vision metadata: unregistered SEI removed anyway"),
+ "frame_from_export_preset": ("warnings", "render.py: frame.width/height taken from the export preset for a frame given only an aspect; {preset, width, height}"),
+ "normalize_on_by_default": ("warnings", "render.py: a platform export preset with no loudness stage got export.py --normalize by default; {preset}"),
+ # degraded or partial results (part 2)
+ "colour_not_converted": ("warnings", "export.py: an HDR or wide-gamut source written by a preset that tags SDR BT.709 without converting it; {preset, hdr_format}"),
+ "output_audio_silent": ("warnings", "export.py: the written audio is silent (-inf LUFS), so no gain meets the platform's loudness; {platform}"),
+ "loudness_outside_spec": ("warnings", "export.py: the written file's loudness is outside the platform's spec (normalized: after --normalize too); {platform, lufs, tp, normalized}"),
+ "camera_not_started": ("fallbacks", "multicam.py: a range asked for a camera that had not started; camera 0 played it ({start, end, requested_camera, camera, starts_at}); `cuts` names the camera rendered"),
+ "loudness_not_measured": ("warnings", "export.py: the written file's loudness could not be read, so the platform's loudness was not verified (loudness.ok null, a failing loudness verification step)"),
+ "loudness_no_audio": ("skipped", "export.py: a platform preset's loudness check, on a source with no audio"),
+ "report_section_missing": ("warnings", "report.py: a loudness, compliance or contact-sheet section could not be built (also a failing verification step naming the section)"),
+ "report_section_dry_run": ("skipped", "report.py --dry-run draws no contact sheet"),
+ "check_dry_run": ("skipped", "render.py --dry-run does not run a check stage or a plan's check step"),
+ "check_no_output": ("skipped", "render.py PLAN: the tool reported no output for the plan's check step"),
+ "check_nothing_to_check": ("skipped", "render.py: a \"check\" stage with neither \"platform\" nor \"content\": true"),
+ "check_output_unparsed": ("warnings", "render.py: check.py printed no JSON document; `check` is a failure document (error.reason check_output_unparsed)"),
+ "cache_entry_damaged": ("warnings", "render.py --cache: a sidecar that does not parse or an artifact of the wrong size (also an audit line in the cache directory); the stage runs again"),
+ "cache_store_failed": ("warnings", "render.py --cache: a stage's artifact could not be stored; the run is unaffected"),
+ "cache_input_unreadable": ("warnings", "render.py --cache: an input could not be fingerprinted and is keyed as \"missing\""),
+ "cache_ffmpeg_banner_unreadable": ("fallbacks", "render.py --cache: the ffprobe banner could not be read; keyed on major.minor, so builds of one release share entries"),
+ "cache_result_unrecorded": ("skipped", "render.py --cache: a hit from an entry that recorded no stage result, so what that stage verified is unknown"),
+ "recipe_project_unreadable": ("warnings", "batch.py: recipe.project could not be read; the cache key covers the recipe only"),
+ "batch_cache_reset": ("warnings", "batch.py: .ffskill_cache.json did not parse and was started empty (also an audit line)"),
+ "batch_cache_output_changed": ("warnings", "batch.py: a cached item's output was replaced or edited since the batch wrote it; the item runs again"),
+ "batch_cache_entry_unverifiable": ("warnings", "batch.py: a cache entry from before output identities; the item runs again"),
+}
+
+
+# `font_used` on the tools that draw text: which font the text was drawn with, so a run that
+# drew with a substitute says so (with a font_substituted fallback)
+FONT_USED_SCHEMA = {
+ "type": "object",
+ "description": "the font the text was drawn with: {asked, file, family, how, substituted, via?, fonts_dir?}. how: font_file (the "
+ "caller's file), script (picked to cover the text's script), resolved (the asked family resolved to a file by "
+ "fc-match or the Windows Fonts folder; for libass, the file fontconfig picks for the family), by_name (no file "
+ "resolved: ffmpeg/libass look the family up themselves, file null), fonts_dir (caption.py: the family looked up "
+ "in --fonts-dir first), ass_styles (caption.py --ass: the file's own styles name the fonts). substituted: the "
+ "file is not the asked family (also a diagnostics.fallbacks entry, font_substituted). overlay.py: with --text; "
+ "grid.py: with --label auto",
}
@@ -442,7 +534,7 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"hw": {"type": "object", "description": "present when VideoToolbox was asked for (--hw, $FFMPEG_SKILL_HW, a render.py stage, a batch.py item), on export.py when $FFMPEG_SKILL_HW=1 is set but its delivery preset does not take it, and when this ffmpeg build has no libx264/libx265 and an encode ran on VideoToolbox instead: {requested: whether this run's encodes were asked onto VideoToolbox (false for that export, and for the build without libx264/libx265), source: flag|env|auto (auto: nobody asked, the build has no libx264/libx265 and this Apple Silicon machine ran VideoToolbox in its place), used: true|false whether the encoder that ran last is VideoToolbox, null under --dry-run (nothing ran) and when this process ran no encode itself (batch.py: its stages are child processes; its top-level hw gathers the items' requested/source/fallback/notes and each results row carries its item's encoder and hw), fallback: a job VideoToolbox refused (encoder open or compression session failure; nothing else is retried) was re-encoded on the CPU, notes: why an encode stayed on or fell back to the CPU, and, when $FFMPEG_SKILL_HW chose the GPU and it ran, what that costs and that --no-hw opts out}. A VideoToolbox encode also adds a top-level notes line: its quality is not CRF-equivalent. A kind: ffmpeg failure under --hw / $FFMPEG_SKILL_HW carries encoder and hw too"},
"plan": {"type": "string", "description": "with --plan FILE: the plan document written (the run itself is a dry run)"},
"verified": {"type": "boolean", "description": "true only when the artifact was written, probed, and every self-check the tool ran (verification) met its target; false under --dry-run"},
- "verification": {"type": "array", "items": {"type": "object", "properties": {"step": {"type": "string"}, "ok": {"type": "boolean"}}}, "description": "what the tool itself verified: probe, plus loudness (loudness.py, export platform presets) or check (render)"},
+ "verification": {"type": "array", "items": {"type": "object", "properties": {"step": {"type": "string"}, "ok": {"type": "boolean"}, "stage": {"type": "string"}}}, "description": "what the tool itself verified: probe, plus loudness (loudness.py, export platform presets; ok false with reason not_measured when it could not be read), check (render), report.py's sections (loudness, compliance, contact_sheet) and batch.py's items (item). render.py also lists every stage's own steps beyond its artifact probe, tagged with `stage` (a cached stage's as recorded), so its `verified` is the conjunction of every stage's"},
"diagnostics": DIAGNOSTICS_SCHEMA}
extra: Dict[str, Any] = {}
if name == "check":
@@ -451,7 +543,7 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"checks": {"type": "array", "items": {"type": "object", "properties": {"check": {"type": "string", "description": "row name; --content adds black, frozen and silence; audio FAILs when the track is present but silent (peak <= -50 dBFS)"}, "status": {"enum": ["PASS", "WARN", "FAIL"]}, "value": {}, "expected": {}, "fix": {"type": "string"}, "kind": {"enum": ["format", "judgement"]}}}}}
elif name == "caption":
extra = {"transcription": {"type": "object", "description": "with --transcribe: {engine, model, language, routing (why auto chose it), detected_language when auto detected it}; when auto ran the English-only Parakeet model on English assumed, notes says so"},
- "caption": {"type": "object", "description": "cue layout: shifted / wrapped / rebalanced / split / extended / dropped counts, plus wrap ('phrase' or 'measured'), phrase_breaks (1.16), broken_inside_word -- atoms hard-sliced at the column edge because they did not fit alone even at the size floor (1.18.4) -- and overlong -- now residual: a single character alone wider than the column. Burn mode: cues_burned -- non-blank cues that overlap [0, duration] and are drawn -- and cues_outside -- non-blank cues wholly outside it (2.2.6); a burn with no visible cue is refused, kind input"},
+ "caption": {"type": "object", "description": "cue layout: shifted / wrapped / rebalanced / split / extended / dropped counts, plus wrap ('phrase' or 'measured'), phrase_breaks (1.16), broken_inside_word -- atoms hard-sliced at the column edge because they did not fit alone even at the size floor (1.18.4) -- and overlong -- now residual: a single character alone wider than the column. Burn mode: cues_burned -- non-blank cues that overlap [0, duration] and are drawn -- and cues_outside -- non-blank cues wholly outside it (2.2.6); a burn with no visible cue is refused, kind input. --ass: ass_malformed -- Dialogue lines that could not be read and are not counted -- and with --offset ass_shifted / ass_unshifted -- lines moved, and lines whose timestamps could not be read, copied as they were (each count above 0 is also a diagnostics.warnings entry)"},
"tracks": {"type": "array", "description": "--mode mux: one entry per subtitle stream in the output ({index, file, language, title, codec, default, cues, kept_from_input}); a stream the input already carried has file null and kept_from_input true (1.16)"},
"subtitle_tracks": {"type": "integer", "description": "--mode mux: how many subtitle streams the output carries"},
"emoji": {"type": "object", "description": "how the emoji in the text were drawn (mode, overlays, missing)"},
@@ -473,15 +565,18 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
elif name == "silence":
extra = {"silences": {"type": "array"}, "keep": {"type": "array"}, "input_duration": {"type": "number"}, "kept_duration": {"type": "number"}, "removed_seconds": {"type": "number"},
"speech_aware": {"type": "object", "description": "--speech-aware (1.18): {min_silence, floor, breaths_kept, breaths_kept_seconds, breaths}"},
- "filler": {"type": "object", "description": "--filler: {lang, source (whisper-json:FILE | whisper:ENGINE | parakeet:ENGINE), engine, transcription (with --transcribe: {engine, model, language, routing}), removed, removed_count, ...}"},
+ "filler": {"type": "object", "description": "--filler: {lang, source (whisper-json:FILE | whisper:ENGINE | parakeet:ENGINE), engine, transcription (with --transcribe: {engine, model, language, routing}), removed, removed_count, matched_untimed (listed words in the transcript with no usable timing, not removed), matched_too_long (listed words longer than a filler, kept), ...}"},
"notes": {"type": "array", "items": {"type": "string"}, "description": "--filler --transcribe under --engine auto: present when the English-only Parakeet model ran on English assumed (no --filler-lang, no language detected, no Whisper engine installed or none that succeeded); the transcript is wrong if the speech is not English"}}
elif name == "sync":
extra = {"reference": {"type": "string"}, "second": {"type": "string"}, "offset_seconds": {"type": "number"}, "confidence": {"type": "number"}, "meaning": {"type": "string"}, "drift": {"type": "object"},
- "sources": {"type": "array", "description": "1.18: [{path, offset_s, confidence, drift_ppm}], one per SOURCE; the only per-source shape once more than one SOURCE is given"}}
+ "sources": {"type": "array", "description": "1.18: [{path, offset_s, confidence, drift_ppm}], one per SOURCE; the only per-source shape once more than one SOURCE is given; with --fix-drift each also carries drift_skipped_reason"},
+ "drift_skipped_reason": {"type": ["string", "null"], "enum": ["too_short", "low_confidence", None], "description": "--fix-drift, one SOURCE: why the drift correction was not applied (too_short: no tail window far enough from the head; low_confidence: the tail did not correlate), null when it was; also a diagnostics.skipped entry, drift_correction_skipped"}}
elif name == "look":
extra = {"outputs": {"type": "array", "items": {"type": "string"}}}
elif name == "render":
- extra = {"stages": {"type": "array", "items": {"type": "string"}}, "check": {"type": ["object", "null"]},
+ extra = {"stages": {"type": "array", "items": {"type": "string"}}, "check": {"type": ["object", "null"], "description": "check.py's document for a \"check\" stage with a platform, \"content\": true, or both; when check.py printed none, a failure document with error.reason check_output_unparsed"},
+ "cache": {"type": ["object", "null"], "description": "--cache: {dir, ffmpeg, hits, misses, miss_reasons: [{stage, reason: not_cached|sidecar_missing|artifact_missing|sidecar_corrupt|artifact_truncated|no_output_path, ...}], saved_seconds, entries, would_hit (--dry-run)}"},
+ "notes": {"type": "array", "items": {"type": "string"}, "description": "each stage's own notes, prefixed with its tool (\"export: loudness ... is outside x's ...\")"},
"plan": {"type": "string", "description": "when the argument was a plan.json (written by <tool> --plan): its path"},
"plan_sha256": {"type": ["string", "null"], "description": "plan execution: sha256 of the plan file as executed"},
"unbound_inputs": {"type": "array", "items": {"type": "string"}, "description": "plan execution: inputs that did not exist when the plan was made, so carry no fingerprint and were not checked (also a diagnostics.warnings entry, code plan_unbound_inputs)"},
@@ -491,16 +586,19 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"transition_frames, music, markers, notes, and not_exported (what the project asks for that an "
"editor timeline cannot carry)"}}
elif name == "verify":
- extra = {"report": {"type": ["string", "null"]}, "files": {"type": "array"}, "failed": {"type": "integer"}, "total": {"type": "integer"}}
+ extra = {"report": {"type": ["string", "null"]},
+ "files": {"type": "array", "description": "one entry per file: {file, probe, steps, skipped}. A step is {step, ok, seconds, error (one line, for the table)}, plus the tool's verified when it completed, and on a failure its own kind, reason, message, details and exit_code (reason step_timeout or step_output_unparsed when the tool printed no failure document); skipped lists the steps that do not apply as {step, reason: quick|no_video|no_audio|not_hdr|not_multichannel|input_unreadable, message}"},
+ "failed": {"type": "integer"}, "total": {"type": "integer"}}
elif name == "batch":
- extra = {"results": {"type": "array", "description": "one row per input; a row whose recipe ran cut.py carries cut_reencoded (each cut.py call's reencoded) and cut_reencode_reasons (each reason those calls named, once per call)"},
+ extra = {"results": {"type": "array", "description": "one row per input, every input (an interrupted batch included): ok (every step completed), verified (every step also verified what it wrote; false under --dry-run) and unverified ([{step, tool, failed}] naming the steps that did not); a failed row's error {step, tool, kind, message, code, reason, details, hint, exit_code} from the failing step's own document; diagnostics from its steps (tagged with step); cache_miss {reason: output_missing|output_changed|entry_unverifiable, ...} when a cache entry was not served; skipped \"timeout\" or \"interrupted\" for an item that never started and interrupted true for one stopped while running. A row whose recipe ran cut.py carries cut_reencoded (each cut.py call's reencoded) and cut_reencode_reasons (each reason those calls named, once per call)"},
"processed": {"type": "integer"}, "total": {"type": "integer"},
"cut_stream_copy": {"type": ["object", "null"], "description": "cut.py calls across the folder: {calls, stream_copy, reencoded, stream_copy_rate}; null when no cut.py step ran"},
"cut_reencode_reasons": {"type": ["object", "null"], "description": "how many cut.py calls named each reencode_reason ({} when none re-encoded); a re-encoded call whose row came from a cache an earlier 2.x batch wrote, which recorded no reasons, counts as unknown; null when no cut.py step ran"}}
elif name == "report":
- extra = {"report": {"type": "string"}, "check": {"type": ["object", "null"]}}
+ extra = {"report": {"type": "string"}, "check": {"type": ["object", "null"], "description": "check.py's document for --platform; null when it was not asked for or could not run (a failing compliance verification step then says why)"}}
elif name == "export":
- extra = {"loudness": {"type": "object", "description": "platform presets with audio: the written file's lufs/tp against the platform's target_lufs/target_tp, ok true when inside the spec; normalized true when --normalize ran loudness.py on the file; silent true (lufs \"-inf\", ok false, no normalize) when the written audio is silent"},
+ extra = {"loudness": {"type": "object", "description": "platform presets with audio: the written file's lufs/tp against the platform's target_lufs/target_tp, ok true when inside the spec; normalized true when --normalize ran loudness.py on the file; silent true (lufs \"-inf\", ok false, no normalize) when the written audio is silent; measured false (lufs, tp and ok null, reason not_measured) when the measurement could not be read"},
+ "trimmed": {"type": "object", "description": "present when the source was longer than the preset's platform maximum and the export was cut to it (no --allow-long): {from, to} in seconds; notes says so too"},
"notes": {"type": "array", "items": {"type": "string"}}}
elif name == "loudness":
extra = {"measured": {"type": "object", "description": "the loudnorm measurement of the input (input_i, input_tp, input_lra, input_thresh, target_offset); with --measure-only it is the whole result"},
@@ -537,6 +635,9 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"lossless_alternative": {"type": ["string", "null"], "description": "when a lossless cut re-encoded because of the keyframe snap: the --start that would stream-copy instead, and how far it is from the request"},
"join_check": {"type": ["object", "null"], "description": "--segments video stream-copy join: the written file measured by demuxing, {packets, expected_packets, max_step_seconds, audio_offset_ms, audio_parts_checked, ok}. expected_packets counts the source's video packets from each part's start keyframe to its end (or the end of the video); max_step_seconds is the largest presentation step (null when vfr_check measured anything but sampled_cfr, e.g. a --vfr-guard off join of variable timing, which checks the count only), and each step at a join must be the step its parts predict within 1 ms; audio_offset_ms is, per part, its sound's shift minus its picture's (+ = sound late) from audio packets matched by CRC32 to the source stream shaped like the joined audio (FFmpeg's default selection may copy one other than a:0; an MPEG-TS source's AAC is hashed without its ADTS headers) (null for a part with no unique match, e.g. digital silence; null as a whole when the source has no audio or the video check failed), and audio_parts_checked counts the measured parts; ok is the conjunction (counts, steps, every measured part within 5 ms), and false means the join was re-cut from the source (concat_fallback) and this is the failed measurement. null for a single segment, audio, --dry-run, --accurate/--codec/a forced re-encode, a segment that re-encoded, or a join that fell back before a copy join was attempted (open GOPs, a later B-frame segment starting between keyframes, parts that differ, a concat error)"},
"segment_end_snap_seconds": {"type": ["array", "null"], "items": {"type": ["number", "null"]}, "description": "--segments stream-copy join: per segment, the signed seconds its end moved to the keyframe the copied part stops at (the first keyframe at or after the requested end, never an earlier one; null for a part that runs to the end of the video, a part of a source with no B-frames, which is cut to its requested end as in every 2.x release, or a Matroska/MPEG-TS part, which is not snapped). null as a whole unless the output is those copied parts: a single segment, audio, --dry-run, or any re-encode or fallback"}}
+ elif name == "multicam":
+ extra = {"cuts": {"type": "array", "description": "[[start, end, camera], ...] on the reference timeline: the camera actually rendered. A range whose camera had not started yet plays camera 0, and diagnostics.fallbacks names the camera asked for (code camera_not_started)"},
+ "offsets_seconds": {"type": "array"}, "confidence": {"type": "array"}}
elif name == "join":
extra = {"mode": {"enum": ["video", "audio"]}, "clips": {"type": "integer"}, "transition": {"type": "string"},
"expected_duration": {"type": ["number", "null"], "description": "clip lengths minus the transitions; null under --dry-run while an input is pending (its length is unknown)"},
@@ -555,6 +656,11 @@ def output_schema(name: str, meta: Dict[str, Any], hw: bool = False) -> Dict[str
"silent": {"type": "array", "description": "--on-silent warn (default), only when a track was silent: [{flag, path, peak_db}] for each --music/--replace/--effects file, or under --duck the voice (flag \"input\"), whose whole-file peak is at or below --silence-threshold (default -50 dBFS); it was mixed anyway. --on-silent fail refuses instead, reason \"silent (peak X dBFS)\" in problems"},
"notes": {"type": "array", "items": {"type": "string"}, "description": "present with `silent`: which tracks were silent and how to refuse them"},
"audio": {"type": "object", "description": "what the mix was built from: voice (null | light | medium | strong), stereo_widen, effects/effects_volume, and with --music the music_volume plus duck (null when --duck was not given, else the threshold in dB and linear, ratio, attack_ms, release_ms, amount_db actually used)"}}
+ if name in ("overlay", "graphics", "caption", "grid"):
+ extra["font_used"] = FONT_USED_SCHEMA
+ if name == "cropdetect":
+ extra.update({"windows": {"type": "integer", "description": "sampled windows (--samples)"},
+ "windows_failed": {"type": "integer", "description": "windows ffmpeg could not decode and that were skipped (a cropdetect_windows_failed warning when > 0); every window failing is a kind ffmpeg failure"}})
props = dict(base)
props.update(extra)
notes = props.get("notes")
@@ -1733,13 +1839,17 @@ def build(detect: bool = True) -> Dict[str, Any]:
"details": "optional, with reason: an object of the facts behind the refusal (paths, expected and actual values)"},
"diagnostics": DIAGNOSTICS_SCHEMA["description"]},
"error_reasons": ERROR_REASONS,
+ "diagnostic_codes": {code: {"bucket": bucket, "meaning": meaning} for code, (bucket, meaning) in DIAGNOSTIC_CODES.items()},
"audit_log": {"format": "JSON lines, appended: {ts, tool, event, pid, plan_sha256 (plan events), details}",
"events": {"plan_rejected": "render.py refused a plan (details: the refusal's error.details plus reason)",
"lock_takeover": "a stale output lock was taken over (details: lock, output, pid, pid_alive, deadline, why)",
"output_unlocked": "no output lock could be created and the run went on without one",
- "output_is_input_refused": "the output-is-input refusal (only when an audit log was named)"},
- "location": "render.py --audit-log FILE when given; otherwise a refused plan's <plan>.audit.jsonl, and a lock event's "
- "hidden .<output>.ffskill-audit.jsonl beside the output. Never written under --dry-run; a write that "
+ "output_is_input_refused": "the output-is-input refusal (only when an audit log was named)",
+ "cache_entry_damaged": "render.py --cache: a damaged entry (details: stage, reason, key, sidecar or artifact, sizes)",
+ "batch_cache_reset": "batch.py: .ffskill_cache.json did not parse and was started empty (details: path, size, error)"},
+ "location": "render.py --audit-log FILE when given; otherwise a refused plan's <plan>.audit.jsonl, a lock event's "
+ "hidden .<output>.ffskill-audit.jsonl beside the output, and a cache event's .ffskill-audit.jsonl in the "
+ "cache directory (render.py --cache DIR) or the batch's output_dir. Never written under --dry-run; a write that "
"fails is a diagnostics.warnings entry (audit_log_unwritable), never a failure"},
"error_kinds": {"input": "missing or unsuitable input, bad arguments", "ffmpeg": "ffmpeg/ffprobe returned an error (message carries the last stderr lines)", "output": "ffmpeg exited 0 but the artifact is missing, empty or unreadable (an empty file is removed)", "missing_tool": "ffmpeg or ffprobe not on PATH", "timeout": "one ffmpeg/ffprobe run exceeded --timeout (default 1800 s, FFMPEG_SKILL_TIMEOUT) and was killed; partial output removed; exit 124", "verification": "the tool ran but its result failed the requested check: check.py platform rows (checks attached), render.py's check stage (output written, check attached), batch.py items (results attached), verify.py steps (files attached), color.py --retag (output written, but ffprobe does not read the requested colour tags on it); exit 1", "interrupted": "the tool received SIGINT or SIGTERM: the running ffmpeg (or sibling script) was stopped and its partial output removed; exit 130 or 143"},
"success_criterion": "exit 0 AND the output exists AND is non-empty AND ffprobe reads a stream from it; only then is status completed printed and the output probe attached",
diff --git a/scripts/audio.py b/scripts/audio.py
index 1aac190..9734966 100755
--- a/scripts/audio.py
+++ b/scripts/audio.py
@@ -26,6 +26,7 @@ from pathlib import Path
from typing import Any, Dict, List
from _common import STATE, add_common, dry_run_input_pending, require_tool, apply_common, audio_codec_for, db_to_linear, default_output, die, emit, ffmpeg_base, info, is_audio_output, measured_level_dbfs, probe, run, run_keeping_subtitles, fmt_secs
+from _common.emit import warn
VOICE_CHAIN = "highpass=f=80,deesser=i=0.4,afftdn=nf=-25:tn=1,acompressor=threshold=-18dB:ratio=3:attack=5:release=80:makeup=2"
@@ -261,7 +262,7 @@ def main() -> int:
notes.append("silent track(s) mixed anyway: " + "; ".join(
f"{t['flag']} {t['path']} peaks at {t['peak_db']:g} dBFS" for t in silent)
+ f" (at or below --silence-threshold {args.silence_threshold:g}); pass --on-silent fail to refuse instead")
- info("warning: " + notes[-1])
+ warn(notes[-1], "silent_track_mixed", tracks=[t["path"] for t in silent])
meta = probe(args.input)
dur = meta.get("duration") or 0.0
diff --git a/scripts/batch.py b/scripts/batch.py
index 81117e2..236330e 100755
--- a/scripts/batch.py
+++ b/scripts/batch.py
@@ -32,9 +32,9 @@ import time
from pathlib import Path
from typing import Any, Dict, List, Optional
-from _common import STATE, add_common, apply_common, child_args, die, emit, info, run_tool, read_text_or_die, MEDIA_EXT as _MEDIA_EXT
-from _common.runner import HW_FORCED_ENV, add_hw_orchestrator_args, env_hw
-from _common.emit import absorb_stage_hw, steps_encoder_report
+from _common import STATE, add_common, apply_common, child_args, die, emit, info, run_tool, MEDIA_EXT as _MEDIA_EXT
+from _common.runner import HW_FORCED_ENV, add_hw_orchestrator_args, env_hw, stop_children
+from _common.emit import absorb_stage_hw, audit, note, steps_encoder_report, warn
HERE = Path(__file__).resolve().parent
MEDIA_EXT = {e for e in _MEDIA_EXT if e not in (".png", ".jpg", ".jpeg", ".webp")} # one list (_common); a batch walks media, not stills
@@ -70,8 +70,11 @@ def recipe_key(recipe: Dict[str, Any]) -> str:
if recipe.get("project"):
try:
project_content = Path(recipe["project"]).read_text(encoding="utf-8")
- except OSError:
- pass
+ except (OSError, UnicodeDecodeError) as exc:
+ # keyed on the recipe alone: an unreadable project cannot invalidate the cache by its
+ # content, so say so (each item that needs it then fails on reading it)
+ warn(f"recipe.project {recipe['project']} could not be read ({exc}); the cache key covers the recipe only",
+ "recipe_project_unreadable", path=str(recipe["project"]), error=str(exc))
return hashlib.sha1((json.dumps(recipe, sort_keys=True) + "\0" + project_content).encode()).hexdigest()[:12]
@@ -90,6 +93,8 @@ def gpu_key() -> str:
JOBS_CAP = 8 # beyond this, concurrent encodes contend for the same cores and memory
_LOG = threading.local()
+# set by batch.py's signal handler: a --jobs worker between two steps must not start the next one
+_STOP = threading.Event()
def log(message: str) -> None:
@@ -102,16 +107,46 @@ def log(message: str) -> None:
buf.append(message)
+class ItemError(Exception):
+ """An item that cannot be processed, with the structured `error` its results row carries.
+ process() raises it instead of die(): die() inside an item printed a second JSON document
+ and left the row with error "1" (str(SystemExit))."""
+
+ def __init__(self, error: Dict[str, Any]) -> None:
+ super().__init__(error.get("message"))
+ self.error = error
+
+
+def step_error(index: Optional[int], tool: Optional[str], doc: Any) -> Dict[str, Any]:
+ """A failed step's own failure, for its item's row: {step, tool, kind, message, code, reason,
+ details, hint, exit_code} from the step's --json document (keys it did not report are left
+ out; step and tool are always there)."""
+ err = doc.get("error") if isinstance(doc, dict) and isinstance(doc.get("error"), dict) else {}
+ out: Dict[str, Any] = {"step": index, "tool": tool}
+ for k in ("kind", "message", "code", "reason", "details", "hint"):
+ if err.get(k) is not None:
+ out[k] = err[k]
+ if isinstance(doc, dict) and doc.get("exit_code") is not None:
+ out["exit_code"] = doc["exit_code"]
+ return out
+
+
def run_step(argv: List[str], per_call: "Optional[float]" = None) -> "tuple[bool, Optional[dict]]":
- """Returns (ok, doc): doc is the step's own --json result document, or None if it couldn't
- be parsed. {in}/{out} are already resolved by the caller before argv reaches here, so forcing
+ """Returns (ok, doc): doc is the step's own --json result document; a failed step that
+ printed none that parses gets one in the failure shape (error.reason step_output_unparsed,
+ the last stderr line as the message), so its row still says why. {in}/{out} are already resolved by the caller before argv reaches here, so forcing
--json on every step (harmless: no step reads another step's stdout, only its output path)
costs nothing and lets a caller like process() read back tool-specific fields, e.g. cut.py's
`reencoded`, without adding a bespoke flag per field it might someday want."""
script = argv[0]
if script not in ALLOWED_STEP_SCRIPTS:
- die(f"recipe step names a script that isn't one of this skill's own tools: {script!r} "
- f"(must be a bare filename like 'silence.py', found in scripts/)")
+ # main() refuses such a recipe before any item runs; this stays as the last line of defence
+ raise ItemError({"step": None, "tool": script, "kind": "input", "reason": "step_script_not_allowed",
+ "message": f"recipe step names a script that isn't one of this skill's own tools: {script!r} "
+ "(must be a bare filename like 'silence.py', found in scripts/)"})
+ if _STOP.is_set():
+ return False, {"status": "failed", "exit_code": 130,
+ "error": {"kind": "interrupted", "message": f"interrupted before {script} started"}}
cmd = [str(HERE / script)] + argv[1:] + child_args() + (["--json"] if "--json" not in argv[1:] else [])
log(" → " + " ".join(os.path.basename(c) if i < 1 else c for i, c in enumerate(cmd)))
proc = run_tool(cmd, per_call=per_call)
@@ -121,6 +156,10 @@ def run_step(argv: List[str], per_call: "Optional[float]" = None) -> "tuple[bool
doc = None
if proc.returncode != 0:
log(" " + "\n ".join(proc.stderr.strip().splitlines()[-4:]))
+ if not isinstance(doc, dict):
+ doc = {"status": "failed", "exit_code": proc.returncode,
+ "error": {"reason": "step_output_unparsed",
+ "message": (proc.stderr.strip().splitlines() or [f"{script} exited {proc.returncode} and printed nothing"])[-1][:300]}}
return False, doc
for line in proc.stderr.splitlines():
if line.startswith("warning:"): # a step's deprecation notice is not swallowed by a success (review 9)
@@ -159,9 +198,13 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
return max(1.0, deadline - time.monotonic())
if recipe.get("project"):
try:
- proj = json.loads(read_text_or_die(str(recipe["project"]), "recipe.project"))
- except ValueError as e:
- die(f"recipe.project: {recipe['project']} is not valid JSON: {e}")
+ proj = json.loads(Path(str(recipe["project"])).read_text(encoding="utf-8"))
+ except (OSError, ValueError) as e:
+ raise ItemError({"step": None, "tool": "render.py", "kind": "input", "reason": "recipe_project_unreadable",
+ "message": f"recipe.project: {recipe['project']} could not be read as JSON: {e}"})
+ if not isinstance(proj, dict):
+ raise ItemError({"step": None, "tool": "render.py", "kind": "input", "reason": "recipe_project_unreadable",
+ "message": f"recipe.project: {recipe['project']} is not a JSON object"})
idx = int(recipe.get("clip_key", 0))
proj.setdefault("clips", [{}])
while len(proj["clips"]) <= idx:
@@ -172,14 +215,17 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
pj.write_text(json.dumps(proj, indent=2), encoding="utf-8")
ok, _doc = run_step(["render.py", str(pj)], budget())
step_docs = [("render", _doc)]
+ failure = None if ok else step_error(0, "render.py", _doc)
cut_reencoded: List[bool] = []
cut_reasons: List[str] = []
else:
steps = recipe.get("steps") or []
if not steps:
- die("recipe needs steps or project")
+ raise ItemError({"step": None, "tool": None, "kind": "input", "reason": "recipe_has_no_steps",
+ "message": "recipe needs steps or project"})
cur = str(src)
ok = True
+ failure = None
cut_reencoded = []
step_docs = []
cut_reasons = []
@@ -198,9 +244,30 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
cut_reasons.extend(r for r in (doc.get("reencode_reason") or []) if isinstance(r, str))
if not step_ok:
ok = False
+ failure = step_error(i, str(argv[0]) if argv else None, doc)
break
cur = out
result: Dict[str, Any] = {"file": str(src), "output": str(final), "ok": ok, "seconds": round(time.time() - t0, 1)}
+ if failure:
+ result["error"] = failure
+ # `ok` is "every step completed" (design-decisions: a spec miss a tool cannot fix is completed +
+ # verified false, not a failure); `verified` is whether every step also verified what it wrote,
+ # and `unverified` names the steps that did not and which of their checks failed
+ unverified = []
+ for idx, (name, doc) in enumerate(step_docs):
+ if ok and isinstance(doc, dict) and doc.get("verified") is not True:
+ unverified.append({"step": idx, "tool": name,
+ "failed": [str(v.get("step")) + (f" ({v['stage']})" if v.get("stage") else "")
+ for v in doc.get("verification") or [] if isinstance(v, dict) and not v.get("ok")]})
+ result["verified"] = bool(ok and not STATE.dry_run and step_docs and not unverified)
+ if unverified and not STATE.dry_run:
+ result["unverified"] = unverified
+ diag: Dict[str, List[Any]] = {}
+ for name, doc in step_docs:
+ for bucket, entries in ((doc or {}).get("diagnostics") or {}).items() if isinstance(doc, dict) else []:
+ diag.setdefault(bucket, []).extend(dict(e, step=name) for e in entries if isinstance(e, dict))
+ if diag:
+ result["diagnostics"] = diag
if cut_reencoded:
result["cut_reencoded"] = cut_reencoded
result["cut_reencode_reasons"] = cut_reasons
@@ -210,6 +277,34 @@ def process(src: Path, recipe: Dict[str, Any], outdir: Path, work: Path,
return result
+def output_identity(path: str) -> "Optional[Dict[str, int]]":
+ """What a cache entry remembers of the output it served: size and mtime_ns (a stat, no read)."""
+ try:
+ st = os.stat(path)
+ except OSError:
+ return None
+ return {"size": st.st_size, "mtime_ns": st.st_mtime_ns}
+
+
+def cached_output_changed(hit: Dict[str, Any]) -> "Optional[Dict[str, Any]]":
+ """None when the entry's output is still the file this batch wrote; else why it is not served:
+ output_missing, output_changed (size or mtime differ: something replaced or edited it) or
+ entry_unverifiable (an entry written by 2.10.11 or earlier recorded no identity, so nothing can say
+ the file is the one it describes)."""
+ out = str(hit.get("output") or "")
+ now = output_identity(out) if out else None
+ if now is None:
+ return {"reason": "output_missing", "output": out, "message": "the output is gone"}
+ expected = hit.get("output_identity")
+ if not isinstance(expected, dict):
+ return {"reason": "entry_unverifiable", "output": out,
+ "message": "the cache entry predates output identities, so the file cannot be checked"}
+ if expected != now:
+ return {"reason": "output_changed", "output": out, "expected": expected, "actual": now,
+ "message": "the output changed since the batch wrote it"}
+ return None
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("folder")
@@ -235,8 +330,20 @@ def main() -> int:
recipe = json.loads(Path(args.recipe).read_text(encoding="utf-8"))
except (OSError, ValueError) as exc:
die(f"cannot read recipe: {exc}")
+ if not isinstance(recipe, dict):
+ die("cannot read recipe: not a JSON object")
if recipe.get("project") and not os.path.isabs(recipe["project"]):
recipe["project"] = str((Path(args.recipe).resolve().parent / recipe["project"]))
+ # a recipe error is the same for every item: refuse it once, before any item runs, rather than
+ # failing each item with it
+ if not recipe.get("project"):
+ if not recipe.get("steps"):
+ die("recipe needs steps or project")
+ for st in recipe["steps"]:
+ name = st[0] if isinstance(st, list) and st else st
+ if not isinstance(name, str) or name not in ALLOWED_STEP_SCRIPTS:
+ die(f"recipe step names a script that isn't one of this skill's own tools: {name!r} "
+ f"(must be a bare filename like 'silence.py', found in scripts/)")
outdir = Path(recipe.get("output_dir") or (folder / "out"))
if not outdir.is_absolute():
outdir = folder / outdir
@@ -263,7 +370,15 @@ def main() -> int:
if cache_path.exists() and not args.force:
try:
cache = json.loads(cache_path.read_text(encoding="utf-8"))
- except ValueError:
+ if not isinstance(cache, dict):
+ raise ValueError("not a JSON object")
+ except (OSError, ValueError) as exc:
+ # every entry is lost: each item runs again, and the next store rewrites the file
+ size = cache_path.stat().st_size if cache_path.exists() else None
+ warn(f"the batch cache {cache_path} could not be read ({exc}); starting with an empty cache, so every item runs again",
+ "batch_cache_reset", path=str(cache_path), size=size, error=str(exc))
+ audit("batch_cache_reset", {"path": str(cache_path), "size": size, "error": str(exc)},
+ default_path=str(outdir / ".ffskill-audit.jsonl"))
cache = {}
rkey = recipe_key(recipe) + gpu_key()
glob = recipe.get("glob") or "*"
@@ -294,10 +409,27 @@ def main() -> int:
deadline = time.monotonic() + STATE.timeout if (shared_budget and STATE.timeout) else None
cache_lock = threading.Lock()
timed_out = {"hit": False}
- interrupted = {"hit": False}
+ interrupted: Dict[str, Any] = {"hit": False, "signal": None}
+ _STOP.clear()
+ # Ctrl-C (and SIGTERM where it is delivered) stops the running steps and reports the table:
+ # the shared handler would exit with "nothing was written" and no rows, and inside an item it
+ # surfaced as that item's failure while the batch went on (spec 4.1)
+ import signal as _signal
+
+ def _interrupt(signum: int, frame: Any) -> None:
+ interrupted["signal"] = signum
+ _STOP.set()
+ stop_children()
+ raise KeyboardInterrupt
+ if threading.current_thread() is threading.main_thread():
+ for sig in (getattr(_signal, "SIGINT", None), getattr(_signal, "SIGTERM", None)):
+ if sig is not None:
+ try:
+ _signal.signal(sig, _interrupt)
+ except (ValueError, OSError):
+ pass
def one_pass() -> List[Dict[str, Any]]:
- results = []
files = sorted(p for p in folder.glob(glob) if p.is_file() and p.suffix.lower() in MEDIA_EXT and outdir not in p.parents)
# Two different sources can compute the same final path (most often a fixed recipe "ext"
# collapsing e.g. clip.mp4 and clip.mov to the same clip_out.mp4) -- catch that before
@@ -332,7 +464,8 @@ def main() -> int:
# the other GPU setting) that names the same file now describes bytes that are gone
for stale in [k for k, v in cache.items() if k != key and isinstance(v, dict) and v.get("output") == r["output"]]:
del cache[stale]
- cache[key] = r
+ # the output as this run left it: a later hit is served only while it is unchanged
+ cache[key] = dict(r, output_identity=output_identity(r["output"]))
# write_text isn't atomic -- a process killed mid-write (or a --watch loop racing
# a concurrent manual run) could leave a truncated file that json.loads() above
# then silently treats as "no cache" (a ValueError -> {}), discarding every prior
@@ -349,13 +482,23 @@ def main() -> int:
# warm, which happens at --jobs 1 too and contradicts the order this tool promises.
slots: "List[Optional[Dict[str, Any]]]" = [None] * len(files)
pending: List[tuple] = []
+ cache_miss: Dict[int, Dict[str, Any]] = {}
for i, src in enumerate(files):
key = f"{file_key(src)}:{rkey}"
hit = cache.get(key)
- if hit and Path(hit.get("output", "")).exists() and not args.force:
- info(f"skip (cached) {src.name}")
- slots[i] = {**hit, "cached": True}
- continue
+ if isinstance(hit, dict) and not args.force:
+ why = cached_output_changed(hit)
+ if why is None:
+ info(f"skip (cached) {src.name}")
+ slots[i] = {**{k: v for k, v in hit.items() if k != "output_identity"}, "cached": True}
+ continue
+ cache_miss[i] = why
+ facts = {k: v for k, v in why.items() if k not in ("reason", "message")}
+ msg = f"{src.name}: the cached output {hit.get('output')} is not served ({why['message']}); the item runs again"
+ if why["reason"] == "output_changed":
+ warn(msg, "batch_cache_output_changed", file=str(src), **facts)
+ elif why["reason"] == "entry_unverifiable":
+ warn(msg, "batch_cache_entry_unverifiable", file=str(src), **facts)
pending.append((i, src, key))
def timed_out_row(src: Path) -> "Dict[str, Any]":
@@ -367,13 +510,41 @@ def main() -> int:
"""A worker that raised is a failed item, not a dead run. A die() inside a thread
raises SystemExit through fut.result() and used to take the whole process down
before the summary and the per-item table were printed -- so the one thing the user
- needed, which item failed and which succeeded, was the thing they did not get."""
- reason = str(exc) or exc.__class__.__name__
+ needed, which item failed and which succeeded, was the thing they did not get.
+ `error` is structured: an ItemError's own, else what the exception says (its class
+ in `exception`), never str(SystemExit) -- which was the exit code, "1"."""
+ if isinstance(exc, ItemError):
+ error = dict(exc.error)
+ elif isinstance(exc, SystemExit):
+ error = {"step": None, "tool": None, "exception": "SystemExit", "exit_code": exc.code,
+ "message": f"the item stopped with exit code {exc.code}; its message is on stderr"}
+ else:
+ error = {"step": None, "tool": None, "exception": exc.__class__.__name__,
+ "message": (str(exc) or exc.__class__.__name__)[:300]}
+ log(f" failed: {error['message']}")
return {"file": str(src), "output": str(final_path(src, recipe, outdir)),
- "ok": False, "seconds": 0.0, "error": reason[:300]}
+ "ok": False, "verified": False, "seconds": 0.0, "error": error}
+
+ def interrupted_row(src: Path, started: bool, r: "Optional[Dict[str, Any]]" = None) -> "Dict[str, Any]":
+ """An item the interrupt reached: `interrupted: true` when it was running (its steps
+ were stopped), `skipped: "interrupted"` when it had not started."""
+ row = dict(r) if r else {"file": str(src), "output": str(final_path(src, recipe, outdir)), "seconds": 0.0}
+ row.update(ok=False, verified=False)
+ if started:
+ row["interrupted"] = True
+ row.setdefault("error", {"step": None, "tool": None, "kind": "interrupted",
+ "message": "interrupted while this item was running; its running step was stopped"})
+ else:
+ row["skipped"] = "interrupted"
+ return row
+
+ def with_miss(i: int, r: Dict[str, Any]) -> Dict[str, Any]:
+ if i in cache_miss:
+ r["cache_miss"] = {k: v for k, v in cache_miss[i].items() if k != "message"}
+ return r
if jobs == 1 or len(pending) < 2:
- for i, src, key in pending:
+ for n, (i, src, key) in enumerate(pending):
if deadline is not None and time.monotonic() >= deadline:
slots[i] = timed_out_row(src)
continue
@@ -382,13 +553,17 @@ def main() -> int:
r = process(src, recipe, outdir, item_work(i, src), deadline)
except KeyboardInterrupt:
interrupted["hit"] = True
+ # the running item, then every one not started: each keeps its row
+ slots[i] = interrupted_row(src, True)
+ for j, later, _k in pending[n + 1:]:
+ slots[j] = interrupted_row(later, False)
break
except BaseException as exc: # noqa: BLE001 - reported, never swallowed
if isinstance(exc, SystemExit) and not exc.code:
raise
- slots[i] = failed_row(src, exc)
+ slots[i] = with_miss(i, failed_row(src, exc))
continue
- slots[i] = r
+ slots[i] = with_miss(i, r)
store(key, r)
return [r for r in slots if r is not None]
@@ -398,8 +573,9 @@ def main() -> int:
_LOG.buffer = [f"=== {src.name}"]
try:
r = process(src, recipe, outdir, item_work(i, src), deadline)
- store(key, r)
- return r
+ if not interrupted["hit"]: # a step the interrupt stopped is no result to keep
+ store(key, r)
+ return with_miss(i, r)
finally:
r_lines, _LOG.buffer = _LOG.buffer, None
setattr(_LOG, "lines", r_lines)
@@ -439,86 +615,34 @@ def main() -> int:
for i, src, key in queue:
slots[i] = timed_out_row(src)
except KeyboardInterrupt:
- # Spec 4.1: cancel what has not started, let the running children be killed by
- # the shared signal handling, and REPORT what completed -- exit 130 with the
- # partial table, never a traceback.
+ # Spec 4.1: cancel what has not started, stop the running steps (the handler
+ # already did), and REPORT every item -- exit 130 with the table, never a
+ # traceback. The pool's exit waits for the stopped items; they are read below.
interrupted["hit"] = True
for fut in futures:
fut.cancel()
- for fut, (i, src) in futures.items():
- if slots[i] is not None or not fut.done():
- continue
- try:
- slots[i], lines[i] = fut.result()
- except BaseException: # noqa: BLE001
- pass
info("interrupted: reporting what had already finished")
+ if interrupted["hit"]:
+ for fut, (i, src) in futures.items():
+ if slots[i] is not None:
+ continue
+ if fut.cancelled():
+ slots[i] = interrupted_row(src, False)
+ continue
+ try:
+ r, lines[i] = fut.result()
+ except BaseException: # noqa: BLE001 - its row says it was interrupted
+ r = None
+ # finished in time: kept as it is; stopped by the interrupt: said so
+ slots[i] = r if (r and r.get("ok")) else interrupted_row(src, True, r)
+ for i, src, _key in queue:
+ if slots[i] is None:
+ slots[i] = interrupted_row(src, False)
for i, src in enumerate(files):
for line in lines.get(i, []):
info(line)
return [r for r in slots if r is not None]
- import concurrent.futures
-
- def work_one(i: int, src: Path, key: str) -> Dict[str, Any]:
- _LOG.buffer = [f"=== {src.name}"]
- try:
- r = process(src, recipe, outdir, item_work(i, src), deadline)
- store(key, r)
- return r
- finally:
- r_lines, _LOG.buffer = _LOG.buffer, None
- setattr(_LOG, "lines", r_lines)
-
- # Submitted in the existing sorted order and collected into a list indexed by submission
- # order, so the summary and the per-item table are identical to a serial run's whatever
- # order the encodes actually finish in.
- ordered: List[Optional[Dict[str, Any]]] = [None] * len(pending)
- lines: List[List[str]] = [[] for _ in pending]
- with concurrent.futures.ThreadPoolExecutor(max_workers=jobs) as pool:
- futures: "Dict[Any, int]" = {}
- queue = list(enumerate(pending))
- in_flight: "set" = set()
- try:
- # The pool is topped up to `jobs` in flight and no further: submitting the whole
- # list up front would put every item past the deadline check before the first one
- # had finished, and the shared budget could then never stop anything.
- while queue or in_flight:
- while queue and len(in_flight) < jobs:
- slot, (i, src, key) = queue[0]
- if deadline is not None and time.monotonic() >= deadline:
- break
- queue.pop(0)
- fut = pool.submit(_run_buffered, work_one, i, src, key)
- futures[fut] = slot
- in_flight.add(fut)
- if not in_flight:
- break
- done_now, in_flight = concurrent.futures.wait(
- in_flight, return_when=concurrent.futures.FIRST_COMPLETED)
- in_flight = set(in_flight)
- for fut in done_now:
- ordered[futures[fut]], lines[futures[fut]] = fut.result()
- for slot, (i, src, key) in queue:
- timed_out["hit"] = True
- ordered[slot] = {"file": str(src),
- "output": str(final_path(src, recipe, outdir)),
- "ok": False, "seconds": 0.0, "skipped": "timeout"}
- except KeyboardInterrupt:
- for fut in futures:
- fut.cancel()
- info("interrupted: finishing what had already started")
- raise
- for slot, (i, src, key) in enumerate(pending):
- for line in lines[slot]:
- info(line)
- if ordered[slot] is None:
- timed_out["hit"] = True
- ordered[slot] = {"file": str(src), "output": str(final_path(src, recipe, outdir)),
- "ok": False, "seconds": 0.0, "skipped": "timeout"}
- results.append(ordered[slot])
- return results
-
results = one_pass()
if args.watch:
info(f"watching {folder} every {args.watch:g}s (Ctrl-C to stop)")
@@ -534,6 +658,8 @@ def main() -> int:
finally:
signal.signal(signal.SIGINT, previous)
results = one_pass()
+ if interrupted["hit"]:
+ break # the pass reported its rows; a watch interrupted mid-pass stops here
except KeyboardInterrupt:
info("watch stopped")
# the items' GPU facts (asked for, fell back and why) reach the top-level `hw`, as render.py's
@@ -547,10 +673,13 @@ def main() -> int:
print(f"{'OK ' if r['ok'] else 'FAIL'} {r['file']} -> {r['output']}" + (" (cached)" if r.get("cached") else ""))
if interrupted["hit"]:
# Ctrl-C on a batch that already produced files: the user needs the partial table, not a
- # traceback and not "nothing was written". Exit 130 with everything that completed.
- die(f"interrupted after {done} of {len(results)} item(s); the finished outputs are kept "
- "and the rest were not started",
- code=130, kind="interrupted", output=None, dry_run=STATE.dry_run, results=results,
+ # traceback and not "nothing was written". Exit 130 (143 for SIGTERM) with every item:
+ # finished, interrupted while running, or not started.
+ stopped = sum(1 for r in results if r.get("interrupted"))
+ not_started = sum(1 for r in results if r.get("skipped") == "interrupted")
+ die(f"interrupted after {done} of {len(results)} item(s): {stopped} stopped while running and "
+ f"{not_started} not started; the finished outputs are kept",
+ code=128 + int(interrupted["signal"] or 2), kind="interrupted", output=None, dry_run=STATE.dry_run, results=results,
processed=done, total=len(results), jobs=jobs, jobs_requested=requested,
timed_out=timed_out["hit"])
if timed_out["hit"]:
@@ -593,7 +722,13 @@ def main() -> int:
why = (" (reencode_reason: " + ", ".join(f"{k} x{v}" for k, v in reasons.items()) + ")") if reasons else ""
info(f"cut.py: {copied}/{len(all_cuts)} call(s) stayed fully lossless stream-copy "
f"({cut_summary['stream_copy_rate']:.0%}), {len(all_cuts) - copied} re-encoded a range or its --segments join{why}")
- emit(None, results=results, processed=done, total=len(results),
+ unverified = [os.path.basename(r["file"]) for r in results if r.get("ok") and not r.get("verified")]
+ if unverified and not STATE.dry_run:
+ note(f"{len(unverified)} item(s) completed but did not verify everything they wrote (see each row's unverified): "
+ + ", ".join(unverified[:5]) + (" ..." if len(unverified) > 5 else ""))
+ emit(None, verification=[{"step": "item", "file": r["file"], "ok": bool(r.get("verified"))} for r in results]
+ if not STATE.dry_run else [],
+ results=results, processed=done, total=len(results),
jobs=jobs, jobs_requested=requested, wall_seconds=round(time.time() - started, 1),
item_seconds_total=round(sum(float(r.get("seconds") or 0) for r in results), 1),
timed_out=timed_out["hit"], cut_stream_copy=cut_summary, cut_reencode_reasons=reasons)
diff --git a/scripts/caption.py b/scripts/caption.py
index f620706..d8028aa 100755
--- a/scripts/caption.py
+++ b/scripts/caption.py
@@ -51,6 +51,8 @@ from _common import emoji_filter_chain, EMOJI_ASSET_HINT, emoji_asset_for, emoji
# caption.parse_srt / caption.transcribe / caption.whisper_word_timings before 1.17 still does.
from _common import asr as _asr
from _common.asr import ENGINE_CHOICES
+from _common.emit import warn
+from _common.fonts import font_choice, record_font_choice
from _common import (ASR_INSTALL_HINT, die_no_engine, parse_srt, transcribe, whisper_word_timings,
write_srt)
from _common import (SAFE_WIDTH_FRACTION, ORPHAN_MIN_EM, WRAP_MODES, wrap_text, wrap_variants, best_break,
@@ -219,8 +221,8 @@ def plan_emoji(cues, args, play_w, play_h, brand=None):
return out, plan
if mode in ("color", "mono"):
if mode == "mono":
- info("warning: emoji rendered monochrome (no colour path on this ffmpeg; "
- "--emoji-assets DIR for colour). " + support["detail"])
+ warn("emoji rendered monochrome (no colour path on this ffmpeg; "
+ "--emoji-assets DIR for colour). " + support["detail"], "emoji_monochrome")
return cues, plan
# --- the PNG overlay route -------------------------------------------------------------
if not play_w or not play_h:
@@ -295,8 +297,8 @@ def plan_emoji(cues, args, play_w, play_h, brand=None):
"--emoji-max raises it); ffmpeg's filter graph and the per-frame cost both grow "
"linearly -- split the job, or use --emoji none", kind="input")
if plan["missing"]:
- info("warning: no PNG in the assets directory for " + ", ".join(plan["missing"]) +
- " -- those clusters are drawn by the text font instead")
+ warn("no PNG in the assets directory for " + ", ".join(plan["missing"]) +
+ " -- those clusters are drawn by the text font instead", "emoji_assets_missing", missing=list(plan["missing"]))
plan["box_px"] = int(round(box_px))
return out_cues, plan
@@ -493,13 +495,16 @@ def parse_ass_dialogue(path: str) -> str:
return "\n".join(text)
-def parse_ass_cues(path: str) -> List[Tuple[float, float, str]]:
- """(start, end, drawn text) of every Dialogue line of an ASS file, override blocks stripped."""
+def parse_ass_cues(path: str, stats: Optional[dict] = None) -> List[Tuple[float, float, str]]:
+ """(start, end, drawn text) of every Dialogue line of an ASS file, override blocks stripped.
+ A Dialogue line with too few fields or an unreadable start/end is left out; `stats`, when
+ given, gets their number as `malformed` so the caller can say how many it could not count."""
def secs(stamp: str) -> float:
h, m, rest = stamp.strip().split(":")
return int(h) * 3600 + int(m) * 60 + float(rest)
cues: List[Tuple[float, float, str]] = []
+ malformed = 0
# the [Events] Format line names the columns; Text is always last (libass reads it the same
# way), and a file without one gets the standard ten
names = ["layer", "start", "end", "style", "name", "marginl", "marginr", "marginv", "effect", "text"]
@@ -511,13 +516,17 @@ def parse_ass_cues(path: str) -> List[Tuple[float, float, str]]:
continue
fields = line.partition(":")[2].split(",", len(names) - 1)
if len(fields) < len(names) or "start" not in names or "end" not in names:
+ malformed += 1
continue
try:
start, end = secs(fields[names.index("start")]), secs(fields[names.index("end")])
except (ValueError, IndexError):
+ malformed += 1
continue
text = re.sub(r"\{[^}]*\}", "", fields[-1]).replace("\\N", "\n").replace("\\n", "\n").replace("\\h", " ")
cues.append((start, end, text))
+ if stats is not None:
+ stats["malformed"] = malformed
return cues
@@ -540,8 +549,17 @@ def count_visible_cues(cues, duration) -> Tuple[int, int]:
return burned, outside
-def shift_ass_file(src: str, dst: str, offset: float) -> int:
- """Copy an ASS file with every Dialogue start/end moved by `offset` seconds."""
+def shift_ass_file(src: str, dst: str, offset: float) -> Tuple[int, int]:
+ """Copy an ASS file with every Dialogue start/end moved by `offset` seconds; (lines shifted,
+ Dialogue lines copied unshifted because their timestamps could not be read)."""
+ text, n, unshifted = shift_ass_text(Path(src).read_text(encoding="utf-8-sig"), offset)
+ Path(dst).write_text(text, encoding="utf-8-sig")
+ return n, unshifted
+
+
+def shift_ass_text(source: str, offset: float) -> Tuple[str, int, int]:
+ """shift_ass_file() on text: (shifted text, lines shifted, Dialogue lines left as they were).
+ Pure, so a dry run can report the counts without writing the copy."""
def shift(stamp: str) -> str:
h, m, rest = stamp.split(":")
secs = int(h) * 3600 + int(m) * 60 + float(rest) + offset
@@ -552,22 +570,25 @@ def shift_ass_file(src: str, dst: str, offset: float) -> int:
ss, cc = divmod(rem, 100)
return f"{hh}:{mm:02d}:{ss:02d}.{cc:02d}"
- n = 0
+ n = unshifted = 0
out = []
- for line in Path(src).read_text(encoding="utf-8-sig").splitlines():
+ for line in source.splitlines():
if line.startswith("Dialogue:"):
head, sep, rest = line.partition(":")
fields = rest.split(",")
+ done = False
if len(fields) >= 3:
try:
fields[1], fields[2] = shift(fields[1].strip()), shift(fields[2].strip())
line = head + sep + ",".join(fields)
n += 1
+ done = True
except (ValueError, IndexError):
pass
+ if not done:
+ unshifted += 1 # copied as it was: counted and reported, not dropped unseen
out.append(line)
- Path(dst).write_text("\n".join(out) + "\n", encoding="utf-8-sig")
- return n
+ return "\n".join(out) + "\n", n, unshifted
def word_durations_from_timings(words: List[Tuple[float, float, str]], start: float, end: float,
@@ -1296,6 +1317,7 @@ def main() -> int:
planned_cues: Optional[List[Tuple[float, float, str]]] = None
# an English-only transcript made on an assumption says so first (asr.assumed_english_note)
side_notes: List[str] = list(getattr(args, "_asr_notes", None) or [])
+ ass_counts: dict = {} # --ass: ass_shifted / ass_unshifted (with --offset) and ass_malformed, into `caption`
ass_sample_path = args.ass
if args.srt and not (args.text or args.transcribe) and os.path.exists(srt_path or ""):
adjusted, changed = lay_out(parse_srt(srt_path))
@@ -1315,10 +1337,18 @@ def main() -> int:
if args.ass and args.offset and os.path.exists(args.ass):
shifted = os.path.splitext(output)[0] + "_offset.ass"
if STATE.dry_run:
- info(f"[dry-run] would write {shifted} (cues shifted by {args.offset:+g} s)")
+ _text, n, unshifted = shift_ass_text(read_text_or_die(args.ass, "--ass").lstrip("\ufeff"), args.offset)
+ info(f"[dry-run] would write {shifted} ({n} cues shifted by {args.offset:+g} s)")
else:
- n = shift_ass_file(args.ass, shifted, args.offset)
+ n, unshifted = shift_ass_file(args.ass, shifted, args.offset)
info(f"wrote {shifted} ({n} cues shifted by {args.offset:+g} s)")
+ # the counts reach the document (caption.ass_shifted / ass_unshifted), and a line
+ # whose timestamps could not be read -- copied unshifted -- is a warning, not a silence
+ ass_counts.update(ass_shifted=n, ass_unshifted=unshifted)
+ if unshifted:
+ warn(f"{unshifted} Dialogue line(s) in {args.ass} have timestamps that could not be read, so they were not "
+ f"shifted by {args.offset:+g} s (copied as they were)", "ass_lines_unshifted",
+ path=args.ass, count=unshifted, offset=args.offset)
args.ass = shifted
side_notes.append(f"the burned subtitles are {shifted}, the offset copy of "
f"{os.path.basename(ass_sample_path)} this run writes"
@@ -1449,6 +1479,19 @@ def main() -> int:
_script, font_file, font_family = script_font_for_text(
sample, lang=args.language, font=args.font, font_explicit=font_explicit,
fonts_dir=args.fonts_dir)
+ # the font libass draws with, reported as `font_used`: a --fonts-dir (or a brand's font
+ # file) is searched first for the family, by libass itself; otherwise fontconfig resolves it
+ if args.ass:
+ # the caller's own ASS names its fonts in its styles; this run does not choose them
+ font_used = {"asked": None, "file": None, "family": None, "how": "ass_styles",
+ "fonts_dir": args.fonts_dir or (os.path.dirname(font_file) if font_file else None), "substituted": False}
+ elif font_file or not args.fonts_dir:
+ # the family libass is given is the sanitised one (ass_font_name): that is what it resolves
+ font_used = record_font_choice(font_choice(ass_font_name(args.font), script_file=font_file, script_family=font_family,
+ renderer="libass"))
+ else:
+ font_used = {"asked": ass_font_name(args.font), "file": None, "family": ass_font_name(args.font), "how": "fonts_dir",
+ "fonts_dir": args.fonts_dir, "substituted": False}
if font_file:
args.font = font_family or args.font
if not args.fonts_dir:
@@ -1508,10 +1551,16 @@ def main() -> int:
# a dry run's pending --ass (an earlier step writes it): nothing to count yet
burn_cues = None
elif args.ass:
+ parsed: dict = {}
if planned_ass: # dry run: the offset copy is not written yet, shift the caller's file
- burn_cues = [(s + args.offset, e + args.offset, t) for s, e, t in parse_ass_cues(ass_sample_path)]
+ burn_cues = [(s + args.offset, e + args.offset, t) for s, e, t in parse_ass_cues(ass_sample_path, parsed)]
else:
- burn_cues = parse_ass_cues(args.ass)
+ burn_cues = parse_ass_cues(args.ass, parsed)
+ ass_counts["ass_malformed"] = parsed.get("malformed", 0)
+ if ass_counts["ass_malformed"]:
+ warn(f"{ass_counts['ass_malformed']} Dialogue line(s) in {ass_sample_path} could not be read (too few fields "
+ "or an unreadable start/end) and are not counted as cues", "ass_lines_malformed",
+ path=ass_sample_path, count=ass_counts["ass_malformed"])
if not burn_cues:
die(f"no Dialogue lines found in {ass_sample_path} -- nothing would be drawn", kind="input")
elif planned_cues is not None:
@@ -1533,7 +1582,7 @@ def main() -> int:
if outside:
msg = (f"{outside} cue(s) fall outside the video (0\u2013{float(meta['duration']):.1f} s) "
"and are not drawn")
- info(f"warning: {msg}")
+ warn(msg, "cues_outside_video", count=outside, duration=float(meta["duration"]))
side_notes.append(msg)
caption_stats["cues_burned"] = burned
caption_stats["cues_outside"] = outside
@@ -1606,7 +1655,8 @@ def main() -> int:
if caption_stats["text_unchanged"]:
info("caption text unchanged: the cues were burned exactly as given (line breaks, timing "
"and type size only)")
- extra["caption"] = dict(caption_stats)
+ extra["caption"] = dict(caption_stats, **ass_counts)
+ extra["font_used"] = font_used
if getattr(args, "_asr", None):
extra["transcription"] = args._asr
if emoji_plan:
diff --git a/scripts/color.py b/scripts/color.py
index fb1850a..c8004c3 100755
--- a/scripts/color.py
+++ b/scripts/color.py
@@ -24,6 +24,7 @@ import sys
from typing import List, Optional
from _common import STATE, add_common, analyze_levels, apply_common, emit, aac_args, cfr_args, default_output, die, escape_filter_path, ffmpeg_base, ffmpeg_bsfs, ffmpeg_exe, info, probe, run, run_analysis, run_keeping_subtitles, x264_args, X264_PRESETS, fmt_secs
+from _common.emit import warn
TONEMAPS = ["hable", "mobius", "reinhard", "bt2390", "clip", "linear", "gamma"]
@@ -388,7 +389,7 @@ def main() -> int:
if v.get("codec") != "hevc":
die("--strip-dovi only applies to HEVC (Dolby Vision) streams")
if not v.get("dolby_vision"):
- info("note: no Dolby Vision metadata detected; removing unregistered SEI anyway")
+ warn("no Dolby Vision metadata detected; removing unregistered SEI anyway", "dovi_not_detected")
cmd = ffmpeg_base() + ["-i", args.input, "-map", "0", "-c", "copy", "-bsf:v", "filter_units=remove_types=62", "-tag:v", "hvc1"]
if os.path.splitext(output)[1].lower() in (".mp4", ".mov", ".m4v"):
cmd += ["-movflags", "+faststart"]
diff --git a/scripts/cropdetect.py b/scripts/cropdetect.py
index b459eb3..ce6f2de 100644
--- a/scripts/cropdetect.py
+++ b/scripts/cropdetect.py
@@ -31,9 +31,10 @@ import argparse
import re
import sys
from collections import Counter
-from typing import Dict, List, Tuple
+from typing import Dict, List, Optional, Tuple
from _common import add_common, apply_common, die, emit, info, print_json, probe, require_tool, run_analysis, decode_gray_frames, frame_flow
+from _common.emit import warn
MOTION_CENTRE_FPS = 2.0
MOTION_CENTRE_W, MOTION_CENTRE_H = 64, 36
@@ -41,7 +42,11 @@ MOTION_CENTRE_W, MOTION_CENTRE_H = 64, 36
CROP_RE = re.compile(r"crop=(\d+):(\d+):(\d+):(\d+)")
-def detect(path: str, seconds: float, samples: int, limit: float, round_to: int, duration: float) -> List[Tuple[int, int, int, int]]:
+def detect(path: str, seconds: float, samples: int, limit: float, round_to: int, duration: float,
+ failed: "Optional[List[Dict]]" = None) -> List[Tuple[int, int, int, int]]:
+ """Crop rectangles from `samples` windows spread over the file. A window ffmpeg cannot decode
+ is skipped, and appended to `failed` as {start, seconds, error} when the caller passes a list,
+ so the result can say how many of the readings it stands on."""
ffmpeg = require_tool("ffmpeg")
per_window = max(0.5, seconds / max(1, samples))
rects: List[Tuple[int, int, int, int]] = []
@@ -56,6 +61,8 @@ def detect(path: str, seconds: float, samples: int, limit: float, round_to: int,
# One window ffmpeg cannot decode (a damaged stretch) is skipped; the other windows
# still measure. Only when every window fails is there nothing to report.
failures.append(proc.stderr.strip().splitlines()[-1:] or ["?"])
+ if failed is not None:
+ failed.append({"start": round(start, 3), "seconds": round(per_window, 3), "error": failures[-1][0][:300]})
continue
for m in CROP_RE.finditer(proc.stderr):
rects.append((int(m.group(1)), int(m.group(2)), int(m.group(3)), int(m.group(4))))
@@ -145,8 +152,16 @@ def main() -> int:
sw, sh = sh, sw
duration = meta.get("duration") or 0.0
- rects = detect(args.input, args.seconds, args.samples, args.limit, args.round_to, duration)
- result: Dict = {"file": args.input, "source_width": sw, "source_height": sh}
+ failed: List[Dict] = []
+ rects = detect(args.input, args.seconds, args.samples, args.limit, args.round_to, duration, failed)
+ # windows_failed: a window ffmpeg could not decode was skipped without a word, so a
+ # crop measured on one window of five read as well-founded as one measured on all five
+ result: Dict = {"file": args.input, "source_width": sw, "source_height": sh,
+ "windows": args.samples, "windows_failed": len(failed)}
+ if failed:
+ warn(f"cropdetect could not decode {len(failed)} of {args.samples} sampled windows; the crop rests on the other "
+ f"{args.samples - len(failed)}: {failed[-1]['error']}", "cropdetect_windows_failed",
+ failed=len(failed), windows=args.samples, starts=[f["start"] for f in failed])
if not rects:
result["crop"] = None
info("no crop bars detected (cropdetect produced no readings -- try --limit higher, or the source may already be full-frame)")
diff --git a/scripts/export.py b/scripts/export.py
index 3d16f88..13e5a99 100755
--- a/scripts/export.py
+++ b/scripts/export.py
@@ -38,6 +38,7 @@ from typing import Dict, List
from _common import STATE, add_common, apply_common, bt709_matrix_args, bt709_tag_args, child_args, emit, cfr_args, default_output, die, encoder_args, ffmpeg_base, info, probe, run, run_tool, validate_color, pad_filters, add_pad_fill_args, fmt_secs
from _common.decision import hw_preset_video, restate_last_swap
+from _common.emit import note, record_skip, warn
from check import SPECS as PLATFORMS, measure_loudness
from _platforms import ALIASES as _ALIASES, PLATFORMS as PLATFORM_TABLE, resolve as resolve_platform
# Frame and duration limit come from the one platform table (scripts/_platforms.py) rather than
@@ -108,6 +109,22 @@ PLATFORM_OF["youtube4k"] = PLATFORM_TABLE["youtube"]["check"]
PRESET_CHOICES: List[str] = sorted(set(PRESETS) | {a for a in _ALIASES if resolve_platform(a) in PRESETS})
+def _loudness_unmeasured(extra: Dict[str, object], notes: List[str], platform: str, spec: Dict, normalized: bool = False) -> None:
+ """The written file's loudness could not be read (ffmpeg's loudnorm JSON did not parse): the
+ platform's loudness was not verified, and the document says so -- a `loudness` block with null
+ values and a reason, a failing `loudness` verification step, a warning -- rather than leaving
+ the block out, which read as "nothing to check". check.py reports the same case as a WARN row."""
+ reason = "not_measured"
+ extra["loudness"] = {"lufs": None, "tp": None, "target_lufs": spec["lufs"], "target_tp": spec["tp"], "ok": None,
+ "measured": False, "reason": reason, **({"normalized": True} if normalized else {})}
+ extra["verification"] = [{"step": "loudness", "ok": False, "platform": platform, "reason": reason}]
+ msg = (f"the written file's loudness could not be measured{' after --normalize' if normalized else ''} "
+ f"(ffmpeg's loudnorm output did not parse), so {platform}'s {spec['lufs']:g} LUFS / {spec['tp']:g} dBTP was not "
+ "verified; run check.py on the file")
+ notes.append(msg)
+ warn(msg, "loudness_not_measured", platform=platform, normalized=normalized)
+
+
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("input", nargs="?")
@@ -154,7 +171,7 @@ def main() -> int:
# not HDR but still needs converting before BT.709 tags describe it (eval 24 h1)
kind = "HDR" if meta["video"].get("hdr") else "wide-gamut SDR, not HDR"
notes.append("source is %s (%s). This preset outputs SDR BT.709 tags without converting it; run color.py --to-sdr first for correct colours." % (kind, meta["video"].get("hdr_format")))
- info("warning: " + notes[-1])
+ warn(notes[-1], "colour_not_converted", preset=args.preset, hdr_format=meta["video"].get("hdr_format"))
has_audio = bool(meta.get("audio"))
output = args.output or default_output(args.input, args.preset, p["ext"])
out_ext = Path(output).suffix.lstrip(".").lower()
@@ -227,15 +244,20 @@ def main() -> int:
if out_ext == "mp4":
cmd += ["-movflags", "+faststart"]
cmd += (p["audio"] if has_audio else ["-an"])
+ trimmed = None
if p["max"] and not args.allow_long and (meta.get("duration") or 0) > p["max"]:
- info(f"trimming to the platform maximum of {p['max']:.0f}s (use --allow-long to keep full length)")
+ # the preset's promise (the platform's maximum), but the caller loses the tail: say how much
+ # in the document (`trimmed`, `notes`), not only on stderr
+ trimmed = {"from": round(float(meta["duration"]), 3), "to": round(float(p["max"]), 3)}
+ note(f"trimmed from {trimmed['from']:.3f}s to the {args.preset} maximum of {p['max']:.0f}s "
+ "(use --allow-long to keep full length)")
cmd += ["-t", f"{p['max']:.3f}"]
cmd.append(output)
run(cmd)
result = probe(output, role="output")
v = result["video"]
info(f"wrote {output} ({fmt_secs(result['duration'])}, {v['width']}x{v['height']}, {v['codec']})")
- extra: Dict[str, object] = {}
+ extra: Dict[str, object] = {"trimmed": trimmed} if trimmed else {}
platform = PLATFORM_OF.get(args.preset)
if platform and has_audio and not STATE.dry_run:
# Every eval run that exported for a platform then had to come back with loudness.py: the
@@ -243,6 +265,8 @@ def main() -> int:
# written file here so the result names the gap and the caller plans one pass, not two.
spec = PLATFORMS[platform]
m = measure_loudness(output)
+ if not m:
+ _loudness_unmeasured(extra, notes, platform, spec)
if m:
ok = abs(m["lufs"] - spec["lufs"]) <= spec["lufs_tol"] and m["tp"] <= spec["tp"]
extra["loudness"] = {"lufs": m["lufs"], "tp": m["tp"], "target_lufs": spec["lufs"], "target_tp": spec["tp"], "ok": ok}
@@ -253,7 +277,7 @@ def main() -> int:
extra["loudness"]["silent"] = True
notes.append(f"output audio is silent (integrated loudness -inf, {platform} expects {spec['lufs']:g} LUFS); "
"check the source's audio or give the export a real track (audio.py --replace)")
- info("warning: " + notes[-1])
+ warn(notes[-1], "output_audio_silent", platform=platform)
elif not ok and args.normalize:
# Eval 7: every platform job ran export -> loudness.py -> export again (two video
# encodes). The levels pass only re-encodes audio, so do it here on the written
@@ -275,17 +299,25 @@ def main() -> int:
die(f"--normalize: loudness.py failed: {err.get('message') or proc.stderr.strip()[-300:]}",
kind=err.get("kind") or "ffmpeg", output=output, hint=err.get("hint"))
os.replace(tmp, output)
- m = measure_loudness(output) or m
- ok = abs(m["lufs"] - spec["lufs"]) <= spec["lufs_tol"] and m["tp"] <= spec["tp"]
- extra["loudness"] = {"lufs": m["lufs"], "tp": m["tp"], "target_lufs": spec["lufs"], "target_tp": spec["tp"], "ok": ok, "normalized": True}
- extra["verification"] = [{"step": "loudness", "ok": ok, "platform": platform}]
+ m = measure_loudness(output)
+ if not m:
+ # the file changed: the measurement before --normalize no longer describes it
+ _loudness_unmeasured(extra, notes, platform, spec, normalized=True)
+ ok = True # nothing more to say about a level that was not measured
+ else:
+ ok = abs(m["lufs"] - spec["lufs"]) <= spec["lufs_tol"] and m["tp"] <= spec["tp"]
+ extra["loudness"] = {"lufs": m["lufs"], "tp": m["tp"], "target_lufs": spec["lufs"], "target_tp": spec["tp"], "ok": ok, "normalized": True}
+ extra["verification"] = [{"step": "loudness", "ok": ok, "platform": platform}]
if not ok:
notes.append(f"loudness is still {m['lufs']:.1f} LUFS / {m['tp']:+.1f} dBTP after loudness.py (target {spec['lufs']:g} LUFS / {spec['tp']:g} dBTP): {child.get('result', {}).get('note') or 'the encoder overshoots the ceiling'}")
- info("warning: " + notes[-1])
+ warn(notes[-1], "loudness_outside_spec", platform=platform, lufs=m["lufs"], tp=m["tp"], normalized=True)
elif not ok:
notes.append(f"loudness {m['lufs']:.1f} LUFS / {m['tp']:+.1f} dBTP is outside {platform}'s {spec['lufs']:g} LUFS / {spec['tp']:g} dBTP; "
f"run loudness.py -I {spec['lufs']:g} --tp {spec['tp']:g} on this file, or export with --normalize")
- info("warning: " + notes[-1])
+ warn(notes[-1], "loudness_outside_spec", platform=platform, lufs=m["lufs"], tp=m["tp"], normalized=False)
+ elif platform and not has_audio and not STATE.dry_run:
+ record_skip(f"no loudness check against {platform}'s spec: the source has no audio, so the export has none",
+ "loudness_no_audio", platform=platform)
elif platform and has_audio and args.normalize:
spec = PLATFORMS[platform]
notes.append(f"[dry-run] --normalize: loudness.py -I {spec['lufs']:g} --tp {spec['tp']:g} would run on the written file if it misses {platform}'s spec")
diff --git a/scripts/graphics.py b/scripts/graphics.py
index a686d00..cf4a2d1 100755
--- a/scripts/graphics.py
+++ b/scripts/graphics.py
@@ -29,17 +29,19 @@ import argparse
import os
import re
import sys
-from typing import List, Optional
+from typing import List, Optional, Tuple
from _platforms import PLATFORMS, PLATFORM_CHOICES, safe_margins_px, resolve as resolve_platform
from _common import (aac_args, add_common, brand_caption_style, script_font_for_text, apply_common, cfr_args,
- color_hex, default_font_file, default_output, die, draws_chroma, drawn_chain, emit, escape_drawtext, escape_filter_path,
+ color_hex, default_output, die, draws_chroma, drawn_chain, emit, escape_drawtext, escape_filter_path,
ffmpeg_base, info, label_chain, load_brand, parse_time, probe, rgb_chain_to_bt709, run, run_keeping_subtitles, video_args,
drawtext_boxborderw, drawtext_center_align, X264_PRESETS, time_arg, fmt_secs, STATE, drawtext_text_opts,
LANGUAGE_NAMES, needs_shaping, detect_script, BIDI_SCRIPTS, font_family_of_file, font_family_for_script, has_emoji,
emoji_clusters, emoji_codepoint_name, char_script, emoji_filter_chain, emoji_asset_for, emoji_support, resolve_emoji_assets,
EMOJI_ASSET_HINT, text_width_em, drawtext_shaping, wrap_text, WRAP_MODES,
SAFE_WIDTH_FRACTION, drawtext_draws_newline_glyph, is_multiline, _DRAWTEXT_PENDING)
+from _common.emit import warn
+from _common.fonts import font_choice, record_font_choice
from _ass_overlay import text_overlay_ass, EMOJI_SENTINEL
# where the template's text chain goes in a filter_complex it builds (graphics.main fills it in)
@@ -52,15 +54,16 @@ def ff_color(hex_rgb: str, alpha: float = 1.0) -> str:
return f"0x{color_hex(hex_rgb)}@{alpha:g}"
-def font_opts(brand: dict, font: Optional[str], font_file: Optional[str], script_file: Optional[str] = None) -> str:
- if font_file or brand.get("font_file"):
- return f"fontfile={escape_filter_path(font_file or brand['font_file'])}"
- if script_file: # a font picked by the script of the text itself (1.12)
- return f"fontfile={escape_filter_path(script_file)}"
- resolved = default_font_file(font or brand.get("font", "DejaVu Sans"))
- if resolved:
- return f"fontfile={escape_filter_path(resolved)}"
- return f"font='{escape_drawtext(font or brand.get('font', 'DejaVu Sans'))}'"
+def font_opts(brand: dict, font: Optional[str], font_file: Optional[str], script_file: Optional[str] = None,
+ script_family: Optional[str] = None, renderer: str = "drawtext") -> "Tuple[str, dict]":
+ """(the drawtext font option, the choice behind it -- the result's `font_used`): the caller's
+ file, a font picked by the script of the text itself (1.12), the asked family resolved to a
+ file, or the family by name when nothing resolves."""
+ choice = font_choice(font or brand.get("font", "DejaVu Sans"), font_file=font_file or brand.get("font_file"),
+ script_file=script_file, script_family=script_family, renderer=renderer)
+ if choice["file"]:
+ return f"fontfile={escape_filter_path(choice['file'])}", choice
+ return f"font='{escape_drawtext(font or brand.get('font', 'DejaVu Sans'))}'", choice
def main(force_route: Optional[str] = None) -> int:
@@ -156,7 +159,7 @@ def main(force_route: Optional[str] = None) -> int:
_script, script_file, _family = script_font_for_text(
" ".join(t for t in (args.name, args.title, args.subtitle, args.text, args.top, args.bottom) if t),
lang=args.lang, font=args.font, font_explicit=bool(args.font), font_file=args.font_file or brand.get("font_file"))
- fo = font_opts(brand, args.font, args.font_file, script_file)
+ fo, font_used = font_opts(brand, args.font, args.font_file, script_file, _family)
meta = probe(args.input)
if not meta.get("video"):
@@ -263,8 +266,8 @@ def main(force_route: Optional[str] = None) -> int:
"be drawn as an empty box -- stripped from the text instead "
"(--text-render auto renders it monochrome through libass)")
if emoji_mode == "mono":
- info("warning: emoji rendered monochrome (no colour path on this ffmpeg; "
- "--emoji-assets DIR for colour). " + support["detail"])
+ warn("emoji rendered monochrome (no colour path on this ffmpeg; "
+ "--emoji-assets DIR for colour). " + support["detail"], "emoji_monochrome")
if emoji_mode == "none":
if not "".join(ch for ch in all_text if char_script(ch) != "emoji").strip():
die("the template's text is nothing but emoji and this machine can draw none of them "
@@ -565,9 +568,9 @@ def main(force_route: Optional[str] = None) -> int:
"(FFmpeg 8.0-8.0.1 and 8.1.0; fixed in 8.0.2/8.1.1), so it is rendered through libass")
_DRAWTEXT_PENDING.clear()
return main(force_route="ass")
- info("warning: this ffmpeg's drawtext draws a box at each line break of a multi-line label "
+ warn("this ffmpeg's drawtext draws a box at each line break of a multi-line label "
"(FFmpeg 8.0-8.0.1 and 8.1.0; fixed in 8.0.2/8.1.1): drop --text-render drawtext to render it "
- "through libass, or upgrade FFmpeg")
+ "through libass, or upgrade FFmpeg", "drawtext_newline_glyph", text_render="drawtext")
output = args.output or default_output(args.input, "gfx")
ass_path = None
@@ -631,7 +634,7 @@ def main(force_route: Optional[str] = None) -> int:
f"(limit {_max}, --emoji-max raises it); ffmpeg's filter graph and the "
"per-frame cost both grow linearly -- split the job, or use --emoji none", kind="input")
if missing:
- info("warning: no PNG in the assets directory for " + ", ".join(missing))
+ warn("no PNG in the assets directory for " + ", ".join(missing), "emoji_assets_missing", missing=list(missing))
if emoji_mode:
emoji_result = {"mode": emoji_mode, "count": len(emoji_clusters(all_text)),
"clusters": sorted(seen) or sorted({emoji_codepoint_name(cl) for _i, cl in emoji_clusters(all_text)}),
@@ -708,8 +711,11 @@ def main(force_route: Optional[str] = None) -> int:
dropped_streams = run_keeping_subtitles(cmd, output)
r = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(r['duration'])}, {args.template})")
+ if route == "ass":
+ # libass looks the family up itself (fontconfig): the same choice, asked for that renderer
+ font_used = font_opts(brand, args.font, args.font_file, script_file, _family, renderer="libass")[1]
extra = {"template": args.template, "dropped_non_av_streams": dropped_streams,
- "text_renderer": route, "script": _script}
+ "text_renderer": route, "script": _script, "font_used": record_font_choice(font_used)}
if ass_path:
extra["ass"] = ass_path
if emoji_result:
diff --git a/scripts/grid.py b/scripts/grid.py
index d314192..0fb672d 100644
--- a/scripts/grid.py
+++ b/scripts/grid.py
@@ -25,8 +25,9 @@ import argparse
import os
import sys
-from _common import add_common, apply_common, aac_args, cfr_args, default_font_file, default_output, die, emit, X264_PRESETS, fmt_secs, \
+from _common import add_common, apply_common, aac_args, cfr_args, default_output, die, emit, X264_PRESETS, fmt_secs, \
draws_chroma, drawn_chain, escape_drawtext, escape_filter_path, ffmpeg_base, info, label_chain, matrix_chain, probe, run, validate_color, video_args
+from _common.fonts import font_choice, record_font_choice
LABEL_MARGIN = 10
@@ -84,7 +85,10 @@ def main() -> int:
durations = [m.get("duration") or 0.0 for m in metas]
target_duration = max(durations) if args.pad else min(durations)
- font_file = default_font_file(args.font)
+ font_used = font_choice(args.font) # the label font, reported as `font_used` when labels are drawn
+ font_file = font_used["file"]
+ if args.label == "auto":
+ record_font_choice(font_used)
output = args.output or default_output(args.inputs[0], "grid")
cmd = ffmpeg_base()
@@ -157,7 +161,7 @@ def main() -> int:
v = result["video"]
info(f"wrote {output} ({fmt_secs(result['duration'])}, {v['width']}x{v['height']}, {args.cols}x{args.rows} grid, "
f"{n} clips, {'padded to longest' if args.pad else 'stopped at shortest'})")
- emit(output, cols=args.cols, rows=args.rows, clips=n)
+ emit(output, cols=args.cols, rows=args.rows, clips=n, **({"font_used": font_used} if args.label == "auto" else {}))
return 0
diff --git a/scripts/join.py b/scripts/join.py
index 1547a9d..1c36d5f 100755
--- a/scripts/join.py
+++ b/scripts/join.py
@@ -40,6 +40,7 @@ import sys
from typing import Any, Dict, List, Optional, Tuple
from _common import STATE, AUDIO_MEDIA_EXT, draws_chroma, drawn_chain, label_chain, matrix_chain, video_args, aac_args, add_common, apply_common, audio_codec_for, default_output, die, dry_run_input_pending, emit, ffmpeg_base, info, measured_level_dbfs, is_audio_output, probe, require_tool, run, validate_color, X264_PRESETS, fmt_secs
+from _common.emit import warn
TRANSITIONS = ["fade", "dissolve", "wipeleft", "wiperight", "wipeup", "wipedown", "slideleft", "slideright",
"circleopen", "circleclose", "fadeblack", "fadewhite", "smoothleft", "smoothright", "radial", "none"]
@@ -113,7 +114,7 @@ def note_short_segments(paths: List[str], metas: List[dict], minimum: float, uni
STATE_SHORT.append({"index": k, "path": p, "duration": round(dur, 4)})
if STATE_SHORT:
listing = ", ".join(f"#{s['index'] + 1} {s['path']} ({s['duration']:g}s)" for s in STATE_SHORT)
- info(f"warning: {len(STATE_SHORT)} input(s) shorter than {unit}: {listing}")
+ warn(f"{len(STATE_SHORT)} input(s) shorter than {unit}: {listing}", "short_inputs", count=len(STATE_SHORT))
STATE_NOTES.append(f"{len(STATE_SHORT)} input(s) are shorter than {unit} and were joined anyway: {listing}")
@@ -361,7 +362,8 @@ def main() -> int:
STATE_SILENT[:] = sorted(intended + (silent if args.on_silent == "warn" else []), key=lambda s: s["index"])
if silent and args.on_silent == "warn":
listing = ", ".join(f"#{s['index'] + 1} {s['path']} (peak {s['peak_db']:.1f} dBFS)" for s in silent)
- info(f"warning: {len(silent)} input(s) silent at or below {args.silence_threshold:g} dBFS, joined anyway: {listing}")
+ warn(f"{len(silent)} input(s) silent at or below {args.silence_threshold:g} dBFS, joined anyway: {listing}",
+ "silent_inputs", count=len(silent))
STATE_NOTES.append(f"{len(silent)} input(s) have no audible sound (peak at or below {args.silence_threshold:g} dBFS) "
f"and were joined anyway: {listing}; pass --on-silent fail or skip to refuse or drop them")
STATE_SKIPPED[:] = skipped
@@ -373,7 +375,7 @@ def main() -> int:
STATE_DUPES[:] = dupes
if dupes:
listing = "; ".join(f"{dd['path']} at #" + ", #".join(str(i + 1) for i in dd["indices"]) for dd in dupes)
- info(f"warning: input(s) listed more than once: {listing}")
+ warn(f"input(s) listed more than once: {listing}", "duplicate_inputs", count=len(dupes))
STATE_NOTES.append(f"{len(dupes)} input(s) are listed more than once and joined each time: {listing}; "
"fine if the repeat is intended")
STATE_PENDING[:] = pending
diff --git a/scripts/loop.py b/scripts/loop.py
index d491c65..0454dd3 100644
--- a/scripts/loop.py
+++ b/scripts/loop.py
@@ -25,6 +25,7 @@ import math
import sys
from _common import add_common, aac_args, apply_common, cfr_args, default_output, die, emit, ffmpeg_base, info, parse_time, probe, run, video_args, X264_PRESETS, time_arg, fmt_secs
+from _common.emit import warn
# The loop filter's size is capped at 32767 frames, so one boomerang cycle (2N-2 frames) must fit.
@@ -51,8 +52,9 @@ def _boomerang_graph(meta, src_dur: float, times) -> str:
bytes_per_frame = v["width"] * v["height"] * 1.5 * (2 if (v.get("bit_depth") or 8) > 8 else 1)
held = bytes_per_frame * frames
if held > 2 * 1024 ** 3:
- info(f"warning: --boomerang holds about {held / 1024 ** 3:.1f} GiB of decoded frames in memory "
- f"({frames} frames of {v['width']}x{v['height']}) -- cut the clip shorter first (cut.py) if RAM is tight")
+ warn(f"--boomerang holds about {held / 1024 ** 3:.1f} GiB of decoded frames in memory "
+ f"({frames} frames of {v['width']}x{v['height']}) -- cut the clip shorter first (cut.py) if RAM is tight",
+ "memory_heavy", bytes=int(held))
repeats = -1 if times is None else times - 1
# Retimed by frame index at the source's nominal rate: a VFR source's reversed half carries
# its gaps mirrored, and a CFR conform after the graph duplicated a frame against them (a
diff --git a/scripts/multicam.py b/scripts/multicam.py
index 1b04a32..467fe7a 100755
--- a/scripts/multicam.py
+++ b/scripts/multicam.py
@@ -32,6 +32,7 @@ import sys
from typing import Dict, List, Tuple
from _common import label_chain, matrix_chain, video_args, aac_args, add_common, apply_common, default_output, die, emit, ffmpeg_base, info, time_arg, probe, run, x264_args, X264_PRESETS, fmt_secs, decode_pcm_mono, rms_envelope, STATE
+from _common.emit import record_fallback, warn
from sync import measure_offset
ENERGY_WINDOW_S = 0.25 # --switch energy: loudness measured over this window on the reference timeline
@@ -199,7 +200,8 @@ def main() -> int:
conf.append(score)
info(f"{p}: offset {off:+.3f}s (confidence {score:.2f})" + (f", drift {(ratio - 1) * 1e6:+.0f} ppm" if args.fix_drift else ""))
if score < 0.1:
- info(f"warning: {p} has low correlation confidence ({score:.2f}); check that it shares an audio event with the reference before trusting this offset")
+ warn(f"{p} has low correlation confidence ({score:.2f}); check that it shares an audio event with the reference before "
+ "trusting this offset", "sync_low_confidence", path=p, confidence=round(score, 3))
report = {"inputs": args.inputs, "offsets_seconds": [round(o, 4) for o in offsets],
"confidence": [round(c, 3) for c in conf]}
@@ -275,13 +277,19 @@ def main() -> int:
convert = [matrix_chain(m, metas[0]) for m in metas]
if any(convert):
convert = [c or label_chain(m, metas[0]) for c, m in zip(convert, metas)]
+ # what is actually rendered: a range whose camera has not started yet plays camera 0, and the
+ # result says so (`cuts` names the camera on screen, diagnostics.fallbacks the one asked for)
+ rendered: List[Tuple[float, float, int]] = []
for i, (s, e, c) in enumerate(filled):
# reference time t maps to source time (t - offset_c) * ratio_c
src_s = (s - offsets[c]) * ratios[c]
src_e = (e - offsets[c]) * ratios[c]
if src_s < 0:
- info(f"warning: camera {c} has not started at reference {s:.2f}s; using camera 0 for that range")
+ record_fallback(f"camera {c} has not started at reference {s:.2f}s; using camera 0 for that range "
+ f"({s:.3f}-{e:.3f}s)", "camera_not_started", start=round(s, 3), end=round(e, 3),
+ requested_camera=c, camera=0, starts_at=round(offsets[c], 3))
c, src_s, src_e = 0, s, e
+ rendered.append((s, e, c))
parts.append(f"[{c}:v]trim=start={src_s:.4f}:end={src_e:.4f},setpts=PTS-STARTPTS,{convert[c]}{geo}[v{i}]")
labels.append(f"[v{i}]")
project_clips.append({"src": args.inputs[c], "in": round(src_s, 3), "out": round(src_e, 3)})
@@ -321,7 +329,7 @@ def main() -> int:
r = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(r['duration'])}, {len(filled)} cuts, audio from input {a})")
extra = {"switch_mode": "energy", "min_shot": args.min_shot} if energy_mode else {}
- emit(output, cuts=[[round(s, 3), round(e, 3), c] for s, e, c in filled], **report, **extra)
+ emit(output, cuts=[[round(s, 3), round(e, 3), c] for s, e, c in rendered], **report, **extra)
return 0
diff --git a/scripts/overlay.py b/scripts/overlay.py
index ae1c6c5..6c20fa5 100755
--- a/scripts/overlay.py
+++ b/scripts/overlay.py
@@ -26,7 +26,9 @@ import sys
from typing import List, Optional
from _platforms import PLATFORMS, PLATFORM_CHOICES, safe_margins_px, resolve as resolve_platform
-from _common import STATE, script_font_for_text, drawtext_clean, drawtext_line_metrics, drawtext_text_opts, drawtext_draws_newline_glyph, is_multiline, needs_shaping, LANGUAGE_NAMES, has_emoji, detect_script, emoji_clusters, load_brand, drawn_chain, label_chain, matrix_chain, rgb_chain_to_bt709, video_args, add_common, apply_common, default_font_file, emit, aac_args, cfr_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, run_keeping_subtitles, validate_color, x264_args, X264_PRESETS, time_arg, fmt_secs
+from _common import STATE, script_font_for_text, drawtext_clean, drawtext_line_metrics, drawtext_text_opts, drawtext_draws_newline_glyph, is_multiline, needs_shaping, LANGUAGE_NAMES, has_emoji, detect_script, emoji_clusters, load_brand, drawn_chain, label_chain, matrix_chain, rgb_chain_to_bt709, video_args, add_common, apply_common, emit, aac_args, cfr_args, default_output, die, escape_drawtext, escape_filter_path, ffmpeg_base, info, parse_time, probe, run, run_keeping_subtitles, validate_color, x264_args, X264_PRESETS, time_arg, fmt_secs
+from _common.emit import warn
+from _common.fonts import font_choice, record_font_choice
# Per-edge margins: a platform's UI does not cover the same fraction of every edge (TikTok's
# like column is 14 % of the width, its description block 22 % of the height), so a position
@@ -240,20 +242,22 @@ def main() -> int:
if args.emoji == "none":
for _i, _cl in list(reversed(emoji_clusters(args.text))):
args.text = args.text[:_i] + args.text[_i + len(_cl):]
- info("emoji: stripped from the drawn text (--emoji none)")
+ warn("emoji: stripped from the drawn text (--emoji none)", "emoji_stripped", mode="none")
else:
- info("warning: emoji are drawn by the text font here (monochrome at best); "
- "caption.py/graphics.py composite colour PNGs with --emoji-assets")
+ warn("emoji are drawn by the text font here (monochrome at best); "
+ "caption.py/graphics.py composite colour PNGs with --emoji-assets", "emoji_monochrome")
+ script_file = script_family = None
if args.text and not args.font_file:
# 1.12: non-Latin overlay text picks a font by script, so a title in Japanese, Korean,
# Arabic ... draws glyphs instead of boxes. drawtext does not shape or reorder RTL text --
# see references/gotchas.md#fonts-by-script.
- _script, script_file, _family = script_font_for_text(
+ _script, script_file, script_family = script_font_for_text(
args.text, font=args.font, font_explicit=args.font != ap.get_default("font"), font_file=args.font_file)
- if script_file:
- args.font_file = script_file
- if not args.font_file:
- args.font_file = default_font_file(args.font)
+ # the font drawtext gets, reported as `font_used` and recorded when it is not the one asked for
+ font_used = font_choice(args.font, font_file=args.font_file, script_file=script_file, script_family=script_family)
+ args.font_file = font_used["file"]
+ if args.text:
+ record_font_choice(font_used)
meta = probe(args.input)
if not meta.get("video"):
die("input has no video stream")
@@ -411,7 +415,7 @@ def main() -> int:
if not STATE.dry_run:
result = probe(output, role="output")
info(f"wrote {output} ({fmt_secs(result['duration'])})")
- emit(output, dropped_non_av_streams=dropped_streams)
+ emit(output, dropped_non_av_streams=dropped_streams, **({"font_used": font_used} if args.text else {}))
return 0
diff --git a/scripts/render.py b/scripts/render.py
index ae2b1b4..f6ca577 100755
--- a/scripts/render.py
+++ b/scripts/render.py
@@ -48,7 +48,11 @@ is "font_size", and "scale" on a text entry is refused.
itself failing to run) exits 1, same as running check.py directly would --
the render is not silently reported as successful just because every stage
up to it completed. The output file is still written and `--json`'s
-`check` field still carries the full row-by-row result either way.
+`check` field still carries the full row-by-row result either way. A check
+stage with only "content": true runs check.py --content without a platform.
+`verified` is every stage's: a stage that completed unverified (an export that
+missed its loudness) leaves the render verified false, its steps and notes in
+the result.
Examples:
python3 render.py --init project.json # write a commented starter project
@@ -72,7 +76,7 @@ from export import PRESETS, PLATFORM_OF
from _platforms import PLATFORMS, caption_defaults, resolve as resolve_platform
from _common import STATE, add_common, aspect_ratio, brand_caption_style, load_brand, apply_common, child_args, die, emit, info, probe, require_tool, run, run_tool, place_output, refuse_output_is_input, _check_existing_output, _check_output_path, fingerprint, PLAN_VERSION, ffmpeg_version, ffmpeg_exe
from _common.runner import HW_ENV, HW_FORCED_ENV, add_hw_orchestrator_args
-from _common.emit import absorb_stage, audit, refuse, warn
+from _common.emit import absorb_stage, audit, record_fallback, record_skip, refuse, warn
import subprocess
from _contract import CONTRACT_VERSION
from batch import file_key
@@ -550,7 +554,7 @@ def sh(script: str, *argv: Any, extra: List[str] = None, stage: str = None) -> s
# would make a second run with the first run's output already on disk miss every time.
key_args = ["<out>" if a == dest else a for a in full]
key = cache_key(stage, script, key_args, inputs, dest)
- if cache_lookup(stage, key, dest):
+ if cache_lookup(stage, key, dest, Path(script).stem):
CACHE["hits"].append(stage)
info(f"→ {script} {stage}: served from --cache")
# No child ran, so there is no document: say so rather than leaving the PREVIOUS
@@ -562,6 +566,7 @@ def sh(script: str, *argv: Any, extra: List[str] = None, stage: str = None) -> s
_refuse_uncached_earlier_stage(stage)
elif stage and CACHE.get("dir"):
CACHE["misses"].append(stage)
+ _cache_miss(stage, "no_output_path") # nothing names where the artifact goes, so there is nothing to keep
started = time.time()
cmd = [str(HERE / script)] + [str(a) for a in argv] + (extra or []) + child_args() + ["--json"]
info("→ " + " ".join(os.path.basename(c) if i < 1 else c for i, c in enumerate(cmd[:-1])))
@@ -589,7 +594,7 @@ def sh(script: str, *argv: Any, extra: List[str] = None, stage: str = None) -> s
absorb_stage(Path(script).stem, doc) # a stage's GPU facts (ran on VideoToolbox, fell back and why) and its diagnostics reach this result
out_path = str(doc.get("output") or "")
if key:
- cache_store(stage, key, out_path or (dest or ""), time.time() - started)
+ cache_store(stage, key, out_path or (dest or ""), time.time() - started, doc)
return out_path
@@ -613,7 +618,7 @@ STAGE_ORDER = ("clips", "audiogram", "join", "silence", "fit", "captions", "grap
"overlays", "audio", "loudness", "export", "chapters")
def _fresh_cache() -> "Dict[str, Any]":
- return {"dir": None, "ffmpeg": None, "hits": [], "misses": [], "saved_seconds": 0.0,
+ return {"dir": None, "ffmpeg": None, "hits": [], "misses": [], "miss_reasons": [], "saved_seconds": 0.0,
"entries": 0, "would_hit": [], "from": None}
@@ -625,10 +630,16 @@ CACHE: Dict[str, Any] = _fresh_cache()
def _content_hash(path: str) -> str:
"""The same cheap content fingerprint batch.py caches on: name, size, mtime, first and last
- MB. Reused rather than reinvented so one file has one identity across the skill."""
+ MB. Reused rather than reinvented so one file has one identity across the skill. A file that
+ cannot be read keys as "missing" -- recorded (cache_input_unreadable), since two unreadable
+ files then share that part of the key."""
try:
return file_key(Path(path))
- except (OSError, ValueError):
+ except (OSError, ValueError) as exc:
+ entry = {"code": "cache_input_unreadable", "path": str(path)}
+ if not any(all(w.get(k) == v for k, v in entry.items()) for w in STATE.warnings):
+ warn(f"cache: could not fingerprint {path} ({exc}); it is keyed as \"missing\"", "cache_input_unreadable",
+ path=str(path), error=str(exc))
return "missing"
@@ -646,9 +657,14 @@ def ffmpeg_banner() -> str:
first = (out or "").strip().splitlines()
if first:
return first[0].strip()
- except (OSError, subprocess.SubprocessError):
- pass
- return ".".join(str(n) for n in ffmpeg_version())
+ why = "ffprobe -version printed nothing"
+ except (OSError, subprocess.SubprocessError) as exc:
+ why = str(exc)
+ pair = ".".join(str(n) for n in ffmpeg_version())
+ # two builds of one major.minor now share cache entries: say so
+ record_fallback(f"cache: the ffmpeg build banner could not be read ({why}); keying on ffmpeg {pair} instead, so builds "
+ "of the same major.minor share entries", "cache_ffmpeg_banner_unreadable", used=pair, error=why)
+ return pair
def cache_key(stage: str, script: str, argv: "Sequence[Any]", inputs: "Sequence[str]",
@@ -684,22 +700,46 @@ def cache_key(stage: str, script: str, argv: "Sequence[Any]", inputs: "Sequence[
return hashlib.sha1(json.dumps(payload, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
-def cache_lookup(stage: str, key: str, dest: str) -> bool:
+def _cache_miss(stage: str, reason: str, **facts: Any) -> bool:
+ """Record why `stage` was not served from the cache (`cache.miss_reasons`); a damaged entry
+ (unreadable sidecar, artifact of the wrong size) is also a warning and an audit line in the
+ cache directory, because something rewrote or truncated a file this tool wrote."""
+ CACHE["miss_reasons"].append(dict({"stage": stage, "reason": reason}, **facts))
+ if reason in ("sidecar_corrupt", "artifact_truncated"):
+ msg = f"cache: the {stage} entry is damaged ({reason.replace('_', ' ')}); the stage runs again"
+ warn(msg, "cache_entry_damaged", stage=stage, reason=reason, **facts)
+ audit("cache_entry_damaged", dict({"stage": stage, "reason": reason}, **facts),
+ default_path=str(Path(CACHE["dir"]) / ".ffskill-audit.jsonl"))
+ return False
+
+
+def cache_lookup(stage: str, key: str, dest: str, tool: Optional[str] = None) -> bool:
"""Put the cached artifact for `key` at `dest` and return True, or return False on any
- mismatch -- silently, because a miss is not an error, it is just work to do."""
+ mismatch. A miss is not an error, it is work to do, but it is not silent either: why each
+ stage missed is in `cache.miss_reasons` (not_cached, sidecar_missing, artifact_missing,
+ sidecar_corrupt, artifact_truncated). A hit replays the stage's recorded result (its
+ verification, notes and diagnostics) into this run, as a stage that ran would have."""
cdir = CACHE.get("dir")
if not cdir:
return False
side = Path(cdir) / f"{key}.json"
art = Path(cdir) / f"{key}{Path(dest).suffix or '.bin'}"
- if not (side.exists() and art.exists()):
- return False
+ if not side.exists() and not art.exists():
+ return _cache_miss(stage, "not_cached", key=key)
+ if not side.exists():
+ return _cache_miss(stage, "sidecar_missing", key=key, artifact=str(art))
+ if not art.exists():
+ return _cache_miss(stage, "artifact_missing", key=key, sidecar=str(side))
try:
meta = json.loads(side.read_text(encoding="utf-8"))
- except (OSError, ValueError):
- return False
- if int(meta.get("size") or -1) != art.stat().st_size:
- return False
+ if not isinstance(meta, dict):
+ raise ValueError("not a JSON object")
+ expected = int(meta.get("size") or -1)
+ except (OSError, ValueError, TypeError) as exc:
+ return _cache_miss(stage, "sidecar_corrupt", key=key, sidecar=str(side), error=str(exc))
+ actual = art.stat().st_size
+ if expected != actual:
+ return _cache_miss(stage, "artifact_truncated", key=key, artifact=str(art), expected_size=expected, actual_size=actual)
if STATE.dry_run:
CACHE["would_hit"].append(stage)
return True
@@ -711,11 +751,26 @@ def cache_lookup(stage: str, key: str, dest: str) -> bool:
except OSError:
shutil.copy2(art, dest) # ... and a copy where it does not. Never a move: the cache
CACHE["saved_seconds"] += float(meta.get("seconds") or 0.0)
+ result = meta.get("result")
+ if isinstance(result, dict):
+ absorb_stage(tool or stage, dict(result, status="completed", dry_run=False)) # tagged as the stage's tool, as a live run is
+ else:
+ # an entry written before the stage's result was kept: what the stage verified is unknown
+ record_skip(f"cache: the {stage} entry predates recorded results, so what that stage verified is not "
+ "known; rerun without --cache to see it", "cache_result_unrecorded", stage=stage)
return True
-def cache_store(stage: str, key: str, produced: str, seconds: float) -> None:
- """Keep `produced` for the next run. Never under --dry-run: a plan writes nothing."""
+def _stage_result(doc: Any) -> Dict[str, Any]:
+ """What a cache entry keeps of the stage's own --json document, replayed on a hit."""
+ if not isinstance(doc, dict):
+ return {}
+ return {k: doc[k] for k in ("verified", "verification", "notes", "diagnostics") if k in doc}
+
+
+def cache_store(stage: str, key: str, produced: str, seconds: float, doc: Any = None) -> None:
+ """Keep `produced` for the next run. Never under --dry-run: a plan writes nothing. A store
+ that fails is a warning (cache_store_failed), never a failure of the run."""
cdir = CACHE.get("dir")
if not cdir or STATE.dry_run or not produced or not os.path.exists(produced):
return
@@ -725,10 +780,11 @@ def cache_store(stage: str, key: str, produced: str, seconds: float) -> None:
Path(cdir, f"{key}.json").write_text(json.dumps({
"stage": stage, "ffmpeg": CACHE.get("ffmpeg"), "skill": SKILL_VERSION,
"created": time.time(), "size": art.stat().st_size, "seconds": round(seconds, 2),
- "artifact": art.name}, indent=2), encoding="utf-8")
+ "artifact": art.name, "result": _stage_result(doc)}, indent=2), encoding="utf-8")
CACHE["entries"] += 1
except OSError as exc:
- info(f"cache: could not store the {stage} artifact ({exc}); the run is unaffected")
+ warn(f"cache: could not store the {stage} artifact ({exc}); the run is unaffected", "cache_store_failed",
+ stage=stage, error=str(exc))
def _utc_now() -> str:
@@ -776,6 +832,28 @@ def plan_input_status(inputs: Sequence[Any]) -> List[Dict[str, Any]]:
return rows
+def run_check(output: str, platform: Optional[str], content: bool) -> Dict[str, Any]:
+ """check.py on `output` for a project's "check" stage or a plan's check step: `platform`
+ (or none: check.py's youtube assumption, judgement rows WARN) and `content` (--content).
+ Returns check.py's own document; when check.py printed none that parses, a failure document
+ in its shape -- status failed, error {kind, reason: check_output_unparsed, message,
+ exit_code, stderr_tail} -- so a caller never reads a bare string."""
+ cp = run_tool([str(HERE / "check.py"), output] + (["--platform", str(platform)] if platform else [])
+ + (["--content"] if content else []) + ["--json"] + child_args())
+ try:
+ doc = json.loads(cp.stdout)
+ except ValueError:
+ doc = None
+ if isinstance(doc, dict):
+ return doc
+ tail = cp.stderr.strip()[-300:]
+ msg = "check.py printed no JSON document" + (f": {tail.splitlines()[-1]}" if tail else "")
+ warn(f"check: {msg}", "check_output_unparsed", exit_code=cp.returncode)
+ return {"status": "failed", "platform": platform, "ok": False,
+ "error": {"kind": "timeout" if cp.returncode == 124 else "verification", "reason": "check_output_unparsed",
+ "message": msg, "exit_code": cp.returncode, "stderr_tail": tail}}
+
+
def execute_plan(plan: Dict[str, Any], path: str) -> int:
"""Run a plan written by `<tool> --plan FILE`: refuse if any fingerprinted input changed since
the plan was made (the plan's commands would then describe a different edit), run the tool
@@ -818,6 +896,10 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
for c in plan.get("commands") or []:
STATE.commands.append(c)
info("[dry-run] " + c)
+ for step in plan.get("verify") or []:
+ if step.get("tool") == "check" and (step.get("platform") or step.get("content") is True):
+ record_skip("[dry-run] the plan's check step is not run: there is no output to check yet",
+ "check_dry_run", platform=step.get("platform"), content=step.get("content") is True)
emit(plan.get("output"), plan=path, plan_sha256=digest, tool=tool, stages=[tool], check=None, unbound_inputs=unbound)
return 0
info(f"executing plan {path}: {tool} " + " ".join(argv))
@@ -844,14 +926,12 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
# a check step names a platform, asks for check.py --content's black/frozen/silence rows
# ("content": true), or both -- the same keys a project's "check" stage takes
if step.get("tool") == "check" and (step.get("platform") or step.get("content") is True) and output:
- cp = run_tool([str(HERE / "check.py"), output] + (["--platform", str(step["platform"])] if step.get("platform") else [])
- + (["--content"] if step.get("content") is True else []) + ["--json"] + child_args())
- try:
- check_result = json.loads(cp.stdout)
- except ValueError:
- check_result = {"error": cp.stderr.strip()[-300:]}
+ check_result = run_check(output, step.get("platform"), step.get("content") is True)
if check_result.get("failed") or check_result.get("status") == "failed" or check_result.get("error"):
exit_code = 1
+ elif step.get("tool") == "check" and not output:
+ record_skip("the plan's check step is not run: the tool reported no output", "check_no_output",
+ platform=step.get("platform"))
# a failed platform check is reported the way the direct path reports it: the tool completed,
# verified is false, the rows say what to fix (review 6: a plan must not fail harder than the
# same command run by hand)
@@ -860,9 +940,10 @@ def execute_plan(plan: Dict[str, Any], path: str) -> int:
info(f"plan done: {output}, but the {check_result.get('platform') or 'content'} check failed" + (f": {', '.join(failed_rows)}" if failed_rows else ""))
else:
info(f"plan done: {output}")
+ # the tool's own steps (export's loudness, ...) come in through absorb_stage() above, tagged
+ # with `stage`, so `verified` is theirs and the check's together
emit(output, plan=path, plan_sha256=digest, unbound_inputs=unbound, tool=tool, stages=[tool] + (["check"] if check_result else []), check=check_result, tool_result=doc,
- verification=([{"step": "check", "ok": not exit_code, "platform": check_result.get("platform")}] if check_result else [])
- + [s for s in (doc.get("verification") or []) if s.get("step") != "probe"])
+ verification=([{"step": "check", "ok": not exit_code, "platform": check_result.get("platform")}] if check_result else []))
return 0
@@ -886,7 +967,8 @@ def frame_from_preset(frame: Dict[str, Any], export: Dict[str, Any]) -> None:
if abs(float(m.group(1)) / float(m.group(2)) - preset["w"] / preset["h"]) > 0.01:
return
frame["width"], frame["height"] = preset["w"], preset["h"]
- info(f"frame: {preset['w']}x{preset['h']} from the {export['preset']} export preset (captions and overlays are sized for delivery)")
+ warn(f"frame: {preset['w']}x{preset['h']} from the {export['preset']} export preset (captions and overlays are sized for delivery)",
+ "frame_from_export_preset", preset=export["preset"], width=preset["w"], height=preset["h"])
def export_timeline(proj: Dict[str, Any], rel, dest: str) -> int:
@@ -1441,7 +1523,8 @@ def main() -> int:
# was rendered again; a platform preset with no loudness stage of its own gets the
# one-export behaviour by default ("normalize": false opts out)
normalize = True
- info(f"export: --normalize on by default for the {ex['preset']} preset (set \"normalize\": false to skip)")
+ warn(f"export: --normalize on by default for the {ex['preset']} preset (set \"normalize\": false to skip)",
+ "normalize_on_by_default", preset=ex["preset"])
if normalize:
argv += ["--normalize"] # one export that meets the platform's loudness (export.py --normalize)
sh("export.py", *argv, stage="export")
@@ -1474,22 +1557,28 @@ def main() -> int:
ck = proj.get("check")
check_result = None
exit_code = 0
- if ck and ck.get("platform") and not STATE.dry_run:
- proc = run_tool([str(HERE / "check.py"), output, "--platform", ck["platform"], "--json"]
- + (["--content"] if ck.get("content") is True else []) + child_args())
- try:
- check_result = json.loads(proc.stdout)
- except ValueError:
- check_result = {"error": proc.stderr.strip()[-300:]}
+ # a platform, --content's rows, or both: the same rule a plan's check step follows (a
+ # {"content": true} stage without a platform used to be skipped without a word)
+ wants_check = bool(ck) and (bool(ck.get("platform")) or ck.get("content") is True)
+ check_label = ck.get("platform") if ck and ck.get("platform") else "content"
+ if wants_check and STATE.dry_run:
+ record_skip("[dry-run] the check stage is not run: a dry run writes no output to check",
+ "check_dry_run", platform=ck.get("platform"), content=ck.get("content") is True)
+ elif wants_check:
+ check_result = run_check(output, ck.get("platform"), ck.get("content") is True)
if check_result.get("failed"):
info(f"check: {check_result['failed']} FAIL — " + "; ".join(f"{r['check']}={r['value']} ({r['fix']})" for r in check_result["checks"] if r["status"] == "FAIL"))
exit_code = 1
elif check_result.get("error") or check_result.get("status") == "failed":
- info(f"check: could not run check.py — {check_result.get('error')}")
+ err = check_result.get("error")
+ info(f"check: could not run check.py — {err.get('message') if isinstance(err, dict) else err}")
exit_code = 1
else:
- info(f"check: OK for {ck['platform']}")
+ info(f"check: OK for {check_label}")
stages_done.append("check")
+ elif "check" in proj:
+ record_skip('the check stage names neither "platform" nor "content": true, so nothing was checked',
+ "check_nothing_to_check", keys=sorted(ck or {}))
if not args.keep and not args.work:
# Also clean up on --dry-run: a dry run still creates this directory (and some steps,
@@ -1502,14 +1591,14 @@ def main() -> int:
# The deliverable is written and verified, but it does not meet the requested platform
# spec (or the check itself could not run): a failed delivery, reported as one.
failed_rows = [r["check"] for r in (check_result or {}).get("checks", []) if r.get("status") == "FAIL"]
- die(f"rendered {output} but the {ck['platform']} check failed" + (f": {', '.join(failed_rows)}" if failed_rows else ""),
+ die(f"rendered {output} but the {check_label} check failed" + (f": {', '.join(failed_rows)}" if failed_rows else ""),
kind="verification", output=output, dry_run=STATE.dry_run, stages=stages_done, check=check_result,
probe=probe(output, role="output"))
info(f"rendered {output} via {' → '.join(stages_done)}")
cache_report = None
if args.cache:
cache_report = {"dir": CACHE["dir"], "ffmpeg": CACHE["ffmpeg"],
- "hits": CACHE["hits"], "misses": CACHE["misses"],
+ "hits": CACHE["hits"], "misses": CACHE["misses"], "miss_reasons": CACHE["miss_reasons"],
"saved_seconds": round(CACHE["saved_seconds"], 1),
"entries": CACHE["entries"]}
if STATE.dry_run:
@@ -1524,7 +1613,7 @@ def main() -> int:
snap_report["clips"] = snap_reports
emit(output, stages=stages_done, check=check_result, snap=snap_report, cache=cache_report,
caption=caption_report,
- verification=[{"step": "check", "ok": True, "platform": ck["platform"]}] if check_result else [])
+ verification=[{"step": "check", "ok": True, "platform": ck.get("platform")}] if check_result else [])
return 0
diff --git a/scripts/report.py b/scripts/report.py
index 0b55449..1701c0d 100755
--- a/scripts/report.py
+++ b/scripts/report.py
@@ -17,32 +17,57 @@ import os
import sys
import tempfile
from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, List, Optional, Tuple
from _common import STATE, add_common, apply_common, child_args, die, emit, info, probe, read_text_or_die, run_tool
+from _common.emit import record_skip, warn
HERE = Path(__file__).resolve().parent
-def sheet_b64(path: str, tiles: str = "4x2", width: int = 1200) -> Optional[str]:
+def _child_failure(proc: Any, doc: Any) -> Dict[str, Any]:
+ """Why a sibling tool produced nothing usable: its own error (kind, reason, message), or the
+ last stderr line and the exit code when it printed no failure document."""
+ err = (doc or {}).get("error") if isinstance(doc, dict) and isinstance(doc.get("error"), dict) else {}
+ out: Dict[str, Any] = {"exit_code": proc.returncode,
+ "message": str(err.get("message") or (proc.stderr.strip().splitlines() or ["?"])[-1])[:300]}
+ for k in ("kind", "reason"):
+ if err.get(k):
+ out[k] = err[k]
+ return out
+
+
+def sheet_b64(path: str, tiles: str = "4x2", width: int = 1200) -> "Tuple[Optional[str], Dict[str, Any]]":
+ """(the contact sheet as base64, or None; why there is none)."""
+ if STATE.dry_run:
+ return None, {"reason": "dry_run", "message": "a dry run draws no contact sheet"}
with tempfile.TemporaryDirectory(prefix="ffskill_report_") as tmp:
png = os.path.join(tmp, "sheet.png")
- proc = run_tool([str(HERE / "look.py"), path, "--tiles", tiles, "--width", str(width), "-o", png] + child_args())
+ proc = run_tool([str(HERE / "look.py"), path, "--tiles", tiles, "--width", str(width), "-o", png, "--json"] + child_args())
+ try:
+ doc = json.loads(proc.stdout)
+ except ValueError:
+ doc = None
if proc.returncode != 0 or not os.path.exists(png):
- return None
- return base64.b64encode(Path(png).read_bytes()).decode("ascii")
+ return None, dict(_child_failure(proc, doc), reason=((doc or {}).get("error") or {}).get("reason") or "look_failed")
+ return base64.b64encode(Path(png).read_bytes()).decode("ascii"), {}
-def loudness(path: str) -> Dict[str, Any]:
+def loudness(path: str) -> "Tuple[Dict[str, Any], Dict[str, Any]]":
+ """(lufs/tp/lra, or {}; why there is no measurement)."""
proc = run_tool([str(HERE / "loudness.py"), path, "--measure-only"] + child_args())
try:
d = json.loads(proc.stdout)
- return {"lufs": round(float(d["input_i"]), 1), "tp": round(float(d["input_tp"]), 1), "lra": round(float(d["input_lra"]), 1)}
- except (ValueError, KeyError):
- return {}
+ except ValueError:
+ d = None
+ try:
+ return {"lufs": round(float(d["input_i"]), 1), "tp": round(float(d["input_tp"]), 1), "lra": round(float(d["input_lra"]), 1)}, {}
+ except (TypeError, ValueError, KeyError):
+ return {}, dict(_child_failure(proc, d), reason="loudness_not_measured")
-def check(path: str, platform: str) -> Optional[Dict[str, Any]]:
+def check(path: str, platform: str) -> "Tuple[Optional[Dict[str, Any]], Dict[str, Any]]":
+ """(check.py's document, or None when it could not run at all; why)."""
proc = run_tool([str(HERE / "check.py"), path, "--platform", platform, "--json"] + child_args())
try:
doc = json.loads(proc.stdout)
@@ -51,10 +76,10 @@ def check(path: str, platform: str) -> Optional[Dict[str, Any]]:
if not isinstance(doc, dict) or "checks" not in doc:
# check.py could not run at all (missing ffmpeg, unreadable file): its failure document
# has no rows to render. A failed *verification* still carries its rows and is shown.
- reason = ((doc or {}).get("error") or {}).get("message") or (proc.stderr.strip().splitlines() or ["?"])[-1]
- info(f"check.py could not run: {reason[:200]}")
- return None
- return doc
+ why = dict(_child_failure(proc, doc), reason=((doc or {}).get("error") or {}).get("reason") or "check_failed")
+ info(f"check.py could not run: {why['message'][:200]}")
+ return None, why
+ return doc, {}
def fmt_dur(sec: Optional[float]) -> str:
@@ -167,15 +192,44 @@ def main() -> int:
die("--after is required (or --pack FILE for a social pack table)")
after = probe(args.after)
before = probe(args.before) if args.before else None
- ld_after = loudness(args.after) if after.get("audio") else {}
- ld_before = loudness(args.before) if before and before.get("audio") else {}
- chk = check(args.after, args.platform) if args.platform else None
- sheets = {}
+ # every section the report would show is either in it or accounted for: a section that could
+ # not be built is a failing verification step (and a warning), never just absent from the page
+ verification: List[Dict[str, Any]] = []
+
+ def section(step: str, which: Optional[str], why: Dict[str, Any], what: str) -> None:
+ entry: Dict[str, Any] = {"step": step, "ok": not why}
+ if which:
+ entry["file"] = which
+ if why:
+ if why.get("reason") == "dry_run":
+ record_skip(f"{what}: {why['message']}", "report_section_dry_run", section=step, file=which)
+ return
+ entry.update({k: v for k, v in why.items() if k != "message"}, message=why.get("message"))
+ warn(f"the report has no {what}: {why.get('message')}", "report_section_missing", section=step, file=which,
+ reason=why.get("reason"))
+ verification.append(entry)
+
+ ld_after: Dict[str, Any] = {}
+ ld_before: Dict[str, Any] = {}
+ if after.get("audio"):
+ ld_after, why = loudness(args.after)
+ section("loudness", "after", why, "loudness for --after")
+ if before and before.get("audio"):
+ ld_before, why = loudness(args.before)
+ section("loudness", "before", why, "loudness for --before")
+ chk = None
+ if args.platform:
+ chk, why = check(args.after, args.platform)
+ section("compliance", None, why, f"{args.platform} compliance table")
+ if chk is not None and verification:
+ # ok: the table is in the report; passed: what the table says about the file
+ verification[-1].update(platform=args.platform, passed=bool(chk.get("ok")))
+ sheets: Dict[str, Optional[str]] = {}
if not args.no_sheets:
- if before and before.get("video"):
- sheets["before"] = sheet_b64(args.before)
- if after.get("video"):
- sheets["after"] = sheet_b64(args.after)
+ for which, path, meta in (("before", args.before, before), ("after", args.after, after)):
+ if meta and meta.get("video"):
+ sheets[which], why = sheet_b64(path)
+ section("contact_sheet", which, why, f"{which} contact sheet")
commands = read_text_or_die(args.commands, "--commands").splitlines() if args.commands else []
notes = read_text_or_die(args.notes, "--notes") if args.notes else ""
title = args.title or f"Delivery report — {Path(args.after).name}"
@@ -235,7 +289,7 @@ def main() -> int:
except OSError as e:
die(f"cannot write {output}: {e}", kind="output")
info(f"wrote {output} ({os.path.getsize(output) / 1024:.0f} KB)")
- emit(None, report=output, check=chk, verification=([{"step": "exists", "ok": True}] if not STATE.dry_run else []))
+ emit(None, report=output, check=chk, verification=([{"step": "exists", "ok": True}] + verification if not STATE.dry_run else []))
if not args.json:
print(output)
return 0
diff --git a/scripts/reverse.py b/scripts/reverse.py
index 480177c..b56bf49 100644
--- a/scripts/reverse.py
+++ b/scripts/reverse.py
@@ -15,6 +15,7 @@ import argparse
import sys
from _common import add_common, apply_common, aac_args, cfr_args, default_output, die, emit, ffmpeg_base, info, probe, run, video_args, X264_PRESETS, fmt_secs
+from _common.emit import warn
def main() -> int:
@@ -35,8 +36,9 @@ def main() -> int:
output = args.output or default_output(args.input, "reverse")
if (meta.get("duration") or 0) > 60:
- info(f"warning: reverse buffers every decoded frame in memory; {meta['duration']:.0f}s of "
- f"{meta['video']['width']}x{meta['video']['height']} can exhaust RAM -- cut the part to reverse first (cut.py)")
+ warn(f"reverse buffers every decoded frame in memory; {meta['duration']:.0f}s of "
+ f"{meta['video']['width']}x{meta['video']['height']} can exhaust RAM -- cut the part to reverse first (cut.py)",
+ "memory_heavy", duration=meta["duration"])
cmd = ffmpeg_base() + ["-i", args.input, "-vf", "reverse"]
if has_audio:
cmd += ["-af", "areverse"]
diff --git a/scripts/sequence.py b/scripts/sequence.py
index 0c10952..367f868 100644
--- a/scripts/sequence.py
+++ b/scripts/sequence.py
@@ -21,6 +21,7 @@ import tempfile
from pathlib import Path
from _common import add_common, apply_common, default_output, die, emit, ffmpeg_base, info, probe, run, video_args, X264_PRESETS, concat_list_line, fmt_secs
+from _common.emit import warn
def even(n: float) -> int:
@@ -97,8 +98,9 @@ def main() -> int:
# to be dropped silently, giving a shorter video; name the gap and what was left out
later = _frames_after(directory, args.pattern, i)
if later:
- info(f"WARNING: frame {args.pattern % i} is missing; {len(later)} later frame(s) "
- f"(from {later[0].name}) are NOT in the sequence -- fill the gap or rename them")
+ warn(f"frame {args.pattern % i} is missing; {len(later)} later frame(s) "
+ f"(from {later[0].name}) are NOT in the sequence -- fill the gap or rename them", "sequence_gap",
+ missing=args.pattern % i, frames_left_out=len(later), first_left_out=later[0].name)
frame_meta = probe(str(frames[0]))
if not frame_meta.get("video"):
diff --git a/scripts/silence.py b/scripts/silence.py
index 3236bc9..e969af5 100755
--- a/scripts/silence.py
+++ b/scripts/silence.py
@@ -22,6 +22,7 @@ from typing import List, Tuple
from _common import (filler_spans, FILLER_WORDS, FILLER_AMBIGUOUS, FILLER_DISCOURSE_MARKERS,
FILLER_PAD, read_text_or_die)
from _common import asr as _asr
+from _common.emit import warn
from _common import detect_silences as detect, STATE, video_args, add_common, apply_common, audio_codec_for, cfr_args, default_output, die, emit, ffmpeg_base, info, is_audio_output, print_json, probe, run, X264_PRESETS, measured_level_dbfs, fmt_secs
@@ -101,13 +102,14 @@ def resolve_filler(args, meta):
"timestamps) or --transcribe. There is no way to find a filler word without them -- "
"cutting the short quiet blips instead would remove real speech.", kind="input")
source, engine, raw, transcription = None, None, None, {}
+ untimed: list = [] # transcript words without a usable timing (a --words file can carry them)
if args.words:
try:
raw = json.loads(read_text_or_die(args.words, "--words"))
except ValueError as exc:
die(f"--words {args.words}: not readable JSON ({exc})", kind="input")
source = f"whisper-json:{args.words}"
- words, had_segments = _words_from_transcript(raw)
+ words, had_segments, untimed = _words_from_transcript(raw)
if not words:
if had_segments:
die(f"the transcript in {args.words} has segment timings but no word timings; "
@@ -186,7 +188,8 @@ def resolve_filler(args, meta):
wordlist |= set(_word_list(args.filler_extra))
wordlist -= set(_word_list(args.filler_keep))
- spans = filler_spans(words, wordlist, pad=args.filler_pad)
+ kept: dict = {}
+ spans = filler_spans(list(words) + untimed, wordlist, pad=args.filler_pad, stats=kept)
removed_words = sorted({s["word"] for s in spans})
warnings = []
ambiguous = sorted(set(FILLER_AMBIGUOUS.get(lang, ())) & set(
@@ -210,12 +213,21 @@ def resolve_filler(args, meta):
"removed_seconds": round(sum(s["end"] - s["start"] for s in spans), 3),
"word_timings": len(words), "list": listname, "removed_words": removed_words,
"warnings": warnings,
+ # listed words this run did not cut, and why: a transcript word with no timing has
+ # nothing to cut on, and one longer than a filler is a held vowel someone meant
+ "matched_untimed": kept.get("matched_untimed", 0),
+ "matched_too_long": kept.get("matched_too_long", 0),
}
+ if block["matched_untimed"]:
+ warn(f"--filler: {block['matched_untimed']} listed word(s) in the transcript carry no usable timing and were "
+ "not removed (nothing measured to cut on)", "filler_words_untimed", count=block["matched_untimed"])
return block, [(s["start"], s["end"]) for s in spans]
def _words_from_transcript(data):
- """([{word,start,end}], whether the document had segments at all) from a whisper JSON."""
+ """([{word,start,end}], whether the document had segments at all, [{word}] of the words
+ that carried no usable timing) from a whisper JSON. The third list lets --filler count a
+ listed word it could not cut instead of dropping it unseen."""
raw, had_segments = [], False
if isinstance(data, dict):
raw = list(data.get("words") or [])
@@ -225,7 +237,7 @@ def _words_from_transcript(data):
raw.extend((seg or {}).get("words") or [])
elif isinstance(data, list):
raw = list(data)
- out = []
+ out, untimed = [], []
for w in raw:
if not isinstance(w, dict):
continue
@@ -233,8 +245,8 @@ def _words_from_transcript(data):
out.append({"word": str(w.get("word") or w.get("text") or ""),
"start": float(w["start"]), "end": float(w["end"])})
except (KeyError, TypeError, ValueError):
- continue
- return out, had_segments
+ untimed.append({"word": str(w.get("word") or w.get("text") or "")})
+ return out, had_segments, untimed
def main() -> int:
@@ -353,7 +365,7 @@ def main() -> int:
f"{filler_info['removed_seconds']:.2f}s, from {filler_info['word_timings']} word timings "
f"({filler_info['lang']}, list {filler_info['list']})")
for warning in filler_info.get("warnings") or []:
- info("warning: " + warning)
+ warn(warning, "filler_removal_caution")
if getattr(args, "_asr_notes", None):
# an English-only transcript made on an assumption (asr.assumed_english_note): the
# bridge already logged the warning; the result document carries it at the top level
diff --git a/scripts/sync.py b/scripts/sync.py
index aee74f5..57188be 100755
--- a/scripts/sync.py
+++ b/scripts/sync.py
@@ -33,6 +33,7 @@ import sys
from typing import Dict, List
from _common import video_args, add_common, apply_common, emit, aac_args, audio_codec_for, default_output, die, ffmpeg_base, info, probe, require_tool, run, run_analysis, x264_args, decode_pcm_mono, rms_envelope
+from _common.emit import record_skip, warn
SR = 8000 # decode sample rate
@@ -218,6 +219,9 @@ def measure_one(reference: str, path: str, args) -> Dict:
offset, score = measure_offset(reference, path, 0.0, args.analyze_seconds, args.step_ms, args.max_offset, args.fine_ms)
drift_ratio = 1.0
drift_info = None
+ # why a requested --fix-drift was not applied: too_short (no tail window far enough from
+ # the head to measure over) or low_confidence (the tail did not correlate); None when applied
+ skipped_reason = None
if args.fix_drift:
ref_dur = probe(reference)["duration"] or 0.0
sec_dur = probe(path)["duration"] or 0.0
@@ -225,7 +229,10 @@ def measure_one(reference: str, path: str, args) -> Dict:
head_len = min(args.analyze_seconds, overlap_end)
tail_start = overlap_end - args.drift_window
if tail_start <= head_len / 2 + 5:
- info(f"warning: {path} too short to measure drift reliably; skipping drift correction")
+ skipped_reason = "too_short"
+ record_skip(f"{path} too short to measure drift reliably; skipping drift correction", "drift_correction_skipped",
+ path=path, reason=skipped_reason, overlap_seconds=round(overlap_end, 3),
+ drift_window=args.drift_window)
else:
ref_start = tail_start
sec_start = tail_start - offset
@@ -251,10 +258,13 @@ def measure_one(reference: str, path: str, args) -> Dict:
abs(drift_ratio - 1) * 1e6, "long/slow" if drift_ratio > 1 else "short/fast", abs(residual), elapsed),
"confidence": round(end_score, 3)}
else:
- info(f"warning: could not measure drift with confidence for {path}; skipping drift correction")
+ skipped_reason = "low_confidence"
+ record_skip(f"could not measure drift with confidence for {path}; skipping drift correction",
+ "drift_correction_skipped", path=path, reason=skipped_reason,
+ confidence=round(max(0.0, end_score), 3), measured_over_seconds=round(elapsed, 2))
return {"path": path, "offset_seconds": round(offset, 4), "offset_s": round(offset, 4),
"confidence": round(max(0.0, min(1.0, score)), 3), "drift_ratio": drift_ratio, "drift": drift_info,
- "drift_ppm": (drift_info["drift_ppm"] if drift_info else None)}
+ "drift_ppm": (drift_info["drift_ppm"] if drift_info else None), "drift_skipped_reason": skipped_reason}
def main() -> int:
@@ -301,10 +311,11 @@ def main() -> int:
measured = [measure_one(args.reference, p, args) for p in args.sources]
for p, m in zip(args.sources, measured):
if m["confidence"] < 0.1:
- info(f"warning: {p}: low correlation confidence; check that it shares an audio event with the reference")
+ warn(f"{p}: low correlation confidence; check that it shares an audio event with the reference",
+ "sync_low_confidence", path=p, confidence=m["confidence"])
sources_block = [{"path": p, "offset_s": m["offset_s"], "confidence": m["confidence"],
- **({"drift_ppm": m["drift_ppm"]} if args.fix_drift else {})}
+ **({"drift_ppm": m["drift_ppm"], "drift_skipped_reason": m["drift_skipped_reason"]} if args.fix_drift else {})}
for p, m in zip(args.sources, measured)]
if n_sources == 1:
@@ -323,8 +334,8 @@ def main() -> int:
}
if drift_info:
result["drift"] = drift_info
- if score < 0.1:
- info("warning: low correlation confidence; check that both files contain the same audio event")
+ if args.fix_drift:
+ result["drift_skipped_reason"] = m["drift_skipped_reason"]
else:
offset = drift_ratio = drift_info = None # not used below; output writing is 1-source only
result = {"reference": args.reference, "sources": sources_block}
diff --git a/scripts/verify.py b/scripts/verify.py
index e811083..2178f04 100755
--- a/scripts/verify.py
+++ b/scripts/verify.py
@@ -15,6 +15,8 @@ Examples:
python3 verify.py clip.mov --keep --out ./verify_out
"""
import argparse
+import contextlib
+import io
import json
import os
import subprocess
@@ -22,7 +24,7 @@ import sys
import tempfile
import time
from pathlib import Path
-from typing import Dict, List
+from typing import Any, Dict, List, Optional, Tuple
from _common import STATE, add_common, apply_common, die, emit, info, probe
@@ -67,26 +69,77 @@ def collect(paths: List[str]) -> List[Path]:
return files
+def _cell(message: str) -> str:
+ """A failure message as one Markdown table cell: one line, no pipes, at most 200 characters."""
+ return " / ".join(line.strip() for line in str(message).splitlines() if line.strip()).replace("|", "/")[:200]
+
+
+def _probe_capturing(path: str) -> "Tuple[Optional[Dict[str, Any]], Dict[str, Any]]":
+ """probe() in this process, with its failure kept: (the probe, {}) or (None, the failure's
+ kind/reason/message). probe() dies on an unreadable file; its failure document is read back
+ instead of a fixed string standing in for it (one JSON document per run: it is not printed)."""
+ was_json, STATE.json = STATE.json, True
+ out, err = io.StringIO(), io.StringIO()
+ try:
+ with contextlib.redirect_stdout(out), contextlib.redirect_stderr(err):
+ return probe(path), {}
+ except SystemExit as exc:
+ try:
+ doc = json.loads(out.getvalue())
+ except ValueError:
+ doc = {}
+ e = doc.get("error") if isinstance(doc.get("error"), dict) else {}
+ failure = {k: e[k] for k in ("kind", "reason", "message") if e.get(k)}
+ failure.setdefault("message", (err.getvalue().strip().splitlines() or [f"probe exited {exc.code}"])[-1])
+ return None, failure
+ finally:
+ STATE.json = was_json
+
+
def step(name: str, argv: List[str], timeout: float) -> Dict:
+ """Run one step and keep what its tool said: `ok` (it completed), and on a failure the tool's
+ own `kind`, `reason` and `message` (its --json failure document), `error` being that message
+ for the table. A completed step also carries the tool's `verified`."""
t0 = time.time()
if argv[0] == "__check_hdr__":
- was_json, STATE.json = STATE.json, False # probe()'s die() would print a JSON document of its own
- try:
- v = probe(argv[1]).get("video") or {}
- ok = bool(v.get("bt2020_or_hdr")) and v.get("bit_depth", 8) >= 10
- err = "" if ok else f"re-encode lost HDR: {v.get('color_transfer')}/{v.get('pix_fmt')}"
- except SystemExit:
- ok, err = False, "output missing"
- finally:
- STATE.json = was_json
- return {"step": name, "ok": ok, "seconds": round(time.time() - t0, 1), "error": err}
+ meta, failure = _probe_capturing(argv[1])
+ if meta is None:
+ return dict({"step": name, "ok": False, "seconds": round(time.time() - t0, 1), "error": _cell(failure.get("message", ""))}, **failure)
+ v = meta.get("video") or {}
+ ok = bool(v.get("bt2020_or_hdr")) and v.get("bit_depth", 8) >= 10
+ r: Dict[str, Any] = {"step": name, "ok": ok, "seconds": round(time.time() - t0, 1),
+ "error": "" if ok else f"re-encode lost HDR: {v.get('color_transfer')}/{v.get('pix_fmt')}"}
+ if not ok:
+ r.update(kind="verification", reason="hdr_lost", message=r["error"])
+ return r
try:
- proc = subprocess.run([sys.executable, str(HERE / argv[0])] + argv[1:], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", errors="replace", timeout=timeout)
- ok = proc.returncode == 0
- err = "" if ok else (proc.stderr.strip().splitlines() or ["?"])[-1][:200]
+ proc = subprocess.run([sys.executable, str(HERE / argv[0])] + argv[1:] + ["--json"], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, encoding="utf-8", errors="replace", timeout=timeout)
except subprocess.TimeoutExpired:
- ok, err = False, f"timeout after {timeout:.0f}s"
- return {"step": name, "ok": ok, "seconds": round(time.time() - t0, 1), "error": err}
+ msg = f"timeout after {timeout:.0f}s"
+ return {"step": name, "ok": False, "seconds": round(time.time() - t0, 1), "error": msg,
+ "kind": "timeout", "reason": "step_timeout", "message": msg}
+ try:
+ doc = json.loads(proc.stdout)
+ except ValueError:
+ doc = None
+ doc = doc if isinstance(doc, dict) else {}
+ ok = proc.returncode == 0
+ r = {"step": name, "ok": ok, "seconds": round(time.time() - t0, 1), "error": ""}
+ if ok:
+ if "verified" in doc:
+ r["verified"] = doc["verified"]
+ return r
+ e = doc.get("error") if isinstance(doc.get("error"), dict) else {}
+ r["message"] = str(e.get("message") or (proc.stderr.strip().splitlines() or [f"exit {proc.returncode}"])[-1])
+ r["error"] = _cell(r["message"])
+ for k in ("kind", "reason", "details"):
+ if e.get(k) is not None:
+ r[k] = e[k]
+ if not e:
+ r["reason"] = "step_output_unparsed"
+ r["exit_code"] = proc.returncode
+ return r
def main() -> int:
@@ -117,16 +170,13 @@ def main() -> int:
results = []
for f in files:
info(f"=== {f}")
- entry: Dict = {"file": str(f), "steps": []}
- was_json, STATE.json = STATE.json, False # same: one JSON document per run, printed at the end
- try:
- meta = probe(str(f))
- except SystemExit:
- entry["steps"].append({"step": "probe", "ok": False, "seconds": 0, "error": "ffprobe failed"})
+ entry: Dict = {"file": str(f), "steps": [], "skipped": []}
+ meta, failure = _probe_capturing(str(f))
+ if meta is None:
+ entry["steps"].append(dict({"step": "probe", "ok": False, "seconds": 0, "error": _cell(failure.get("message", ""))}, **failure))
+ entry["skipped"].append({"step": "*", "reason": "input_unreadable", "message": "no other step runs on a file probe cannot read"})
results.append(entry)
continue
- finally:
- STATE.json = was_json
entry["probe"] = {k: meta.get(k) for k in ("duration", "format")}
entry["probe"]["video"] = {k: (meta.get("video") or {}).get(k) for k in ("codec", "width", "height", "fps", "pix_fmt", "hdr_format", "rotation", "variable_frame_rate_suspected")}
entry["probe"]["audio"] = {k: (meta.get("audio") or {}).get(k) for k in ("codec", "channels", "sample_rate")}
@@ -138,6 +188,10 @@ def main() -> int:
seg_end = min(dur, args.seconds) if dur else args.seconds
fast = ["--fast"]
plan = []
+ skipped: List[Dict[str, str]] = entry["skipped"]
+
+ def skip(step_name: str, reason: str, message: str) -> None:
+ skipped.append({"step": step_name, "reason": reason, "message": message})
plan.append(("cut copy", ["cut.py", str(f), "--start", "0", "--end", f"{seg_end:.2f}", "-o", cut]))
if has_v:
plan.append(("cut accurate", ["cut.py", str(f), "--start", "0", "--end", f"{seg_end:.2f}", "--accurate", "-o", f"{stem}_acc.mp4"] + fast))
@@ -152,12 +206,30 @@ def main() -> int:
plan.append(("color to-sdr", ["color.py", f"{stem}_acc.mp4", "--to-sdr", "-o", f"{stem}_sdr.mp4"] + fast))
plan.append(("hdr preserved", ["__check_hdr__", f"{stem}_acc.mp4"]))
plan.append(("probe analyze", ["probe.py", cut, "--analyze"]))
+ else:
+ for n in ("overlay text", "look sheet", "color to-sdr", "hdr preserved", "probe analyze"):
+ skip(n, "quick", "--quick runs probe, copy cut, fit, caption and export only")
+ if not args.quick and not (meta.get("video") or {}).get("bt2020_or_hdr"):
+ for n in ("color to-sdr", "hdr preserved"):
+ skip(n, "not_hdr", "the file is not HDR")
plan.append(("export x", ["export.py", cut, "--preset", "x", "-o", f"{stem}_x.mp4"]))
+ else:
+ for n in ("cut accurate", "fit 9:16", "caption", "overlay text", "look sheet", "color to-sdr", "hdr preserved",
+ "probe analyze", "export x"):
+ skip(n, "no_video", "the file has no video stream")
if has_a and not args.quick:
plan.append(("loudness measure", ["loudness.py", str(f), "--measure-only"]))
plan.append(("silence list", ["silence.py", cut, "--list"]))
if (meta.get("audio") or {}).get("channels", 0) > 2:
plan.append(("audio downmix", ["audio.py", cut, "--downmix", "-o", f"{stem}_st.mp4"]))
+ else:
+ skip("audio downmix", "not_multichannel", "the audio has 2 channels or fewer")
+ elif not has_a:
+ for n in ("loudness measure", "silence list", "audio downmix"):
+ skip(n, "no_audio", "the file has no audio stream")
+ else:
+ for n in ("loudness measure", "silence list", "audio downmix"):
+ skip(n, "quick", "--quick runs probe, copy cut, fit, caption and export only")
for name, argv in plan:
r = step(name, argv, args.timeout)
entry["steps"].append(r)
@@ -179,6 +251,8 @@ def main() -> int:
lines.append("|---|---|---|---|")
for s in e["steps"]:
lines.append(f"| {s['step']} | {'PASS' if s['ok'] else 'FAIL'} | {s['seconds']}s | {s['error']} |")
+ for s in e.get("skipped") or []:
+ lines.append(f"| {s['step']} | SKIP | - | {s['message']} |")
lines.append("")
report = "\n".join(lines)
if args.report:
diff --git a/scripts/waveform.py b/scripts/waveform.py
index 5628c70..2fc57ed 100644
--- a/scripts/waveform.py
+++ b/scripts/waveform.py
@@ -38,7 +38,7 @@ from _platforms import PLATFORMS, PLATFORM_CHOICES, resolve as resolve_platform
from _common import (add_common, apply_common, aac_args, default_output, die, draws_chroma, emit, ffmpeg_base,
info, load_brand, pad_filters, probe, rgb_chain_to_bt709, run, STATE, validate_color, video_args, X264_PRESETS,
fmt_secs, child_args, run_tool, measured_level_dbfs)
-from _common.emit import absorb_stage_hw
+from _common.emit import absorb_stage_hw, warn
WAVEFORM_MODES = ["point", "line", "p2p", "cline"]
@@ -272,7 +272,8 @@ def main() -> int:
f"{overlap}px: the captions or the app's own UI will sit over the waveform "
f"(lower --vis-height to {max(0.05, (args.height - safe_px) / args.height):.2f})")
if silent_peak is not None:
- info(f"warning: {args.input} is silent (peak {silent_peak:g} dBFS); the visualisation is a flat line")
+ warn(f"{args.input} is silent (peak {silent_peak:g} dBFS); the visualisation is a flat line", "silent_input",
+ path=args.input, peak_db=silent_peak)
notes.append(f"the input audio is silent (peak {silent_peak:g} dBFS, at or below --silence-threshold "
f"{args.silence_threshold:g}): the visualisation is a flat line; pass --on-silent fail to refuse instead")
duration_ok = True
diff --git a/tests/test_analysis.py b/tests/test_analysis.py
index 942598a..b9ba712 100644
--- a/tests/test_analysis.py
+++ b/tests/test_analysis.py
@@ -6,6 +6,7 @@
"""
import json
import os
+import re
import shutil
import subprocess
import sys
@@ -531,11 +532,14 @@ class AnalysisTests(MediaFixtures):
proc = script("verify.py", folder, "--seconds", "2", "--keep", "--out", out, "--report", report, expect_fail=True)
text = report.read_text(encoding="utf-8")
self.assertIn(text.strip(), proc.stdout.strip(), "without --json the report is printed to stdout")
- self.assertIn("| probe | FAIL | 0s | ffprobe failed |", text)
+ # the probe row carries probe's own message (nothing happens silently, part 2), not a fixed "ffprobe failed"
+ probe_row = re.search(r"^\| probe \| FAIL \| 0s \| (ffprobe failed on .*broken\.mp4.*) \|$", text, re.M)
+ self.assertTrue(probe_row, text)
+ self.assertIn("| * | SKIP |", text)
for step in ("overlay text", "look sheet", "probe analyze", "loudness measure", "silence list",
"color to-sdr", "hdr preserved", "audio downmix", "export x"):
self.assertIn(f"| {step} | PASS |", text, f"{step} missing or failed:\n{text}")
- self.assertNotIn("| FAIL |", text.replace("| probe | FAIL | 0s | ffprobe failed |", ""))
+ self.assertNotIn("| FAIL |", text.replace(probe_row.group(0), ""))
self.assertTrue((out / f"{Path(self.hdr).stem}_sdr.mp4").exists())
self.assertTrue((out / f"{Path(self.surround).stem}_st.mp4").exists())
# a per-step timeout is reported per step, never raised
diff --git a/tests/test_contract.py b/tests/test_contract.py
index cc76e14..330ecde 100644
--- a/tests/test_contract.py
+++ b/tests/test_contract.py
@@ -5,6 +5,8 @@ verification policy) holds when the tool actually runs.
python3 tests/test_contract.py
"""
+import contextlib
+import io
import json
import os
import platform
@@ -1583,8 +1585,14 @@ class ContractTests(unittest.TestCase):
"params": {"name": "silence", "arguments": {"argv": [str(self.src), "--list"]}}}) + "\n",
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
self.assertEqual(proc.returncode, 0, proc.stderr)
- resps = {r["id"]: r for r in (json.loads(l) for l in proc.stdout.strip().splitlines())}
- self.assertEqual(sorted(resps), [1, 2, 3, 4, 5, 6], "garbage and notifications produce no response; nothing else is lost")
+ lines = [json.loads(l) for l in proc.stdout.strip().splitlines()]
+ # the line that is not JSON is answered with a JSON-RPC parse error (-32700, id null)
+ # instead of being dropped; a JSON scalar and a notification still get no response
+ parse_errors = [r for r in lines if r["id"] is None]
+ self.assertEqual([r["error"]["code"] for r in parse_errors], [-32700], "a non-JSON line is answered with -32700")
+ self.assertIn("parse error", parse_errors[0]["error"]["message"])
+ resps = {r["id"]: r for r in lines if r["id"] is not None}
+ self.assertEqual(sorted(resps), [1, 2, 3, 4, 5, 6], "scalars and notifications produce no response; nothing else is lost")
self.assertEqual(resps[1]["result"], {})
self.assertEqual(resps[2]["error"]["code"], -32601)
self.assertEqual(resps[3]["error"]["code"], -32000)
@@ -2907,7 +2915,10 @@ class ContractTests(unittest.TestCase):
doc = json.loads(tool("cut", self.src, "--start", "0", "--end", "1", "-o", self.out("vf_dry.mp4"), "--dry-run", "--json").stdout)
self.assertEqual((doc["verified"], doc["verification"]), (False, []))
doc = json.loads(tool("report", "--after", self.src, "-o", self.out("vf_rep.html"), "--json").stdout)
- self.assertTrue(doc["verified"]); self.assertEqual(doc["verification"], [{"step": "exists", "ok": True}])
+ # the report file exists, and (since part 2 of "nothing happens silently") each section it
+ # built is a step too, so a section that could not be built cannot hide behind verified
+ self.assertTrue(doc["verified"]); self.assertEqual(doc["verification"][0], {"step": "exists", "ok": True})
+ self.assertEqual([s["step"] for s in doc["verification"][1:]], ["loudness", "contact_sheet"])
doc = json.loads(tool("look", self.src, "-o", self.out("vf_look.png"), "--json").stdout)
self.assertEqual((doc["verified"], doc["verification"]), (True, [{"step": "probe", "ok": True}]))
@@ -3738,6 +3749,693 @@ class SilentPathTests(unittest.TestCase):
self.assertNotRegex(src, r"(?<![\w.])die\(", f"{fn.__qualname__} refuses in prose only (use refuse())")
+class SilentResultTests(unittest.TestCase):
+ """docs/design-decisions.md, "Nothing happens silently" (part 2): a degraded or partial result
+ is never reported as the asked-for one. Each test fails on 2.10.11's code."""
+
+ @classmethod
+ def setUpClass(cls):
+ require_ffmpeg_or_skip("ffmpeg", "ffprobe")
+ cls.work = Path(tempfile.mkdtemp(prefix="ffskill_silent2_"))
+ cls.src = cls.work / "src.mp4"
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=160x90:rate=25", "-f", "lavfi", "-i", "sine=f=440:r=48000",
+ "-t", "4", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p", "-c:a", "aac", cls.src)
+ cls.late = cls.work / "late.mp4" # the same scene, starting 0.8 s later on the reference timeline
+ ffmpeg("-ss", "0.8", "-i", cls.src, "-c:v", "libx264", "-preset", "ultrafast", "-c:a", "aac", cls.late)
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.work, ignore_errors=True)
+
+ def out(self, name):
+ return self.work / name
+
+ def patched(self, module, patch, *argv):
+ """Run scripts/<module>.py's main() in a child Python with `patch` (source) applied to the
+ imported module first: the narrowest way to make one measurement fail, on every OS."""
+ code = ("import sys; sys.path.insert(0, %r); import %s as m\n%s\nsys.argv = [%r] + %r\nsys.exit(m.main())"
+ % (str(SCRIPTS), module, patch, module + ".py", [str(a) for a in argv]))
+ return subprocess.run([sys.executable, "-c", code], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
+ text=True, encoding="utf-8", errors="replace")
+
+ def project(self, name, **proj):
+ p = self.out(name + ".json")
+ p.write_text(json.dumps(dict({"output": str(self.out(name + ".mp4")), "clips": [{"src": str(self.src)}]}, **proj)),
+ encoding="utf-8")
+ return p
+
+ # ---------------------------------------------------------------- multicam
+ def test_multicam_names_the_camera_it_rendered_not_the_one_asked_for(self):
+ """A range whose camera had not started played camera 0, but `cuts` still named the camera
+ asked for. `cuts` is now what was rendered, and the fallback is recorded."""
+ doc = json.loads(tool("multicam", self.src, self.late, "--switch", "0-2:1,2-4:0", "--dry-run", "--json",
+ "-o", self.out("mc.mp4")).stdout)
+ self.assertEqual(doc["cuts"][0][2], 0, doc["cuts"])
+ fb = doc["diagnostics"]["fallbacks"][0]
+ self.assertEqual((fb["code"], fb["requested_camera"], fb["camera"], fb["start"]), ("camera_not_started", 1, 0, 0.0))
+
+ # ---------------------------------------------------------------- export
+ def test_export_says_it_trimmed_to_the_platform_maximum(self):
+ """A clip over the preset's maximum was cut with only a stderr line."""
+ long_src = self.out("long.mp4")
+ ffmpeg("-f", "lavfi", "-i", "color=c=gray:s=32x32:r=2", "-t", "92", "-c:v", "libx264", "-preset", "ultrafast", long_src)
+ doc = json.loads(tool("export", long_src, "--preset", "reels", "--dry-run", "--json", "-o", self.out("long_reels.mp4")).stdout)
+ self.assertEqual(doc["trimmed"], {"from": 92.0, "to": 90.0})
+ self.assertTrue(any("trimmed from 92.000s" in n for n in doc["notes"]), doc.get("notes"))
+ doc = json.loads(tool("export", long_src, "--preset", "reels", "--allow-long", "--dry-run", "--json",
+ "-o", self.out("long_reels2.mp4")).stdout)
+ self.assertNotIn("trimmed", doc)
+
+ def test_export_an_unmeasured_loudness_is_a_failing_step_not_an_absence(self):
+ """When loudnorm's JSON did not parse, the loudness block and step were left out and the
+ export read verified: true against a platform it never measured."""
+ proc = self.patched("export", "m.measure_loudness = lambda path: {}", self.src, "--preset", "x", "--fast",
+ "-o", self.out("x_unmeasured.mp4"), "--json")
+ self.assertEqual(proc.returncode, 0, proc.stderr[-500:])
+ doc = json.loads(proc.stdout)
+ self.assertEqual((doc["status"], doc["verified"]), ("completed", False))
+ self.assertEqual({k: doc["loudness"][k] for k in ("lufs", "ok", "measured", "reason")},
+ {"lufs": None, "ok": None, "measured": False, "reason": "not_measured"})
+ self.assertIn({"step": "loudness", "ok": False, "platform": "x", "reason": "not_measured"}, doc["verification"])
+ self.assertEqual(doc["diagnostics"]["warnings"][0]["code"], "loudness_not_measured")
+
+ # ---------------------------------------------------------------- report
+ def test_report_records_a_section_it_could_not_build(self):
+ """A contact sheet, loudness or --platform check that failed was left out of the report and
+ the document said verified: true."""
+ patch = ("import subprocess\nreal = m.run_tool\n"
+ "def fake(argv, **k):\n"
+ " if str(argv[0]).endswith(('look.py', 'loudness.py', 'check.py')):\n"
+ " return subprocess.CompletedProcess(argv, 1, 'not json', 'error: injected failure\\n')\n"
+ " return real(argv, **k)\n"
+ "m.run_tool = fake")
+ proc = self.patched("report", patch, "--after", self.src, "--platform", "x", "-o", self.out("rep.html"), "--json")
+ self.assertEqual(proc.returncode, 0, proc.stderr[-500:])
+ doc = json.loads(proc.stdout)
+ self.assertFalse(doc["verified"])
+ self.assertIsNone(doc["check"])
+ failed = {s["step"]: s for s in doc["verification"] if not s["ok"]}
+ self.assertEqual(sorted(failed), ["compliance", "contact_sheet", "loudness"])
+ self.assertEqual(failed["loudness"]["message"], "error: injected failure")
+ self.assertEqual({w["section"] for w in doc["diagnostics"]["warnings"]}, {"compliance", "contact_sheet", "loudness"})
+ # every section built: each is a passing step, and the compliance step says what the table found
+ doc = json.loads(tool("report", "--after", self.src, "--platform", "x", "-o", self.out("rep_ok.html"), "--json").stdout)
+ self.assertTrue(doc["verified"], doc["verification"])
+ self.assertEqual([s["step"] for s in doc["verification"]], ["exists", "loudness", "compliance", "contact_sheet"])
+ self.assertIn("passed", doc["verification"][2])
+
+ # ---------------------------------------------------------------- render
+ def test_render_verified_is_every_stages_and_carries_their_notes(self):
+ """export's loudness miss (completed, verified: false) became render's verified: true, and
+ the stage's note naming the fix was dropped."""
+ proj = self.project("rv", export={"preset": "x", "normalize": False})
+ doc = json.loads(tool("render", proj, "--fast", "--json").stdout)
+ self.assertEqual((doc["status"], doc["verified"]), ("completed", False))
+ self.assertIn({"step": "loudness", "ok": False, "platform": "x", "stage": "export"}, doc["verification"])
+ self.assertTrue(any(n.startswith("export: loudness ") for n in doc["notes"]), doc.get("notes"))
+
+ def test_render_runs_a_content_only_check_and_records_a_dry_run_skip(self):
+ """"check": {"content": true} without a platform was skipped without a word (a plan's
+ check step honoured it); a check skipped by --dry-run left no record."""
+ proj = self.project("rc", check={"content": True})
+ doc = json.loads(tool("render", proj, "--fast", "--json").stdout)
+ self.assertIn("check", doc["stages"])
+ self.assertTrue({"black", "frozen", "silence"} <= {r["check"] for r in doc["check"]["checks"]})
+ self.assertIn({"step": "check", "ok": True, "platform": None}, doc["verification"])
+ dry = json.loads(tool("render", proj, "--dry-run", "--overwrite", "--json").stdout)
+ self.assertEqual([s["code"] for s in dry["diagnostics"]["skipped"]], ["check_dry_run"])
+ proj = self.project("rc_empty", check={})
+ doc = json.loads(tool("render", proj, "--fast", "--json").stdout)
+ self.assertEqual([s["code"] for s in doc["diagnostics"]["skipped"]], ["check_nothing_to_check"])
+
+ def test_check_output_that_does_not_parse_is_a_structured_failure(self):
+ """execute_plan and the check stage stored check.py's unparsed stderr as a bare string."""
+ import render
+ from unittest import mock
+ fake = subprocess.CompletedProcess(["check.py"], 1, "garbage", "Traceback ...\nValueError: boom\n")
+ with mock.patch.object(render, "run_tool", return_value=fake), contextlib.redirect_stderr(io.StringIO()):
+ saved = list(_common.STATE.warnings)
+ try:
+ doc = render.run_check(str(self.src), "x", False)
+ finally:
+ _common.STATE.warnings[:] = saved
+ self.assertEqual((doc["status"], doc["ok"]), ("failed", False))
+ self.assertEqual((doc["error"]["reason"], doc["error"]["exit_code"]), ("check_output_unparsed", 1))
+ self.assertIn("ValueError: boom", doc["error"]["message"])
+
+ def test_render_cache_says_why_a_stage_missed_and_replays_what_a_hit_verified(self):
+ """--cache missed silently (no sidecar, a corrupt one, a truncated artifact), stored
+ silently on failure, and a hit carried none of the stage's verification."""
+ cdir = self.out("rcache")
+ proj = self.project("rcache", clips=[{"src": str(self.src), "in": 0, "out": 3}], export={"preset": "x", "normalize": False})
+
+ def entry(stage):
+ """The newest sidecar the cache holds for `stage`."""
+ side = [p for p in Path(cdir).glob("*.json") if json.loads(p.read_text(encoding="utf-8") or "{}").get("stage") == stage]
+ return max(side, key=lambda p: p.stat().st_mtime_ns)
+ render = lambda: json.loads(tool("render", proj, "--fast", "--json", "--overwrite", "--cache", cdir).stdout) # noqa: E731
+ first = render()
+ self.assertEqual({m["reason"] for m in first["cache"]["miss_reasons"]}, {"not_cached"})
+ again = render()
+ self.assertEqual(again["cache"]["hits"], ["clips", "export"])
+ self.assertFalse(again["verified"], "a cached export that missed its loudness is still unverified")
+ self.assertIn({"step": "loudness", "ok": False, "platform": "x", "stage": "export"}, again["verification"])
+ entry("export").write_text("{not json", encoding="utf-8")
+ doc = render()
+ self.assertEqual([(m["stage"], m["reason"]) for m in doc["cache"]["miss_reasons"]], [("export", "sidecar_corrupt")])
+ self.assertEqual(doc["diagnostics"]["warnings"][0]["code"], "cache_entry_damaged")
+ self.assertEqual(json.loads((Path(cdir) / ".ffskill-audit.jsonl").read_text(encoding="utf-8").splitlines()[-1])["event"],
+ "cache_entry_damaged")
+ with open(entry("export").with_suffix(".mp4"), "r+b") as fh:
+ fh.truncate(100)
+ doc = render()
+ self.assertEqual([(m["stage"], m["reason"]) for m in doc["cache"]["miss_reasons"]], [("export", "artifact_truncated")])
+
+ def test_render_cache_store_failure_banner_fallback_and_unreadable_input_are_recorded(self):
+ import render
+ from unittest import mock
+ saved = {b: list(getattr(_common.STATE, b)) for b in ("warnings", "fallbacks")}
+ saved_cache = dict(render.CACHE)
+ try:
+ with contextlib.redirect_stderr(io.StringIO()):
+ render.CACHE.update(dir=str(self.out("no_such_cache_dir")), ffmpeg="x")
+ render.cache_store("export", "k" * 40, str(self.src), 1.0, {"verified": True})
+ self.assertEqual(_common.STATE.warnings[-1]["code"], "cache_store_failed")
+ with mock.patch.object(render.subprocess, "run", side_effect=OSError("no ffprobe")):
+ pair = render.ffmpeg_banner()
+ self.assertEqual((_common.STATE.fallbacks[-1]["code"], _common.STATE.fallbacks[-1]["used"]),
+ ("cache_ffmpeg_banner_unreadable", pair))
+ with mock.patch.object(render, "file_key", side_effect=PermissionError(13, "denied")):
+ self.assertEqual(render._content_hash(str(self.src)), "missing")
+ self.assertEqual(_common.STATE.warnings[-1]["code"], "cache_input_unreadable")
+ finally:
+ for b, v in saved.items():
+ setattr(_common.STATE, b, v)
+ render.CACHE.clear()
+ render.CACHE.update(saved_cache)
+
+ # ---------------------------------------------------------------- batch
+ def batch_folder(self, name, steps):
+ folder = self.out(name)
+ folder.mkdir()
+ shutil.copy(self.src, folder / "a.mp4")
+ recipe = folder / "batch.json"
+ recipe.write_text(json.dumps({"glob": "*.mp4", "output_dir": "out", "suffix": "_b", "steps": steps}), encoding="utf-8")
+ return folder, recipe
+
+ def test_batch_a_failed_item_says_which_step_failed_and_why(self):
+ """A failed item was {"ok": false} with no step, kind or message."""
+ folder, recipe = self.batch_folder("bfail", [["cut.py", "{in}", "--start", "0", "--end", "1", "--quality", "99", "-o", "{out}"]])
+ doc = json.loads(tool("batch", folder, "--recipe", recipe, "--json", check=False).stdout)
+ err = doc["results"][0]["error"]
+ self.assertEqual((err["step"], err["tool"], err["kind"]), (0, "cut.py", "input"))
+ self.assertIn("--quality", err["message"])
+
+ def test_batch_counts_a_completed_but_unverified_step_as_ok_and_says_it_is_unverified(self):
+ """design-decisions: a spec miss is completed + verified false, not a failure. The row
+ said ok and nothing more; it now says verified: false and which step's check failed, and
+ the batch's own verified is the conjunction of its items'."""
+ folder, recipe = self.batch_folder("bunver", [["export.py", "{in}", "--preset", "x", "-o", "{out}"]])
+ doc = json.loads(tool("batch", folder, "--recipe", recipe, "--fast", "--json").stdout)
+ row = doc["results"][0]
+ self.assertEqual((row["ok"], row["verified"]), (True, False))
+ self.assertEqual(row["unverified"], [{"step": 0, "tool": "export", "failed": ["loudness"]}])
+ self.assertEqual((doc["processed"], doc["verified"]), (1, False))
+ self.assertEqual(doc["verification"], [{"step": "item", "file": row["file"], "ok": False}])
+
+ def test_batch_serves_a_cached_output_only_while_it_is_unchanged_and_records_a_reset_cache(self):
+ """A cache hit checked only that the output existed, so a replaced output was served as
+ "cached ok"; a corrupt cache file was reset without a word."""
+ folder, recipe = self.batch_folder("bcache", [["fit.py", "{in}", "--height", "64", "-o", "{out}"]])
+ run = lambda: json.loads(tool("batch", folder, "--recipe", recipe, "--fast", "--overwrite", "--json").stdout) # noqa: E731
+ run()
+ self.assertTrue(run()["results"][0]["cached"])
+ out = Path(run()["results"][0]["output"])
+ shutil.copy(self.src, out) # someone replaced the delivered file
+ doc = run()
+ row = doc["results"][0]
+ self.assertFalse(row.get("cached"))
+ self.assertEqual(row["cache_miss"]["reason"], "output_changed")
+ self.assertEqual(doc["diagnostics"]["warnings"][0]["code"], "batch_cache_output_changed")
+ cache = folder / "out" / ".ffskill_cache.json"
+ cache.write_text("{truncated", encoding="utf-8")
+ doc = run()
+ self.assertEqual(doc["diagnostics"]["warnings"][0]["code"], "batch_cache_reset")
+ rec = json.loads((folder / "out" / ".ffskill-audit.jsonl").read_text(encoding="utf-8").splitlines()[-1])
+ self.assertEqual(rec["event"], "batch_cache_reset")
+ # the same file, not the same spelling: Windows' temp dir comes back as RUNNER~1 or runneradmin
+ self.assertTrue(os.path.samefile(rec["details"]["path"], cache), rec["details"]["path"])
+
+ def run_batch_in_process(self, folder, recipe, process):
+ """batch.main() with process() replaced; returns (exit code, document)."""
+ import batch
+ import signal
+ from unittest import mock
+ out = io.StringIO()
+ handlers = {s: signal.getsignal(s) for s in (signal.SIGINT, getattr(signal, "SIGTERM", signal.SIGINT))}
+ code = 0
+ try:
+ with mock.patch.object(batch, "process", process), \
+ mock.patch.object(sys, "argv", ["batch.py", str(folder), "--recipe", str(recipe), "--force", "--json"]), \
+ contextlib.redirect_stdout(out), contextlib.redirect_stderr(io.StringIO()):
+ _common.STATE.reset()
+ try:
+ batch.main()
+ except SystemExit as exc:
+ code = exc.code
+ finally:
+ for s, h in handlers.items():
+ signal.signal(s, h)
+ _common.STATE.reset()
+ return code, json.loads(out.getvalue())
+
+ def test_batch_interrupted_reports_every_item_and_the_right_total(self):
+ """Ctrl-C dropped the running and the not-started items: `total` counted only what had
+ finished."""
+ folder, recipe = self.batch_folder("bint", [["fit.py", "{in}", "--height", "64", "-o", "{out}"]])
+ for name in ("b.mp4", "c.mp4"):
+ shutil.copy(self.src, folder / name)
+ calls = []
+
+ def process(src, recipe, outdir, work, deadline=None):
+ calls.append(src.name)
+ if len(calls) == 2:
+ raise KeyboardInterrupt
+ return {"file": str(src), "output": str(outdir / src.name), "ok": True, "verified": True, "seconds": 0.0}
+ code, doc = self.run_batch_in_process(folder, recipe, process)
+ self.assertEqual((code, doc["error"]["kind"], doc["total"], doc["processed"]), (130, "interrupted", 3, 1))
+ rows = doc["results"]
+ self.assertEqual([r["ok"] for r in rows], [True, False, False])
+ self.assertTrue(rows[1]["interrupted"])
+ self.assertEqual(rows[2]["skipped"], "interrupted")
+
+ def test_batch_a_worker_that_raised_names_its_error_not_its_exit_code(self):
+ """failed_row stored str(SystemExit), which is the exit code: "1"."""
+ folder, recipe = self.batch_folder("bexit", [["fit.py", "{in}", "--height", "64", "-o", "{out}"]])
+
+ def process(src, recipe, outdir, work, deadline=None):
+ raise SystemExit(1)
+ code, doc = self.run_batch_in_process(folder, recipe, process)
+ err = doc["results"][0]["error"]
+ self.assertIsInstance(err, dict)
+ self.assertEqual((err["exception"], err["exit_code"]), ("SystemExit", 1))
+ self.assertNotEqual(err["message"], "1")
+
+ def test_batch_recipe_key_records_an_unreadable_project(self):
+ import batch
+ saved = list(_common.STATE.warnings)
+ try:
+ with contextlib.redirect_stderr(io.StringIO()):
+ batch.recipe_key({"project": str(self.out("no_such_project.json"))})
+ self.assertEqual(_common.STATE.warnings[-1]["code"], "recipe_project_unreadable")
+ finally:
+ _common.STATE.warnings[:] = saved
+
+ # ---------------------------------------------------------------- verify
+ def test_verify_keeps_the_tools_own_failure_and_lists_the_steps_it_skipped(self):
+ """A step's error was its last stderr line or a fixed string ("ffprobe failed"), its kind was
+ lost, and steps that did not apply were dropped."""
+ garbage = self.out("garbage.mp4")
+ garbage.write_bytes(b"\0not media" * 100)
+ wav = self.out("tone.wav")
+ ffmpeg("-i", self.src, "-vn", "-c:a", "pcm_s16le", wav)
+ proc = tool("verify", garbage, wav, "--quick", "--out", self.out("verify_out"), "--json", check=False)
+ doc = json.loads(proc.stdout)
+ bad, audio = doc["files"]
+ self.assertEqual(bad["steps"][0]["kind"], "input")
+ self.assertNotEqual(bad["steps"][0]["error"], "ffprobe failed")
+ self.assertEqual(bad["skipped"][0]["reason"], "input_unreadable")
+ self.assertIn(("export x", "no_video"), [(s["step"], s["reason"]) for s in audio["skipped"]])
+ import verify
+ r = verify.step("cut", ["cut.py", str(self.out("missing.mp4")), "--start", "0", "--end", "1", "-o", str(self.out("v.mp4"))], 120)
+ self.assertEqual((r["ok"], r["kind"]), (False, "input"))
+ self.assertIn("missing.mp4", r["message"])
+
+
+class SilentFallbackTests(unittest.TestCase):
+ """docs/design-decisions.md, "Nothing happens silently" (part 3): the fallbacks, skips and
+ stderr-only warnings that remained, each now a `diagnostics` record or a structured field.
+ In-process where the decision is a function; one small fixture for the tools that run."""
+
+ BUCKETS = ("warnings", "fallbacks", "skipped", "notes")
+
+ @classmethod
+ def setUpClass(cls):
+ cls.tmp = Path(tempfile.mkdtemp(prefix="ffskill_silent3_"))
+
+ @classmethod
+ def tearDownClass(cls):
+ shutil.rmtree(cls.tmp, ignore_errors=True)
+
+ def setUp(self):
+ runner = sys.modules["_common.runner"]
+ self.saved = (_common.STATE.dry_run, _common.STATE.timeout, runner._ENCODERS, runner._BSFS, dict(getattr(runner, "_LIST_ERRORS", {})))
+ _common.STATE.dry_run = False
+ for bucket in self.BUCKETS:
+ setattr(_common.STATE, bucket, [])
+
+ def tearDown(self):
+ runner = sys.modules["_common.runner"]
+ _common.STATE.dry_run, _common.STATE.timeout, runner._ENCODERS, runner._BSFS, errors = self.saved
+ if hasattr(runner, "_LIST_ERRORS"):
+ runner._LIST_ERRORS.clear()
+ runner._LIST_ERRORS.update(errors)
+ for bucket in self.BUCKETS:
+ setattr(_common.STATE, bucket, [])
+
+ def codes(self, bucket):
+ return [e["code"] for e in getattr(_common.STATE, bucket)]
+
+ def media(self):
+ """A 6 s 320x180 clip with a tone, made once per class (each test that runs a tool needs one)."""
+ src = self.tmp / "src.mp4"
+ if not src.exists():
+ require_ffmpeg_or_skip("ffmpeg", "ffprobe")
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=320x180:rate=30", "-f", "lavfi", "-i", f"aevalsrc='{TONE}':s=48000",
+ "-t", "6", "-c:v", "libx264", "-preset", "veryfast", "-pix_fmt", "yuv420p", "-c:a", "aac", src)
+ return src
+
+ class Refused(Exception):
+ pass
+
+ def refuse_raises(self):
+ def refuse(kind, reason, msg, **kw):
+ raise self.Refused(dict(kw, kind=kind, reason=reason, message=msg))
+ return refuse
+
+ # ------------------------------------------------------------------ the runner
+ def test_an_odd_sized_source_scaled_to_even_is_a_recorded_fallback(self):
+ """runner.run()'s even_retry re-ran an encode that failed on odd dimensions with an even
+ scale and said so on stderr only; the output is a pixel smaller than the source."""
+ odd = self.tmp / "odd.mp4"
+ require_ffmpeg_or_skip("ffmpeg", "ffprobe")
+ ffmpeg("-f", "lavfi", "-i", "testsrc=size=161x91:rate=25:duration=1", "-c:v", "libx264", "-pix_fmt", "yuv444p", odd)
+ doc = json.loads(tool("denoise", odd, "-o", self.tmp / "odd_dn.mp4", "--fast", "--json").stdout)
+ fb = doc["diagnostics"]["fallbacks"]
+ self.assertEqual([f["code"] for f in fb], ["odd_dimensions_scaled_even"])
+ self.assertEqual(fb[0]["scale"], _common.EVEN_SCALE)
+ self.assertEqual((doc["probe"]["video"]["width"], doc["probe"]["video"]["height"]), (160, 90))
+
+ def test_a_measurement_skipped_for_a_pending_input_reaches_the_document(self):
+ """dry_run_input_pending(): "measurement skipped" went to stderr only, though
+ design-decisions promised a note. It is a diagnostics.skipped entry, once per path."""
+ _common.STATE.dry_run = True
+ missing = str(self.tmp / "not_yet.mp4")
+ self.assertTrue(_common.dry_run_input_pending(missing))
+ self.assertTrue(_common.dry_run_input_pending(missing))
+ self.assertEqual(self.codes("skipped"), ["measurement_input_pending"])
+ self.assertEqual(_common.STATE.skipped[0]["path"], missing)
+ _common.STATE.dry_run = False
+ self.assertFalse(_common.dry_run_input_pending(missing))
+ # end to end: a dry run of loudness.py on a file an earlier stage would write
+ doc = json.loads(tool("loudness", missing, "-o", self.tmp / "ln.mp4", "--dry-run", "--json").stdout)
+ self.assertEqual([s["code"] for s in doc["diagnostics"]["skipped"]], ["measurement_input_pending"])
+
+ def test_an_unreadable_encoder_list_is_named_not_taken_for_a_missing_encoder(self):
+ """ffmpeg_encoders()/ffmpeg_bsfs() turned a list that could not be read into an empty set,
+ and --codec av1/prores then refused with "this build has neither", --hw's note with "this
+ build has no h264_videotoolbox". The read failure is kept, recorded once, and named."""
+ from unittest import mock
+ runner = sys.modules["_common.runner"]
+ decision = sys.modules["_common.decision"]
+ runner._ENCODERS = runner._BSFS = None
+ with mock.patch.object(runner.subprocess, "run", side_effect=OSError("exec format error")):
+ self.assertEqual(_common.ffmpeg_encoders(), set())
+ self.assertEqual(_common.ffmpeg_bsfs(), set())
+ self.assertIn("exec format error", _common.ffmpeg_list_error("encoders"))
+ self.assertEqual([(w["code"], w["list"]) for w in _common.STATE.warnings],
+ [("ffmpeg_list_unreadable", "encoders"), ("ffmpeg_list_unreadable", "bsfs")])
+ with mock.patch.object(decision, "refuse", self.refuse_raises()):
+ for codec, encoders in (("av1", ["libsvtav1", "libaom-av1"]), ("prores", ["prores_ks"])):
+ with self.assertRaises(self.Refused) as r:
+ decision._cpu_encoder_args(codec, 23, "medium", None)
+ got = r.exception.args[0]
+ self.assertEqual((got["kind"], got["reason"], got["encoders"]), ("ffmpeg", "encoder_list_unreadable", encoders))
+ self.assertIn("exec format error", got["message"])
+ self.assertNotIn("has neither", got["message"])
+ # a list that was read and lacks the encoder is the missing_tool refusal it always was
+ runner._ENCODERS = {"libx264"}
+ with self.assertRaises(self.Refused) as r:
+ decision._cpu_encoder_args("av1", 23, "medium", None)
+ self.assertEqual((r.exception.args[0]["kind"], r.exception.args[0]["reason"]), ("missing_tool", "encoder_missing"))
+ # --hw's CPU note says the list could not be read, not that the build lacks the encoder
+ runner._ENCODERS = set()
+ saved_notes = list(_common.STATE.hw_notes)
+ try:
+ with mock.patch.object(decision, "hw_platform_reason", return_value=None):
+ self.assertIsNone(decision._vt_args("h264", 23, None, True))
+ self.assertTrue(any("could not be confirmed" in n and "exec format error" in n for n in _common.STATE.hw_notes),
+ _common.STATE.hw_notes)
+ finally:
+ _common.STATE.hw_notes[:] = saved_notes
+
+ def test_the_libaom_fallback_and_a_clamped_crf_are_recorded(self):
+ """decision.py chose libaom-av1 when SVT-AV1 was missing, and capped the HEVC HDR line's
+ CRF+2 at 51, without a word."""
+ from unittest import mock
+ decision = sys.modules["_common.decision"]
+ with mock.patch.object(decision, "ffmpeg_encoders", return_value={"libaom-av1"}):
+ args = decision._cpu_encoder_args("av1", 30, "medium", None)
+ decision._cpu_encoder_args("av1", 30, "medium", None)
+ self.assertEqual(args[args.index("-c:v") + 1], "libaom-av1")
+ self.assertEqual(self.codes("fallbacks"), ["av1_encoder_libaom"], "recorded once however many encodes take it")
+ hdr = {"video": {"bt2020_or_hdr": True, "color_transfer": "smpte2084"}}
+ args = decision._cpu_encoder_args("hevc", 50, "medium", hdr)
+ self.assertEqual(args[args.index("-crf") + 1], "51")
+ w = _common.STATE.warnings[-1]
+ self.assertEqual((w["code"], w["requested"], w["asked"], w["used"]), ("crf_clamped", 50, 52, 51))
+ _common.STATE.warnings.clear()
+ decision._cpu_encoder_args("hevc", 20, "medium", hdr)
+ self.assertEqual(self.codes("warnings"), [], "an unclamped CRF records nothing")
+
+ def test_run_keeping_subtitles_records_the_first_attempts_error(self):
+ """A subtitle stream the container cannot hold made the first attempt fail; the retry
+ without it was reported only as dropped_non_av_streams: true, with no reason."""
+ src = self.media()
+ srt = self.tmp / "a.srt"
+ srt.write_text("1\n00:00:00,000 --> 00:00:02,000\nHello\n", encoding="utf-8")
+ subbed = self.tmp / "subbed.mkv"
+ ffmpeg("-i", src, "-i", srt, "-map", "0", "-map", "1", "-c", "copy", "-c:s", "srt", subbed)
+ doc = json.loads(tool("denoise", subbed, "-o", self.tmp / "subbed_dn.mp4", "--fast", "--json").stdout)
+ self.assertTrue(doc["dropped_non_av_streams"])
+ fb = doc["diagnostics"]["fallbacks"][0]
+ self.assertEqual(fb["code"], "non_av_streams_dropped")
+ self.assertNotEqual(fb["returncode"], 0)
+ self.assertTrue(fb["error"], "the first attempt's error is kept")
+
+ def test_an_invalid_ffmpeg_skill_timeout_is_a_recorded_warning(self):
+ """FFMPEG_SKILL_TIMEOUT=abc silently became 1800 s, and -5 silently meant no limit."""
+ src = self.media()
+ for value, limit in (("abc", 1800.0), ("nan", 1800.0), ("-5", 0.0)):
+ env = dict(os.environ, FFMPEG_SKILL_TIMEOUT=value)
+ doc = json.loads(tool("cut", src, "--start", "0", "--end", "1", "-o", self.tmp / "t.mp4", "--dry-run", "--json", env=env).stdout)
+ w = doc["diagnostics"]["warnings"]
+ self.assertEqual([(x["code"], x["value"], x["timeout"]) for x in w], [("timeout_env_invalid", value, limit)])
+ # an explicit --timeout makes the variable irrelevant, and a valid value says nothing
+ env = dict(os.environ, FFMPEG_SKILL_TIMEOUT="abc")
+ doc = json.loads(tool("cut", src, "--start", "0", "--end", "1", "-o", self.tmp / "t.mp4", "--dry-run", "--json", "--timeout", "60", env=env).stdout)
+ self.assertNotIn("diagnostics", doc)
+ doc = json.loads(tool("cut", src, "--start", "0", "--end", "1", "-o", self.tmp / "t.mp4", "--dry-run", "--json",
+ env=dict(os.environ, FFMPEG_SKILL_TIMEOUT="90")).stdout)
+ self.assertNotIn("diagnostics", doc)
+
+ # ------------------------------------------------------------------ tools
+ def test_filler_words_with_no_timing_are_counted(self):
+ """filler_spans() skipped a listed word with no measured timing; silence.py's word reader
+ dropped it even before that. Both are counted (matched_untimed / matched_too_long)."""
+ stats = {}
+ spans = _common.filler_spans([{"word": "um", "start": 1.0, "end": 1.2}, {"word": "um"},
+ {"word": "uh", "start": 2.0, "end": 1.0}, {"word": "um", "start": 3.0, "end": 5.0}],
+ {"um", "uh"}, stats=stats)
+ self.assertEqual(len(spans), 1)
+ self.assertEqual(stats, {"matched_untimed": 2, "matched_too_long": 1})
+ src = self.media()
+ words = self.tmp / "words.json"
+ words.write_text(json.dumps({"words": [{"word": "um", "start": 1.0, "end": 1.2}, {"word": "uh"},
+ {"word": "hello", "start": 2.0, "end": 2.4}]}), encoding="utf-8")
+ doc = json.loads(tool("silence", src, "--filler", "--words", words, "--filler-lang", "en", "--list", "--json").stdout)
+ self.assertEqual((doc["filler"]["matched_untimed"], doc["filler"]["matched_too_long"]), (1, 0))
+ self.assertEqual(doc["filler"]["word_timings"], 2)
+ self.assertEqual([w["code"] for w in doc["diagnostics"]["warnings"]], ["filler_words_untimed"])
+
+ def test_a_skipped_drift_correction_says_why(self):
+ """sync.py --fix-drift on a pair too short to measure skipped the correction with a
+ stderr warning only; drift_skipped_reason and a diagnostics.skipped entry now say so."""
+ src = self.media()
+ mic = self.tmp / "mic.wav"
+ ffmpeg("-ss", "1.5", "-i", src, "-vn", "-c:a", "pcm_s16le", mic)
+ doc = json.loads(tool("sync", src, mic, "--fix-drift", "--json").stdout)
+ self.assertEqual(doc["drift_skipped_reason"], "too_short")
+ self.assertEqual(doc["sources"][0]["drift_skipped_reason"], "too_short")
+ sk = doc["diagnostics"]["skipped"][0]
+ self.assertEqual((sk["code"], sk["reason"], sk["path"]), ("drift_correction_skipped", "too_short", str(mic)))
+ self.assertNotIn("drift_skipped_reason", json.loads(tool("sync", src, mic, "--json").stdout), "only with --fix-drift")
+
+ def test_cropdetect_counts_the_windows_it_could_not_decode(self):
+ """A window ffmpeg could not decode was skipped unseen; the crop then rested on fewer
+ readings than --samples said."""
+ from unittest import mock
+ import cropdetect
+ real = cropdetect.run_analysis
+ calls = []
+
+ def flaky(cmd, check=True, **kw):
+ calls.append(cmd)
+ if len(calls) == 2:
+ return subprocess.CompletedProcess(cmd, 1, "", "Invalid data found when processing input\n")
+ return real(cmd, check=check, **kw)
+ src = self.media()
+ failed = []
+ with mock.patch.object(cropdetect, "run_analysis", flaky):
+ rects = cropdetect.detect(str(src), 5.0, 5, 0.0941176, 16, 6.0, failed)
+ self.assertTrue(rects)
+ self.assertEqual(len(failed), 1)
+ self.assertIn("Invalid data", failed[0]["error"])
+ doc = json.loads(tool("cropdetect", src, "--json").stdout)
+ self.assertEqual((doc["windows"], doc["windows_failed"]), (5, 0))
+
+ def test_caption_counts_ass_lines_it_could_not_shift_or_read(self):
+ """caption.py --ass --offset copied a Dialogue line with unreadable timestamps unshifted,
+ parse_ass_cues() left it out of the visible-cue count, and the shift count went to stderr
+ only. Each count is in `caption`, and each non-zero one is a warning."""
+ src = self.media()
+ ass = self.tmp / "s.ass"
+ ass.write_text("[Script Info]\nScriptType: v4.00+\n\n[Events]\n"
+ "Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text\n"
+ "Dialogue: 0,0:00:00.00,0:00:02.00,Default,,0,0,0,,Hello\n"
+ "Dialogue: 0,bad,0:00:03.00,Default,,0,0,0,,Broken\n"
+ "Dialogue: 0,0:00:02.00,0:00:04.00,Default,,0,0,0,,World\n", encoding="utf-8")
+ doc = json.loads(tool("caption", src, "--ass", ass, "--offset", "1", "-o", self.tmp / "cap.mp4", "--dry-run", "--json").stdout)
+ cap = doc["caption"]
+ self.assertEqual((cap["ass_shifted"], cap["ass_unshifted"], cap["ass_malformed"]), (2, 1, 1))
+ self.assertEqual(sorted(w["code"] for w in doc["diagnostics"]["warnings"]), ["ass_lines_malformed", "ass_lines_unshifted"])
+ self.assertEqual(doc["font_used"]["how"], "ass_styles")
+
+ def test_a_timeline_that_does_not_parse_keeps_the_reason(self):
+ """timeline.parses() returned False and discarded the exception, so render.py's parse
+ verification step said ok: false with no reason anywhere."""
+ from _common import timeline as tlmod
+ bad = self.tmp / "bad.fcpxml"
+ bad.write_text("<fcpxml><unclosed>", encoding="utf-8")
+ self.assertFalse(tlmod.parses(str(bad), "fcpxml"))
+ w = _common.STATE.warnings[-1]
+ self.assertEqual((w["code"], w["format"]), ("timeline_parse_failed", "fcpxml"))
+ self.assertIn("ParseError", w["error"])
+ other = self.tmp / "other.otio"
+ other.write_text("[1, 2]", encoding="utf-8")
+ self.assertFalse(tlmod.parses(str(other), "otio"), "a JSON list is not a timeline (and no longer an AttributeError)")
+ good = self.tmp / "good.otio"
+ good.write_text(json.dumps({"OTIO_SCHEMA": "Timeline.1"}), encoding="utf-8")
+ self.assertTrue(tlmod.parses(str(good), "otio"))
+ self.assertEqual(len(_common.STATE.warnings), 2)
+
+ def test_the_font_drawn_with_is_reported_and_a_substitute_recorded(self):
+ """fonts.default_font_file() returned whatever fontconfig (or the Windows Fonts folder)
+ answered; a family that is not installed was drawn in another without a word."""
+ from unittest import mock
+ fonts = sys.modules["_common.fonts"]
+ face = self.tmp / "Other.ttf"
+ face.write_bytes(b"\0")
+ answer = subprocess.CompletedProcess([], 0, f"{face}\nOther Sans,Other Sans Book\n", "")
+ with mock.patch.object(fonts.platform, "system", return_value="Linux"), \
+ mock.patch.object(fonts.shutil, "which", return_value="/usr/bin/fc-match"), \
+ mock.patch.object(fonts.subprocess, "run", return_value=answer):
+ got = fonts.resolve_font("Wanted Serif")
+ same = fonts.resolve_font("Other Sans")
+ alias = fonts.resolve_font("sans-serif")
+ self.assertEqual((got["file"], got["family"], got["substituted"], got["via"]), (str(face), "Other Sans", True, "fc-match"))
+ self.assertFalse(same["substituted"])
+ self.assertFalse(alias["substituted"], "a generic alias is resolved, not substituted")
+ with mock.patch.object(fonts, "resolve_font", return_value=got):
+ choice = fonts.record_font_choice(fonts.font_choice("Wanted Serif"))
+ self.assertEqual((choice["how"], choice["file"], choice["substituted"]), ("resolved", str(face), True))
+ self.assertEqual(self.codes("fallbacks"), ["font_substituted"])
+ self.assertEqual(fonts.font_choice("X", font_file="/f.ttf")["how"], "font_file")
+ # end to end: overlay.py reports the font it handed drawtext
+ src = self.media()
+ doc = json.loads(tool("overlay", src, "--text", "Hi", "--font", "No Such Family Qx", "-o", self.tmp / "ov.mp4", "--dry-run", "--json").stdout)
+ used = doc["font_used"]
+ self.assertEqual(used["asked"], "No Such Family Qx")
+ if used["how"] == "by_name":
+ self.skipTest("no font resolves to a file on this machine (no fc-match, no Windows Arial)")
+ self.assertTrue(used["substituted"])
+ self.assertEqual([f["code"] for f in doc["diagnostics"]["fallbacks"]], ["font_substituted"])
+
+ def test_stderr_only_warnings_reach_the_document(self):
+ """overlay.py's emoji lines, color.py --strip-dovi with no Dolby Vision, and render.py's
+ frame-from-preset and --normalize defaults were stderr only."""
+ src = self.media()
+ doc = json.loads(tool("overlay", src, "--text", "Hi \U0001F600", "--emoji", "none", "-o", self.tmp / "e1.mp4", "--dry-run", "--json").stdout)
+ self.assertIn("emoji_stripped", [w["code"] for w in doc["diagnostics"]["warnings"]])
+ doc = json.loads(tool("overlay", src, "--text", "Hi \U0001F600", "-o", self.tmp / "e2.mp4", "--dry-run", "--json").stdout)
+ self.assertIn("emoji_monochrome", [w["code"] for w in doc["diagnostics"]["warnings"]])
+ hevc = self.tmp / "plain.mp4"
+ ffmpeg("-f", "lavfi", "-i", "testsrc2=size=160x90:rate=30", "-t", "1", "-c:v", "libx265", "-preset", "ultrafast",
+ "-x265-params", "log-level=error", "-tag:v", "hvc1", hevc)
+ doc = json.loads(tool("color", hevc, "--strip-dovi", "-o", self.tmp / "nodv.mp4", "--dry-run", "--json").stdout)
+ self.assertEqual([w["code"] for w in doc["diagnostics"]["warnings"]], ["dovi_not_detected"])
+ proj = self.tmp / "proj.json"
+ proj.write_text(json.dumps({"output": str(self.tmp / "r.mp4"), "clips": [{"src": str(src), "in": "0:01", "out": "0:03"}],
+ "frame": {"aspect": "9:16"}, "export": {"preset": "reels"}}), encoding="utf-8")
+ proc = tool("render", proj, "--dry-run", "--json")
+ warned = {w["code"]: w for w in json.loads(proc.stdout)["diagnostics"]["warnings"]}
+ self.assertEqual((warned["frame_from_export_preset"]["width"], warned["frame_from_export_preset"]["height"]), (1080, 1920))
+ self.assertEqual(warned["normalize_on_by_default"]["preset"], "reels")
+ self.assertIn("frame: 1080x1920 from the reels export preset", proc.stderr, "the stderr line stays")
+
+ def test_graphics_pinned_to_drawtext_on_a_newline_glyph_build_is_a_warning(self):
+ """graphics.py --text-render drawtext with a multi-line label on FFmpeg 8.0-8.0.1/8.1.0
+ draws a box at each line break; the warning was stderr only. Runs where the build has
+ the bug (the 8.0.1 CI job, or a local 8.0.1 build first on PATH)."""
+ if not _common.drawtext_draws_newline_glyph():
+ self.skipTest("this ffmpeg's drawtext does not draw a box at a line break")
+ src = self.media()
+ doc = json.loads(tool("graphics", src, "--template", "title", "--title", "one\ntwo", "--text-render", "drawtext",
+ "-o", self.tmp / "g.mp4", "--dry-run", "--json").stdout)
+ self.assertIn("drawtext_newline_glyph", [w["code"] for w in doc["diagnostics"]["warnings"]])
+
+ # ------------------------------------------------------------------ source checks
+ def test_no_stderr_only_warning_is_left_in_the_sources(self):
+ """A `warning:` line printed with info() reaches stderr and no document. Every one goes
+ through warn() (stderr AND diagnostics.warnings), with no exception left."""
+ pattern = re.compile(r"""\b(?:info|print)\(\s*f?["']warning\b|sys\.stderr\.write\(\s*f?["']warning\b""", re.I)
+ left = {}
+ for path in script_sources() + [ROOT / "mcp" / "server.py"]:
+ for line in path.read_text(encoding="utf-8").splitlines():
+ code = line.split("#", 1)[0]
+ if pattern.search(code) and not code.strip().startswith(("(", '"', "'")):
+ left[path.name] = left.get(path.name, 0) + 1
+ self.assertEqual(left, {}, "stderr-only warnings: use warn(msg, code, ...) so the line reaches --json too")
+
+ def test_every_diagnostic_code_is_in_the_contract_under_its_bucket(self):
+ """json_output.diagnostic_codes is the list a caller switches on: every warn() /
+ record_fallback() / record_skip() call under scripts/ names a code listed there under the
+ bucket the call writes to, and the contract lists no code nothing records."""
+ bucket_of = {"warn": "warnings", "record_fallback": "fallbacks", "record_skip": "skipped"}
+ used = {}
+ for path in script_sources():
+ text = path.read_text(encoding="utf-8")
+ for m in re.finditer(r"(?<![\w.])(warn|record_fallback|record_skip)\(", text):
+ if text[max(0, m.start() - 4):m.start()] == "def ":
+ continue
+ depth, i = 1, m.end()
+ while depth and i < len(text):
+ depth += {"(": 1, ")": -1}.get(text[i], 0)
+ i += 1
+ body = text[m.end():i - 1]
+ literals = re.findall(r"""["']([a-z][a-z0-9_]+)["']""", body)
+ if not body.strip() or not literals:
+ continue # a mention in prose ("warn()"), or a call whose code is a variable
+ listed = [c for c in literals if c in _contract.DIAGNOSTIC_CODES]
+ self.assertTrue(listed, f"{path.name}: {m.group(1)}({body[:70]}...) names no code in _contract.DIAGNOSTIC_CODES")
+ code = listed[0]
+ self.assertEqual(_contract.DIAGNOSTIC_CODES[code][0], bucket_of[m.group(1)], f"{code}: recorded as {m.group(1)}")
+ used[code] = path.name
+ self.assertEqual(sorted(set(_contract.DIAGNOSTIC_CODES) - set(used)), [], "the contract lists codes nothing records")
+ published = _contract.build(detect=False)["json_output"]["diagnostic_codes"]
+ self.assertEqual(set(published), set(_contract.DIAGNOSTIC_CODES))
+
+
class FfmpegSecurityTableTests(unittest.TestCase):
"""doctor's `ffmpeg_security` table and comparison, without running ffmpeg (every OS)."""
diff --git a/tests/test_editing.py b/tests/test_editing.py
index 9dad8f4..c18aeb6 100644
--- a/tests/test_editing.py
+++ b/tests/test_editing.py
@@ -909,6 +909,11 @@ class EditingTests(MediaFixtures):
proc = script("sequence.py", "--dir", frames_dir, "--pattern", "frame_%04d.png", "--fps", "5",
"-o", OUT / "seq_gap.mp4")
self.assertIn("frame_0002.png is missing; 2 later frame(s)", proc.stderr)
+ # the gap reaches the document too, not only stderr
+ proc = script("sequence.py", "--dir", frames_dir, "--pattern", "frame_%04d.png", "--fps", "5",
+ "-o", OUT / "seq_gap_plan.mp4", "--dry-run", "--json")
+ gap = json.loads(proc.stdout)["diagnostics"]["warnings"][0]
+ self.assertEqual((gap["code"], gap["missing"], gap["frames_left_out"]), ("sequence_gap", "frame_0002.png", 2))
def test_sequence_glob_pattern(self):
frames_dir = OUT / "seqframes_glob"