Skip to content

Local video editor for coding agents: 42 FFmpeg tools with a machine-readable contract and an MCP server, for Claude Code, Cursor and Codex. Cut, captions, reframe, loudness, platform exports. No cloud, no API keys.

fix(asr): decode whisper.cpp transcripts with -mc 0 (#323)

Shogo Kajiyama9687a0a433da
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 959ba46..962f891 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,10 @@ _Automated release: version and notes generated from pull requests merged since
- **`cut.py --segments --accurate` no longer hangs when a later segment crosses a mid-stream colour-tag change.** The concat re-cut opens the source once per segment; on FFmpeg 8.1.2 a source whose tags change partway (untagged to bt470bg/tv on a new SPS) deadlocked at 0% CPU in the second input's graph rebuild and ended at `--timeout` with nothing written. Every segment input now carries `-reinit_filter 0`, so the graph keeps its first format; the output's colour tags were already written explicitly, so no pixels change. A source whose frame size changes mid-stream is no longer refused cleanly by the concat filter (see `docs/design-decisions.md`) (#316).
- fix(cut): -reinit_filter 0 on every --segments --accurate input (#322)
+### Changed
+
+- **whisper.cpp transcripts are decoded with `-mc 0` (no previous-text context).** This is a decoding change for whisper.cpp users: `caption.py --transcribe` and `silence.py --filler --transcribe` pass `-mc 0` to the `whisper-cli` calls that write the SRT and the word timings (not to language detection), so a transcript can differ from 2.8.3's. Whisper conditions on its own previous text, and a hallucinated line can then repeat to the end of the file; in the measurement large-v3-turbo repeated one sentence ~300 times on two of six sets and lost about 7 minutes of speech on one. With `-mc 0` both loops are gone and every model's word-weighted WER over the six sets falls: base 19.87 to 18.33, small 17.07 to 15.42, large-v3-turbo 69.12 to 12.52. It is not better everywhere: gigaspeech is slightly worse for all three models, and some single sets are for some models. faster-whisper, openai-whisper and Parakeet are unchanged. Measured and proposed by [@rymalia](https://github.com/rymalia) in #318 (whisper.cpp 1.9.5, 58 min of English from the ESB test sets).
+
## 2.8.3
_Automated release: version and notes generated from pull requests merged since 2.8.2._
diff --git a/docs/contract.md b/docs/contract.md
index c1c938d..1285f19 100644
--- a/docs/contract.md
+++ b/docs/contract.md
@@ -379,7 +379,9 @@ and no detector can name it, a Whisper engine that can run takes it (whisper.cpp
`-l auto`: its own default is English). Parakeet then runs only on English *assumed* -- when no
Whisper engine is installed, or every one failed, or `--model` names an English-only Parakeet
model -- and the result's top-level `notes` says the transcript is wrong if the speech is not
-English; a multilingual (v3) Parakeet model assumes nothing and gets no note. The result's
+English; a multilingual (v3) Parakeet model assumes nothing and gets no note. Every whisper.cpp
+command that writes a transcript is also passed `-mc 0` (no previous-text context, so one
+hallucinated line is not repeated to the end of the file); its language detection is not. The result's
`transcription` says which engine, model and routing decision produced the cues (in every
`--mode`, `mux` included).
Every engine transcribes on the machine running the skill, and the audio is never uploaded. The
diff --git a/docs/design-decisions.md b/docs/design-decisions.md
index ede8fd6..1a8038c 100644
--- a/docs/design-decisions.md
+++ b/docs/design-decisions.md
@@ -1036,6 +1036,28 @@ not a new file format this tool would have to maintain.
each engine's source and process, so `network: false` cannot be read as "a first
`--transcribe` is network-free". Tests: `InstallHintTests`,
`test_the_network_claim_matches_where_each_engine_runs`.
+- **whisper.cpp transcribes with `-mc 0`, by default.** Whisper conditions each window on the text
+ before it, and once a hallucinated line is in that context it can repeat it for the rest of the
+ file: in the contributor's benchmark (#318, six English sets of ~10 min, whisper.cpp 1.9.5)
+ large-v3-turbo repeated one sentence ~300 times on two sets, and on common_voice lost about the
+ last 7 minutes of speech. `-mc 0` (`--max-context 0`, previous-text tokens kept: none) removed both
+ loops and lowered every model's word-weighted WER: base 19.87 -> 18.33, small 17.07 -> 15.42,
+ large-v3-turbo 69.12 -> 12.52. It is a decoding change, not a free one: gigaspeech is worse for all
+ three models (13.16 -> 14.11 for large-v3-turbo) and some single cells are (small on
+ common_voice 16.44 -> 19.28, large-v3-turbo on ami 25.72 -> 27.55). It is the default rather than
+ an opt-in because a loop that silently loses minutes of speech is a defect, which the 2.x promise
+ ("behaviour may change to fix a defect, with a CHANGELOG line") allows, and because the net is
+ better for every model measured; an opt-in would leave the loss with everyone who does not know to ask.
+ It is given to the two whisper.cpp commands that write a transcript (the SRT of `caption.py
+ --transcribe` and the `--output-json-full` word timings of `silence.py --filler --transcribe`) and
+ not to language detection (`-dl`: it reads the first 30 s and writes no transcript), nor to any other
+ engine: faster-whisper, openai-whisper and Parakeet are not whisper.cpp and were not measured. No
+ repetition detector was added: the flag removes the cause, a detector would be a second decoder to
+ maintain. Every whisper.cpp binary name the skill runs (`whisper-cli`, `whisper-cpp`, `main`) is
+ documented to take `-mc` (the flag is years old); a build that rejected it would write no file, so
+ the run reads as "whisper.cpp found but failed" with the build's last output line and the next
+ engine is tried (`engine_failed` when whisper.cpp was named). Reverse this if a measurement on other material shows `-mc 0` costing more than the
+ loops it prevents. Test: `WhisperContextTests`.
## Unreleased — cuts that say what they did
diff --git a/references/scripts.md b/references/scripts.md
index e29ebde..b020664 100644
--- a/references/scripts.md
+++ b/references/scripts.md
@@ -1023,7 +1023,8 @@ top-level `notes` then says so: that transcript is wrong for non-English speech,
`--engine parakeet-mlx|parakeet.cpp|whisper.cpp|faster-whisper|openai-whisper` forces one (a
missing one is `kind: missing_tool`; an installed one that fails is `kind: input`, `reason:
engine_failed`, with its own error line in `detail`). Without a language whisper.cpp is passed
-`-l auto` (its own default is English), else the named or detected language. `--model` is the
+`-l auto` (its own default is English), else the named or detected language, and `-mc 0` (no
+previous-text context, so one hallucinated line is not repeated to the end of the file; #318). `--model` is the
whisper model; a Parakeet engine takes it only when it names a Parakeet model
(`mlx-community/parakeet-*` or a `.gguf`), else `PARAKEET_MODEL` / `PARAKEET_CPP_MODEL`, else the
English `tdt-0.6b-v2`. A non-English `--language` on an English-only Parakeet model is refused; a
diff --git a/scripts/_common/asr.py b/scripts/_common/asr.py
index 9812ef0..df4e148 100644
--- a/scripts/_common/asr.py
+++ b/scripts/_common/asr.py
@@ -28,6 +28,11 @@ WHISPER_ENGINES = ("whisper.cpp", "faster-whisper", "openai-whisper")
ASR_ENGINES = PARAKEET_ENGINES + WHISPER_ENGINES
ENGINE_CHOICES = ("auto",) + ASR_ENGINES
ASR_ENGINE_ENV = "FFMPEG_SKILL_ASR_ENGINE"
+# whisper.cpp conditions every window on the text before it, and once a hallucinated line is in that
+# context it can repeat it to the end of the file (#318: one sentence ~300 times, ~7 minutes of speech
+# lost). -mc / --max-context 0 keeps no previous text. Given to the commands that write a transcript,
+# not to language detection (-dl), which writes none.
+WHISPER_CPP_NO_CONTEXT = ["-mc", "0"]
PARAKEET_MLX_DEFAULT_MODEL = "mlx-community/parakeet-tdt-0.6b-v2"
# Every engine transcribes on this machine: the audio is never uploaded. Three of them fetch
# their model weights the first time they run, which the hint says, so "offline" is never
@@ -342,7 +347,7 @@ def _transcribe_in(tmpdir: str, video: str, out_srt: str, language: Optional[str
model_path = _whisper_cpp_model(model)
base = os.path.join(tmpdir, "out")
# whisper.cpp's own default is -l en: without "auto" it decodes any speech as English
- cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base, "-l", lang or "auto"]
+ cmd = [cli, "-m", model_path, "-f", wav, "-osrt", "-of", base, "-l", lang or "auto", *WHISPER_CPP_NO_CONTEXT]
proc = _asr_run(cmd, subprocess, "whisper.cpp")
if proc.returncode == 0 and os.path.exists(base + ".srt"):
info(f"transcribed with whisper.cpp ({os.path.basename(cli)}, model {os.path.basename(model_path)})")
@@ -995,7 +1000,8 @@ def transcribe_words_result(video: str, language: "Optional[str]" = None, model:
model_path = _whisper_cpp_model(model)
base = os.path.join(tmpdir, "out")
# whisper.cpp's own default is -l en: without "auto" it decodes any speech as English
- cmd = [cli, "-m", model_path, "-f", wav, "--output-json-full", "-of", base, "-l", lang or "auto"]
+ cmd = [cli, "-m", model_path, "-f", wav, "--output-json-full", "-of", base, "-l", lang or "auto",
+ *WHISPER_CPP_NO_CONTEXT]
proc = _asr_run(cmd, _subprocess, "whisper.cpp")
if proc.returncode == 0 and os.path.exists(base + ".json"):
words = _words_from_whisper_cpp_json(base + ".json")
diff --git a/tests/test_asr.py b/tests/test_asr.py
index aeb5ab3..87843b3 100644
--- a/tests/test_asr.py
+++ b/tests/test_asr.py
@@ -237,10 +237,10 @@ class _FakeEngines:
template = (kw["env"] if kw.get("env") is not None else os.environ).get("PARAKEET_OUTPUT_TEMPLATE") or "{filename}"
name = template.format(filename="audio", parent=out, date="20261009", index="1")
Path(out, name + ".json").write_text(json.dumps(MLX_DOC), encoding="utf-8")
- elif exe == "whisper-cli" and "-osrt" in cmd:
+ elif exe in ("whisper-cli", "whisper-cpp") and "-osrt" in cmd:
Path(cmd[cmd.index("-of") + 1] + ".srt").write_text(
"1\n00:00:00,000 --> 00:00:01,500\nBonjour tout le monde\n", encoding="utf-8")
- elif exe == "whisper-cli" and "--output-json-full" in cmd:
+ elif exe in ("whisper-cli", "whisper-cpp") and "--output-json-full" in cmd:
Path(cmd[cmd.index("-of") + 1] + ".json").write_text(json.dumps({"transcription": [{"tokens": [
{"text": " Bonjour", "offsets": {"from": 0, "to": 600}},
{"text": " euh", "offsets": {"from": 700, "to": 900}}]}]}), encoding="utf-8")
@@ -599,6 +599,65 @@ class WhisperLanguageTests(_SrtDir, _BridgeHarness, unittest.TestCase):
self.assertTrue(asr._whisper_ready(openai, "base"))
+class WhisperContextTests(_SrtDir, _BridgeHarness, unittest.TestCase):
+ """#318: whisper.cpp conditions each window on the text before it, and once a hallucinated line is in
+ that context it can repeat it for the rest of the file (large-v3-turbo: one sentence 293 and 303
+ times, about 7 minutes of speech lost on one set). `-mc 0` (--max-context) keeps no previous text.
+ Both commands that produce a transcript carry it; the language detector writes none and does not."""
+
+ @staticmethod
+ def _flags(argv):
+ return [argv[i + 1] for i, a in enumerate(argv) if a == "-mc"]
+
+ def test_the_srt_command_keeps_no_previous_text(self):
+ for cli in ("whisper-cli", "whisper-cpp"):
+ fake = _FakeEngines([cli])
+ self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "en", "base"), fake=fake)
+ argv = fake.ran(cli)[-1]
+ self.assertIn("-osrt", argv)
+ self.assertEqual(self._flags(argv), ["0"], argv)
+
+ def test_the_word_timing_command_keeps_no_previous_text(self):
+ for cli in ("whisper-cli", "whisper-cpp"):
+ fake = _FakeEngines([cli])
+ self._with([], lambda: asr.transcribe_words_result("talk.mp4", "en"), fake=fake)
+ argv = fake.ran(cli)[-1]
+ self.assertIn("--output-json-full", argv)
+ self.assertEqual(self._flags(argv), ["0"], argv)
+
+ def test_the_flag_does_not_replace_the_language(self):
+ fake = _FakeEngines(["whisper-cli"])
+ self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "fr", "base"), fake=fake)
+ argv = fake.ran("whisper-cli")[-1]
+ self.assertEqual((argv[argv.index("-l") + 1], self._flags(argv)), ("fr", ["0"]))
+
+ def test_language_detection_is_left_as_it_was(self):
+ """-dl reads the first 30 s and writes no transcript: there is no context to drop."""
+ fake = _FakeEngines(["whisper-cli"])
+ with tempfile.TemporaryDirectory() as d:
+ tiny = Path(d) / "ggml-tiny.bin"
+ tiny.write_bytes(b"")
+ with mock.patch.object(shutil, "which", fake.which), mock.patch.object(subprocess, "run", fake.run), \
+ mock.patch.object(asr, "_whisper_cpp_model", lambda m: str(tiny) if m == "tiny" else m):
+ asr.detect_language("a.wav", shutil, subprocess)
+ argv = fake.ran("whisper-cli")[-1]
+ self.assertIn("-dl", argv)
+ self.assertNotIn("-mc", argv)
+
+ def test_other_engines_are_not_given_the_flag(self):
+ """The change is whisper.cpp's: faster-whisper and openai-whisper take no -mc, Parakeet no such option."""
+ fake = _FakeEngines(["whisper", "parakeet-mlx"])
+ # the fake writes nothing for openai-whisper, so the run is refused; what it was handed is recorded
+ with mock.patch.object(asr, "detect_language", return_value="fr"), self.assertRaises(SystemExit):
+ self._with([], lambda: asr.transcribe_result("talk.mp4", self.srt, "fr", "base", engine="openai-whisper"),
+ fake=fake)
+ self.assertTrue(fake.ran("whisper"))
+ for exe in ("whisper", "parakeet-mlx"):
+ for argv in fake.ran(exe):
+ self.assertNotIn("-mc", argv)
+ self.assertNotIn("--max-context", argv)
+
+
class AutoRoutingFallbackTests(_SrtDir, _BridgeHarness, unittest.TestCase):
"""Review of #305: what --engine auto does when its first choice fails, and what it then says."""
@@ -961,6 +1020,7 @@ class ParakeetEngineTests(MediaFixtures):
# whisper.cpp's own default is -l en: "handles any language" only holds when it is told to detect
for argv in (caption_args, filler_args):
self.assertEqual(argv[argv.index("-l") + 1], "auto", argv)
+ self.assertEqual(argv[argv.index("-mc") + 1], "0", argv) # #318: no previous-text conditioning
self.assertIn("Bonjour tout le monde", (OUT / "pk_whisperfirst.srt").read_text())
self.assertFalse(_assumed(doc))
self.assertEqual(fil["filler"]["source"], "whisper:whisper.cpp")