Skip to content

The truly free, open-source, local-first, cross-platform CapCut replacement. Scriptable through a JSON-RPC API & MCP.

fix(captions): backport upstream whisper timestamp gap correction

MDf1a5f4d334a0
diff --git a/src/patches/README.md b/src/patches/README.md
index 0d019d4c..871a3ac2 100644
--- a/src/patches/README.md
+++ b/src/patches/README.md
@@ -1,6 +1,6 @@
# Patched dependencies
-Crates copied from crates.io with one change each, applied through the
+Crates copied from crates.io with the changes documented below, applied through the
workspace's `[patch.crates-io]`. Each is the published version the
`Cargo.lock` names, so an upgrade is: copy the new version in, re-apply
the change, and read it again.
@@ -62,5 +62,27 @@ SYCL, Vulkan, WebGPU, zDNN, ZenDNN) are left out of
`whisper.cpp/ggml/src`, so the crate's `cuda`, `hipblas`, `vulkan` and
`intel-sycl` features no longer build. CMake only enters a backend's folder
when that backend is turned on. `Cargo.lock` and the crate's registry
-metadata files are left out too. Nothing else differs from the published
-crate.
+metadata files are left out too.
+
+The bundled whisper.cpp 1.8.3 also carries the upstream timestamp fix
+[c33c5618b72bb345df029b730b36bc0e369845a3](https://github.com/ggml-org/whisper.cpp/commit/c33c5618b72bb345df029b730b36bc0e369845a3)
+(whisper.cpp #2279, MIT licence), backported without changes to
+`whisper.cpp/src/whisper.cpp`. The upstream commit linked above contains
+the complete original diff.
+
+The old segment parser skipped consecutive timestamp tokens after a
+segment and reused its end as the next segment's start. Captions after a
+pause could therefore begin at the end of the preceding caption, covering
+the silence. The corrected parser uses the last consecutive timestamp as
+the next start. The upstream patch also preserves these tokens when
+printing special tokens and flushes realtime output consistently.
+
+This changes the common segment parser for every Whisper model and
+language; no model download, decoding option or application setting changes.
+It does not guarantee exact alignment, remove leading silence from the
+first caption, or add word timestamps or word-by-word animation. The
+existing shared-library build and CPU backend selection are unchanged.
+
+Apart from the changes documented here, the copy matches the published
+crate. When refreshing it, check whether the new whisper.cpp already
+includes this fix before reapplying it.
diff --git a/src/patches/whisper-rs-sys/whisper.cpp/src/whisper.cpp b/src/patches/whisper-rs-sys/whisper.cpp/src/whisper.cpp
index 5b6e4b4b..5ec82fe4 100644
--- a/src/patches/whisper-rs-sys/whisper.cpp/src/whisper.cpp
+++ b/src/patches/whisper-rs-sys/whisper.cpp/src/whisper.cpp
@@ -7659,11 +7659,14 @@ int whisper_full_with_state(
}
}
text = "";
- while (i < (int) tokens_cur.size() && tokens_cur[i].id > whisper_token_beg(ctx)) {
+ t0 = t1;
+ while (i + 1 < (int) tokens_cur.size() && tokens_cur[i + 1].id > whisper_token_beg(ctx)) {
i++;
+ if (params.print_special) {
+ text += whisper_token_to_str(ctx, tokens_cur[i].id);
+ }
+ t0 = seek + 2 * (tokens_cur[i].tid - whisper_token_beg(ctx));
}
- i--;
- t0 = t1;
i0 = i + 1;
speaker_turn_next = false;
}
@@ -7680,8 +7683,8 @@ int whisper_full_with_state(
printf("[%s --> %s] %s\n", to_timestamp(tt0).c_str(), to_timestamp(tt1).c_str(), text.c_str());
} else {
printf("%s", text.c_str());
- fflush(stdout);
}
+ fflush(stdout);
}
result_all.push_back({ tt0, tt1, text, state->no_speech_prob, {}, speaker_turn_next });