Skip to content

🎥 Make videos programmatically with React

Document Whisper WebGPU options

JonnyBurgerf0946fb04e05
diff --git a/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx b/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx
index 97816d81dc..ce9103a7fc 100644
--- a/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx
+++ b/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx
@@ -10,6 +10,8 @@ Removes cached model files that are no longer returned by [`getAvailableModels()
Call this function when the page loads to remove models discontinued by a newer version of `@remotion/whisper-webgpu`.
+Model files downloaded from Hugging Face by versions before 4.0.522 are not removed by this function. They remain until the browser evicts them or its cache is cleared.
+
```ts twoslash title="clear-stale-models.ts"
import {clearStaleModels} from '@remotion/whisper-webgpu';
diff --git a/packages/docs/docs/whisper-webgpu/get-available-models.mdx b/packages/docs/docs/whisper-webgpu/get-available-models.mdx
index cd5123de6a..5e1748350c 100644
--- a/packages/docs/docs/whisper-webgpu/get-available-models.mdx
+++ b/packages/docs/docs/whisper-webgpu/get-available-models.mdx
@@ -15,7 +15,13 @@ const models = getAvailableModels();
console.log(models);
```
-Each item contains the short `name`, Hugging Face `modelId`, parameter count, whether the model is multilingual, and the estimated WebGPU download size.
+Each item contains the short `name`, upstream Hugging Face `modelId`, parameter count, whether the model is multilingual, whether it supports translation, and the estimated WebGPU download size.
+
+The `modelId` identifies the upstream model repository for provenance. Model files are served from `remotion.media`; `modelId` is not the download location.
+
+### supportsTranslation<AvailableFrom v="4.0.522" />
+
+Whether the model supports `task: 'translate'` in [`transcribe()`](/docs/whisper-webgpu/transcribe). Translation converts speech into English. It is supported by multilingual non-turbo models only.
Available sizes are `tiny`, `base`, `small`, `medium`, and `large-v3-turbo`. English-only `.en` variants are available through `medium`.
diff --git a/packages/docs/docs/whisper-webgpu/index.mdx b/packages/docs/docs/whisper-webgpu/index.mdx
index ba72fd62f7..0c55add22f 100644
--- a/packages/docs/docs/whisper-webgpu/index.mdx
+++ b/packages/docs/docs/whisper-webgpu/index.mdx
@@ -12,7 +12,9 @@ import {TableOfContents} from './table-of-contents';
## Installation
-<Installation pkg="@remotion/whisper-webgpu" />
+<Installation pkg="@remotion/whisper-webgpu @huggingface/[email protected]" />
+
+`@huggingface/transformers` is a peer dependency. Remotion recommends and tests version `4.2.0` so that multiple browser-based AI workflows can share one Transformers.js runtime.
## Example
@@ -36,6 +38,14 @@ console.log(captions);
The model is downloaded on the first call and cached by the browser. Call [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model) first to display model download progress.
+## Model downloads<AvailableFrom v="4.0.522" />
+
+Model files are downloaded from `remotion.media`.
+
+`@remotion/whisper-webgpu` temporarily sets `env.remoteHost` while loading a model and restores it afterwards. This keeps the shared Transformers.js runtime available to other AI workflows. If your site uses a Content Security Policy, allow `https://remotion.media` in `connect-src`.
+
+Versions before 4.0.522 downloaded models from Hugging Face. After upgrading, the browser downloads a previously cached model once more because the download URL, and therefore its cache key, has changed. With the default Transformers.js cache settings, subsequent loads use the new cached copy.
+
## APIs
<TableOfContents />
diff --git a/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx b/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx
index 0a9ffe50b2..e46ca71f77 100644
--- a/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx
+++ b/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx
@@ -6,7 +6,7 @@ crumb: '@remotion/whisper-webgpu'
# isWhisperModelCached()<AvailableFrom v="4.0.518" />
-Checks whether all files needed for a timestamped Whisper model are present in the browser cache. The check does not download or initialize the model.
+Checks whether all files needed for a timestamped Whisper model from `remotion.media` are present in the browser cache. The check does not download or initialize the model.
```ts twoslash title="model-cache.ts"
import {isWhisperModelCached} from '@remotion/whisper-webgpu';
@@ -26,6 +26,8 @@ One of the names returned by [`getAvailableModels()`](/docs/whisper-webgpu/get-a
A `Promise<boolean>` resolving to `true` if every required model file is cached.
+Models cached from Hugging Face by versions before 4.0.522 do not satisfy this check. Loading the model downloads and caches a new copy from `remotion.media`.
+
## Compatibility
<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />
diff --git a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
index 21644e91a5..30908f3c34 100644
--- a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
+++ b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
@@ -6,7 +6,7 @@ crumb: '@remotion/whisper-webgpu'
# loadWhisperModel()<AvailableFrom v="4.0.518" />
-Downloads and initializes a timestamped Whisper model. Calling it before [`transcribe()`](/docs/whisper-webgpu/transcribe) lets you show model-loading progress.
+Downloads a timestamped Whisper model from `remotion.media` and initializes it. Calling it before [`transcribe()`](/docs/whisper-webgpu/transcribe) lets you show model-loading progress.
```ts twoslash title="load-model.ts"
import {loadWhisperModel} from '@remotion/whisper-webgpu';
@@ -37,6 +37,8 @@ Returns whether the same model pipeline was already loaded in memory.
The browser manages its persistent [Transformers.js](https://huggingface.co/docs/transformers.js) cache. This function also retains the initialized model in memory until [`disposeWhisperModel()`](/docs/whisper-webgpu/dispose-whisper-model) is called or the page closes.
+Versions before 4.0.522 downloaded models from Hugging Face. A model cached by an earlier version is downloaded once more from `remotion.media`, then reused from the new cache entry when using the default Transformers.js cache settings.
+
## Compatibility
<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />
diff --git a/packages/docs/docs/whisper-webgpu/transcribe.mdx b/packages/docs/docs/whisper-webgpu/transcribe.mdx
index e0f5d43b17..7b903644b3 100644
--- a/packages/docs/docs/whisper-webgpu/transcribe.mdx
+++ b/packages/docs/docs/whisper-webgpu/transcribe.mdx
@@ -36,6 +36,12 @@ One of the timestamped models returned by [`getAvailableModels()`](/docs/whisper
The spoken language name or language code. This option is required for multilingual models because automatic language detection is not supported. Omit it for English-only models such as `small.en`.
+### task?<AvailableFrom v="4.0.522" />
+
+Either `transcribe` or `translate`. Default: `transcribe`.
+
+`translate` translates speech into English and is supported by multilingual non-turbo models only. Check the model's `supportsTranslation` property before using it. Word timestamps may be less reliable when translating.
+
### chunkLengthInSeconds?
Length of long-audio chunks. Default: `30`. Must be a finite number greater than `0`.
@@ -44,6 +50,30 @@ Length of long-audio chunks. Default: `30`. Must be a finite number greater than
Overlap on both sides of a chunk. Default: `5`. Must be finite, non-negative, and less than half of `chunkLengthInSeconds`.
+### forceFullSequences?<AvailableFrom v="4.0.522" />
+
+Whether every sequence must end with a timestamp. Default: `false`. Must be a boolean. When set to `true`, transcription throws if the model returns a trailing sequence without an ending timestamp.
+
+### doSample?<AvailableFrom v="4.0.522" />
+
+Whether to sample tokens probabilistically instead of always choosing the most likely token. Default: `false`. Must be a boolean. Sampling can produce different results between runs.
+
+### temperature?<AvailableFrom v="4.0.522" />
+
+Controls randomness when `doSample` is `true`. Default: `1`. Must be a finite number greater than `0`; lower values favor more likely tokens.
+
+### topK?<AvailableFrom v="4.0.522" />
+
+Limits sampling to the tokens with the highest probabilities when `doSample` is `true`. Default: `50`. Must be a non-negative integer; `0` disables the limit.
+
+### repetitionPenalty?<AvailableFrom v="4.0.522" />
+
+Adjusts the probability of tokens that have already been generated. Default: `1`, which applies no penalty. Must be a finite number greater than `0`; values above `1` discourage repetition and values below `1` encourage it.
+
+### noRepeatNgramSize?<AvailableFrom v="4.0.522" />
+
+Prevents a generated token sequence of this length from occurring more than once. Default: `0`, which disables the constraint. Must be a non-negative integer.
+
### onModelLoadProgress?
Called while the model is loaded. To load independently, use [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model).