🎥 Make videos programmatically with React
Document Whisper WebGPU options
JonnyBurgerf0946fb04e05
diff --git a/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx b/packages/docs/docs/whisper-webgpu/clear-stale-models.mdxindex 97816d81dc..ce9103a7fc 100644--- a/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx+++ b/packages/docs/docs/whisper-webgpu/clear-stale-models.mdx@@ -10,6 +10,8 @@ Removes cached model files that are no longer returned by [`getAvailableModels()Call this function when the page loads to remove models discontinued by a newer version of `@remotion/whisper-webgpu`.+Model files downloaded from Hugging Face by versions before 4.0.522 are not removed by this function. They remain until the browser evicts them or its cache is cleared.+```ts twoslash title="clear-stale-models.ts"import {clearStaleModels} from '@remotion/whisper-webgpu';diff --git a/packages/docs/docs/whisper-webgpu/get-available-models.mdx b/packages/docs/docs/whisper-webgpu/get-available-models.mdxindex cd5123de6a..5e1748350c 100644--- a/packages/docs/docs/whisper-webgpu/get-available-models.mdx+++ b/packages/docs/docs/whisper-webgpu/get-available-models.mdx@@ -15,7 +15,13 @@ const models = getAvailableModels();console.log(models);```-Each item contains the short `name`, Hugging Face `modelId`, parameter count, whether the model is multilingual, and the estimated WebGPU download size.+Each item contains the short `name`, upstream Hugging Face `modelId`, parameter count, whether the model is multilingual, whether it supports translation, and the estimated WebGPU download size.++The `modelId` identifies the upstream model repository for provenance. Model files are served from `remotion.media`; `modelId` is not the download location.++### supportsTranslation<AvailableFrom v="4.0.522" />++Whether the model supports `task: 'translate'` in [`transcribe()`](/docs/whisper-webgpu/transcribe). Translation converts speech into English. It is supported by multilingual non-turbo models only.Available sizes are `tiny`, `base`, `small`, `medium`, and `large-v3-turbo`. English-only `.en` variants are available through `medium`.diff --git a/packages/docs/docs/whisper-webgpu/index.mdx b/packages/docs/docs/whisper-webgpu/index.mdxindex ba72fd62f7..0c55add22f 100644--- a/packages/docs/docs/whisper-webgpu/index.mdx+++ b/packages/docs/docs/whisper-webgpu/index.mdx@@ -12,7 +12,9 @@ import {TableOfContents} from './table-of-contents';## Installation-<Installation pkg="@remotion/whisper-webgpu" />+<Installation pkg="@remotion/whisper-webgpu @huggingface/[email protected]" />++`@huggingface/transformers` is a peer dependency. Remotion recommends and tests version `4.2.0` so that multiple browser-based AI workflows can share one Transformers.js runtime.## Example@@ -36,6 +38,14 @@ console.log(captions);The model is downloaded on the first call and cached by the browser. Call [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model) first to display model download progress.+## Model downloads<AvailableFrom v="4.0.522" />++Model files are downloaded from `remotion.media`.++`@remotion/whisper-webgpu` temporarily sets `env.remoteHost` while loading a model and restores it afterwards. This keeps the shared Transformers.js runtime available to other AI workflows. If your site uses a Content Security Policy, allow `https://remotion.media` in `connect-src`.++Versions before 4.0.522 downloaded models from Hugging Face. After upgrading, the browser downloads a previously cached model once more because the download URL, and therefore its cache key, has changed. With the default Transformers.js cache settings, subsequent loads use the new cached copy.+## APIs<TableOfContents />diff --git a/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx b/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdxindex 0a9ffe50b2..e46ca71f77 100644--- a/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx+++ b/packages/docs/docs/whisper-webgpu/is-whisper-model-cached.mdx@@ -6,7 +6,7 @@ crumb: '@remotion/whisper-webgpu'# isWhisperModelCached()<AvailableFrom v="4.0.518" />-Checks whether all files needed for a timestamped Whisper model are present in the browser cache. The check does not download or initialize the model.+Checks whether all files needed for a timestamped Whisper model from `remotion.media` are present in the browser cache. The check does not download or initialize the model.```ts twoslash title="model-cache.ts"import {isWhisperModelCached} from '@remotion/whisper-webgpu';@@ -26,6 +26,8 @@ One of the names returned by [`getAvailableModels()`](/docs/whisper-webgpu/get-aA `Promise<boolean>` resolving to `true` if every required model file is cached.+Models cached from Hugging Face by versions before 4.0.522 do not satisfy this check. Loading the model downloads and caches a new copy from `remotion.media`.+## Compatibility<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />diff --git a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdxindex 21644e91a5..30908f3c34 100644--- a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx+++ b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx@@ -6,7 +6,7 @@ crumb: '@remotion/whisper-webgpu'# loadWhisperModel()<AvailableFrom v="4.0.518" />-Downloads and initializes a timestamped Whisper model. Calling it before [`transcribe()`](/docs/whisper-webgpu/transcribe) lets you show model-loading progress.+Downloads a timestamped Whisper model from `remotion.media` and initializes it. Calling it before [`transcribe()`](/docs/whisper-webgpu/transcribe) lets you show model-loading progress.```ts twoslash title="load-model.ts"import {loadWhisperModel} from '@remotion/whisper-webgpu';@@ -37,6 +37,8 @@ Returns whether the same model pipeline was already loaded in memory.The browser manages its persistent [Transformers.js](https://huggingface.co/docs/transformers.js) cache. This function also retains the initialized model in memory until [`disposeWhisperModel()`](/docs/whisper-webgpu/dispose-whisper-model) is called or the page closes.+Versions before 4.0.522 downloaded models from Hugging Face. A model cached by an earlier version is downloaded once more from `remotion.media`, then reused from the new cache entry when using the default Transformers.js cache settings.+## Compatibility<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />diff --git a/packages/docs/docs/whisper-webgpu/transcribe.mdx b/packages/docs/docs/whisper-webgpu/transcribe.mdxindex e0f5d43b17..7b903644b3 100644--- a/packages/docs/docs/whisper-webgpu/transcribe.mdx+++ b/packages/docs/docs/whisper-webgpu/transcribe.mdx@@ -36,6 +36,12 @@ One of the timestamped models returned by [`getAvailableModels()`](/docs/whisperThe spoken language name or language code. This option is required for multilingual models because automatic language detection is not supported. Omit it for English-only models such as `small.en`.+### task?<AvailableFrom v="4.0.522" />++Either `transcribe` or `translate`. Default: `transcribe`.++`translate` translates speech into English and is supported by multilingual non-turbo models only. Check the model's `supportsTranslation` property before using it. Word timestamps may be less reliable when translating.+### chunkLengthInSeconds?Length of long-audio chunks. Default: `30`. Must be a finite number greater than `0`.@@ -44,6 +50,30 @@ Length of long-audio chunks. Default: `30`. Must be a finite number greater thanOverlap on both sides of a chunk. Default: `5`. Must be finite, non-negative, and less than half of `chunkLengthInSeconds`.+### forceFullSequences?<AvailableFrom v="4.0.522" />++Whether every sequence must end with a timestamp. Default: `false`. Must be a boolean. When set to `true`, transcription throws if the model returns a trailing sequence without an ending timestamp.++### doSample?<AvailableFrom v="4.0.522" />++Whether to sample tokens probabilistically instead of always choosing the most likely token. Default: `false`. Must be a boolean. Sampling can produce different results between runs.++### temperature?<AvailableFrom v="4.0.522" />++Controls randomness when `doSample` is `true`. Default: `1`. Must be a finite number greater than `0`; lower values favor more likely tokens.++### topK?<AvailableFrom v="4.0.522" />++Limits sampling to the tokens with the highest probabilities when `doSample` is `true`. Default: `50`. Must be a non-negative integer; `0` disables the limit.++### repetitionPenalty?<AvailableFrom v="4.0.522" />++Adjusts the probability of tokens that have already been generated. Default: `1`, which applies no penalty. Must be a finite number greater than `0`; values above `1` discourage repetition and values below `1` encourage it.++### noRepeatNgramSize?<AvailableFrom v="4.0.522" />++Prevents a generated token sequence of this length from occurring more than once. Default: `0`, which disables the constraint. Must be a non-negative integer.+### onModelLoadProgress?Called while the model is loaded. To load independently, use [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model).