Skip to content

🎥 Make videos programmatically with React

Refine video matting and Whisper documentation

JonnyBurger3ac5dc7ae0aa
diff --git a/packages/docs/docs/video-matting/load-video-matting-model.mdx b/packages/docs/docs/video-matting/load-video-matting-model.mdx
index b7731f7e4f..be9348af2f 100644
--- a/packages/docs/docs/video-matting/load-video-matting-model.mdx
+++ b/packages/docs/docs/video-matting/load-video-matting-model.mdx
@@ -6,7 +6,9 @@ crumb: '@remotion/video-matting'
# loadVideoMattingModel()<AvailableFrom v="4.0.523" />
-Downloads a video matting model from an immutable path on `remotion.media` and initializes it. Calling it before [`separateVideoLayers()`](/docs/video-matting/separate-video-layers) lets you show model-loading progress.
+Downloads a video matting model from our `remotion.media` CDN.
+
+Call it before [`separateVideoLayers()`](/docs/video-matting/separate-video-layers).
```ts twoslash title="load-model.ts"
import {loadVideoMattingModel} from '@remotion/video-matting';
@@ -67,10 +69,6 @@ Releases this call's hold on the initialized model. The pipeline is disposed aft
Use `await using` to call this method automatically when leaving the current scope. Call [`disposeVideoMattingModel()`](/docs/video-matting/dispose-video-matting-model) to dispose the pipeline explicitly regardless of retained load handles.
-The browser manages the persistent [Transformers.js](https://huggingface.co/docs/transformers.js) cache. Releasing the initialized pipeline does not delete the downloaded model files.
-
-The hosted files are byte-identical to files from the pinned upstream Hugging Face revision.
-
## Compatibility
<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />
diff --git a/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx b/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx
index 2b948b91be..9dd3b75bf9 100644
--- a/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx
+++ b/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx
@@ -8,8 +8,6 @@ crumb: '@remotion/whisper-webgpu'
Asynchronously checks whether WebGPU transcription is supported in the current environment.
-In a browser, WebGPU, a usable adapter, and a secure context (or localhost) are required. Node.js support is available from <AvailableFrom v="4.0.528" inline />. In Node.js, the function creates and disposes a tiny ONNX Runtime session using the WebGPU execution provider. This checks whether ONNX Runtime can acquire a WebGPU adapter without downloading a Whisper model or running inference.
-
```ts twoslash title="support.ts"
import {canUseWhisperWebGpu} from '@remotion/whisper-webgpu';
diff --git a/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx
index 7156e7e15a..247e04da72 100644
--- a/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx
+++ b/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx
@@ -8,8 +8,6 @@ crumb: '@remotion/whisper-webgpu'
Releases the memory used by initialized Whisper pipelines. Persistently cached model files are not deleted.
-Calling this function disposes the selected pipeline regardless of retained handles returned by [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model).
-
```ts twoslash title="dispose.ts"
import {disposeWhisperModel} from '@remotion/whisper-webgpu';
diff --git a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
index c90ceace81..4177571a60 100644
--- a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
+++ b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx
@@ -45,8 +45,6 @@ Use `await using` to call this method automatically when leaving the current sco
[Transformers.js](https://huggingface.co/docs/transformers.js) manages the persistent model cache. Releasing the initialized pipeline does not delete the downloaded model files.
-Versions before 4.0.522 downloaded models from Hugging Face. A model cached by an earlier version is downloaded once more from `remotion.media`, then reused from the new cache entry when using the default Transformers.js cache settings.
-
## Compatibility
<CompatibilityTable chrome firefox safari nodejs={true} bun={true} serverlessFunctions={false} clientSideRendering={false} serverSideRendering player studio />
diff --git a/packages/docs/docs/whisper-webgpu/node.mdx b/packages/docs/docs/whisper-webgpu/node.mdx
index 883a36e2eb..b97d4ebefd 100644
--- a/packages/docs/docs/whisper-webgpu/node.mdx
+++ b/packages/docs/docs/whisper-webgpu/node.mdx
@@ -6,11 +6,9 @@ crumb: '@remotion/whisper-webgpu'
# Using @remotion/whisper-webgpu in Node.js<AvailableFrom v="4.0.528" />
-`@remotion/whisper-webgpu` can transcribe a 16kHz mono waveform in Node.js using [Transformers.js](https://huggingface.co/docs/transformers.js) and [ONNX Runtime's WebGPU execution provider](https://onnxruntime.ai/docs/execution-providers/WebGPU-ExecutionProvider.html).
+You can use the `@remotion/whisper-webgpu` package in Node.js to transcribe audio.
-Node.js does not expose the browser `navigator.gpu` API. Transformers.js uses the native `onnxruntime-node` addon instead, whose experimental WebGPU execution provider uses Dawn to access Metal, Vulkan, or Direct3D 12.
-
-The following example uses [`@mediabunny/server`](https://mediabunny.dev/guide/extensions/server) to decode and resample a local audio or video file before transcribing it.
+The following example uses [`@mediabunny/server`](https://mediabunny.dev/guide/extensions/server) to decode and resample a local audio or video file to 16Khz.
## Installation
@@ -39,61 +37,65 @@ import {writeFile} from 'node:fs/promises';
registerMediabunnyServer();
-const chunks: Array<{startFrame: number; waveform: Float32Array}> = [];
-
-{
- using input = new Input({
- formats: ALL_FORMATS,
- source: new FilePathSource('audio.mp3'),
- });
- const audioTrack = await input.getPrimaryAudioTrack();
- if (audioTrack === null) {
- throw new Error('The media does not contain an audio track.');
- }
-
- const conversion = await Conversion.init({
- input,
- output: new Output({
- format: new WavOutputFormat(),
- target: new NullTarget(),
- }),
- video: {discard: true},
- audio: (track) => {
- if (track.id !== audioTrack.id) {
- return {discard: true};
- }
-
- return {
- codec: 'pcm-f32',
- forceTranscode: true,
- numberOfChannels: 1,
- sampleFormat: 'f32',
- sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,
- process: (sample) => {
- const waveform = new Float32Array(
- sample.allocationSize({format: 'f32', planeIndex: 0}) /
- Float32Array.BYTES_PER_ELEMENT,
- );
- sample.copyTo(waveform, {format: 'f32', planeIndex: 0});
- chunks.push({
- startFrame: Math.round(
- sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE,
- ),
- waveform,
- });
- return sample;
- },
- };
- },
- });
-
- if (!conversion.isValid) {
- throw new Error('The audio track cannot be decoded.');
- }
+type WaveformChunk = {
+ startFrame: number;
+ waveform: Float32Array;
+};
+const chunks: WaveformChunk[] = [];
+
+// In long-running apps, put this code in an async function rather than using
+// `using` at the top level: top-level resources live until the module finishes.
+using input = new Input({
+ formats: ALL_FORMATS,
+ source: new FilePathSource('audio.mp3'),
+});
+const audioTrack = await input.getPrimaryAudioTrack();
+if (audioTrack === null) {
+ throw new Error('The media does not contain an audio track.');
+}
- await conversion.execute();
+const conversion = await Conversion.init({
+ input,
+ output: new Output({
+ format: new WavOutputFormat(),
+ target: new NullTarget(),
+ }),
+ video: {discard: true},
+ audio: (track) => {
+ if (track.id !== audioTrack.id) {
+ return {discard: true};
+ }
+
+ return {
+ codec: 'pcm-f32',
+ forceTranscode: true,
+ numberOfChannels: 1,
+ sampleFormat: 'f32',
+ sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,
+ process: (sample) => {
+ const waveform = new Float32Array(
+ sample.allocationSize({format: 'f32', planeIndex: 0}) /
+ Float32Array.BYTES_PER_ELEMENT,
+ );
+ sample.copyTo(waveform, {format: 'f32', planeIndex: 0});
+ chunks.push({
+ startFrame: Math.round(
+ sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE,
+ ),
+ waveform,
+ });
+ return sample;
+ },
+ };
+ },
+});
+
+if (!conversion.isValid) {
+ throw new Error('The audio track cannot be decoded.');
}
+await conversion.execute();
+
const waveformLength = chunks.reduce(
(max, chunk) => Math.max(max, chunk.startFrame + chunk.waveform.length),
0,
@@ -122,13 +124,17 @@ const {captions} = toCaptions({whisperWebGpuOutput: transcription});
await writeFile('captions.json', JSON.stringify(captions, null, 2));
```
-The model is downloaded on the first run and cached on the file system. `await using` releases it from memory when the script exits while keeping the downloaded files cached. The input to [`transcribe()`](/docs/whisper-webgpu/transcribe) must be a mono `Float32Array` sampled at 16kHz.
-
Call [`canUseWhisperWebGpu()`](/docs/whisper-webgpu/can-use-whisper-webgpu) before loading the model if the script may run on different hardware. It creates and disposes a tiny WebGPU-backed ONNX session, so it verifies that ONNX Runtime can acquire an adapter without downloading a Whisper model.
-## Supported platforms
+## Implementation notes
+
+A compatible GPU is required for this to work.
+
+Node.js does not have WebGPU APIs available by default.
+
+Transformers.js uses the `onnxruntime-node` addon instead, whose experimental WebGPU execution provider uses [Dawn](https://github.com/google/dawn).
-ONNX Runtime bundles its experimental WebGPU execution provider for Node.js on macOS x64 and Arm64, Windows x64 and Arm64, and Linux x64. A usable GPU adapter and compatible driver are still required.
+ONNX Runtime does not work on Linux arm64.
## See also