🎥 Make videos programmatically with React
Refine video matting and Whisper documentation
JonnyBurger3ac5dc7ae0aa
diff --git a/packages/docs/docs/video-matting/load-video-matting-model.mdx b/packages/docs/docs/video-matting/load-video-matting-model.mdxindex b7731f7e4f..be9348af2f 100644--- a/packages/docs/docs/video-matting/load-video-matting-model.mdx+++ b/packages/docs/docs/video-matting/load-video-matting-model.mdx@@ -6,7 +6,9 @@ crumb: '@remotion/video-matting'# loadVideoMattingModel()<AvailableFrom v="4.0.523" />-Downloads a video matting model from an immutable path on `remotion.media` and initializes it. Calling it before [`separateVideoLayers()`](/docs/video-matting/separate-video-layers) lets you show model-loading progress.+Downloads a video matting model from our `remotion.media` CDN.++Call it before [`separateVideoLayers()`](/docs/video-matting/separate-video-layers).```ts twoslash title="load-model.ts"import {loadVideoMattingModel} from '@remotion/video-matting';@@ -67,10 +69,6 @@ Releases this call's hold on the initialized model. The pipeline is disposed aftUse `await using` to call this method automatically when leaving the current scope. Call [`disposeVideoMattingModel()`](/docs/video-matting/dispose-video-matting-model) to dispose the pipeline explicitly regardless of retained load handles.-The browser manages the persistent [Transformers.js](https://huggingface.co/docs/transformers.js) cache. Releasing the initialized pipeline does not delete the downloaded model files.--The hosted files are byte-identical to files from the pinned upstream Hugging Face revision.-## Compatibility<CompatibilityTable chrome firefox safari nodejs="" bun="" serverlessFunctions="" clientSideRendering={false} serverSideRendering player studio hideServers />diff --git a/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx b/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdxindex 2b948b91be..9dd3b75bf9 100644--- a/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx+++ b/packages/docs/docs/whisper-webgpu/can-use-whisper-webgpu.mdx@@ -8,8 +8,6 @@ crumb: '@remotion/whisper-webgpu'Asynchronously checks whether WebGPU transcription is supported in the current environment.-In a browser, WebGPU, a usable adapter, and a secure context (or localhost) are required. Node.js support is available from <AvailableFrom v="4.0.528" inline />. In Node.js, the function creates and disposes a tiny ONNX Runtime session using the WebGPU execution provider. This checks whether ONNX Runtime can acquire a WebGPU adapter without downloading a Whisper model or running inference.-```ts twoslash title="support.ts"import {canUseWhisperWebGpu} from '@remotion/whisper-webgpu';diff --git a/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdxindex 7156e7e15a..247e04da72 100644--- a/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx+++ b/packages/docs/docs/whisper-webgpu/dispose-whisper-model.mdx@@ -8,8 +8,6 @@ crumb: '@remotion/whisper-webgpu'Releases the memory used by initialized Whisper pipelines. Persistently cached model files are not deleted.-Calling this function disposes the selected pipeline regardless of retained handles returned by [`loadWhisperModel()`](/docs/whisper-webgpu/load-whisper-model).-```ts twoslash title="dispose.ts"import {disposeWhisperModel} from '@remotion/whisper-webgpu';diff --git a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdxindex c90ceace81..4177571a60 100644--- a/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx+++ b/packages/docs/docs/whisper-webgpu/load-whisper-model.mdx@@ -45,8 +45,6 @@ Use `await using` to call this method automatically when leaving the current sco[Transformers.js](https://huggingface.co/docs/transformers.js) manages the persistent model cache. Releasing the initialized pipeline does not delete the downloaded model files.-Versions before 4.0.522 downloaded models from Hugging Face. A model cached by an earlier version is downloaded once more from `remotion.media`, then reused from the new cache entry when using the default Transformers.js cache settings.-## Compatibility<CompatibilityTable chrome firefox safari nodejs={true} bun={true} serverlessFunctions={false} clientSideRendering={false} serverSideRendering player studio />diff --git a/packages/docs/docs/whisper-webgpu/node.mdx b/packages/docs/docs/whisper-webgpu/node.mdxindex 883a36e2eb..b97d4ebefd 100644--- a/packages/docs/docs/whisper-webgpu/node.mdx+++ b/packages/docs/docs/whisper-webgpu/node.mdx@@ -6,11 +6,9 @@ crumb: '@remotion/whisper-webgpu'# Using @remotion/whisper-webgpu in Node.js<AvailableFrom v="4.0.528" />-`@remotion/whisper-webgpu` can transcribe a 16kHz mono waveform in Node.js using [Transformers.js](https://huggingface.co/docs/transformers.js) and [ONNX Runtime's WebGPU execution provider](https://onnxruntime.ai/docs/execution-providers/WebGPU-ExecutionProvider.html).+You can use the `@remotion/whisper-webgpu` package in Node.js to transcribe audio.-Node.js does not expose the browser `navigator.gpu` API. Transformers.js uses the native `onnxruntime-node` addon instead, whose experimental WebGPU execution provider uses Dawn to access Metal, Vulkan, or Direct3D 12.--The following example uses [`@mediabunny/server`](https://mediabunny.dev/guide/extensions/server) to decode and resample a local audio or video file before transcribing it.+The following example uses [`@mediabunny/server`](https://mediabunny.dev/guide/extensions/server) to decode and resample a local audio or video file to 16Khz.## Installation@@ -39,61 +37,65 @@ import {writeFile} from 'node:fs/promises';registerMediabunnyServer();-const chunks: Array<{startFrame: number; waveform: Float32Array}> = [];--{- using input = new Input({- formats: ALL_FORMATS,- source: new FilePathSource('audio.mp3'),- });- const audioTrack = await input.getPrimaryAudioTrack();- if (audioTrack === null) {- throw new Error('The media does not contain an audio track.');- }-- const conversion = await Conversion.init({- input,- output: new Output({- format: new WavOutputFormat(),- target: new NullTarget(),- }),- video: {discard: true},- audio: (track) => {- if (track.id !== audioTrack.id) {- return {discard: true};- }-- return {- codec: 'pcm-f32',- forceTranscode: true,- numberOfChannels: 1,- sampleFormat: 'f32',- sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,- process: (sample) => {- const waveform = new Float32Array(- sample.allocationSize({format: 'f32', planeIndex: 0}) /- Float32Array.BYTES_PER_ELEMENT,- );- sample.copyTo(waveform, {format: 'f32', planeIndex: 0});- chunks.push({- startFrame: Math.round(- sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE,- ),- waveform,- });- return sample;- },- };- },- });-- if (!conversion.isValid) {- throw new Error('The audio track cannot be decoded.');- }+type WaveformChunk = {+ startFrame: number;+ waveform: Float32Array;+};+const chunks: WaveformChunk[] = [];++// In long-running apps, put this code in an async function rather than using+// `using` at the top level: top-level resources live until the module finishes.+using input = new Input({+ formats: ALL_FORMATS,+ source: new FilePathSource('audio.mp3'),+});+const audioTrack = await input.getPrimaryAudioTrack();+if (audioTrack === null) {+ throw new Error('The media does not contain an audio track.');+}- await conversion.execute();+const conversion = await Conversion.init({+ input,+ output: new Output({+ format: new WavOutputFormat(),+ target: new NullTarget(),+ }),+ video: {discard: true},+ audio: (track) => {+ if (track.id !== audioTrack.id) {+ return {discard: true};+ }++ return {+ codec: 'pcm-f32',+ forceTranscode: true,+ numberOfChannels: 1,+ sampleFormat: 'f32',+ sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,+ process: (sample) => {+ const waveform = new Float32Array(+ sample.allocationSize({format: 'f32', planeIndex: 0}) /+ Float32Array.BYTES_PER_ELEMENT,+ );+ sample.copyTo(waveform, {format: 'f32', planeIndex: 0});+ chunks.push({+ startFrame: Math.round(+ sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE,+ ),+ waveform,+ });+ return sample;+ },+ };+ },+});++if (!conversion.isValid) {+ throw new Error('The audio track cannot be decoded.');}+await conversion.execute();+const waveformLength = chunks.reduce((max, chunk) => Math.max(max, chunk.startFrame + chunk.waveform.length),0,@@ -122,13 +124,17 @@ const {captions} = toCaptions({whisperWebGpuOutput: transcription});await writeFile('captions.json', JSON.stringify(captions, null, 2));```-The model is downloaded on the first run and cached on the file system. `await using` releases it from memory when the script exits while keeping the downloaded files cached. The input to [`transcribe()`](/docs/whisper-webgpu/transcribe) must be a mono `Float32Array` sampled at 16kHz.-Call [`canUseWhisperWebGpu()`](/docs/whisper-webgpu/can-use-whisper-webgpu) before loading the model if the script may run on different hardware. It creates and disposes a tiny WebGPU-backed ONNX session, so it verifies that ONNX Runtime can acquire an adapter without downloading a Whisper model.-## Supported platforms+## Implementation notes++A compatible GPU is required for this to work.++Node.js does not have WebGPU APIs available by default.++Transformers.js uses the `onnxruntime-node` addon instead, whose experimental WebGPU execution provider uses [Dawn](https://github.com/google/dawn).-ONNX Runtime bundles its experimental WebGPU execution provider for Node.js on macOS x64 and Arm64, Windows x64 and Arm64, and Linux x64. A usable GPU adapter and compatible driver are still required.+ONNX Runtime does not work on Linux arm64.## See also