Code-first video & image, rendered natively on WebGPU.
@framefields/node-vision
2.14.0
2.13.1
2.13.0
2.12.0
2.11.0
2.9.3
2.9.2
Minor Changes
- a1e37b8: A
withVision({ mode: "skeleton" })layer of a clip thatcomp.visionanalyses now draws the pose that clip matched to its subject, for the frame on screen, instead of running its own pass and drawing the first person detected. A skeleton layer of a SAM clip needspose: trueon the clip and fails clearly without it. Skeleton mode draws only the skeleton: a frame without a pose is empty rather than showing the footage. SAM clip caches written before this change claimed pose they did not track; they are invalidated, so SAM clips re-analyse once.
2.9.0
Patch Changes
- 7af1f91: Add prompted SAM 2.1 Tiny and SAM 3 video segmentation and semantic text selection to clip vision, decoupled into a dedicated
@framefields/vision-sampackage under the Meta SAM License. Includes temporal memory, point/box corrections, semantic text selection, cached frame seeking, and time-remapped matte reuse. Selected-object masks, geometry, motion, and optional matched pose are exposed through standard vision signals. Model assets download dynamically at runtime to local cache with SHA-256 verification upon explicit license agreement.
2.8.1
2.8.0
2.7.0
Patch Changes
- 18aece7: Add forward and reverse media time remapping for video and audio layers, including smooth ramps, video holds, silent audio holds, source range checks and clip vision synchronization. Reverse video decoding uses a bounded frame cache.
2.6.2
2.6.1
2.6.0
2.5.5
Patch Changes
- demos
2.5.4
2.5.3
Patch Changes
- docs-fixes
2.5.2
2.5.1
2.5.0
2.4.3
2.4.2
2.4.1
2.4.0
2.3.2
2.3.1
2.3.0
Minor Changes
- 303c410:
withVision({ mode: "matte" })(andmask/crop) on a video layer now reusescomp.vision(clip)'s per-frame results when the composition already analyses that clip, instead of segmenting every frame a second time during render. The clip keeps each frame's full-resolution subject mask (merged the same way as the matte), persisted beside its other results in the vision cache, and the cutout uses the mask for the frame being drawn, so it no longer runs one frame behind the plate. Layers whose settings would give a different subject (classes,keyBackground,matteSource: "selfie", a differentvariant/confidence/maskThreshold/featherRadius, or an attached vision bundle) keep their own segmentation, and so do frames the clip has not analysed.
2.2.0
Patch Changes
5c55a7b: Vision layers no longer copy frames to the CPU. ONNX Runtime's WebGPU sessions now run on the renderer's own device (
setVisionDevice), and a compute shader turns the layer's texture straight into each model's input tensor. It does the same resize, letterbox, channel order and normalisation asimageToTensor, within float32 rounding. A small GPU checksum replaces hashing the pixels on the CPU, so layers showing the same frame still share a pass. Frames come back to the CPU only for the opt-in background key (keyBackground).VisionRunnermethods accept aVisionGpuImage({ texture, device, width, height }) as well as RGBA pixels. When its sessions run on another device, or on the CPU, the runner reads the texture back itself.WebGPUProvidertakes adeviceoption and feedsgpuBufferinputs withTensor.fromGpuBuffer.60c35a0: Faster on-device vision, on the GPU by default:
- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
classes: ["person"]and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions throughVisionRunner.derive(). This also fixes thethread constructor failed: Resource temporarily unavailablecrash withonnxruntime-webinstalled. - Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.
onnxruntime-webis now an optional dependency offramefields, so inference runs on WebGPU when there is a hardware adapter. Dawn'sGPUSupportedFeaturesgains theforEachthat onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores.
FRAMEFIELDS_VISION_THREADSoverrides it. FRAMEFIELDS_VISION_PROVIDER(auto,webgpu,cpu,cuda) forces a backend;cudauses onnxruntime-node's CUDA EP on Linux x64.
The smoke project's
visionexport (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after. In the renderer worker on a RunPod RTX 3070: 58.7 s before, 22–25 s after.- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
2.2.0-beta.4
2.2.0-beta.3
2.2.0-beta.2
Patch Changes
5c55a7b: Vision layers no longer copy frames to the CPU. ONNX Runtime's WebGPU sessions now run on the renderer's own device (
setVisionDevice), and a compute shader turns the layer's texture straight into each model's input tensor. It does the same resize, letterbox, channel order and normalisation asimageToTensor, within float32 rounding. A small GPU checksum replaces hashing the pixels on the CPU, so layers showing the same frame still share a pass. Frames come back to the CPU only for the opt-in background key (keyBackground).VisionRunnermethods accept aVisionGpuImage({ texture, device, width, height }) as well as RGBA pixels. When its sessions run on another device, or on the CPU, the runner reads the texture back itself.WebGPUProvidertakes adeviceoption and feedsgpuBufferinputs withTensor.fromGpuBuffer.
2.2.0-beta.1
Patch Changes
60c35a0: Faster on-device vision, on the GPU by default:
- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
classes: ["person"]and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions throughVisionRunner.derive(). This also fixes thethread constructor failed: Resource temporarily unavailablecrash withonnxruntime-webinstalled. - Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.
onnxruntime-webis now an optional dependency offramefields, so inference runs on WebGPU when there is a hardware adapter. Dawn'sGPUSupportedFeaturesgains theforEachthat onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores.
FRAMEFIELDS_VISION_THREADSoverrides it. FRAMEFIELDS_VISION_PROVIDER(auto,webgpu,cpu,cuda) forces a backend;cudauses onnxruntime-node's CUDA EP on Linux x64.
The smoke project's
visionexport (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after. In the renderer worker on a RunPod RTX 3070: 58.7 s before, 22–25 s after.- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
2.2.0-beta.0
2.1.3
2.1.2
2.1.1
Patch Changes
- rm aws bundle
2.1.0
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
2.0.2
2.0.1
2.0.0
1.4.3
1.4.2
1.4.1
1.4.0
1.3.2
1.3.1
Patch Changes
- init