Code-first video & image, rendered natively on WebGPU.
@framefields/vision
2.14.0
2.13.1
2.13.0
2.12.0
2.11.0
2.9.3
2.9.2
Minor Changes
- a1e37b8: A
withVision({ mode: "skeleton" })layer of a clip thatcomp.visionanalyses now draws the pose that clip matched to its subject, for the frame on screen, instead of running its own pass and drawing the first person detected. A skeleton layer of a SAM clip needspose: trueon the clip and fails clearly without it. Skeleton mode draws only the skeleton: a frame without a pose is empty rather than showing the footage. SAM clip caches written before this change claimed pose they did not track; they are invalidated, so SAM clips re-analyse once.
2.9.0
Minor Changes
- dac3694:
@framefields/vision-samis now an optional peer offramefields, loaded only when a clip usessam2orsam3, so the SDK no longer installs or imports SAM-licensed code by default. Install it next toframefieldsto use prompted tracking, and importSam2Tracker/Sam3Trackerfrom@framefields/vision-sam. Mask-derived object geometry moved to@framefields/visionasmaskObject(formerlysam2Object). - 7af1f91: Add prompted SAM 2.1 Tiny and SAM 3 video segmentation and semantic text selection to clip vision, decoupled into a dedicated
@framefields/vision-sampackage under the Meta SAM License. Includes temporal memory, point/box corrections, semantic text selection, cached frame seeking, and time-remapped matte reuse. Selected-object masks, geometry, motion, and optional matched pose are exposed through standard vision signals. Model assets download dynamically at runtime to local cache with SHA-256 verification upon explicit license agreement. - 72f6aaa: SAM 3 downloads the image encoder its backend runs fastest: fp16 on WebGPU and CUDA, q4 on the CPU (
SAM3_ENCODER, override withsam3.precision). Providers report their backend before any download (SessionProvider.backend(),sessionCompute); the auto provider decides with a one-node probe graph and, on Linux with the NVIDIA driver, tries CUDA first. All SAM 3 trackers in a process share one encoder session and reuse its features per frame. Text-seeded objects now output and remember their seed mask on the prompt frame, as SAM does, instead of a speckled re-decode.
Patch Changes
- f1623c0: SAM 3 tracks several objects at once:
sam3.maxObjectslets a text prompt select up to N instances (tracks 1…N by score) through one encoder pass per frame, and points prompts takeobjectto seed or correct one track.comp.visionpublishes every track insignals.objectsandsignals.masks; the clip matte is their union. Fixes concurrentcomp.visionclips of one source dropping decoded frames (each clip now owns its decoder) and concurrent ORT WebGPU session creation falling back to the CPU (creation is serialized; SAM trackers share the process provider).
2.8.1
2.8.0
2.7.0
2.6.2
2.6.1
2.6.0
2.5.5
Patch Changes
- demos
2.5.4
2.5.3
Patch Changes
- docs-fixes
2.5.2
2.5.1
2.5.0
2.4.3
Patch Changes
- Strict compile-time type safety across the engine harness:
- Branded types for coordinate and unit spaces (
Pixels,Normalized01,Degrees,Radians,Frames,Seconds,Hertz,Decibels). - Discriminated union types and capability-safe layer ports on
NodeEffect,Layernodes, and effect classes. - Compile-time exhaustiveness checking with
assertNeverand type-narrowed animation property targeting. - Schema and pipeline unification with generic node operations.
- Branded types for coordinate and unit spaces (
2.4.2
2.4.1
2.4.0
2.3.2
2.3.1
2.3.0
Minor Changes
- 303c410:
withVision({ mode: "matte" })(andmask/crop) on a video layer now reusescomp.vision(clip)'s per-frame results when the composition already analyses that clip, instead of segmenting every frame a second time during render. The clip keeps each frame's full-resolution subject mask (merged the same way as the matte), persisted beside its other results in the vision cache, and the cutout uses the mask for the frame being drawn, so it no longer runs one frame behind the plate. Layers whose settings would give a different subject (classes,keyBackground,matteSource: "selfie", a differentvariant/confidence/maskThreshold/featherRadius, or an attached vision bundle) keep their own segmentation, and so do frames the clip has not analysed.
2.2.0
Minor Changes
5c55a7b: Vision layers no longer copy frames to the CPU. ONNX Runtime's WebGPU sessions now run on the renderer's own device (
setVisionDevice), and a compute shader turns the layer's texture straight into each model's input tensor. It does the same resize, letterbox, channel order and normalisation asimageToTensor, within float32 rounding. A small GPU checksum replaces hashing the pixels on the CPU, so layers showing the same frame still share a pass. Frames come back to the CPU only for the opt-in background key (keyBackground).VisionRunnermethods accept aVisionGpuImage({ texture, device, width, height }) as well as RGBA pixels. When its sessions run on another device, or on the CPU, the runner reads the texture back itself.WebGPUProvidertakes adeviceoption and feedsgpuBufferinputs withTensor.fromGpuBuffer.60c35a0: Faster on-device vision, on the GPU by default:
- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
classes: ["person"]and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions throughVisionRunner.derive(). This also fixes thethread constructor failed: Resource temporarily unavailablecrash withonnxruntime-webinstalled. - Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.
onnxruntime-webis now an optional dependency offramefields, so inference runs on WebGPU when there is a hardware adapter. Dawn'sGPUSupportedFeaturesgains theforEachthat onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores.
FRAMEFIELDS_VISION_THREADSoverrides it. FRAMEFIELDS_VISION_PROVIDER(auto,webgpu,cpu,cuda) forces a backend;cudauses onnxruntime-node's CUDA EP on Linux x64.
The smoke project's
visionexport (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after. In the renderer worker on a RunPod RTX 3070: 58.7 s before, 22–25 s after.- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
Patch Changes
25edbba: A
mask/matte/cropcutout withclasseskeeps the whole subject again. The class filter ran before the subject was assembled, so the parts COCO labels as other classes were thrown away before they could merge: a flowing dress comes back as a "kite", "umbrella" or "horse", and aclasses: ["person"]cutout of a dancer kept only head, arms and torso (about 3% of the frame instead of 12%). The text behind the dancer then showed through the dress on most frames, on every platform.classesnow only decides which instance is the subject; overlapping parts of any class merge into it, and the masks a vision bundle receives are still filtered byclasses.mergeSubjectMask(masks, options)takes{ classes, overlap, maxGrowth }instead of positionaloverlapandmaxGrowth, andselectSubjectMask(masks, classes)takes an optional class list.931435f: Instance segmentation does far less work per frame. The RTMDet-Ins export ran NMS on the CPU in the middle of its graph and returned a full 640×640 mask for every detection: about 40 per frame, 65 MB of floats, computed on the GPU and copied back. The runner now cuts the graph at load time before its first data-dependent step and does the rest itself: top-k, per-class NMS with the export's thresholds, and each mask's three dynamic 1×1 layers at 80×80, only for the detections a layer keeps.
detect()computes no masks at all.Results match the end2end graph: the same detections, scores and boxes, and masks with IoU above 0.98 on the reference frames (edges differ by one interpolation step). The model file and its SHA-256 check are unchanged; a model without the expected head tensors still runs end to end.
The smoke project's
visionexport on an Apple Silicon Mac: 22.3 s → 15 s.34f497a: Fix duplicate pose skeletons on the GPU path. In its default NHWC layout, onnxruntime-web (1.30)'s WebGPU execution provider computes grouped convolutions wrong when a group has more than one channel and the count isn't a multiple of 4. That corrupts RTMO's head (18 channels per group) and turned one person into several detections. Models with such a convolution (RTMO) now run in NCHW layout; the others (RTMDet-Ins, Selfie, whose grouped convolutions are depthwise) keep NHWC, which is about 30% faster. Outputs match the CPU provider on every pinned model.
Vision now reports its inference backend once per process, and the renderer logs it:
[vision] Inference on the GPU (WebGPU)withsharedDevice, or[vision] Inference on the CPUwith the reason.
2.2.0-beta.4
2.2.0-beta.3
Patch Changes
25edbba: A
mask/matte/cropcutout withclasseskeeps the whole subject again. The class filter ran before the subject was assembled, so the parts COCO labels as other classes were thrown away before they could merge: a flowing dress comes back as a "kite", "umbrella" or "horse", and aclasses: ["person"]cutout of a dancer kept only head, arms and torso (about 3% of the frame instead of 12%). The text behind the dancer then showed through the dress on most frames, on every platform.classesnow only decides which instance is the subject; overlapping parts of any class merge into it, and the masks a vision bundle receives are still filtered byclasses.mergeSubjectMask(masks, options)takes{ classes, overlap, maxGrowth }instead of positionaloverlapandmaxGrowth, andselectSubjectMask(masks, classes)takes an optional class list.
2.2.0-beta.2
Minor Changes
5c55a7b: Vision layers no longer copy frames to the CPU. ONNX Runtime's WebGPU sessions now run on the renderer's own device (
setVisionDevice), and a compute shader turns the layer's texture straight into each model's input tensor. It does the same resize, letterbox, channel order and normalisation asimageToTensor, within float32 rounding. A small GPU checksum replaces hashing the pixels on the CPU, so layers showing the same frame still share a pass. Frames come back to the CPU only for the opt-in background key (keyBackground).VisionRunnermethods accept aVisionGpuImage({ texture, device, width, height }) as well as RGBA pixels. When its sessions run on another device, or on the CPU, the runner reads the texture back itself.WebGPUProvidertakes adeviceoption and feedsgpuBufferinputs withTensor.fromGpuBuffer.
Patch Changes
931435f: Instance segmentation does far less work per frame. The RTMDet-Ins export ran NMS on the CPU in the middle of its graph and returned a full 640×640 mask for every detection: about 40 per frame, 65 MB of floats, computed on the GPU and copied back. The runner now cuts the graph at load time before its first data-dependent step and does the rest itself: top-k, per-class NMS with the export's thresholds, and each mask's three dynamic 1×1 layers at 80×80, only for the detections a layer keeps.
detect()computes no masks at all.Results match the end2end graph: the same detections, scores and boxes, and masks with IoU above 0.98 on the reference frames (edges differ by one interpolation step). The model file and its SHA-256 check are unchanged; a model without the expected head tensors still runs end to end.
The smoke project's
visionexport on an Apple Silicon Mac: 22.3 s → 15 s.34f497a: Fix duplicate pose skeletons on the GPU path. In its default NHWC layout, onnxruntime-web (1.30)'s WebGPU execution provider computes grouped convolutions wrong when a group has more than one channel and the count isn't a multiple of 4. That corrupts RTMO's head (18 channels per group) and turned one person into several detections. Models with such a convolution (RTMO) now run in NCHW layout; the others (RTMDet-Ins, Selfie, whose grouped convolutions are depthwise) keep NHWC, which is about 30% faster. Outputs match the CPU provider on every pinned model.
Vision now reports its inference backend once per process, and the renderer logs it:
[vision] Inference on the GPU (WebGPU)withsharedDevice, or[vision] Inference on the CPUwith the reason.
2.2.0-beta.1
Minor Changes
60c35a0: Faster on-device vision, on the GPU by default:
- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
classes: ["person"]and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions throughVisionRunner.derive(). This also fixes thethread constructor failed: Resource temporarily unavailablecrash withonnxruntime-webinstalled. - Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.
onnxruntime-webis now an optional dependency offramefields, so inference runs on WebGPU when there is a hardware adapter. Dawn'sGPUSupportedFeaturesgains theforEachthat onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores.
FRAMEFIELDS_VISION_THREADSoverrides it. FRAMEFIELDS_VISION_PROVIDER(auto,webgpu,cpu,cuda) forces a backend;cudauses onnxruntime-node's CUDA EP on Linux x64.
The smoke project's
visionexport (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after. In the renderer worker on a RunPod RTX 3070: 58.7 s before, 22–25 s after.- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with
2.2.0-beta.0
2.1.3
2.1.2
2.1.1
Patch Changes
- rm aws bundle
2.1.0
2.0.11
2.0.10
2.0.9
2.0.8
2.0.7
2.0.6
2.0.5
2.0.4
2.0.3
Patch Changes
- 124f92e: Headless (Node) vision now prefers the GPU: when no provider is passed, runners try onnxruntime-web's WebGPU execution provider on the process's WebGPU device (the Dawn device the renderer already creates) and fall back to the CPU
onnxruntime-nodeprovider when a WebGPU session can't be created. The newAutoSessionProvidermakes that choice, andNodeWebGPUProviderexposes the WebGPU-only path. On example 19's track scene this lifts the render from 2.37 to 3.49 fps with pixel-identical output (PSNR ≈ 103 dB vs CPU).
2.0.2
2.0.1
2.0.0
1.4.3
1.4.2
1.4.1
1.4.0
1.3.2
1.3.1
Patch Changes
- init