Code-first video & image, rendered natively on WebGPU.
name: framefields-vision description: On-device computer vision in framefields — object detection and tracking, instance segmentation, pose, person mattes and cutouts (RTMDet-Ins, RTMO, Selfie Segmenter), soft alpha matting of hair and motion blur (ViTMatte), and prompted video tracking of a specific object or several people with SAM 2.1 Tiny or Small (point/box prompts) and SAM 3 (text prompts such as "man in white shirt", multiple objects at once). Covers which API to use (withVision, comp.vision, analyzeVisionSequence), every pinned ONNX model with sizes and licenses, GPU/CUDA/CPU backends, model downloads and caching, performance, and troubleshooting. Use when a framefields composition needs to know where people or objects are, cut a subject out, follow someone through a clip, or select one person in a crowd.
framefields-vision
Vision runs ONNX models locally from framefields; nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked by size and SHA-256, and are cached in $FRAMEFIELDS_MODELS_DIR (default ~/.cache/framefields/models). Effects that consume vision are in framefields-effects; frame checks in framefields-render.
Which API
| Need | API | Timing |
|---|---|---|
| Cut out, box, or draw a skeleton on a layer while rendering | layer.withVision({ mode }) |
Reads the layer's previous rendered frame: results trail by one frame and exist only while rendering |
| Signals for the whole composition (boxes, masks, pose) | comp.withVision({ … }) |
Same, on the composited frame |
| Know the motion at a frame (physics, layout, keyframes), or reuse results across renders | comp.vision(clipOrLayer, options) |
Analyses the clip's own frames on demand, before they draw; persisted on disk |
| Follow one specific object or person, or select by description | comp.vision(…, { sam2 }) / { sam3 } |
As above, with SAM's temporal memory |
| Find out what is in a clip before authoring | comp.analyzeVisionSequence(path, …) |
One-shot report |
Models
Core models (Apache-2.0, bundled with framefields)
| Task | Model | t |
s (default) |
m |
Input |
|---|---|---|---|---|---|
| Detect + segment (one shared pass), COCO-80 | RTMDet-Ins | 24 MB | 43 MB | 116 MB | 640² |
| Pose, 17 COCO keypoints | RTMO | 27 MB (416²) | 40 MB | 89 MB | 640² |
| Person matte | MediaPipe Selfie Segmenter | square 0.5 MB (256²), landscape 0.5 MB (256×144) for wide frames | |||
Relative depth (depth: true) |
Depth Anything V2 Small | 99 MB, one size | short side 518, frame aspect | ||
Alpha matting (matting: true) |
ViTMatte Small (Distinctions-646) | 104 MB, one size | the subject's edge, long side ≤ 1024 |
About 585 MB for all of them. variant: "t" | "s" | "m" picks the size.
SAM 2.1 Tiny and Small (sam2, Apache-2.0)
Meta SAM 2.1 Hiera checkpoints (Diffusion Studio fp16 video exports; memory of the prompted frame, the 6 latest frames, 16 object pointers). Point and box prompts only. sam2: { model }:
model |
Input | Image encoder | Memory attention | Total | Per frame, M1 Pro WebGPU |
|---|---|---|---|---|---|
"tiny" (default) |
512² (32×32 features) | 58 MB | 13 MB | ~83 MB | ~0.27 s |
"small" |
1024², SAM 2's native size (64×64) | 84 MB | 16 MB | ~112 MB | ~1.1 s |
SAM 3 (sam3, weights under Meta's SAM License)
Video tracking at 1008², plus text ("concept") selection. Only the image encoder the backend runs fastest is downloaded:
| Graph | Size | Source |
|---|---|---|
| Image encoder, fp16 (WebGPU, CUDA) | 935 MB | onnx-community/sam3-tracker-ONNX |
| Image encoder, q4 (CPU) | 369 MB | same |
| Image encoder, q4f16 (opt-in) | 296 MB | same |
| Mask decoder, memory encoder, memory attention | 18 + 6 + 35 MB | jax-image-tools/sam3-video-onnx |
| Dense-prompt and pointer heads | 67 MB | RamRom/sam3-onnx |
| Text selection: image encoder (int4, fp16 attention), text encoder (int8), decoder (int8), tokenizer | 293 + 357 + 27 + 4 MB | danilobukvic/sam3-text-onnx |
That is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.
fp16 is the fastest and most precise on WebGPU (one 1008² frame on an M1 Pro: fp16 3.65 s, q4f16 4.43 s, q4 6.63 s; 4-bit saves memory, not time). SAM3_ENCODER in framefields/sam maps backend to precision; sam3.precision: "fp16" | "q4f16" | "q4" overrides it.
Backends
Inference picks its backend once per process, before any model downloads:
| Machine | Backend |
|---|---|
| Mac (Apple Silicon) | WebGPU through onnxruntime-web on the Metal device |
| Linux with an NVIDIA GPU | CUDA through onnxruntime-node when the NVIDIA driver is loaded and the CUDA 12 / cuDNN 9 libraries are present, else WebGPU (Vulkan), else CPU |
| No usable GPU (CI, lavapipe, SwiftShader) | CPU through onnxruntime-node |
The preview (startPreview) |
WebGPU in the browser; models are served from the local cache by the preview server |
| Variable | Effect |
|---|---|
FRAMEFIELDS_VISION_PROVIDER |
auto (default; the order above), webgpu, cuda or cpu. A forced backend that fails throws instead of falling back. |
FRAMEFIELDS_VISION_THREADS |
CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default. |
FRAMEFIELDS_MODELS_DIR / FRAMEFIELDS_MODELS_BASE_URL |
Model cache directory, or a mirror to download from (offline and CI renders) |
FRAMEFIELDS_VISION_CACHE_DIR |
Where comp.vision persists results (default ~/.cache/framefields/vision) |
The renderer logs [vision] Inference on the GPU (WebGPU), or the backend it fell back to.
Detection, segmentation, pose, mattes
| Option | Model | Gives you |
|---|---|---|
enableDetection (default) |
RTMDet-Ins | COCO-80 boxes, tracked over time → vision.objects |
enableSegmentation |
RTMDet-Ins (same pass) | Soft per-instance masks → vision.masks, vision.segmentation |
enablePose |
RTMO | 17 COCO keypoints per person → vision.poseLandmarks, track.pose |
enableMatte |
Selfie Segmenter | Fast person alpha → vision.segmentation.matte |
// Whole composition: returns the reactive bundle
const vision = comp.withVision({ enableSegmentation: true, enablePose: true });
// One layer: renders through a node mode
Layer.video("assets/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });
Node modes: passthrough, mask, matte, crop, skeleton, boxes, tracking, depth. depth draws the frame's relative depth as gray (white = near); it always reads a comp.vision clip with depth: true, made for it when there is none, so it is never a frame behind. skeleton draws only the pose on transparency: the first person detected, or, for a layer of a comp.vision clip, the pose that clip matched to its selection. For mask / matte / crop, matteSource: "instance" (default; any COCO class, overlapping parts such as a dress merged into the subject) or "selfie" (people only, fastest). confidence (default 0.3) and classes: ["person", …] filter detections.
Measured per 1280–2048 px frame on CPU (onnxruntime-node); the GPU is faster:
| Task | t |
s |
m |
|---|---|---|---|
| detect / segment (one shared pass) | 150–300 ms | 200–340 ms | 420–700 ms |
| pose | ~50 ms | ~120 ms | ~280 ms |
| matte (Selfie Segmenter) | ~20 ms | ~20 ms | ~20 ms |
- Start with
"s". Use"m"for hero shots with fast motion blur or busy backgrounds;"t"mainly saves time on pose. - Use the selfie matte only for close framing. It misses distant figures and can report a "person" in close-ups with nobody in them.
- Raise
confidence(e.g.0.5) on abstract or stylized footage. At 0.3 the model puts loose labels ("teddy bear", "donut") on smoke, eyes and planets. - Restrict
classeswhen you only care about one thing. It also stops the subject switching to another object when the person leaves frame.
At render time:
- One-frame delay.
withVisionreads the previous rendered frame, so the first frame has no results (a cutout renders transparent) and masks trail the plate by a frame. A matte of a clip the composition also reads withcomp.visionhas neither problem. - Render vision frames in order.
renderVideodoes. For stills, render at least two consecutive frames with the same renderer and keep the last; jumping to a later frame shows a ghosted double of the subject. - Subject choice for
mask/matte/crop: the largest person, else the most confident instance. Instances overlapping it and no more than twice its size are merged in (a dress, a held instrument); large containers (a tunnel, a window frame) are not.classeslimits what the subject can be, not what merges into it. - Each vision layer tracks independently; track IDs from two layers are unrelated. Layers with the same
variantshare models, and layers showing the same pixels share one detect/segment pass.
Clip vision (comp.vision)
const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)
const person = dancer.person.place({ width: W, height: H, fit: "cover" });
person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed over ±2 frames
person.signals("nose").x; // a signal for .signal("x", …)
person.sample(48, x, y); // silhouette alpha + inward gradient
await dancer.ready(0); // analyse a frame (or ready(0, 120)) to read it at build time
- Each frame is analysed just before it draws, in every render path and the preview; nothing runs up front. Results persist, keyed by file and settings (
cacheDir: falseopts out), so later renders are instant. - Options:
startFrame/trimStartSec(where the clip plays; passtrimStartSecyourself when you give a path instead of the layer),fps(analysis rate),pose,mask,maskCell(silhouette grid, default 8 px),smoothing(frames, default 2),variant. A lost person's last pose is held for up to a second. depth: trueadds each frame's relative depth (Depth Anything V2 Small; 0 far … 1 near, per-frame range, not metric) at about 924×518 for 16:9. A depth clip tracks no pose or silhouette unlesspose/maskask. Read it withclip.depthMap(frame)({ data, width, height }), or draw it withwithVision({ mode: "depth" })and lift a 3D layer with it viawithRelief(framefields-compose). Depth frames persist one file each, keyed by the file only, so other options never invalidate them. About 0.4 s per frame on an M1 Pro GPU.matting: trueresolves the subject's edge to soft alpha with ViTMatte Small: hair, motion blur and see-through parts that a hard mask cuts off. It refines whatever the clip selects (the automatic subject, or thesam2/sam3tracks), so it never picks a subject itself. Each frame's mask becomes a trimap: certain inside and outside, plus an unknown band ofbandpx either side of the edge (default about 1.2% of the short side, 13 px at 1080p). One model pass per frame covers the band of every track. The window around the band is scaled toresolution(default 1024) on its long side, and each prompted track keeps its own alpha. Matte layers,signalsmasks and the clip's subject then read the soft alpha. Mattes persist beside the masks, keyed bybandandresolution, so turning matting on or off never re-runs SAM. Widenbandwhen hair reaches well past the mask. Only the band changes: matting does not fix a wrong selection, and an edge more thanbandpx off stays off.- A
withVision({ mode: "matte" })cutout (ormask/crop) of the same clip reuses the clip's subject for the frame being drawn: segmented once, aligned with the plate. Layers withclasses,keyBackground,matteSource: "selfie"or different model settings keep their own pass. With severalcomp.visionclips of one source, the matte links to the first one created.
Prompted tracking with SAM
Detection picks "the largest person". Use SAM when the subject is a specific person or object, especially in a crowd, or when you can only describe it.
SAM 2.1 Tiny (sam2) |
SAM 2.1 Small (sam2: { model: "small" }) |
SAM 3 (sam3) |
|
|---|---|---|---|
| Select by | points and boxes | points and boxes | points, boxes, or text |
| Objects per clip | 1 | 1 | 1 to maxObjects |
| Resolution | 512² | 1024² | 1008² |
| Cost per frame, M1 Pro | ~0.27 s | ~1.1 s | ~3.6 s image encoder (shared by every SAM 3 clip of the source) + ~0.7 s per tracked object |
| Download | ~83 MB | ~112 MB | ~1.75 GB (GPU) / ~1.2 GB (CPU) |
| Weights license | Apache-2.0 | Apache-2.0 | Meta SAM License |
Choosing (tell the user which and why):
- Tiny: one subject you can point at, large in frame (close-up, dancer, speaker). Fastest; masks match Small's on such shots (IoU 0.98).
- Small: one subject that is small, thin or detailed (ball, hand, hair, distant runner), or cuts and look-alikes that fool Tiny. ~4× Tiny's cost.
- SAM 3: the subject must be found from a description, or several must be tracked. Most expensive: trim to the stretch you need.
- Unsure: prototype with Tiny on a short stretch, switch to Small if edges, small parts or identity fail.
License
SAM code ships in framefields (Apache-2.0) and loads only when a clip uses it. SAM 2.1 weights are Apache-2.0. SAM 3 weights are under Meta's SAM License and need explicit acceptance before download: acceptSamLicense: true on the clip (or acceptLicense: true inside sam3), or FRAMEFIELDS_ACCEPT_SAM_LICENSE=1. Ask the user before accepting it on their behalf. Import Sam2Tracker / Sam3Tracker from framefields/sam when you drive SAM directly.
Prompts
const plate = Layer.video("assets/source.mp4", { width: 1920, height: 1080, fit: "contain" })
.withVision({ mode: "matte" });
comp.add(plate);
// SAM 3 + matting: hair and motion blur keep their soft edge in the cutout
comp.vision(layer, { sam3: { text: "woman with long hair" }, acceptSamLicense: true, matting: true });
// SAM 3: one person by description
const hero = comp.vision(plate, {
acceptSamLicense: true,
sam3: { category: "person", text: "man in white shirt" },
pose: true, // optional RTMO pass matched to the selection
});
// SAM 3: everyone else at once, tracks 1…N by score
const crowd = comp.vision("assets/source.mp4", {
acceptSamLicense: true,
sam3: { category: "person", text: "student in blazer", maxObjects: 5 },
});
// SAM 3: every match as ONE object — one mask, one memory, the cost of a single track
const students = Layer.video("assets/source.mp4", { width: 1920, height: 1080, fit: "contain" })
.withVision({ mode: "matte" }); // cuts out all of them as one entity
comp.add(students);
comp.vision(students, {
acceptSamLicense: true,
sam3: { category: "person", text: "student in black jacket", merge: true },
});
// SAM 2.1 Small: a small subject, a box plus a point, and a correction later
const ball = comp.vision(plate, {
sam2: {
model: "small",
category: "ball",
prompts: [
{ frame: 0, points: [
{ x: 0.40, y: 0.55, label: 2 }, // box top-left
{ x: 0.48, y: 0.66, label: 3 }, // box bottom-right
{ x: 0.44, y: 0.60, label: 1 }, // foreground point
] },
{ frame: 120, points: [{ x: 0.61, y: 0.52, label: 1 }] },
],
},
});
const target = hero.signals.objects.get(1); // request signals before rendering
Layer.box({
position: "absolute", x: 0, y: 0, width: 1, height: 1,
background: "transparent", borderWidth: 6, borderColor: "#00ffc3",
}).animate(
LayerAnimation.create()
.signal("x", target.bounds.screenX)
.signal("y", target.bounds.screenY)
.signal("width", target.bounds.screenWidth)
.signal("height", target.bounds.screenHeight)
.signal("opacity", target.active),
);
- Coordinates are normalized to the source image, before layout. Labels:
1foreground,0background,2/3box corners. - Prompt frames are increasing integers on the source analysis clock, before trimming. SAM propagates forward from the first prompt; earlier frames are empty and are not analysed. To track only a stretch of a long clip, trim the layer and prompt on the stretch's first source frame.
- Text prompts are concepts. Name what sets the target apart in that shot ("man in white shirt" fails where every student shows a white collar). Check the first frame of every prompt.
- Several objects (
maxObjects): one encoder pass per frame serves every track, and each track keeps its own memory. Where tracks overlap, the more confident one keeps the pixel.signals.objects.get(k)andsignals.maskscarry tracks1…N; the clip's matte and subject are all tracks merged. A later text prompt re-selects every track (IDs renumber); a points prompt withobject: kseeds or corrects trackkonly (the next unused ID adds one). - One entity for a whole group (
merge: true): every match (up tomaxObjects, default all) becomes one object with one memory, at the cost of one track. Use it for a group's mask or matte (cut out a crowd); usemaxObjectswithoutmergewhen each member needs its own track (~0.7 s per member per frame). - People who appear later (
redetect: N): text selects once, on the prompt frame.redetect: Nre-runs it every N frames and adds what no track covers (into the merged object, or as new IDs up tomaxObjects). Each pass costs about a prompt-frame selection (~50 s on an M1 Pro): pick N by how fast people enter (10–25), not 1. - Two text-prompted clips of one shot can claim the same person (a "student" track on the hero). Prefer prompts that exclude each other.
- Corrections and cuts. A prompt on a later frame resets that object's memory. Re-prompt after hard cuts or long occlusions; appearance tracking cannot guarantee re-identification.
- An explicit selection never falls back to another detected person. If its mask cannot align with a matte layer (a different aspect ratio), the render fails. Matte at the source aspect ratio and crop afterwards.
- SAM gives masks, not skeletons:
pose: true(opt-in for SAM clips) matches an RTMO pose to the selection, the person whose keypoints fall most inside its mask. AwithVision({ mode: "skeleton" })layer of the same clip draws that matched pose for the frame on screen; withoutpose: trueon a SAM clip it fails instead of drawing whoever is detected first. Default smoothing is 0, and an absent object stays empty. clip.release()frees the SAM sessions. Analysed frames stay readable.
Singular or plural: one selection per group
Read the request's number before writing any vision code.
| The user asks for | Means | Write |
|---|---|---|
| One thing: "the man in white", "the ball", "her" | one object | one sam3 text prompt (default maxObjects: 1), or sam2 points |
| A group of look-alikes as a whole: "the students", "all dancers", "everyone in black jackets", "the crowd", "the cars" | one entity | one comp.vision clip, one text prompt, merge: true |
| Each member separately: "box every student", "label each car", "count them" | N objects | one clip, one text prompt, maxObjects: N |
- Never select a group one member at a time. Several clips with the same or overlapping prompts re-detect the same people: duplicate masks, members claimed twice or missed, N× the cost. A group is one prompt on one clip.
- Grammatical number in the prompt doesn't matter;
mergeandmaxObjectsdecide. Name what members share and others lack ("in black jackets"); if someone wrong is included, describe the group more precisely instead of adding a clip to subtract them. - A group plus a different individual (the crowd and the main character) is two clips with prompts that exclude each other:
mergeon the group, a single object for the individual.
Cost and caching
- Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.
- On an M1 Pro: ~2 min setup (models, text selection), then a 20-frame stretch with 6 tracks takes ~5 min; a 15 s clip at 25 fps ~55 min. CUDA is much faster; the CPU encoder ~9× slower than WebGPU.
- Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.
- Offline or CI: download the weights on a connected machine once, then copy
~/.cache/framefields/models, or pointFRAMEFIELDS_MODELS_BASE_URLat a mirror.SAM_MODELS(framefields/sam) lists every file with its URL, size and SHA-256.
Recipes
Subject sandwich (text behind the subject)
comp.addSubjectSandwich({
source: "assets/dancer.mp4",
behind: [Layer.text("HEADLINE BEHIND", { fontSize: 120, fill: "#FF5A1F", fontWeight: 900 })],
feather: 4,
fit: "cover",
});
Smart re-framing (16:9 to 9:16)
const vision = comp.withVision({ enableDetection: true });
comp.addSmartFraming({
source: "assets/action.mp4",
target: vision.objects.primary,
targetAspect: 9 / 16,
damping: 0.15,
leadHeadroom: 0.1,
});
Subject outline glow
const vision = comp.withVision({ enableSegmentation: true });
comp.addSubjectOutline(vision.segmentation.subject, {
source: "assets/character.mp4",
color: "#FF5A1F",
width: 6,
blur: 12,
});
Tracked region blur (faces, plates)
const vision = comp.withVision({ classes: ["person"] });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });
A 3D character copies a performer
Layer.glb("robot.glb", { aim: person.retarget({ rig: "mixamo" }) }) on a placed person (comp.vision(src).person.place({ width, height, fit })) aims the character's spine, neck, arms and legs along the performer's keypoint segments every frame (2D: in the plane facing the camera). Presets mixamo, humanoid (Unity/VRM), rigify, or a { leftUpperArm: "…" } map; bones limits it (arms over a walk clip), mirror swaps sides, minVisibility lets unsure limbs fall back to the clip. Bind the root with person.signals("leftHip"). See framefields-compose → Characters.
Analyse before authoring
const report = await comp.analyzeVisionSequence("assets/street.mp4", {
tasks: ["detect", "pose"],
categories: ["person"],
});
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]
Troubleshooting
| Symptom | Fix |
|---|---|
| Cutout or box is on the wrong person | classes: ["person"]; for a specific person use SAM |
| SAM 3 text picks the wrong one | Name what is unique in that shot; or use a box prompt (points with labels 2/3) |
| Part of a fast-moving garment drops out | variant: "m", or keyBackground: true |
| Faint halo on dark backgrounds | Lower featherRadius, or raise maskThreshold (e.g. 0.6) |
| "Meta SAM License agreement required" (SAM 3) | acceptSamLicense: true or FRAMEFIELDS_ACCEPT_SAM_LICENSE=1, once the user accepts the license |
| "A prompted clip matte requires the source aspect ratio" | Matte at the source aspect, crop afterwards |
Log says Inference on the CPU on a GPU machine |
The log's reason says why; on Linux NVIDIA check the CUDA 12 / cuDNN 9 libraries, or force FRAMEFIELDS_VISION_PROVIDER to see the error |
| A preview of a clip outside the project shows nothing | Add its folder to startPreview(…, { root: [process.cwd(), dir] }) |
| Renders offline or in CI | Copy the model cache, or set FRAMEFIELDS_MODELS_BASE_URL |
| SAM 2.1 Tiny mask loses a small object, its edges, or the subject after a cut | sam2: { model: "small" } |