Skip to content

Code-first video & image, rendered natively on WebGPU.

23.6 KB
Raw Blame

name: framefields-vision description: On-device computer vision in framefields — object detection and tracking, instance segmentation, pose, person mattes and cutouts (RTMDet-Ins, RTMO, Selfie Segmenter), soft alpha matting of hair and motion blur (ViTMatte), and prompted video tracking of a specific object or several people with SAM 2.1 Tiny or Small (point/box prompts) and SAM 3 (text prompts such as "man in white shirt", multiple objects at once). Covers which API to use (withVision, comp.vision, analyzeVisionSequence), every pinned ONNX model with sizes and licenses, GPU/CUDA/CPU backends, model downloads and caching, performance, and troubleshooting. Use when a framefields composition needs to know where people or objects are, cut a subject out, follow someone through a clip, or select one person in a crowd.

framefields-vision

Vision runs ONNX models locally from framefields; nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked by size and SHA-256, and are cached in $FRAMEFIELDS_MODELS_DIR (default ~/.cache/framefields/models). Effects that consume vision are in framefields-effects; frame checks in framefields-render.

Which API

Need API Timing
Cut out, box, or draw a skeleton on a layer while rendering layer.withVision({ mode }) Reads the layer's previous rendered frame: results trail by one frame and exist only while rendering
Signals for the whole composition (boxes, masks, pose) comp.withVision({ … }) Same, on the composited frame
Know the motion at a frame (physics, layout, keyframes), or reuse results across renders comp.vision(clipOrLayer, options) Analyses the clip's own frames on demand, before they draw; persisted on disk
Follow one specific object or person, or select by description comp.vision(…, { sam2 }) / { sam3 } As above, with SAM's temporal memory
Find out what is in a clip before authoring comp.analyzeVisionSequence(path, …) One-shot report

Models

Core models (Apache-2.0, bundled with framefields)

Task Model t s (default) m Input
Detect + segment (one shared pass), COCO-80 RTMDet-Ins 24 MB 43 MB 116 MB 640²
Pose, 17 COCO keypoints RTMO 27 MB (416²) 40 MB 89 MB 640²
Person matte MediaPipe Selfie Segmenter square 0.5 MB (256²), landscape 0.5 MB (256×144) for wide frames
Relative depth (depth: true) Depth Anything V2 Small 99 MB, one size short side 518, frame aspect
Alpha matting (matting: true) ViTMatte Small (Distinctions-646) 104 MB, one size the subject's edge, long side ≤ 1024

About 585 MB for all of them. variant: "t" | "s" | "m" picks the size.

SAM 2.1 Tiny and Small (sam2, Apache-2.0)

Meta SAM 2.1 Hiera checkpoints (Diffusion Studio fp16 video exports; memory of the prompted frame, the 6 latest frames, 16 object pointers). Point and box prompts only. sam2: { model }:

model Input Image encoder Memory attention Total Per frame, M1 Pro WebGPU
"tiny" (default) 512² (32×32 features) 58 MB 13 MB ~83 MB ~0.27 s
"small" 1024², SAM 2's native size (64×64) 84 MB 16 MB ~112 MB ~1.1 s

SAM 3 (sam3, weights under Meta's SAM License)

Video tracking at 1008², plus text ("concept") selection. Only the image encoder the backend runs fastest is downloaded:

Graph Size Source
Image encoder, fp16 (WebGPU, CUDA) 935 MB onnx-community/sam3-tracker-ONNX
Image encoder, q4 (CPU) 369 MB same
Image encoder, q4f16 (opt-in) 296 MB same
Mask decoder, memory encoder, memory attention 18 + 6 + 35 MB jax-image-tools/sam3-video-onnx
Dense-prompt and pointer heads 67 MB RamRom/sam3-onnx
Text selection: image encoder (int4, fp16 attention), text encoder (int8), decoder (int8), tokenizer 293 + 357 + 27 + 4 MB danilobukvic/sam3-text-onnx

That is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.

fp16 is the fastest and most precise on WebGPU (one 1008² frame on an M1 Pro: fp16 3.65 s, q4f16 4.43 s, q4 6.63 s; 4-bit saves memory, not time). SAM3_ENCODER in framefields/sam maps backend to precision; sam3.precision: "fp16" | "q4f16" | "q4" overrides it.

Backends

Inference picks its backend once per process, before any model downloads:

Machine Backend
Mac (Apple Silicon) WebGPU through onnxruntime-web on the Metal device
Linux with an NVIDIA GPU CUDA through onnxruntime-node when the NVIDIA driver is loaded and the CUDA 12 / cuDNN 9 libraries are present, else WebGPU (Vulkan), else CPU
No usable GPU (CI, lavapipe, SwiftShader) CPU through onnxruntime-node
The preview (startPreview) WebGPU in the browser; models are served from the local cache by the preview server
Variable Effect
FRAMEFIELDS_VISION_PROVIDER auto (default; the order above), webgpu, cuda or cpu. A forced backend that fails throws instead of falling back.
FRAMEFIELDS_VISION_THREADS CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default.
FRAMEFIELDS_MODELS_DIR / FRAMEFIELDS_MODELS_BASE_URL Model cache directory, or a mirror to download from (offline and CI renders)
FRAMEFIELDS_VISION_CACHE_DIR Where comp.vision persists results (default ~/.cache/framefields/vision)

The renderer logs [vision] Inference on the GPU (WebGPU), or the backend it fell back to.

Detection, segmentation, pose, mattes

Option Model Gives you
enableDetection (default) RTMDet-Ins COCO-80 boxes, tracked over time → vision.objects
enableSegmentation RTMDet-Ins (same pass) Soft per-instance masks → vision.masks, vision.segmentation
enablePose RTMO 17 COCO keypoints per person → vision.poseLandmarks, track.pose
enableMatte Selfie Segmenter Fast person alpha → vision.segmentation.matte
// Whole composition: returns the reactive bundle
const vision = comp.withVision({ enableSegmentation: true, enablePose: true });

// One layer: renders through a node mode
Layer.video("assets/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });

Node modes: passthrough, mask, matte, crop, skeleton, boxes, tracking, depth. depth draws the frame's relative depth as gray (white = near); it always reads a comp.vision clip with depth: true, made for it when there is none, so it is never a frame behind. skeleton draws only the pose on transparency: the first person detected, or, for a layer of a comp.vision clip, the pose that clip matched to its selection. For mask / matte / crop, matteSource: "instance" (default; any COCO class, overlapping parts such as a dress merged into the subject) or "selfie" (people only, fastest). confidence (default 0.3) and classes: ["person", …] filter detections.

Measured per 1280–2048 px frame on CPU (onnxruntime-node); the GPU is faster:

Task t s m
detect / segment (one shared pass) 150–300 ms 200–340 ms 420–700 ms
pose ~50 ms ~120 ms ~280 ms
matte (Selfie Segmenter) ~20 ms ~20 ms ~20 ms
  • Start with "s". Use "m" for hero shots with fast motion blur or busy backgrounds; "t" mainly saves time on pose.
  • Use the selfie matte only for close framing. It misses distant figures and can report a "person" in close-ups with nobody in them.
  • Raise confidence (e.g. 0.5) on abstract or stylized footage. At 0.3 the model puts loose labels ("teddy bear", "donut") on smoke, eyes and planets.
  • Restrict classes when you only care about one thing. It also stops the subject switching to another object when the person leaves frame.

At render time:

  • One-frame delay. withVision reads the previous rendered frame, so the first frame has no results (a cutout renders transparent) and masks trail the plate by a frame. A matte of a clip the composition also reads with comp.vision has neither problem.
  • Render vision frames in order. renderVideo does. For stills, render at least two consecutive frames with the same renderer and keep the last; jumping to a later frame shows a ghosted double of the subject.
  • Subject choice for mask / matte / crop: the largest person, else the most confident instance. Instances overlapping it and no more than twice its size are merged in (a dress, a held instrument); large containers (a tunnel, a window frame) are not. classes limits what the subject can be, not what merges into it.
  • Each vision layer tracks independently; track IDs from two layers are unrelated. Layers with the same variant share models, and layers showing the same pixels share one detect/segment pass.

Clip vision (comp.vision)

const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 });   // pose + silhouette (person)
const person = dancer.person.place({ width: W, height: H, fit: "cover" });
person.point("rightWrist", 48);   // { x, y, visibility } in composition px, smoothed over ±2 frames
person.signals("nose").x;          // a signal for .signal("x", …)
person.sample(48, x, y);           // silhouette alpha + inward gradient
await dancer.ready(0);             // analyse a frame (or ready(0, 120)) to read it at build time
  • Each frame is analysed just before it draws, in every render path and the preview; nothing runs up front. Results persist, keyed by file and settings (cacheDir: false opts out), so later renders are instant.
  • Options: startFrame / trimStartSec (where the clip plays; pass trimStartSec yourself when you give a path instead of the layer), fps (analysis rate), pose, mask, maskCell (silhouette grid, default 8 px), smoothing (frames, default 2), variant. A lost person's last pose is held for up to a second.
  • depth: true adds each frame's relative depth (Depth Anything V2 Small; 0 far … 1 near, per-frame range, not metric) at about 924×518 for 16:9. A depth clip tracks no pose or silhouette unless pose / mask ask. Read it with clip.depthMap(frame) ({ data, width, height }), or draw it with withVision({ mode: "depth" }) and lift a 3D layer with it via withRelief (framefields-compose). Depth frames persist one file each, keyed by the file only, so other options never invalidate them. About 0.4 s per frame on an M1 Pro GPU.
  • matting: true resolves the subject's edge to soft alpha with ViTMatte Small: hair, motion blur and see-through parts that a hard mask cuts off. It refines whatever the clip selects (the automatic subject, or the sam2 / sam3 tracks), so it never picks a subject itself. Each frame's mask becomes a trimap: certain inside and outside, plus an unknown band of band px either side of the edge (default about 1.2% of the short side, 13 px at 1080p). One model pass per frame covers the band of every track. The window around the band is scaled to resolution (default 1024) on its long side, and each prompted track keeps its own alpha. Matte layers, signals masks and the clip's subject then read the soft alpha. Mattes persist beside the masks, keyed by band and resolution, so turning matting on or off never re-runs SAM. Widen band when hair reaches well past the mask. Only the band changes: matting does not fix a wrong selection, and an edge more than band px off stays off.
  • A withVision({ mode: "matte" }) cutout (or mask / crop) of the same clip reuses the clip's subject for the frame being drawn: segmented once, aligned with the plate. Layers with classes, keyBackground, matteSource: "selfie" or different model settings keep their own pass. With several comp.vision clips of one source, the matte links to the first one created.

Prompted tracking with SAM

Detection picks "the largest person". Use SAM when the subject is a specific person or object, especially in a crowd, or when you can only describe it.

SAM 2.1 Tiny (sam2) SAM 2.1 Small (sam2: { model: "small" }) SAM 3 (sam3)
Select by points and boxes points and boxes points, boxes, or text
Objects per clip 1 1 1 to maxObjects
Resolution 512² 1024² 1008²
Cost per frame, M1 Pro ~0.27 s ~1.1 s ~3.6 s image encoder (shared by every SAM 3 clip of the source) + ~0.7 s per tracked object
Download ~83 MB ~112 MB ~1.75 GB (GPU) / ~1.2 GB (CPU)
Weights license Apache-2.0 Apache-2.0 Meta SAM License

Choosing (tell the user which and why):

  • Tiny: one subject you can point at, large in frame (close-up, dancer, speaker). Fastest; masks match Small's on such shots (IoU 0.98).
  • Small: one subject that is small, thin or detailed (ball, hand, hair, distant runner), or cuts and look-alikes that fool Tiny. ~4× Tiny's cost.
  • SAM 3: the subject must be found from a description, or several must be tracked. Most expensive: trim to the stretch you need.
  • Unsure: prototype with Tiny on a short stretch, switch to Small if edges, small parts or identity fail.

License

SAM code ships in framefields (Apache-2.0) and loads only when a clip uses it. SAM 2.1 weights are Apache-2.0. SAM 3 weights are under Meta's SAM License and need explicit acceptance before download: acceptSamLicense: true on the clip (or acceptLicense: true inside sam3), or FRAMEFIELDS_ACCEPT_SAM_LICENSE=1. Ask the user before accepting it on their behalf. Import Sam2Tracker / Sam3Tracker from framefields/sam when you drive SAM directly.

Prompts

const plate = Layer.video("assets/source.mp4", { width: 1920, height: 1080, fit: "contain" })
	.withVision({ mode: "matte" });
comp.add(plate);

// SAM 3 + matting: hair and motion blur keep their soft edge in the cutout
comp.vision(layer, { sam3: { text: "woman with long hair" }, acceptSamLicense: true, matting: true });

// SAM 3: one person by description
const hero = comp.vision(plate, {
	acceptSamLicense: true,
	sam3: { category: "person", text: "man in white shirt" },
	pose: true, // optional RTMO pass matched to the selection
});

// SAM 3: everyone else at once, tracks 1…N by score
const crowd = comp.vision("assets/source.mp4", {
	acceptSamLicense: true,
	sam3: { category: "person", text: "student in blazer", maxObjects: 5 },
});

// SAM 3: every match as ONE object — one mask, one memory, the cost of a single track
const students = Layer.video("assets/source.mp4", { width: 1920, height: 1080, fit: "contain" })
	.withVision({ mode: "matte" }); // cuts out all of them as one entity
comp.add(students);
comp.vision(students, {
	acceptSamLicense: true,
	sam3: { category: "person", text: "student in black jacket", merge: true },
});

// SAM 2.1 Small: a small subject, a box plus a point, and a correction later
const ball = comp.vision(plate, {
	sam2: {
		model: "small",
		category: "ball",
		prompts: [
			{ frame: 0, points: [
				{ x: 0.40, y: 0.55, label: 2 }, // box top-left
				{ x: 0.48, y: 0.66, label: 3 }, // box bottom-right
				{ x: 0.44, y: 0.60, label: 1 }, // foreground point
			] },
			{ frame: 120, points: [{ x: 0.61, y: 0.52, label: 1 }] },
		],
	},
});

const target = hero.signals.objects.get(1); // request signals before rendering
Layer.box({
	position: "absolute", x: 0, y: 0, width: 1, height: 1,
	background: "transparent", borderWidth: 6, borderColor: "#00ffc3",
}).animate(
	LayerAnimation.create()
		.signal("x", target.bounds.screenX)
		.signal("y", target.bounds.screenY)
		.signal("width", target.bounds.screenWidth)
		.signal("height", target.bounds.screenHeight)
		.signal("opacity", target.active),
);
  • Coordinates are normalized to the source image, before layout. Labels: 1 foreground, 0 background, 2 / 3 box corners.
  • Prompt frames are increasing integers on the source analysis clock, before trimming. SAM propagates forward from the first prompt; earlier frames are empty and are not analysed. To track only a stretch of a long clip, trim the layer and prompt on the stretch's first source frame.
  • Text prompts are concepts. Name what sets the target apart in that shot ("man in white shirt" fails where every student shows a white collar). Check the first frame of every prompt.
  • Several objects (maxObjects): one encoder pass per frame serves every track, and each track keeps its own memory. Where tracks overlap, the more confident one keeps the pixel. signals.objects.get(k) and signals.masks carry tracks 1…N; the clip's matte and subject are all tracks merged. A later text prompt re-selects every track (IDs renumber); a points prompt with object: k seeds or corrects track k only (the next unused ID adds one).
  • One entity for a whole group (merge: true): every match (up to maxObjects, default all) becomes one object with one memory, at the cost of one track. Use it for a group's mask or matte (cut out a crowd); use maxObjects without merge when each member needs its own track (~0.7 s per member per frame).
  • People who appear later (redetect: N): text selects once, on the prompt frame. redetect: N re-runs it every N frames and adds what no track covers (into the merged object, or as new IDs up to maxObjects). Each pass costs about a prompt-frame selection (~50 s on an M1 Pro): pick N by how fast people enter (10–25), not 1.
  • Two text-prompted clips of one shot can claim the same person (a "student" track on the hero). Prefer prompts that exclude each other.
  • Corrections and cuts. A prompt on a later frame resets that object's memory. Re-prompt after hard cuts or long occlusions; appearance tracking cannot guarantee re-identification.
  • An explicit selection never falls back to another detected person. If its mask cannot align with a matte layer (a different aspect ratio), the render fails. Matte at the source aspect ratio and crop afterwards.
  • SAM gives masks, not skeletons: pose: true (opt-in for SAM clips) matches an RTMO pose to the selection, the person whose keypoints fall most inside its mask. A withVision({ mode: "skeleton" }) layer of the same clip draws that matched pose for the frame on screen; without pose: true on a SAM clip it fails instead of drawing whoever is detected first. Default smoothing is 0, and an absent object stays empty.
  • clip.release() frees the SAM sessions. Analysed frames stay readable.

Singular or plural: one selection per group

Read the request's number before writing any vision code.

The user asks for Means Write
One thing: "the man in white", "the ball", "her" one object one sam3 text prompt (default maxObjects: 1), or sam2 points
A group of look-alikes as a whole: "the students", "all dancers", "everyone in black jackets", "the crowd", "the cars" one entity one comp.vision clip, one text prompt, merge: true
Each member separately: "box every student", "label each car", "count them" N objects one clip, one text prompt, maxObjects: N
  • Never select a group one member at a time. Several clips with the same or overlapping prompts re-detect the same people: duplicate masks, members claimed twice or missed, N× the cost. A group is one prompt on one clip.
  • Grammatical number in the prompt doesn't matter; merge and maxObjects decide. Name what members share and others lack ("in black jackets"); if someone wrong is included, describe the group more precisely instead of adding a clip to subtract them.
  • A group plus a different individual (the crowd and the main character) is two clips with prompts that exclude each other: merge on the group, a single object for the individual.

Cost and caching

  • Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.
  • On an M1 Pro: ~2 min setup (models, text selection), then a 20-frame stretch with 6 tracks takes ~5 min; a 15 s clip at 25 fps ~55 min. CUDA is much faster; the CPU encoder ~9× slower than WebGPU.
  • Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.
  • Offline or CI: download the weights on a connected machine once, then copy ~/.cache/framefields/models, or point FRAMEFIELDS_MODELS_BASE_URL at a mirror. SAM_MODELS (framefields/sam) lists every file with its URL, size and SHA-256.

Recipes

Subject sandwich (text behind the subject)

comp.addSubjectSandwich({
	source: "assets/dancer.mp4",
	behind: [Layer.text("HEADLINE BEHIND", { fontSize: 120, fill: "#FF5A1F", fontWeight: 900 })],
	feather: 4,
	fit: "cover",
});

Smart re-framing (16:9 to 9:16)

const vision = comp.withVision({ enableDetection: true });
comp.addSmartFraming({
	source: "assets/action.mp4",
	target: vision.objects.primary,
	targetAspect: 9 / 16,
	damping: 0.15,
	leadHeadroom: 0.1,
});

Subject outline glow

const vision = comp.withVision({ enableSegmentation: true });
comp.addSubjectOutline(vision.segmentation.subject, {
	source: "assets/character.mp4",
	color: "#FF5A1F",
	width: 6,
	blur: 12,
});

Tracked region blur (faces, plates)

const vision = comp.withVision({ classes: ["person"] });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });

A 3D character copies a performer

Layer.glb("robot.glb", { aim: person.retarget({ rig: "mixamo" }) }) on a placed person (comp.vision(src).person.place({ width, height, fit })) aims the character's spine, neck, arms and legs along the performer's keypoint segments every frame (2D: in the plane facing the camera). Presets mixamo, humanoid (Unity/VRM), rigify, or a { leftUpperArm: "…" } map; bones limits it (arms over a walk clip), mirror swaps sides, minVisibility lets unsure limbs fall back to the clip. Bind the root with person.signals("leftHip"). See framefields-compose → Characters.

Analyse before authoring

const report = await comp.analyzeVisionSequence("assets/street.mp4", {
	tasks: ["detect", "pose"],
	categories: ["person"],
});
report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]

Troubleshooting

Symptom Fix
Cutout or box is on the wrong person classes: ["person"]; for a specific person use SAM
SAM 3 text picks the wrong one Name what is unique in that shot; or use a box prompt (points with labels 2/3)
Part of a fast-moving garment drops out variant: "m", or keyBackground: true
Faint halo on dark backgrounds Lower featherRadius, or raise maskThreshold (e.g. 0.6)
"Meta SAM License agreement required" (SAM 3) acceptSamLicense: true or FRAMEFIELDS_ACCEPT_SAM_LICENSE=1, once the user accepts the license
"A prompted clip matte requires the source aspect ratio" Matte at the source aspect, crop afterwards
Log says Inference on the CPU on a GPU machine The log's reason says why; on Linux NVIDIA check the CUDA 12 / cuDNN 9 libraries, or force FRAMEFIELDS_VISION_PROVIDER to see the error
A preview of a clip outside the project shows nothing Add its folder to startPreview(…, { root: [process.cwd(), dir] })
Renders offline or in CI Copy the model cache, or set FRAMEFIELDS_MODELS_BASE_URL
SAM 2.1 Tiny mask loses a small object, its edges, or the subject after a cut sam2: { model: "small" }