Code-first video & image, rendered natively on WebGPU.
@framefields/vision-sam
2.14.0
2.13.1
2.13.0
2.12.0
2.11.0
2.9.3
2.9.2
Minor Changes
- c785aa1:
sam3: { redetect: N }runs the last text prompt again every N frames and adds the instances no tracked object covers yet: people who walk in, step out from behind someone, or were lost. Withmergethey join the one object (what it already tracks stays at full resolution); otherwise they become new track IDs up tomaxObjects. The text features are encoded once per prompt.
2.9.0
Minor Changes
- dac3694:
@framefields/vision-samis now an optional peer offramefields, loaded only when a clip usessam2orsam3, so the SDK no longer installs or imports SAM-licensed code by default. Install it next toframefieldsto use prompted tracking, and importSam2Tracker/Sam3Trackerfrom@framefields/vision-sam. Mask-derived object geometry moved to@framefields/visionasmaskObject(formerlysam2Object). - 7af1f91: Add prompted SAM 2.1 Tiny and SAM 3 video segmentation and semantic text selection to clip vision, decoupled into a dedicated
@framefields/vision-sampackage under the Meta SAM License. Includes temporal memory, point/box corrections, semantic text selection, cached frame seeking, and time-remapped matte reuse. Selected-object masks, geometry, motion, and optional matched pose are exposed through standard vision signals. Model assets download dynamically at runtime to local cache with SHA-256 verification upon explicit license agreement. - 645dd13: SAM 2.1 (Apache-2.0 weights) no longer asks for Meta's SAM License; only SAM 3 does (
acceptSamLicense/sam3.acceptLicense/FRAMEFIELDS_ACCEPT_SAM_LICENSE=1).sam2.acceptLicenseandSam2Tracker'sacceptLicenseare removed.@framefields/vision-samis Apache-2.0 like every framefields package (SAM 3's weights stay under Meta's SAM License, accepted at runtime); its sources no longer carry a Meta copyright header, and the model is named SAM 3 throughout. - 9a82bdb: SAM 2.1 Small:
sam2: { model: "small" }tracks at SAM 2's native 1024² from Diffusion Studio's pinned fp16 export (~112 MB, ~1.1 s/frame on an M1 Pro against Tiny's ~0.27 s), for small, thin or detailed subjects and identity after cuts.Sam2Trackerderives every size from the model's constants;sam2Revision(model)replacesSAM2_TINY_REVISION. The framefields-vision skill says when to choose Tiny, Small or SAM 3. - 72f6aaa: SAM 3 downloads the image encoder its backend runs fastest: fp16 on WebGPU and CUDA, q4 on the CPU (
SAM3_ENCODER, override withsam3.precision). Providers report their backend before any download (SessionProvider.backend(),sessionCompute); the auto provider decides with a one-node probe graph and, on Linux with the NVIDIA driver, tries CUDA first. All SAM 3 trackers in a process share one encoder session and reuse its features per frame. Text-seeded objects now output and remember their seed mask on the prompt frame, as SAM does, instead of a speckled re-decode. - aec869b:
sam3: { text, merge: true }selects every instance the text matches as one object: one mask and one memory, so a whole crowd costs the same as a single track.maxObjectscaps the instances merged (default: all). - f1623c0: SAM 3 tracks several objects at once:
sam3.maxObjectslets a text prompt select up to N instances (tracks 1…N by score) through one encoder pass per frame, and points prompts takeobjectto seed or correct one track.comp.visionpublishes every track insignals.objectsandsignals.masks; the clip matte is their union. Fixes concurrentcomp.visionclips of one source dropping decoded frames (each clip now owns its decoder) and concurrent ORT WebGPU session creation falling back to the CPU (creation is serialized; SAM trackers share the process provider).