Code-first video & image, rendered natively on WebGPU.
feat(vision): add prompted SAM 2.1 Tiny video tracking
okanasl7af1f91056cc
diff --git a/.changeset/prompted-sam2-vision.md b/.changeset/prompted-sam2-vision.mdnew file mode 100644index 00000000..c4b819f8--- /dev/null+++ b/.changeset/prompted-sam2-vision.md@@ -0,0 +1,8 @@+---+"@framefields/vision": minor+"framefields": minor+"@framefields/node-vision": patch+"@framefields/webgpu-renderers": patch+---++Add prompted SAM 2.1 Tiny video segmentation to clip vision, including temporal memory, point/box corrections, cached frame seeking and time-remapped matte reuse. Selected-object masks, geometry, motion and optional matched pose are available through the existing vision signals. Model assets are lazily downloaded and verified, and ONNX session providers support named float and integer multi-input graphs.diff --git a/AGENTS.md b/AGENTS.mdindex 68241e35..6b5131d6 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -18,6 +18,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).+- `packages/vision/src/sam2`: prompted SAM 2.1 Tiny video graph pipeline; `comp.vision(..., { sam2 })` shares clip mattes and exposes object/mask/pose signals. Model files are pinned in the vision registry.- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).diff --git a/README.md b/README.mdindex 3b2ff6b2..141fb6b4 100644--- a/README.md+++ b/README.md@@ -192,6 +192,25 @@ Installed components import only `framefields` and the shared toolkit in `framef---+## Select and track a subject++Use SAM 2.1 Tiny to select a specific object with a point or box and track it through video. Its masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts.++```ts+const subject = Layer.video("assets/source.mp4", {+ width: 1920, height: 1080, fit: "contain",+}).withVision({ mode: "matte" });+comp.add(subject);+const tracked = comp.vision(subject, {+ sam2: { prompts: [{ frame: 0, points: [{ x: 0.5, y: 0.5, label: 1 }] }] },+});+const target = tracked.signals.objects.get(1);+```++Prompts use normalized source coordinates. Model files download lazily, inference runs locally, and cached source-frame results support seeking and time remapping. See the [vision guide](https://framefields.dev/docs/guides/vision/#select-a-specific-object-with-sam-21-tiny) for corrections, signal tracks, model/runtime requirements and mask alignment limits.++---+## AI GatewayFramefields provides built-in access to the [**AI Gateway**](https://framefields.dev/models), allowing agents and developers to generate cinematic video, images, music, and voiceovers directly from code using hundreds of hosted models. Explore the full model catalog in the [**AI Gateway**](https://framefields.dev/models) or read the integration guide in the [**documentation**](https://framefields.dev/docs/guides/ai/).@@ -209,7 +228,7 @@ Everything else—including full API references, interactive tutorials, and guid- **[WebGPU 2D VFX](https://framefields.dev/docs)** — 50+ shaders: 3D LUTs, Curves, Film Grain, Glassmorphism, and Blur- **[Unified 3D Scenes](https://framefields.dev/docs)** — LookAt cameras, PBR materials, studio lighting, shadows, and glTF/GLB models- **[Audio & Procedural SFX](https://framefields.dev/docs)** — BeatGrids, procedural whooshes/impacts, and audio-reactive signals-- **[On-Device Vision](https://framefields.dev/docs)** — Apache-2.0 ONNX models for object tracking, pose estimation, and subject cutouts+- **[On-Device Vision](https://framefields.dev/docs/guides/vision/)** — Apache-2.0 ONNX models for detection, pose, cutouts, and prompted SAM 2.1 video tracking- **[Live Browser Preview](https://framefields.dev/docs)** — Localhost WebGPU player with scrubbing, waveform timeline, and frame-stepping---diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.mdindex 171756d4..1da235f7 100644--- a/nodes/node-compositor/SKILL.md+++ b/nodes/node-compositor/SKILL.md@@ -28,6 +28,8 @@ layout nodes (`flex` / `block` / `box` / `text` / `media`) with HTML-like auto-l(`dir`, `gap`, `padding`, `justify`, `align`, `wrap`) and per-node keyframe animations.Rendering is deterministic: `(document, frame) → pixels`, identical in preview and final render.+When a video layer uses a cached clip-vision matte, the source frame's mask is authoritative: a known absent subject renders empty rather than holding an earlier silhouette. Prompted SAM clip mattes require matching source aspect ratio; apply layout cropping after source-space matting rather than silently detecting another subject.+## When to Use- **Title cards / hero layouts:** Flex stacks with title + subtitle text and box chips (see example below).- **Overlays / Watermarks:** Absolute-positioned text or media over video/image backgrounds.diff --git a/nodes/node-vision/src/renderers/webgpu-renderer.ts b/nodes/node-vision/src/renderers/webgpu-renderer.tsindex d865ad7c..055bc4fa 100644--- a/nodes/node-vision/src/renderers/webgpu-renderer.ts+++ b/nodes/node-vision/src/renderers/webgpu-renderer.ts@@ -748,7 +748,10 @@ export const VisionWebGPURenderer: WebGPUNodeRenderer = async (args) => {let subject: InstanceMask | undefined;if (fromClip) {- subject = selectSubject(clipSubject ?? undefined, nodeKeyStr, frameIdx);+ subject = clipSubject ?? undefined;+ if (subject)+ lastSubjectByNode.set(nodeKeyStr, { mask: subject, frame: frameIdx });+ else lastSubjectByNode.delete(nodeKeyStr);} else if (cached) {frameObjects = cached.objects;if (wantsDetection) {@@ -1077,7 +1080,12 @@ function clipSubjectFor(const subject = source?.subject(frame, fps);if (!subject) return subject;const aspect = subject.width / subject.height / (width / height);- return Math.abs(aspect - 1) < 0.01 ? subject : undefined;+ if (Math.abs(aspect - 1) < 0.01) return subject;+ if (source?.exact)+ throw new Error(+ "A prompted clip matte requires the source aspect ratio. Apply layout cropping after the source matte.",+ );+ return undefined;}/** Adapts a Selfie Segmenter matte to the instance-mask shape the composite path takes. */diff --git a/packages/framefields/src/clip-vision-matte.test.ts b/packages/framefields/src/clip-vision-matte.test.tsindex 1f7eb9a5..073bee61 100644--- a/packages/framefields/src/clip-vision-matte.test.ts+++ b/packages/framefields/src/clip-vision-matte.test.ts@@ -68,6 +68,58 @@ const cutout = (clip: string, vision: Record<string, unknown> = {}) =>});describe("matte reuses comp.vision", () => {+ it("fails clearly when an explicit prompted mask cannot align with the layer aspect", async () => {+ const file = await greenClip();+ const c = stage();+ const layer = Layer.video(file, {+ width: SIZE,+ height: SIZE / 2,+ fit: "cover",+ muted: true,+ }).withVision({ mode: "matte" });+ c.add(layer);+ const clip = c.vision(layer, {+ cacheDir: false,+ sam2: {+ prompts: [{ frame: 0, points: [{ x: 0.25, y: 0.5, label: 1 }] }],+ },+ });+ seedClipVision(+ clip,+ { width: SIZE, height: SIZE },+ Array.from({ length: 40 }, () => ({+ kp: null,+ mask: null,+ subject: leftHalf(),+ })),+ );+ await expect(c.renderFrame({ frame: 0 })).rejects.toThrow(/aspect ratio/);+ });++ it("renders an authoritative absent subject as empty instead of holding an earlier mask", async () => {+ const file = await greenClip();+ const c = stage("#0000ff");+ const layer = cutout(file);+ c.add(layer);+ const clip = c.vision(layer, { cacheDir: false, smoothing: 0 });+ seedClipVision(+ clip,+ { width: SIZE, height: SIZE },+ Array.from({ length: 40 }, (_, frame) => ({+ kp: null,+ mask: null,+ subject: frame === 0 ? leftHalf() : null,+ })),+ );+ const renderer = new HeadlessMediaRenderer();+ await c.renderFrame({ frame: 0, renderer });+ const image = await decodePngPixels(+ await c.renderFrame({ frame: 1, renderer }),+ );+ expect(getPixel(image, 50, 100).b).toBeGreaterThan(240);+ expect(getPixel(image, 50, 100).g).toBeLessThan(15);+ });+it("follows a clip's subject while its media plays in reverse", async () => {const file = await greenClip();const c = stage("#0000ff");diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.tsindex 1f42dccd..ddf7add4 100644--- a/packages/framefields/src/clip-vision.ts+++ b/packages/framefields/src/clip-vision.ts@@ -10,7 +10,14 @@ import {COCO17_KEYPOINT_NAMES,type InstanceMask,mergeSubjectMask,+ SAM2_TINY_REVISION,+ type Sam2FrameResult,+ type Sam2Options,+ Sam2Tracker,+ sam2Object,+ VisionBundle,VisionRunner,+ validateSam2Prompts,} from "@framefields/vision";import { mediaDecoderCache } from "@framefields/webgpu-renderers";import type { VisionProps } from "./effects/index.js";@@ -18,6 +25,8 @@ import type { VisionProps } from "./effects/index.js";export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];export interface ClipVisionOptions {+ /** Prompted SAM 2.1 Tiny video segmentation. Pose is opt-in in this mode. */+ sam2?: Sam2Options;/** Composition frame the clip's first frame plays on (default 0). */startFrame?: number;/** Seconds trimmed off the clip's start, as on its media layer (default 0). */@@ -58,6 +67,7 @@ export interface PersonPlacement {}interface FrameResult {+ score?: number;/** 17 × [x, y, visibility], source px; null when no person was found. */kp: Float32Array | null;/** cellW × cellH, 0–255; null when no person was found. */@@ -115,6 +125,12 @@ export class ClipVision implements ClipSubjectSource {private readonly cell: number;private queue: Promise<void> = Promise.resolve();private runner?: VisionRunner;+ private sam2Tracker?: Sam2Tracker;+ private sam2Frame = -1;+ private signalBundle?: VisionBundle;+ private publishedSignals = new Set<number>();+ private signalMaskFrame = -1;+ private signalMask?: InstanceMask;private cacheLoaded?: Promise<void>;private unsaved = 0;private unsavedSubjects = 0;@@ -129,13 +145,72 @@ export class ClipVision implements ClipSubjectSource {/** @internal Use `comp.vision(source, options)`. */constructor(source: string, fps: number, options: ClipVisionOptions = {}) {this.source = source;- this.options = options;+ this.options = options.sam2+ ? {+ ...options,+ sam2: {+ ...options.sam2,+ prompts: options.sam2.prompts.map((prompt) => ({+ ...prompt,+ points: prompt.points.map((point) => ({ ...point })),+ })),+ },+ }+ : options;this.compositionFps = fps;this.fps = options.fps ?? fps;this.startFrame = Math.round(options.startFrame ?? 0);this.trimStartSec = options.trimStartSec ?? 0;- this.smoothing = Math.max(0, Math.round(options.smoothing ?? 2));+ this.smoothing = Math.max(+ 0,+ Math.round(options.smoothing ?? (options.sam2 ? 0 : 2)),+ );this.cell = Math.max(1, Math.round(options.maskCell ?? 8));+ if (options.sam2) {+ validateSam2Prompts(options.sam2.prompts);+ this.wantSubjects = true;+ }+ }++ /** @internal Explicit selections must never fall back to another detected subject. */+ get exact(): boolean {+ return this.options.sam2 !== undefined;+ }++ /** Object, mask and optional pose signals on the composition clock, in source geometry. */+ get signals(): VisionBundle {+ this.keepSubjects();+ this.signalBundle ??= new VisionBundle({+ fps: this.compositionFps,+ totalFrames: Number.MAX_SAFE_INTEGER,+ maskSource: (frame) => {+ const mask = this.maskAt(frame);+ return mask && mask.area > 0 ? [mask] : [];+ },+ matteSource: (frame) => {+ const mask = this.maskAt(frame);+ return mask+ ? {+ mask: mask.mask,+ width: mask.width,+ height: mask.height,+ coverage: mask.coverage,+ }+ : undefined;+ },+ });+ return this.signalBundle;+ }++ /** Free inference sessions. Analysed frames remain readable; later frames replay memory. */+ release(): void {+ this.sam2Tracker?.release();+ this.sam2Tracker = undefined;+ this.sam2Frame = -1;+ this.runner?.close();+ this.runner = undefined;+ this.signalMaskFrame = -1;+ this.publishedSignals.clear();}/** The tracked person in source pixels; `.place()` maps it into the composition. */@@ -151,14 +226,24 @@ export class ClipVision implements ClipSubjectSource {*/ready(frame: number, toFrame: number = frame): Promise<void> {const [from, to] = this.window(frame, toFrame);- this.queue = this.queue.then(() => this.compute(from, to));- return this.queue;+ const run = this.queue.then(async () => {+ try {+ await this.compute(from, to);+ } catch (error) {+ this.release();+ throw error;+ }+ if (this.signalBundle)+ this.publishSignals(Math.min(frame, toFrame), Math.max(frame, toFrame));+ });+ this.queue = run.catch(() => undefined);+ return run;}/** True when composition frame `frame` can be read synchronously. */isReady(frame: number): boolean {const [from, to] = this.window(frame, frame);- for (let c = from; c <= to; c++) if (!this.results.has(c)) return false;+ for (let c = from; c <= to; c++) if (!this.analysed(c)) return false;return true;}@@ -178,11 +263,16 @@ export class ClipVision implements ClipSubjectSource {*/subject(frame: number, fps: number): InstanceMask | null | undefined {const c = (frame * this.fps) / fps;- if (!this.options.timeRemap && Math.abs(c - Math.round(c)) > 1e-6)+ if (+ !this.options.sam2 &&+ !this.options.timeRemap &&+ Math.abs(c - Math.round(c)) > 1e-6+ )return undefined;- const sourceFrame = this.options.timeRemap- ? Math.floor(c + 1e-6)- : Math.round(c);+ const sourceFrame =+ this.options.timeRemap || this.options.sam2+ ? Math.floor(c + 1e-6)+ : Math.round(c);const s = this.subjects.get(sourceFrame);if (s === undefined) return s;if (s === null || !this.sourceWidth) return null;@@ -258,6 +348,10 @@ export class ClipVision implements ClipSubjectSource {for (let c = from; c <= to; c++) if (!this.analysed(c)) missing = true;if (!missing) return;+ if (this.options.sam2) {+ await this.computeSam2(to);+ return;+ }const wantPose = this.options.pose ?? true;const wantMask = this.options.mask ?? true;this.runner ??= VisionRunner.create({@@ -339,6 +433,201 @@ export class ClipVision implements ClipSubjectSource {if (this.unsavedSubjects > 0) await this.saveSubjects();}+ private async computeSam2(to: number): Promise<void> {+ const settings = this.options.sam2;+ if (!settings) return;+ const first = settings.prompts[0]?.frame ?? 0;+ for (let frame = 0; frame < Math.min(first, to + 1); frame++) {+ this.results.set(frame, { kp: null, mask: null, score: 0 });+ this.subjects.set(frame, null);+ }+ if (to < first) return;+ const decoder = mediaDecoderCache.getVideo(+ this.source,+ true,+ "clip-vision",+ );+ if (!this.sam2Tracker) {+ const frameCount = Math.max(+ 1,+ Math.ceil((await decoder.durationSec()) * this.fps - 1e-6),+ );+ if (settings.prompts.some((prompt) => prompt.frame >= frameCount)) {+ throw new Error("SAM 2 prompt frames must be inside the source video.");+ }+ this.sam2Tracker = await Sam2Tracker.create({+ modelsDir: this.options.modelsDir,+ baseUrl: this.options.baseUrl,+ frameCount,+ });+ this.sam2Frame = first - 1;+ }+ for (let frame = this.sam2Frame + 1; frame <= to; frame++) {+ const image = await decodedPixels(+ await decoder.getFrame(frame / this.fps),+ );+ if (!image) throw new Error(`Cannot decode SAM 2 source frame ${frame}.`);+ if (!this.sourceWidth) this.setSize(image.width, image.height);+ const prompt = settings.prompts.find((item) => item.frame === frame);+ const result = await this.sam2Tracker.process(+ image,+ frame,+ prompt?.points,+ );+ const kp = this.options.pose+ ? await this.matchedPose(image, result.mask)+ : null;+ this.storeSam2Frame(frame, result, kp);+ if (this.unsaved >= 24) {+ await this.saveCache();+ await this.saveSubjects();+ }+ }+ await this.saveCache();+ await this.saveSubjects();+ }++ private storeSam2Frame(+ frame: number,+ result: Sam2FrameResult,+ kp: Float32Array | null,+ ): void {+ const grid = new Uint8Array(this.cellW * this.cellH);+ downsampleMean(+ result.mask,+ result.width,+ result.height,+ this.cell,+ grid,+ this.cellW,+ this.cellH,+ );+ let area = 0;+ for (const alpha of result.mask) area += alpha / 255;+ this.results.set(frame, {+ kp,+ mask: grid,+ score: 1 / (1 + Math.exp(-result.score)),+ });+ this.subjects.set(+ frame,+ area > 0+ ? {+ category: this.options.sam2?.category ?? "object",+ area,+ ...encodeRuns(result.mask),+ }+ : null,+ );+ this.blurred.delete(frame);+ this.sam2Frame = frame;+ this.unsaved++;+ this.unsavedSubjects++;+ }++ private async matchedPose(+ image: {+ data: Uint8Array | Uint8ClampedArray;+ width: number;+ height: number;+ },+ mask: Uint8Array,+ ): Promise<Float32Array | null> {+ this.runner ??= VisionRunner.create({+ modelsDir: this.options.modelsDir,+ baseUrl: this.options.baseUrl,+ variant: this.options.variant,+ });+ const { people } = await this.runner.pose(image);+ const ranked = people+ .map((person) => ({+ person,+ overlap: person.keypoints.reduce((sum, point) => {+ const x = Math.max(0, Math.min(image.width - 1, Math.round(point.x)));+ const y = Math.max(+ 0,+ Math.min(image.height - 1, Math.round(point.y)),+ );+ return (+ sum + ((mask[y * image.width + x] ?? 0) / 255) * point.visibility+ );+ }, 0),+ }))+ .sort((a, b) => b.overlap - a.overlap);+ const match = ranked[0];+ if (!match || match.overlap < 2) return null;+ return Float32Array.from(+ match.person.keypoints.flatMap((point) => [+ point.x,+ point.y,+ point.visibility,+ ]),+ );+ }++ private maskAt(frame: number): InstanceMask | undefined {+ const c = Math.max(0, Math.floor(this.clipFrame(frame)));+ if (this.signalMaskFrame === c) return this.signalMask;+ if (!this.results.has(c) || !this.sourceWidth) return undefined;+ const stored = this.subjects.get(c);+ const data = stored+ ? decodeRuns(stored, this.sourceWidth * this.sourceHeight)+ : new Uint8Array(this.sourceWidth * this.sourceHeight);+ this.signalMaskFrame = c;+ this.signalMask = {+ category: stored?.category ?? this.options.sam2?.category ?? "object",+ mask: data,+ width: this.sourceWidth,+ height: this.sourceHeight,+ area: stored?.area ?? 0,+ coverage: (stored?.area ?? 0) / data.length,+ detectionIndex: 0,+ trackId: 1,+ };+ return this.signalMask;+ }++ private publishSignals(from: number, to: number): void {+ const bundle = this.signalBundle;+ if (!bundle || !this.sourceWidth) return;+ bundle.setFrameSize(this.sourceWidth, this.sourceHeight);+ for (let frame = from; frame <= to; frame++) {+ if (this.publishedSignals.has(frame)) continue;+ this.publishedSignals.add(frame);+ const c = Math.max(0, Math.floor(this.clipFrame(frame)));+ const mask = this.maskAt(frame);+ if (!mask) continue;+ const previous = bundle+ .getObjectResult(frame - 1)+ .objects.find((object) => object.trackId === 1);+ const object = sam2Object(+ mask,+ frame,+ this.compositionFps,+ this.results.get(c)?.score ?? 1,+ previous,+ );+ bundle.setObjectResult(frame, [object]);+ const kp = this.results.get(c)?.kp;+ bundle.setPoseResult(frame, {+ people: kp+ ? [+ {+ score: object.score,+ boundingBox: object.boundingBox,+ trackId: 1,+ keypoints: Array.from({ length: 17 }, (_, i) => ({+ x: (kp[i * 3] ?? 0) / mask.width,+ y: (kp[i * 3 + 1] ?? 0) / mask.height,+ visibility: kp[i * 3 + 2] ?? 0,+ })),+ },+ ]+ : [],+ });+ }+ }+private setSize(width: number, height: number): void {this.sourceWidth = width;this.sourceHeight = height;@@ -392,7 +681,9 @@ export class ClipVision implements ClipSubjectSource {c = Math.max(0, c);if (this.blurred.has(c)) return this.blurred.get(c) ?? null;this.mustHave(c);- const mask = bridgedMask(this.results, c, this.smoothing);+ const mask = this.options.sam2+ ? (this.results.get(c)?.mask ?? null)+ : bridgedMask(this.results, c, this.smoothing);const grid = mask ? boxBlur(mask, this.cellW, this.cellH, 2) : null;this.blurred.set(c, grid);return grid;@@ -451,6 +742,7 @@ export class ClipVision implements ClipSubjectSource {this.cell,o.variant ?? "s",o.confidence ?? null,+ ...(o.sam2 ? [[SAM2_TINY_REVISION, o.sam2]] : []),]);const hash = crypto.createHash("sha256")@@ -478,13 +770,17 @@ export class ClipVision implements ClipSubjectSource {width: number;height: number;cell: number;- frames: Record<string, { kp: number[] | null; mask: string | null }>;+ frames: Record<+ string,+ { kp: number[] | null; mask: string | null; score?: number }+ >;};if (json.cell !== this.cell) return;this.setSize(json.width, json.height);for (const [c, r] of Object.entries(json.frames)) {this.results.set(Number(c), {kp: r.kp ? Float32Array.from(r.kp) : null,+ score: r.score,mask: r.mask ? new Uint8Array(Buffer.from(r.mask, "base64")) : null,});}@@ -497,12 +793,15 @@ export class ClipVision implements ClipSubjectSource {this.unsaved = 0;const file = await this.cachePath();if (!file || !this.sourceWidth) return;- const frames: Record<string, { kp: number[] | null; mask: string | null }> =- {};+ const frames: Record<+ string,+ { kp: number[] | null; mask: string | null; score?: number }+ > = {};for (const [c, r] of this.results) {frames[c] = {kp: r.kp ? Array.from(r.kp, (v) => Math.round(v * 100) / 100) : null,mask: r.mask ? Buffer.from(r.mask).toString("base64") : null,+ score: r.score,};}const fs = await import("node:fs");diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.tsindex 19d456d6..43697409 100644--- a/packages/framefields/src/index.ts+++ b/packages/framefields/src/index.ts@@ -123,6 +123,12 @@ export {PoseSkeletonRenderer,pinNodeToLandmark,pinNodeToObject,+ type Sam2FrameResult,+ type Sam2Options,+ type Sam2Point,+ type Sam2Prompt,+ Sam2Tracker,+ type Sam2TrackerOptions,type SegmentationResult,SegmentationTexturePool,SpatialLandmarkTransformer,diff --git a/packages/framefields/src/sam2-clip.test.ts b/packages/framefields/src/sam2-clip.test.tsnew file mode 100644index 00000000..5a6c7d84--- /dev/null+++ b/packages/framefields/src/sam2-clip.test.ts@@ -0,0 +1,86 @@+import { describe, expect, it } from "vitest";+import { ClipVision, seedClipVision } from "./clip-vision.js";++function selectedClip(reverse = false) {+ const clip = new ClipVision("selected.mp4", 24, {+ cacheDir: false,+ sam2: {+ category: "person",+ prompts: [{ frame: 0, points: [{ x: 0.2, y: 0.5, label: 1 }] }],+ },+ ...(reverse+ ? {+ timeRemap: {+ kind: "rate" as const,+ sourceStartSec: 2 / 24,+ rate: -1,+ },+ }+ : {}),+ });+ seedClipVision(+ clip,+ { width: 8, height: 4 },+ Array.from({ length: 3 }, (_, frame) => {+ const mask = Uint8Array.from({ length: 32 }, (_, i) =>+ i % 8 >= frame && i % 8 < frame + 2 ? 255 : 0,+ );+ return { kp: null, mask: null, subject: mask };+ }),+ );+ return clip;+}++describe("prompted clip vision signals", () => {+ it("reuses the selected mask at fractional analysis timestamps", () => {+ const clip = new ClipVision("selected.mp4", 24, {+ fps: 12,+ sam2: { prompts: [{ frame: 0, points: [{ x: 0.2, y: 0.5, label: 1 }] }] },+ });+ seedClipVision(clip, { width: 2, height: 1 }, [+ { kp: null, mask: null, subject: new Uint8Array([255, 0]) },+ { kp: null, mask: null, subject: new Uint8Array([0, 255]) },+ ]);+ expect(clip.subject(1, 24)?.mask).toEqual(new Uint8Array([255, 0]));+ expect(clip.subject(3, 24)?.mask).toEqual(new Uint8Array([0, 255]));+ });++ it("exposes selected-object geometry and frame-exact lazy masks", async () => {+ const clip = selectedClip();+ const signals = clip.signals;+ await clip.ready(0, 2);+ expect(signals.getObjectResult(0).objects[0]).toMatchObject({+ trackId: 1,+ boundingBox: { originX: 0, width: 2 },+ active: true,+ });+ expect(signals.getObjectResult(2).objects[0]?.velocity.vx).toBe(24);+ expect(signals.getMaskResult(2)[0]?.mask[2]).toBe(255);+ expect(signals.getMaskResult(0)[0]?.mask[2]).toBe(0);+ expect(signals.segmentation.matte.at(2)?.coverage).toBe(0.25);+ expect(+ signals.objects.get(1).active.get({ frame: 2, fps: 24 } as never),+ ).toBe(1);+ clip.release();+ expect(signals.getMaskResult(1)[0]?.mask[1]).toBe(255);+ });++ it("maps reverse playback into masks and object velocity without rewinding model state", async () => {+ const clip = selectedClip(true);+ const signals = clip.signals;+ await clip.ready(0, 2);+ expect(signals.getObjectResult(0).objects[0]?.boundingBox.originX).toBe(2);+ expect(signals.getObjectResult(1).objects[0]?.velocity.vx).toBe(-24);+ expect(signals.segmentation.matte.at(2)?.mask[0]).toBe(255);+ });++ it("copies prompt inputs so later edits cannot silently change the cache recipe", () => {+ const points = [{ x: 0.2, y: 0.5, label: 1 as const }];+ const clip = new ClipVision("selected.mp4", 24, {+ sam2: { prompts: [{ frame: 0, points }] },+ });+ const point = points[0];+ if (point) point.x = 0.9;+ expect(clip.options.sam2?.prompts[0]?.points[0]?.x).toBe(0.2);+ });+});diff --git a/packages/framefields/src/sam2-scheduling.test.ts b/packages/framefields/src/sam2-scheduling.test.tsnew file mode 100644index 00000000..07f8b88a--- /dev/null+++ b/packages/framefields/src/sam2-scheduling.test.ts@@ -0,0 +1,83 @@+import { mkdtemp, rm, writeFile } from "node:fs/promises";+import { tmpdir } from "node:os";+import path from "node:path";+import { type Sam2Point, Sam2Tracker } from "@framefields/vision";+import { mediaDecoderCache } from "@framefields/webgpu-renderers";+import { afterEach, describe, expect, it, vi } from "vitest";+import { ClipVision } from "./clip-vision.js";++const points = [{ x: 0.25, y: 0.5, label: 1 as const }];+afterEach(() => vi.restoreAllMocks());++describe("SAM source scheduling and persistence", () => {+ it("propagates skipped frames, applies correction prompts, and reuses disk masks on seek", async () => {+ const dir = await mkdtemp(path.join(tmpdir(), "framefields-sam-clip-"));+ const source = path.join(dir, "source.mp4");+ await writeFile(source, "decoder test fixture");+ const process = vi.fn(+ async (+ _image: unknown,+ frame: number,+ _points?: readonly Sam2Point[],+ ) => ({+ mask: Uint8Array.from({ length: 32 }, (_, i) =>+ i % 8 === frame ? 255 : 0,+ ),+ width: 8,+ height: 4,+ score: 4,+ iou: 0.9,+ }),+ );+ const release = vi.fn();+ vi.spyOn(Sam2Tracker, "create").mockResolvedValue({+ process,+ release,+ } as unknown as Sam2Tracker);+ vi.spyOn(mediaDecoderCache, "getVideo").mockReturnValue({+ durationSec: async () => 10,+ getFrame: async () => ({+ width: 8,+ height: 4,+ buffer: new Uint8Array(128),+ }),+ } as unknown as ReturnType<typeof mediaDecoderCache.getVideo>);+ const options = {+ cacheDir: dir,+ sam2: {+ prompts: [+ { frame: 1, points },+ { frame: 3, points },+ ],+ },+ };+ try {+ const clip = new ClipVision(source, 24, options);+ const signals = clip.signals;+ await clip.ready(4);+ expect(process.mock.calls.map((call) => call[1])).toEqual([1, 2, 3, 4]);+ expect(process.mock.calls[0]?.[2]).toEqual(points);+ expect(process.mock.calls[1]?.[2]).toBeUndefined();+ expect(process.mock.calls[2]?.[2]).toEqual(points);+ expect(clip.subject(0, 24)).toBeNull();+ expect(signals.getMaskResult(4)[0]?.mask[4]).toBe(255);+ clip.release();+ const restored = new ClipVision(source, 24, options);+ await restored.ready(2);+ expect(restored.subject(2, 24)?.mask[2]).toBe(255);+ expect(process).toHaveBeenCalledTimes(4);+ const invalid = new ClipVision(source, 24, {+ cacheDir: false,+ sam2: {+ prompts: [+ { frame: 0, points },+ { frame: 300, points },+ ],+ },+ });+ await expect(invalid.ready(0)).rejects.toThrow("inside the source video");+ } finally {+ await rm(dir, { recursive: true, force: true });+ }+ });+});diff --git a/packages/vision/package.json b/packages/vision/package.jsonindex b7d2137a..cba331b1 100644--- a/packages/vision/package.json+++ b/packages/vision/package.json@@ -1,6 +1,6 @@{"name": "@framefields/vision",- "description": "On-device vision for framefields: RTMDet-Ins detection + instance masks, RTMO pose, and Selfie Segmenter mattes over ONNX Runtime.",+ "description": "On-device vision for framefields: RTMDet-Ins detection + instance masks, RTMO pose, Selfie Segmenter mattes, and prompted SAM 2.1 Tiny video tracking over ONNX Runtime.","version": "2.8.1","private": false,"publishConfig": {@@ -39,6 +39,7 @@"dev": "tsdown --watch","clean": "rm -rf .turbo && rm -rf node_modules && rm -rf dist","test": "vitest run",+ "test:sam2": "FRAMEFIELDS_SAM2_E2E=1 vitest run src/sam2/sam2.e2e.test.ts","test:models": "FRAMEFIELDS_VISION_E2E=1 vitest run src/models.e2e.test.ts"},"peerDependencies": {diff --git a/packages/vision/src/analysis/analyze-sequence.test.ts b/packages/vision/src/analysis/analyze-sequence.test.tsindex 6e069767..1588e2a0 100644--- a/packages/vision/src/analysis/analyze-sequence.test.ts+++ b/packages/vision/src/analysis/analyze-sequence.test.ts@@ -28,6 +28,7 @@ class MovingProvider implements SessionProvider {step++;return out;},+ runMany: async () => rtmdetInsOutput(),release: () => {},};}diff --git a/packages/vision/src/index.ts b/packages/vision/src/index.tsindex a60109da..81151781 100644--- a/packages/vision/src/index.ts+++ b/packages/vision/src/index.ts@@ -13,6 +13,9 @@ export * from "./runner/canonical-baselines.js";export * from "./runner/vision-runner.js";export * from "./runtime/node-webgpu-provider.js";export * from "./runtime/session-provider.js";+export * from "./sam2/signal-results.js";+export * from "./sam2/tracker.js";+export * from "./sam2/types.js";export * from "./segmentation/clip-subject.js";export * from "./segmentation/subject.js";export * from "./signals/vision-bundle.js";diff --git a/packages/vision/src/model/registry.ts b/packages/vision/src/model/registry.tsindex 84b23b68..ffbd5599 100644--- a/packages/vision/src/model/registry.ts+++ b/packages/vision/src/model/registry.ts@@ -30,10 +30,16 @@ export type VisionModelKey =| "rtmo-s"| "rtmo-m"| "selfie-square"- | "selfie-landscape";+ | "selfie-landscape"+ | "sam2.1-tiny-constants"+ | "sam2.1-tiny-vision_encoder"+ | "sam2.1-tiny-mask_decoder"+ | "sam2.1-tiny-memory_encoder"+ | "sam2.1-tiny-memory_attention"+ | "sam2.1-tiny-pointer_tpos";/** Which decoder a model's outputs feed (see runner/vision-runner.ts). */-export type VisionModelFamily = "rtmdet-ins" | "rtmo" | "selfie";+export type VisionModelFamily = "rtmdet-ins" | "rtmo" | "selfie" | "sam2";export interface VisionModelDescriptor {readonly key: VisionModelKey;@@ -66,6 +72,72 @@ const SELFIE_SOURCE =export const VISION_MODELS: Readonly<Record<VisionModelKey, VisionModelDescriptor>> = {+ "sam2.1-tiny-constants": {+ key: "sam2.1-tiny-constants",+ family: "sam2",+ filename: "sam2.1-tiny-constants.json",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/constants.json",+ bytes: 9781,+ sha256: "67583810f4b80b762285377a5cc1066bbe19cdf8ef7fc87ff6aa78fc79830ef2",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },+ "sam2.1-tiny-vision_encoder": {+ key: "sam2.1-tiny-vision_encoder",+ family: "sam2",+ filename: "sam2.1-tiny-vision_encoder.onnx",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/onnx/vision_encoder.onnx",+ bytes: 58385353,+ sha256: "24e9ccf5994be5c8dd93ebe771a4cc4f8bb122d0524555ac3ba4e7141710b133",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },+ "sam2.1-tiny-mask_decoder": {+ key: "sam2.1-tiny-mask_decoder",+ family: "sam2",+ filename: "sam2.1-tiny-mask_decoder.onnx",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/onnx/mask_decoder.onnx",+ bytes: 8898805,+ sha256: "6bc158b0d028777ca2e25203ece0f27429c436de39a5c7fe301b3858d6059789",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },+ "sam2.1-tiny-memory_encoder": {+ key: "sam2.1-tiny-memory_encoder",+ family: "sam2",+ filename: "sam2.1-tiny-memory_encoder.onnx",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/onnx/memory_encoder.onnx",+ bytes: 2807753,+ sha256: "8b2bb6810444cee684ef938b7464c5645653f236014f5497c3db3654818cc7ce",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },+ "sam2.1-tiny-memory_attention": {+ key: "sam2.1-tiny-memory_attention",+ family: "sam2",+ filename: "sam2.1-tiny-memory_attention.onnx",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/onnx/memory_attention.onnx",+ bytes: 13029470,+ sha256: "67c4e4da01bc7f501b88294bb19a8200362915ece04e9d7e367a437658f342c9",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },+ "sam2.1-tiny-pointer_tpos": {+ key: "sam2.1-tiny-pointer_tpos",+ family: "sam2",+ filename: "sam2.1-tiny-pointer_tpos.onnx",+ url: "https://huggingface.co/diffusionstudio/sam2.1-tiny-video-onnx-fp16/resolve/66673b5db39371b7dd7847f4d3bc0d4f4179b79e/onnx/pointer_tpos.onnx",+ bytes: 34228,+ sha256: "3ab5bfce40ed01208301b2d8f596b9c9f7f493dcd7e09ef01ec22e5be3694bd7",+ input: [512, 512],+ source:+ "Meta SAM 2.1 Hiera Tiny (Apache-2.0), Diffusion Studio 512px fp16 video export",+ },// ── Detection + instance segmentation (COCO 80) ─────────────────────"rtmdet-ins-t": {key: "rtmdet-ins-t",diff --git a/packages/vision/src/models.e2e.test.ts b/packages/vision/src/models.e2e.test.tsindex 111a23d7..8b54e2bb 100644--- a/packages/vision/src/models.e2e.test.ts+++ b/packages/vision/src/models.e2e.test.ts@@ -62,7 +62,9 @@ describe.skipIf(!enabled)("vision models (real weights)", () => {});it.each(- Object.keys(VISION_MODELS) as VisionModelKey[],+ (Object.keys(VISION_MODELS) as VisionModelKey[]).filter(+ (key) => VISION_MODELS[key].family !== "sam2",+ ),)("%s downloads, verifies and matches its registry input shape", async (key) => {const bytes = await new VisionModelStore().ensure(key);const ort = await import("onnxruntime-node");diff --git a/packages/vision/src/runner/vision-runner.ts b/packages/vision/src/runner/vision-runner.tsindex 8afe5300..dc7a03f5 100644--- a/packages/vision/src/runner/vision-runner.ts+++ b/packages/vision/src/runner/vision-runner.ts@@ -361,6 +361,8 @@ export class VisionRunner {): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {const session = await this.session(key);const desc = VISION_MODELS[key];+ if (desc.family === "sam2")+ throw new Error("Use Sam2Tracker for prompted video segmentation.");const spec = preprocessSpec(desc.family, desc.input);const dims = [1, 3, spec.height, spec.width];let image: VisionImageInput;diff --git a/packages/vision/src/runtime/session-provider.ts b/packages/vision/src/runtime/session-provider.tsindex e6aee9f9..c7dd1100 100644--- a/packages/vision/src/runtime/session-provider.ts+++ b/packages/vision/src/runtime/session-provider.ts@@ -15,7 +15,7 @@ export interface VisionTensor {export interface VisionTensorInput {readonly name: string;/** Ignored (may be empty) when `gpuBuffer` is set. */- readonly data: Float32Array;+ readonly data: Float32Array | Int32Array;/** The input already on the GPU; only for providers with a `device` (it must be that device's). */readonly gpuBuffer?: GPUBuffer;readonly dims: readonly number[];@@ -25,6 +25,8 @@ export type VisionTensorOutputs = Readonly<Record<string, VisionTensor>>;export interface VisionSession {run(input: VisionTensorInput): Promise<VisionTensorOutputs>;+ /** Multi-input graphs; optional for compatibility with existing custom providers. */+ runMany?(inputs: readonly VisionTensorInput[]): Promise<VisionTensorOutputs>;release(): void;}@@ -294,24 +296,35 @@ export class NodeSessionProvider implements SessionProvider {});type OrtTensor = InstanceType<typeof ort.Tensor>;- return {- run: async (input: VisionTensorInput): Promise<VisionTensorOutputs> => {- const inputName = session.inputNames.includes(input.name)- ? input.name- : (session.inputNames[0] ?? input.name);- const feeds: Record<string, OrtTensor> = {- [inputName]: new ort.Tensor("float32", input.data, [...input.dims]),+ const runMany = async (+ inputs: readonly VisionTensorInput[],+ ): Promise<VisionTensorOutputs> => {+ const feeds: Record<string, OrtTensor> = {};+ for (const input of inputs) {+ const name =+ inputs.length === 1 && !session.inputNames.includes(input.name)+ ? (session.inputNames[0] ?? input.name)+ : input.name;+ if (input.gpuBuffer)+ throw new Error("CPU vision sessions do not accept GPU buffers.");+ feeds[name] =+ input.data instanceof Int32Array+ ? new ort.Tensor("int32", input.data, [...input.dims])+ : new ort.Tensor("float32", input.data, [...input.dims]);+ }+ const results = await session.run(feeds);+ const outputs: Record<string, VisionTensor> = {};+ for (const [name, tensor] of Object.entries(results)) {+ outputs[name] = {+ data: tensor.data as VisionTensor["data"],+ dims: [...tensor.dims],};- const results = await session.run(feeds);- const outputs: Record<string, VisionTensor> = {};- for (const [name, tensor] of Object.entries(results)) {- outputs[name] = {- data: tensor.data as VisionTensor["data"],- dims: [...tensor.dims],- };- }- return outputs;- },+ }+ return outputs;+ };+ return {+ run: (input) => runMany([input]),+ runMany,release: () => {void session.release();},diff --git a/packages/vision/src/runtime/webgpu-provider.test.ts b/packages/vision/src/runtime/webgpu-provider.test.tsindex 6505b8f8..fa5da870 100644--- a/packages/vision/src/runtime/webgpu-provider.test.ts+++ b/packages/vision/src/runtime/webgpu-provider.test.ts@@ -58,6 +58,44 @@ function fakeModule(opts: { failWebGPU?: boolean } = {}) {}describe("WebGPUProvider", () => {+ it("passes multiple named float and integer tensors without renaming", async () => {+ const { module } = fakeModule();+ let feeds: Record<string, unknown> = {};+ const baseCreate = module.InferenceSession.create;+ const provider = new WebGPUProvider({+ loader: async () => ({+ ...module,+ InferenceSession: {+ create: async (bytes, options) => {+ const session = await baseCreate(bytes, options);+ return {+ ...session,+ inputNames: ["features", "labels"],+ run: async (input: Record<string, unknown>) => {+ feeds = input;+ return {};+ },+ release: async () => {},+ };+ },+ },+ }),+ });+ const session = await provider.createSession(new Uint8Array([1]));+ await session.runMany?.([+ { name: "features", data: new Float32Array([1]), dims: [1] },+ { name: "labels", data: new Int32Array([2]), dims: [1] },+ ]);+ expect(feeds.features).toMatchObject({+ type: "float32",+ data: new Float32Array([1]),+ });+ expect(feeds.labels).toMatchObject({+ type: "int32",+ data: new Int32Array([2]),+ });+ });+it("runs sessions on a given device and feeds GPU-buffer inputs", async () => {const { module, calls, fromGpuBuffer } = fakeModule();const device = { label: "renderer" } as unknown as GPUDevice;diff --git a/packages/vision/src/runtime/webgpu-provider.ts b/packages/vision/src/runtime/webgpu-provider.tsindex 2c0cdd12..35a4188e 100644--- a/packages/vision/src/runtime/webgpu-provider.ts+++ b/packages/vision/src/runtime/webgpu-provider.ts@@ -52,8 +52,8 @@ export interface OrtWebModule {};/** ORT tensor constructor (called with `new`), with `fromGpuBuffer` on WebGPU builds. */readonly Tensor: (new (- type: "float32",- data: Float32Array,+ type: "float32" | "int32",+ data: Float32Array | Int32Array,dims: readonly number[],) => unknown) & {fromGpuBuffer?(@@ -138,28 +138,43 @@ export class WebGPUProvider implements SessionProvider {});}- return {- run: async (input: VisionTensorInput): Promise<VisionTensorOutputs> => {- const inputName = session.inputNames.includes(input.name)- ? input.name- : (session.inputNames[0] ?? input.name);- const tensor =+ const runMany = async (+ inputs: readonly VisionTensorInput[],+ ): Promise<VisionTensorOutputs> => {+ const feeds: Record<string, unknown> = {};+ for (const input of inputs) {+ const name =+ inputs.length === 1 && !session.inputNames.includes(input.name)+ ? (session.inputNames[0] ?? input.name)+ : input.name;+ if (input.gpuBuffer && input.data instanceof Int32Array) {+ throw new Error("GPU-buffer vision inputs must be float32.");+ }+ feeds[name] =input.gpuBuffer && ort.Tensor.fromGpuBuffer? ort.Tensor.fromGpuBuffer(input.gpuBuffer, {dataType: "float32",dims: [...input.dims],})- : new ort.Tensor("float32", input.data, [...input.dims]);- const results = await session.run({ [inputName]: tensor });- const outputs: Record<string, VisionTensorOutputs[string]> = {};- for (const [name, tensor] of Object.entries(results)) {- outputs[name] = {- data: tensor.data,- dims: [...tensor.dims],- };- }- return outputs;- },+ : new ort.Tensor(+ input.data instanceof Int32Array ? "int32" : "float32",+ input.data,+ [...input.dims],+ );+ }+ const results = await session.run(feeds);+ const outputs: Record<string, VisionTensorOutputs[string]> = {};+ for (const [name, tensor] of Object.entries(results)) {+ outputs[name] = {+ data: tensor.data,+ dims: [...tensor.dims],+ };+ }+ return outputs;+ };+ return {+ run: (input) => runMany([input]),+ runMany,release: () => {void session.release();},diff --git a/packages/vision/src/sam2/memory.ts b/packages/vision/src/sam2/memory.tsnew file mode 100644index 00000000..d6c6992f--- /dev/null+++ b/packages/vision/src/sam2/memory.ts@@ -0,0 +1,82 @@+/** CPU-owned memory bank; neural inference uses the configured vision execution provider. */+export interface Sam2Memory {+ readonly frame: number;+ readonly tokens: Float32Array;+ readonly positions: Float32Array;+ readonly pointer: Float32Array;+}++export class Sam2MemoryBank {+ private seed?: Sam2Memory;+ private recent: Sam2Memory[] = [];+ constructor(private readonly temporal: readonly (readonly number[])[]) {}++ remember(memory: Sam2Memory, prompted: boolean): void {+ if (prompted) {+ this.seed = memory;+ this.recent = [];+ } else {+ this.recent.push(memory);+ if (this.recent.length > 15) this.recent.shift();+ }+ }++ assemble(+ frame: number,+ frameCount = 16,+ ): {+ memory: Float32Array;+ positions: Float32Array;+ distances: Float32Array;+ } {+ if (!this.seed)+ throw new Error("SAM 2 must be prompted before propagation.");+ const newest = [...this.recent].reverse();+ const blocks = [this.seed, ...newest.slice(0, 6)];+ while (blocks.length < 7) blocks.push(blocks[1] ?? this.seed);+ const pointers = [this.seed, ...newest].slice(0, 16);+ while (pointers.length < 16)+ pointers.push(pointers[pointers.length - 1] as Sam2Memory);+ const length = this.seed.tokens.length;+ const memory = new Float32Array(length * 7 + 16 * 256);+ const positions = new Float32Array(memory.length);+ for (const [slot, block] of blocks.entries()) {+ memory.set(block.tokens, slot * length);+ const available = Math.min(slot, this.recent.length);+ const row =+ this.temporal[slot === 0 || available === 0 ? 6 : available - 1];+ writePositions(positions, slot * length, block.positions, row);+ }+ const distances = new Float32Array(16);+ for (const [i, entry] of pointers.entries()) {+ memory.set(entry.pointer, length * 7 + i * 256);+ distances[i] =+ Math.abs(frame - entry.frame) /+ Math.max(1, Math.min(frameCount, 16) - 1);+ }+ return { memory, positions, distances };+ }+}++/** NCHW features to the sequence-major layout consumed by memory attention. */+export function sam2Tokens(data: Float32Array): Float32Array {+ const count = data.length / 256;+ const result = new Float32Array(data.length);+ for (let channel = 0; channel < 256; channel++) {+ for (let token = 0; token < count; token++)+ result[token * 256 + channel] = data[channel * count + token] ?? 0;+ }+ return result;+}++function writePositions(+ target: Float32Array,+ offset: number,+ spatial: Float32Array,+ temporal?: readonly number[],+): void {+ if (!temporal || temporal.length !== 64)+ throw new Error("Invalid SAM 2 temporal encoding.");+ for (let i = 0; i < spatial.length; i++)+ target[offset + i] = (spatial[i] ?? 0) + (temporal[i % 64] ?? 0);+}diff --git a/packages/vision/src/sam2/sam2.e2e.test.ts b/packages/vision/src/sam2/sam2.e2e.test.tsnew file mode 100644index 00000000..c4d45df3--- /dev/null+++ b/packages/vision/src/sam2/sam2.e2e.test.ts@@ -0,0 +1,50 @@+import { describe, expect, it } from "vitest";+import { NodeWebGPUProvider } from "../runtime/node-webgpu-provider.js";+import { Sam2Tracker } from "./tracker.js";++const enabled = process.env.FRAMEFIELDS_SAM2_E2E === "1";++/** No external video fixture: a moving selected shape and a stationary distractor. */+function picture(frame: number) {+ const width = 128,+ height = 96;+ const data = new Uint8Array(width * height * 4);+ for (let y = 0; y < height; y++)+ for (let x = 0; x < width; x++) {+ const selected =+ x >= 20 + frame * 3 && x < 52 + frame * 3 && y >= 16 && y < 84;+ const distractor = x >= 100 && x < 120 && y >= 16 && y < 84;+ data.set(+ selected+ ? [230, 40, 40, 255]+ : distractor+ ? [40, 220, 80, 255]+ : [30, 50, 120, 255],+ (y * width + x) * 4,+ );+ }+ return { width, height, data };+}++describe.skipIf(!enabled)("SAM 2.1 Tiny on native Dawn", () => {+ it("runs every graph and propagates a selected object without selecting a distractor", async () => {+ const tracker = await Sam2Tracker.create({+ provider: new NodeWebGPUProvider(),+ frameCount: 8,+ });+ try {+ for (let frame = 0; frame < 8; frame++) {+ const result = await tracker.process(+ picture(frame),+ frame,+ frame === 0 ? [{ x: 0.28, y: 0.5, label: 1 }] : undefined,+ );+ expect(result.score).toBeGreaterThan(0);+ expect(result.mask[48 * 128 + 36 + frame * 3]).toBe(255);+ expect(result.mask[48 * 128 + 110]).toBe(0);+ }+ } finally {+ tracker.release();+ }+ }, 120_000);+});diff --git a/packages/vision/src/sam2/sam2.test.ts b/packages/vision/src/sam2/sam2.test.tsnew file mode 100644index 00000000..ffb5d673--- /dev/null+++ b/packages/vision/src/sam2/sam2.test.ts@@ -0,0 +1,165 @@+import { describe, expect, it } from "vitest";+import { VisionModelStore } from "../model/model-store.js";+import { ScriptedProvider } from "../test-support.js";+import { Sam2MemoryBank, sam2Tokens } from "./memory.js";+import { decodeSam2Mask, Sam2Tracker } from "./tracker.js";+import { validateSam2Prompts } from "./types.js";++const float = (data: Float32Array, dims: readonly number[]) => ({ data, dims });+const frame = { data: new Uint8Array(8 * 4 * 4), width: 8, height: 4 };+const points = [{ x: 0.25, y: 0.5, label: 1 as const }];++async function pipeline() {+ const temporal = Array.from(+ { length: 7 },+ () => Array(64).fill(0) as number[],+ );+ const constants = JSON.stringify({+ image_size: 512,+ memory_frames: 7,+ image_mean: [0.485, 0.456, 0.406],+ image_std: [0.229, 0.224, 0.225],+ memory_temporal_positional_encoding: temporal,+ });+ const store = new VisionModelStore({+ modelsDir: `/tmp/framefields-sam2-test-${crypto.randomUUID()}`,+ verify: false,+ fetchImpl: async (url) =>+ new Response(+ String(url).endsWith("constants.json")+ ? constants+ : new Uint8Array([1]),+ ),+ });+ const mask = new Float32Array(512 * 512).fill(-1);+ for (let y = 0; y < 512; y++)+ for (let x = 0; x < 256; x++) mask[y * 512 + x] = 1;+ const feature = float(new Float32Array(256 * 1024), [1, 256, 32, 32]);+ const provider = new ScriptedProvider([+ {+ feats0: float(new Float32Array(32 * 128 * 128), [1, 32, 128, 128]),+ feats1: float(new Float32Array(64 * 64 * 64), [1, 64, 64, 64]),+ feats2: feature,+ feats2_no_mem: feature,+ vision_pos_embed: feature,+ },+ {+ high_res_mask: float(mask, [1, 1, 512, 512]),+ object_score_logits: float(new Float32Array([5]), [1, 1, 1]),+ object_pointer: float(new Float32Array(256).fill(7), [1, 1, 256]),+ iou: float(new Float32Array([0.9]), [1, 1]),+ },+ {+ memory_tokens: float(new Float32Array(1024 * 64).fill(3), [1024, 1, 64]),+ memory_pos: float(new Float32Array(1024 * 64), [1024, 1, 64]),+ },+ { conditioned_feats: feature },+ { pointer_pos: float(new Float32Array(16 * 64), [16, 64]) },+ ]);+ return { tracker: await Sam2Tracker.create({ store, provider }), provider };+}++describe("SAM 2 video pipeline", () => {+ it("uses point labels, remembers the prompt, and conditions subsequent frames on memory", async () => {+ const { tracker, provider } = await pipeline();+ const first = await tracker.process(frame, 4, points);+ expect(first.mask).toEqual(+ Uint8Array.from({ length: 32 }, (_, i) => (i % 8 < 4 ? 255 : 0)),+ );+ await tracker.process(frame, 5);+ const decoderInputs = provider.created[1]?.inputs ?? [];+ expect(+ decoderInputs.find((entry) => entry.name === "input_labels")?.data,+ ).toBeInstanceOf(Int32Array);+ expect(+ decoderInputs.find((entry) => entry.name === "input_points")?.data,+ ).toEqual(new Float32Array([128, 256]));+ const memory = provider.created[3]?.inputs.find(+ (entry) => entry.name === "memory",+ );+ expect(memory?.dims).toEqual([7232, 1, 64]);+ expect(memory?.data[0]).toBe(3);+ expect(memory?.data[7 * 1024 * 64]).toBe(7);+ expect(provider.created[4]?.inputs[0]?.data[0]).toBeCloseTo(1 / 15);+ await expect(tracker.process(frame, 7)).rejects.toThrow("consecutive");+ tracker.release();+ await expect(tracker.process(frame, 6)).rejects.toThrow("released");+ });++ it("requires explicit selection and accepts a correction on a later frame", async () => {+ const { tracker, provider } = await pipeline();+ await expect(tracker.process(frame, 0)).rejects.toThrow("prompt");+ await tracker.process(frame, 0, points);+ await tracker.process(frame, 1, points);+ expect(provider.created[3]?.inputs).toHaveLength(0);+ expect(+ provider.created[2]?.inputs+ .filter((entry) => entry.name === "binarize")+ .map((entry) => entry.data[0]),+ ).toEqual([1, 1]);+ tracker.release();+ });++ it("reports an absent object as an empty source-sized mask", () => {+ expect(+ decodeSam2Mask(new Float32Array(512 * 512).fill(2), 13, 7, -2, 0.4).mask,+ ).toEqual(new Uint8Array(91));+ });++ it("validates prompt ordering, coordinates, and box corners", () => {+ expect(() => validateSam2Prompts([])).toThrow();+ expect(() =>+ validateSam2Prompts([+ { frame: 0, points },+ { frame: 0, points },+ ]),+ ).toThrow();+ expect(() =>+ validateSam2Prompts([{ frame: 0, points: [{ x: NaN, y: 0, label: 1 }] }]),+ ).toThrow();+ expect(() =>+ validateSam2Prompts([{ frame: 0, points: [{ x: 0, y: 0, label: 2 }] }]),+ ).toThrow();+ expect(() =>+ validateSam2Prompts([+ {+ frame: 0,+ points: [+ { x: 0, y: 0, label: 2 },+ { x: 1, y: 1, label: 3 },+ ],+ },+ ]),+ ).not.toThrow();+ });+});++describe("SAM 2 temporal geometry", () => {+ it("transposes channels to sequence tokens without mixing feature channels", () => {+ const data = Float32Array.from({ length: 512 }, (_, i) => i);+ const tokens = sam2Tokens(data);+ expect(tokens[0]).toBe(0);+ expect(tokens[1]).toBe(2);+ expect(tokens[256]).toBe(1);+ });+ it("keeps the conditioning frame and resets its history after correction", () => {+ const bank = new Sam2MemoryBank(+ Array.from({ length: 7 }, (_, i) => Array(64).fill(i) as number[]),+ );+ const memory = (frame: number) => ({+ frame,+ tokens: new Float32Array(64).fill(frame),+ positions: new Float32Array(64),+ pointer: new Float32Array(256).fill(frame),+ });+ bank.remember(memory(0), true);+ for (let i = 1; i < 20; i++) bank.remember(memory(i), false);+ const assembled = bank.assemble(20);+ expect(assembled.memory[0]).toBe(0);+ expect(assembled.memory[64]).toBe(19);+ expect(assembled.positions[0]).toBe(6);+ expect(assembled.distances[0]).toBeCloseTo(20 / 15);+ bank.remember(memory(20), true);+ expect(bank.assemble(21).memory[64]).toBe(20);+ });+});diff --git a/packages/vision/src/sam2/signal-results.ts b/packages/vision/src/sam2/signal-results.tsnew file mode 100644index 00000000..6256fae8--- /dev/null+++ b/packages/vision/src/sam2/signal-results.ts@@ -0,0 +1,60 @@+import type { TrackedObject } from "@framefields/core";+import type { InstanceMask } from "../types.js";++/** Derive tracked-object geometry from the selected mask, never another person's detector. */+export function sam2Object(+ mask: InstanceMask,+ frame: number,+ fps: number,+ score: number,+ previous?: TrackedObject,+): TrackedObject {+ let left = mask.width,+ top = mask.height,+ right = -1,+ bottom = -1;+ for (let y = 0; y < mask.height; y++)+ for (let x = 0; x < mask.width; x++) {+ if ((mask.mask[y * mask.width + x] ?? 0) < 128) continue;+ left = Math.min(left, x);+ top = Math.min(top, y);+ right = Math.max(right, x);+ bottom = Math.max(bottom, y);+ }+ const active = right >= 0;+ const x = active ? left : 0,+ y = active ? top : 0;+ const width = active ? right - left + 1 : 0,+ height = active ? bottom - top + 1 : 0;+ const centerX = x + width / 2,+ centerY = y + height / 2;+ const vx =+ active && previous?.active ? (centerX - previous.centerX) * fps : 0;+ const vy =+ active && previous?.active ? (centerY - previous.centerY) * fps : 0;+ return {+ trackId: 1,+ category: mask.category,+ score,+ boundingBox: {+ originX: x,+ originY: y,+ width,+ height,+ normalizedX: x / mask.width,+ normalizedY: y / mask.height,+ normalizedWidth: width / mask.width,+ normalizedHeight: height / mask.height,+ },+ centerX,+ centerY,+ normalizedCenterX: centerX / mask.width,+ normalizedCenterY: centerY / mask.height,+ velocity: { vx, vy },+ speed: Math.hypot(vx, vy),+ age: frame + 1,+ hits: frame + 1,+ active,+ isCoasting: false,+ };+}diff --git a/packages/vision/src/sam2/tracker.ts b/packages/vision/src/sam2/tracker.tsnew file mode 100644index 00000000..b0d3d7b2--- /dev/null+++ b/packages/vision/src/sam2/tracker.ts@@ -0,0 +1,300 @@+import { z } from "zod";+import { imageToTensor } from "../decode/preprocess.js";+import {+ VisionModelStore,+ type VisionModelStoreOptions,+} from "../model/model-store.js";+import {+ AutoSessionProvider,+ type SessionProvider,+ type VisionSession,+ type VisionTensor,+ type VisionTensorInput,+ type VisionTensorOutputs,+} from "../runtime/session-provider.js";+import type { VisionImageInput } from "../types.js";+import { Sam2MemoryBank, sam2Tokens } from "./memory.js";+import {+ type Sam2FrameResult,+ type Sam2Point,+ validateSam2Prompts,+} from "./types.js";++const constantsSchema = z.object({+ image_size: z.literal(512),+ memory_frames: z.literal(7),+ image_mean: z.tuple([z.number(), z.number(), z.number()]),+ image_std: z.tuple([+ z.number().positive(),+ z.number().positive(),+ z.number().positive(),+ ]),+ memory_temporal_positional_encoding: z+ .array(z.array(z.number()).length(64))+ .length(7),+});+type Constants = z.infer<typeof constantsSchema>;+type Graph =+ | "vision_encoder"+ | "mask_decoder"+ | "memory_encoder"+ | "memory_attention"+ | "pointer_tpos";+const GRAPHS: readonly Graph[] = [+ "vision_encoder",+ "mask_decoder",+ "memory_encoder",+ "memory_attention",+ "pointer_tpos",+];+export const SAM2_TINY_REVISION = "66673b5db39371b7dd7847f4d3bc0d4f4179b79e";++export interface Sam2TrackerOptions extends VisionModelStoreOptions {+ readonly provider?: SessionProvider;+ readonly store?: VisionModelStore;+ /** Source sequence length; used to normalize object-pointer time positions for short videos. */+ readonly frameCount?: number;+}++function floats(outputs: VisionTensorOutputs, name: string): Float32Array {+ const tensor = outputs[name];+ if (!tensor || !(tensor.data instanceof Float32Array))+ throw new Error(`Missing SAM 2 float output: ${name}`);+ return tensor.data;+}+function input(name: string, tensor: VisionTensor): VisionTensorInput {+ if (tensor.data instanceof BigInt64Array)+ throw new Error(`Unsupported SAM 2 input type: ${name}`);+ return { name, data: tensor.data, dims: tensor.dims };+}+function tensor(+ name: string,+ data: Float32Array | Int32Array,+ dims: readonly number[],+): VisionTensorInput {+ return { name, data, dims };+}++/** Complete pinned 512px SAM 2.1 Tiny video graph pipeline, one explicitly selected object. */+export class Sam2Tracker {+ private readonly sessions = new Map<Graph, VisionSession>();+ private readonly bank: Sam2MemoryBank;+ private lastFrame = -1;+ private released = false;+ private constructor(+ private readonly constants: Constants,+ private readonly frameCount = 16,+ ) {+ this.bank = new Sam2MemoryBank(+ constants.memory_temporal_positional_encoding,+ );+ }++ static async create(options: Sam2TrackerOptions = {}): Promise<Sam2Tracker> {+ const store = options.store ?? new VisionModelStore(options);+ const bytes = await store.ensure("sam2.1-tiny-constants");+ const constants = constantsSchema.parse(+ JSON.parse(new TextDecoder().decode(bytes)),+ );+ if (+ options.frameCount !== undefined &&+ (!Number.isSafeInteger(options.frameCount) || options.frameCount < 1)+ )+ throw new Error("SAM 2 frameCount must be a positive integer.");+ const tracker = new Sam2Tracker(constants, options.frameCount);+ const provider = options.provider ?? new AutoSessionProvider();+ try {+ for (const graph of GRAPHS) {+ const session = await provider.createSession(+ await store.ensure(`sam2.1-tiny-${graph}`),+ );+ tracker.sessions.set(graph, session);+ }+ return tracker;+ } catch (error) {+ tracker.release();+ throw error;+ }+ }++ private run(+ graph: Graph,+ inputs: readonly VisionTensorInput[],+ ): Promise<VisionTensorOutputs> {+ const session = this.sessions.get(graph);+ if (!session) throw new Error("SAM 2 tracker has been released.");+ if (session.runMany) return session.runMany(inputs);+ if (inputs.length === 1 && inputs[0]) return session.run(inputs[0]);+ throw new Error("SAM 2 requires a session provider with runMany support.");+ }++ async process(+ image: VisionImageInput,+ frame: number,+ points?: readonly Sam2Point[],+ ): Promise<Sam2FrameResult> {+ if (this.released) throw new Error("SAM 2 tracker has been released.");+ if (+ !Number.isSafeInteger(frame) ||+ frame < 0 ||+ (this.lastFrame >= 0 && frame !== this.lastFrame + 1)+ )+ throw new Error("SAM 2 propagation requires consecutive source frames.");+ if (points) validateSam2Prompts([{ frame, points }]);+ if (this.lastFrame < 0 && !points)+ throw new Error("SAM 2 needs a prompt on its first frame.");+ const { tensor: pixels } = imageToTensor(image, {+ width: 512,+ height: 512,+ fit: "stretch",+ channels: "rgb",+ padValue: 0,+ mean: this.constants.image_mean.map((v) => v * 255) as [+ number,+ number,+ number,+ ],+ std: this.constants.image_std.map((v) => v * 255) as [+ number,+ number,+ number,+ ],+ });+ const vision = await this.run("vision_encoder", [+ tensor("pixel_values", pixels, [1, 3, 512, 512]),+ ]);+ const conditioned = points+ ? vision.feats2_no_mem+ : await this.condition(vision, frame);+ if (!conditioned || !vision.feats0 || !vision.feats1)+ throw new Error("Invalid SAM 2 vision outputs.");+ const prompt = points ?? [{ x: 0, y: 0, label: -1 }];+ const decoded = await this.run("mask_decoder", [+ input("feats0", vision.feats0),+ input("feats1", vision.feats1),+ input("feats2_cond", conditioned),+ tensor(+ "input_points",+ Float32Array.from(prompt.flatMap((p) => [p.x * 512, p.y * 512])),+ [1, 1, prompt.length, 2],+ ),+ tensor("input_labels", Int32Array.from(prompt.map((p) => p.label)), [+ 1,+ 1,+ prompt.length,+ ]),+ ]);+ await this.remember(vision, decoded, frame, points !== undefined);+ this.lastFrame = frame;+ return decodeSam2Mask(+ floats(decoded, "high_res_mask"),+ image.width,+ image.height,+ floats(decoded, "object_score_logits")[0] ?? -Infinity,+ floats(decoded, "iou")[0] ?? 0,+ );+ }++ private async condition(+ vision: VisionTensorOutputs,+ frame: number,+ ): Promise<VisionTensor> {+ const bank = this.bank.assemble(frame, this.frameCount);+ const pointer = await this.run("pointer_tpos", [+ tensor("normalized_diffs", bank.distances, [16]),+ ]);+ const positions = floats(pointer, "pointer_pos");+ for (let i = 0; i < 16; i++)+ for (let token = 0; token < 4; token++)+ bank.positions.set(+ positions.subarray(i * 64, (i + 1) * 64),+ 7 * 1024 * 64 + (i * 4 + token) * 64,+ );+ const result = await this.run("memory_attention", [+ tensor(+ "current_vision_features",+ sam2Tokens(floats(vision, "feats2")),+ [1024, 1, 256],+ ),+ tensor(+ "current_vision_position_embeddings",+ sam2Tokens(floats(vision, "vision_pos_embed")),+ [1024, 1, 256],+ ),+ tensor("memory", bank.memory, [7232, 1, 64]),+ tensor("memory_pos", bank.positions, [7232, 1, 64]),+ ]);+ const conditioned = result.conditioned_feats;+ if (!conditioned) throw new Error("Missing SAM 2 conditioned features.");+ return conditioned;+ }++ private async remember(+ vision: VisionTensorOutputs,+ decoded: VisionTensorOutputs,+ frame: number,+ prompted: boolean,+ ): Promise<void> {+ if (!vision.feats2 || !decoded.high_res_mask)+ throw new Error("Invalid SAM 2 memory inputs.");+ const memory = await this.run("memory_encoder", [+ input("feats2", vision.feats2),+ input("high_res_mask", decoded.high_res_mask),+ tensor(+ "object_score_logits",+ floats(decoded, "object_score_logits"),+ [1, 1],+ ),+ tensor("binarize", new Float32Array([prompted ? 1 : 0]), []),+ ]);+ this.bank.remember(+ {+ frame,+ tokens: floats(memory, "memory_tokens").slice(),+ positions: floats(memory, "memory_pos").slice(),+ pointer: floats(decoded, "object_pointer").slice(),+ },+ prompted,+ );+ }++ release(): void {+ for (const session of this.sessions.values()) session.release();+ this.sessions.clear();+ this.released = true;+ }+}++/** Bilinear resize of logits before thresholding, matching SAM's source-image mask contract. */+export function decodeSam2Mask(+ logits: Float32Array,+ width: number,+ height: number,+ score: number,+ iou: number,+): Sam2FrameResult {+ if (logits.length !== 512 * 512) throw new Error("Invalid SAM 2 mask shape.");+ const mask = new Uint8Array(width * height);+ if (score > 0) {+ for (let y = 0; y < height; y++) {+ const sy = Math.max(0, Math.min(511, ((y + 0.5) * 512) / height - 0.5));+ const y0 = Math.floor(sy),+ y1 = Math.min(511, y0 + 1),+ fy = sy - y0;+ for (let x = 0; x < width; x++) {+ const sx = Math.max(0, Math.min(511, ((x + 0.5) * 512) / width - 0.5));+ const x0 = Math.floor(sx),+ x1 = Math.min(511, x0 + 1),+ fx = sx - x0;+ const a =+ (logits[y0 * 512 + x0] ?? 0) * (1 - fx) ++ (logits[y0 * 512 + x1] ?? 0) * fx;+ const b =+ (logits[y1 * 512 + x0] ?? 0) * (1 - fx) ++ (logits[y1 * 512 + x1] ?? 0) * fx;+ mask[y * width + x] = a * (1 - fy) + b * fy > 0 ? 255 : 0;+ }+ }+ }+ return { mask, width, height, score, iou };+}diff --git a/packages/vision/src/sam2/types.ts b/packages/vision/src/sam2/types.tsnew file mode 100644index 00000000..8b46e020--- /dev/null+++ b/packages/vision/src/sam2/types.ts@@ -0,0 +1,83 @@+import type { VisionImageInput } from "../types.js";++/** Normalized source-image coordinates. Labels 2/3 form a box's opposite corners. */+export interface Sam2Point {+ readonly x: number;+ readonly y: number;+ readonly label: 0 | 1 | 2 | 3;+}++/** Frame is on the source analysis clock, before trim or time remapping. */+export interface Sam2Prompt {+ readonly frame: number;+ readonly points: readonly Sam2Point[];+}++export interface Sam2Options {+ readonly prompts: readonly Sam2Prompt[];+ /** Metadata for object and class signals; SAM does not classify objects. */+ readonly category?: string;+}++export interface Sam2FrameResult {+ readonly mask: Uint8Array;+ readonly width: number;+ readonly height: number;+ readonly score: number;+ readonly iou: number;+}++export interface Sam2FrameTracker {+ process(+ image: VisionImageInput,+ frame: number,+ points?: readonly Sam2Point[],+ ): Promise<Sam2FrameResult>;+ release(): void;+}++export function validateSam2Prompts(prompts: readonly Sam2Prompt[]): void {+ if (prompts.length === 0)+ throw new Error("SAM 2 requires at least one prompt.");+ let previous = -1;+ for (const prompt of prompts) {+ if (!Number.isSafeInteger(prompt.frame) || prompt.frame <= previous) {+ throw new Error(+ "SAM 2 prompt frames must be nonnegative, unique and increasing.",+ );+ }+ previous = prompt.frame;+ if (prompt.points.length === 0)+ throw new Error("SAM 2 prompts need points or a box.");+ for (const point of prompt.points) validatePoint(point);+ const corners = prompt.points.filter((point) => point.label >= 2);+ if (+ corners.length &&+ (corners.length !== 2 ||+ corners[0]?.label !== 2 ||+ corners[1]?.label !== 3 ||+ corners[0].x >= corners[1].x ||+ corners[0].y >= corners[1].y)+ ) {+ throw new Error(+ "SAM 2 boxes need top-left (2) and bottom-right (3) corners.",+ );+ }+ }+}++function validatePoint(point: Sam2Point): void {+ if (+ ![0, 1, 2, 3].includes(point.label) ||+ !Number.isFinite(point.x) ||+ !Number.isFinite(point.y) ||+ point.x < 0 ||+ point.x > 1 ||+ point.y < 0 ||+ point.y > 1+ ) {+ throw new Error(+ "SAM 2 points require normalized coordinates and labels 0, 1, 2 or 3.",+ );+ }+}diff --git a/packages/vision/src/segmentation/clip-subject.ts b/packages/vision/src/segmentation/clip-subject.tsindex 41471e8c..cc11edae 100644--- a/packages/vision/src/segmentation/clip-subject.ts+++ b/packages/vision/src/segmentation/clip-subject.ts@@ -11,6 +11,8 @@ export const CLIP_SUBJECT: unique symbol = Symbol.for() as never;export interface ClipSubjectSource {+ /** An explicit selection must not silently fall back to automatic subject detection. */+ readonly exact?: boolean;/*** The merged subject mask for a source frame of the clip (`frame` at `fps`),* in source pixels; `null` when the frame was analysed and holds no subject;diff --git a/packages/vision/src/signals/vision-bundle.ts b/packages/vision/src/signals/vision-bundle.tsindex f5fe472a..3d8e8465 100644--- a/packages/vision/src/signals/vision-bundle.ts+++ b/packages/vision/src/signals/vision-bundle.ts@@ -37,6 +37,9 @@ export interface VisionBundleOptions {readonly height?: number;readonly cameraFov?: number;readonly config?: VisionConfig;+ /** Lazy source-frame masks, for clips retaining compressed masks instead of RGBA-sized histories. */+ readonly maskSource?: (frame: number) => readonly InstanceMask[] | undefined;+ readonly matteSource?: (frame: number) => PersonMatte | undefined;}// ── Signal surface types ────────────────────────────────────────────────────@@ -225,6 +228,9 @@ export class VisionBundle {private _height: number;private _transformer: SpatialLandmarkTransformer;private _stencilTexture?: GPUTexture;+ private readonly _maskSource?: VisionBundleOptions["maskSource"];+ private readonly _matteSource?: VisionBundleOptions["matteSource"];+ private readonly _cameraFov: number;private _poseCache = new Map<number, PoseResult>();private _objectCache = new Map<@@ -245,6 +251,9 @@ export class VisionBundle {private _registeredSignals = new Set<ProgrammaticSignal>();constructor(options: VisionBundleOptions = {}) {+ this._maskSource = options.maskSource;+ this._matteSource = options.matteSource;+ this._cameraFov = options.cameraFov ?? 60;this._totalFrames = options.totalFrames ?? 120;this._fps = options.fps ?? 24;this._width = options.width ?? 1920;@@ -1109,6 +1118,19 @@ export class VisionBundle {};}+ /** Update source geometry once a lazily decoded clip's size is known. */+ public setFrameSize(width: number, height: number): void {+ if (this._width === width && this._height === height) return;+ this._width = width;+ this._height = height;+ this._transformer = new SpatialLandmarkTransformer({+ width,+ height,+ fov: this._cameraFov,+ });+ this.invalidateSignals();+ }+public setStencilTexture(texture: GPUTexture): void {this._stencilTexture = texture;}@@ -1193,7 +1215,11 @@ export class VisionBundle {}public getMaskResult(frame: number): readonly InstanceMask[] {- return this.latestAtOrBefore(this._maskCache, frame) ?? [];+ return (+ this._maskSource?.(frame) ??+ this.latestAtOrBefore(this._maskCache, frame) ??+ []+ );}public setMatteResult(frame: number, matte: PersonMatte): void {@@ -1202,7 +1228,10 @@ export class VisionBundle {}public getMatteResult(frame: number): PersonMatte | undefined {- return this.latestAtOrBefore(this._matteCache, frame);+ return (+ this._matteSource?.(frame) ??+ this.latestAtOrBefore(this._matteCache, frame)+ );}// ── Agent DX snapshot ─────────────────────────────────────────────────diff --git a/packages/vision/src/test-support.ts b/packages/vision/src/test-support.tsindex 3bef6262..0738e82b 100644--- a/packages/vision/src/test-support.ts+++ b/packages/vision/src/test-support.ts@@ -27,6 +27,17 @@ export class ScriptedSession implements VisionSession {});return this.outputs;}+ async runMany(+ inputs: readonly VisionTensorInput[],+ ): Promise<VisionTensorOutputs> {+ for (const input of inputs)+ this.inputs.push({+ ...input,+ data: input.data.slice(),+ dims: [...input.dims],+ });+ return this.outputs;+ }release(): void {}}diff --git a/packages/webgpu-renderers/src/nodes/media-decoder.test.ts b/packages/webgpu-renderers/src/nodes/media-decoder.test.tsindex ef863152..598e28b8 100644--- a/packages/webgpu-renderers/src/nodes/media-decoder.test.ts+++ b/packages/webgpu-renderers/src/nodes/media-decoder.test.ts@@ -55,7 +55,9 @@ vi.mock("mediabunny", () => {// Mock inputStorevi.mock("../input-store.js", () => {const mockInput = {- getPrimaryVideoTrack: vi.fn().mockResolvedValue({ id: "video-track" }),+ getPrimaryVideoTrack: vi+ .fn()+ .mockResolvedValue({ id: "video-track", computeDuration: async () => 3 }),getPrimaryAudioTrack: vi.fn().mockResolvedValue({ id: "audio-track" }),dispose: vi.fn(),};@@ -68,6 +70,12 @@ vi.mock("../input-store.js", () => {});describe("Media Decoder", () => {+ it("reads video-track duration without decoding frames", async () => {+ const decoder = new VideoDecoder("duration-test", true);+ expect(await decoder.durationSec()).toBe(3);+ decoder.destroy();+ });+beforeEach(() => {vi.clearAllMocks();});diff --git a/packages/webgpu-renderers/src/nodes/media-decoder.ts b/packages/webgpu-renderers/src/nodes/media-decoder.tsindex 6c6e09ba..490697d2 100644--- a/packages/webgpu-renderers/src/nodes/media-decoder.ts+++ b/packages/webgpu-renderers/src/nodes/media-decoder.ts@@ -106,6 +106,14 @@ export class VideoDecoder extends BaseMediaDecoder<reject: (err: unknown) => void;} | null = null;+ /** Precise video-track duration, independent of a potentially longer audio track. */+ async durationSec(): Promise<number> {+ await this.init();+ const track = await this.input?.getPrimaryVideoTrack();+ if (!track) throw new Error(`No video track in ${this.url}`);+ return track.computeDuration();+ }+private pushHeadlessQueue(sample: VideoSample) {const index = this.headlessQueue.findIndex((s) => s.timestamp > sample.timestamp,diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 9394234e..c5e655be 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -222,6 +222,42 @@ Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optiona| Renders offline / in CI | Pre-download with `runner.preload([...])` into `$FRAMEFIELDS_MODELS_DIR`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror || Check the models themselves | `await runner.preload([...])` on a fresh models directory; it downloads each model and verifies its SHA-256 (about 380 MB for all of them) |+### Prompted object tracking with SAM 2.1 Tiny++Use SAM for a specific object among similar objects. `comp.vision(layer, { sam2 })` runs the complete 512px SAM 2.1 Tiny video pipeline, including temporal memory, through the existing ONNX execution provider. The six pinned model assets (five graphs and constants, about 83 MB) download lazily and are size/SHA-256 verified in the normal vision model cache. `modelsDir`, `baseUrl` and the normal vision provider environment settings apply. Neural inference on the shared Dawn/WebGPU runtime is supported; frame history is held in bounded host arrays and uploaded to the model sessions.++```ts+const subject = Layer.video("assets/source.mp4", {+ width: 1920, height: 1080, fit: "contain", muted: false,+}).withVision({ mode: "matte" });+comp.add(subject);+const tracked = comp.vision(subject, {+ sam2: {+ category: "person", // supplied metadata, not model classification+ prompts: [+ { frame: 0, points: [+ { x: 0.25, y: 0.10, label: 2 }, // box top-left+ { x: 0.70, y: 0.95, label: 3 }, // box bottom-right+ { x: 0.48, y: 0.50, label: 1 }, // foreground+ ] },+ // Add a new point/box prompt at another source frame to correct tracking.+ ],+ },+ pose: true, // optional RTMO pass, matched to the selected silhouette+});+const signals = tracked.signals; // stable VisionBundle; request before rendering+const target = signals.objects.get(1);+// target.bounds, anchors, confidence, active, kinematics and pose use the existing signals API.+await tracked.ready(0);+const mask = signals.segmentation.matte.at(0); // source-sized selected-object alpha+```++Coordinates are normalized to the original source image, before layer placement. Labels are foreground `1`, background `0`, and box corners `2`/`3`. Prompt frames are nonnegative increasing integers on the source analysis clock (`options.fps`, default composition FPS), before trimming/time remapping. SAM is forward propagation from the earliest prompt; frames before it are empty. Corrections reset the conditioning memory. Reset/re-prompt across cuts or when identity is lost; appearance tracking cannot guarantee re-identification after occlusion.++SAM masks do not merge other detected people. Default smoothing is zero and a missing object stays empty. Pose is disabled unless explicitly requested; SAM itself provides no skeleton. `tracked.person` continues to expose silhouette sampling and optional matched pose; `.place(...)` maps those into a layer's layout. The `tracked.signals` bundle uses source geometry on the composition clock; object and mask signals refer to track ID `1`. Full masks are reconstructed lazily from compressed history, avoiding an uncompressed mask per frame.++Frame requests analyze intervening source frames in order, so a seek forward can take time; a backward seek, hold, reverse, and time-remapped playback reuse source-frame results. When analysis FPS is lower than playback FPS, intermediate frames reuse the preceding analysis mask. Source masks are persisted separately from low-resolution silhouette/pose data; cache identity includes all prompts and the model revision. `pnpm --filter @framefields/vision test:sam2` runs a real-model test on native Dawn (requires GPU access; downloads are cached). A matte/mask/crop layer of the same clip reuses the selected masks through the existing clip-matte link. Use the default matte settings and matching source aspect ratio for that link; crop after source-space matting. An explicit selection fails clearly instead of falling back to another detected person when the mask cannot align. `tracked.release()` frees SAM sessions, keeps results readable, and replays the source memory if further uncached frames are requested.+### 1. Subject Sandwich ("Text Behind Subject")Cuts out the foreground subject from footage and sandwiches typography or graphics directly behind them: