Code-first video & image, rendered natively on WebGPU.
feat(framefields): load @framefields/vision-sam lazily as an optional peer
okanasldac36943c770
diff --git a/.changeset/lazy-vision-sam.md b/.changeset/lazy-vision-sam.mdnew file mode 100644index 00000000..4c7e4477--- /dev/null+++ b/.changeset/lazy-vision-sam.md@@ -0,0 +1,7 @@+---+"framefields": minor+"@framefields/vision": minor+"@framefields/vision-sam": minor+---++`@framefields/vision-sam` is now an optional peer of `framefields`, loaded only when a clip uses `sam2` or `sam3`, so the SDK no longer installs or imports SAM-licensed code by default. Install it next to `framefields` to use prompted tracking, and import `Sam2Tracker`/`Sam3Tracker` from `@framefields/vision-sam`. Mask-derived object geometry moved to `@framefields/vision` as `maskObject` (formerly `sam2Object`).diff --git a/AGENTS.md b/AGENTS.mdindex d67888b1..cf63f299 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -18,7 +18,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).-- `packages/vision-sam`: segregated package under Meta SAM License for prompted SAM 2.1 Tiny and SAM 3 / 3.1 video tracking and text selection. `comp.vision(..., { sam2 | sam3 })` shares clip mattes and exposes object/mask/pose signals. Model weights download dynamically at runtime with license confirmation.+- `packages/vision-sam`: SAM 2.1 Tiny / SAM 3 trackers under Meta's SAM License, an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).diff --git a/CLAUDE.md b/CLAUDE.mdindex e47c63d3..da680b7b 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -18,6 +18,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).+- `packages/vision-sam`: SAM 2.1 Tiny / SAM 3 trackers under Meta's SAM License, an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).diff --git a/README.md b/README.mdindex 141fb6b4..e16b93d6 100644--- a/README.md+++ b/README.md@@ -194,7 +194,11 @@ Installed components import only `framefields` and the shared toolkit in `framef## Select and track a subject-Use SAM 2.1 Tiny to select a specific object with a point or box and track it through video. Its masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts.+Use SAM 2.1 Tiny to select a specific object with a point or box and track it through video. Its masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. SAM ships separately under Meta's SAM License, so install it next to framefields:++```bash+pnpm add @framefields/vision-sam+``````tsconst subject = Layer.video("assets/source.mp4", {diff --git a/packages/framefields/package.json b/packages/framefields/package.jsonindex bd331f90..473db5ce 100644--- a/packages/framefields/package.json+++ b/packages/framefields/package.json@@ -118,10 +118,17 @@"yoga-layout": "^3.2.1","zod": "^4.3.6"},+ "peerDependencies": {+ "@framefields/vision-sam": "workspace:*"+ },+ "peerDependenciesMeta": {+ "@framefields/vision-sam": {+ "optional": true+ }+ },"optionalDependencies": {"@dimforge/rapier2d-deterministic-compat": "0.21.0","@dimforge/rapier3d-deterministic-compat": "0.21.0",- "@framefields/vision-sam": "workspace:*","onnxruntime-web": "^1.24.0"},"devDependencies": {diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.tsindex 8756c75b..0bdb05ae 100644--- a/packages/framefields/src/clip-vision.ts+++ b/packages/framefields/src/clip-vision.ts@@ -9,21 +9,17 @@ import {type ClipSubjectSource,COCO17_KEYPOINT_NAMES,type InstanceMask,+ maskObject,mergeSubjectMask,VisionBundle,VisionRunner,} from "@framefields/vision";-import {- SAM2_TINY_REVISION,- SAM3_REVISION,- type Sam2FrameResult,- type Sam2Options,+import type {+ Sam2FrameResult,+ Sam2Options,Sam2Tracker,- type Sam3Options,+ Sam3Options,Sam3Tracker,- sam2Object,- validateSam2Prompts,- validateSam3Prompts,} from "@framefields/vision-sam";import { mediaDecoderCache } from "@framefields/webgpu-renderers";import type { VisionProps } from "./effects/index.js";@@ -105,6 +101,25 @@ const HOLD = 24;const CACHE_VERSION = 1;+type VisionSam = typeof import("@framefields/vision-sam");++let visionSam: Promise<VisionSam> | undefined;++/**+ * SAM lives in `@framefields/vision-sam`, an optional peer under Meta's SAM License, so it+ * is imported only when a clip asks for `sam2` or `sam3`.+ */+function loadVisionSam(): Promise<VisionSam> {+ visionSam ??= import("@framefields/vision-sam").catch((cause: unknown) => {+ visionSam = undefined;+ throw new Error(+ "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package (Meta SAM License). Install it next to framefields.",+ { cause },+ );+ });+ return visionSam;+}+/*** Vision results for one clip, computed per frame on demand and shared by* everything that reads the clip: physics, layout, signals. Unlike the@@ -139,6 +154,7 @@ export class ClipVision implements ClipSubjectSource {private sam2Frame = -1;private sam3Tracker?: Sam3Tracker;private sam3Frame = -1;+ private sam?: Promise<VisionSam>;private signalBundle?: VisionBundle;private publishedSignals = new Set<number>();private signalMaskFrame = -1;@@ -202,14 +218,7 @@ export class ClipVision implements ClipSubjectSource {Math.round(options.smoothing ?? (options.sam2 || options.sam3 ? 0 : 2)),);this.cell = Math.max(1, Math.round(options.maskCell ?? 8));- if (options.sam2) {- validateSam2Prompts(options.sam2.prompts);- this.wantSubjects = true;- }- if (options.sam3) {- validateSam3Prompts(options.sam3);- this.wantSubjects = true;- }+ if (options.sam2 || options.sam3) this.wantSubjects = true;}/** @internal Explicit selections must never fall back to another detected subject. */@@ -380,7 +389,18 @@ export class ClipVision implements ClipSubjectSource {return this.results.has(c) && (!this.wantSubjects || this.subjects.has(c));}+ /** Loads `@framefields/vision-sam` and validates the prompts once, on first use. */+ private prompted(): Promise<VisionSam> {+ this.sam ??= loadVisionSam().then((sam) => {+ if (this.options.sam2) sam.validateSam2Prompts(this.options.sam2.prompts);+ if (this.options.sam3) sam.validateSam3Prompts(this.options.sam3);+ return sam;+ });+ return this.sam;+ }+private async compute(from: number, to: number): Promise<void> {+ if (this.exact) await this.prompted();this.cacheLoaded ??= this.loadCache();await this.cacheLoaded;if (this.wantSubjects) {@@ -502,6 +522,7 @@ export class ClipVision implements ClipSubjectSource {if (settings.prompts.some((prompt) => prompt.frame >= frameCount)) {throw new Error("SAM 2 prompt frames must be inside the source video.");}+ const { Sam2Tracker } = await this.prompted();this.sam2Tracker = await Sam2Tracker.create({modelsDir: this.options.modelsDir,baseUrl: this.options.baseUrl,@@ -558,6 +579,7 @@ export class ClipVision implements ClipSubjectSource {if (settings.prompts.some((prompt) => prompt.frame >= frameCount)) {throw new Error("SAM 3 prompt frames must be inside the source video.");}+ const { Sam3Tracker } = await this.prompted();this.sam3Tracker = await Sam3Tracker.create({modelsDir: this.options.modelsDir,baseUrl: this.options.baseUrl,@@ -710,7 +732,7 @@ export class ClipVision implements ClipSubjectSource {const previous = bundle.getObjectResult(frame - 1).objects.find((object) => object.trackId === 1);- const object = sam2Object(+ const object = maskObject(mask,frame,this.compositionFps,@@ -841,6 +863,7 @@ export class ClipVision implements ClipSubjectSource {process.env.FRAMEFIELDS_VISION_CACHE_DIR ??path.join(os.homedir(), ".cache", "framefields", "vision");const o = this.options;+ const sam = this.exact ? await this.prompted() : undefined;const key = JSON.stringify([CACHE_VERSION,path.resolve(this.source),@@ -852,8 +875,8 @@ export class ClipVision implements ClipSubjectSource {this.cell,o.variant ?? "s",o.confidence ?? null,- ...(o.sam2 ? [[SAM2_TINY_REVISION, o.sam2]] : []),- ...(o.sam3 ? [[SAM3_REVISION, o.sam3]] : []),+ ...(o.sam2 ? [[sam?.SAM2_TINY_REVISION, o.sam2]] : []),+ ...(o.sam3 ? [[sam?.SAM3_REVISION, o.sam3]] : []),]);const hash = crypto.createHash("sha256")diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.tsindex 2a6a24d0..d9309cd4 100644--- a/packages/framefields/src/index.ts+++ b/packages/framefields/src/index.ts@@ -136,17 +136,14 @@ export {type VisionVariant,} from "@framefields/vision";-export {- type Sam2FrameResult,- type Sam2Options,- type Sam2Point,- type Sam2Prompt,- Sam2Tracker,- type Sam2TrackerOptions,- type Sam3Options,- type Sam3Prompt,- Sam3Tracker,- type Sam3TrackerOptions,+// Types only: the trackers ship in the optional @framefields/vision-sam peer (Meta SAM License).+export type {+ Sam2FrameResult,+ Sam2Options,+ Sam2Point,+ Sam2Prompt,+ Sam3Options,+ Sam3Prompt,} from "@framefields/vision-sam";import fs from "node:fs/promises";diff --git a/packages/vision-sam/src/index.ts b/packages/vision-sam/src/index.tsindex 0db26075..2dccbcca 100644--- a/packages/vision-sam/src/index.ts+++ b/packages/vision-sam/src/index.ts@@ -9,7 +9,6 @@export * from "./license.js";export * from "./models/registry.js";export * from "./sam2/memory.js";-export * from "./sam2/signal-results.js";export * from "./sam2/tracker.js";export * from "./sam2/types.js";export * from "./sam3/graphs.js";diff --git a/packages/vision/src/index.ts b/packages/vision/src/index.tsindex 0241805c..7ba5d173 100644--- a/packages/vision/src/index.ts+++ b/packages/vision/src/index.ts@@ -16,6 +16,7 @@ export * from "./runtime/node-webgpu-provider.js";export * from "./runtime/session-provider.js";export * from "./segmentation/clip-subject.js";export * from "./segmentation/subject.js";+export * from "./signals/mask-object.js";export * from "./signals/vision-bundle.js";export * from "./spatial/camera-space-transformer.js";export * from "./spatial/spatial-pin.js";diff --git a/packages/vision-sam/src/sam2/signal-results.ts b/packages/vision/src/signals/mask-object.tssimilarity index 78%rename from packages/vision-sam/src/sam2/signal-results.tsrename to packages/vision/src/signals/mask-object.tsindex e932c760..dafaae1d 100644--- a/packages/vision-sam/src/sam2/signal-results.ts+++ b/packages/vision/src/signals/mask-object.ts@@ -1,16 +1,8 @@-/**- * Copyright (c) Meta Platforms, Inc. and affiliates.- * All rights reserved.- *- * This source code is licensed under the SAM License found in the- * LICENSE file in the root directory of this source tree.- */-import type { TrackedObject } from "@framefields/core";-import type { InstanceMask } from "@framefields/vision";+import type { InstanceMask } from "../types.js";-/** Derive tracked-object geometry from the selected mask, never another person's detector. */-export function sam2Object(+/** Tracked-object geometry of one selected mask (bounds, center, velocity), with no detector involved. */+export function maskObject(mask: InstanceMask,frame: number,fps: number,diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 6c985f7f..602fea81 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -224,7 +224,7 @@ Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optiona### Prompted object tracking with SAM 2.1 Tiny and SAM 3 / 3.1-Use SAM for a specific object among similar objects, or to select an object via semantic text prompt (e.g. "performer in black suit"). SAM tracking and text selection run via `@framefields/vision-sam` (decoupled under Meta's SAM License). To use SAM at runtime, license acceptance is required via `acceptSamLicense: true`, `sam2: { acceptLicense: true, ... }`, `sam3: { acceptLicense: true, ... }`, or the environment variable `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`.+Use SAM for a specific object among similar objects, or to select an object via semantic text prompt (e.g. "performer in black suit"). SAM tracking and text selection run via `@framefields/vision-sam`, an optional peer under Meta's SAM License: install it next to `framefields` (`pnpm add @framefields/vision-sam`); `framefields` imports it only when a clip uses `sam2` or `sam3`, and prompts are validated on that first use. Import `Sam2Tracker`/`Sam3Tracker` from `@framefields/vision-sam`. To use SAM at runtime, license acceptance is required via `acceptSamLicense: true`, `sam2: { acceptLicense: true, ... }`, `sam3: { acceptLicense: true, ... }`, or the environment variable `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`.`comp.vision(layer, { sam2 })` runs the complete 512px SAM 2.1 Tiny video pipeline, including temporal memory. `comp.vision(layer, { sam3 })` runs SAM 3 / 3.1 quantized temporal tracking with optional CLIP/tokenizer-driven concept text selection. The pinned model assets download dynamically to the local cache (`~/.cache/framefields/models`) with size/SHA-256 integrity verification.diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 4baa159a..6ba382f7 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -3005,6 +3005,9 @@ importers:'@framefields/vision':specifier: workspace:*version: link:../vision+ '@framefields/vision-sam':+ specifier: workspace:*+ version: link:../vision-sam'@framefields/webgpu-renderers':specifier: workspace:*version: link:../webgpu-renderers@@ -3039,9 +3042,6 @@ importers:'@dimforge/rapier3d-deterministic-compat':specifier: 0.21.0version: 0.21.0- '@framefields/vision-sam':- specifier: workspace:*- version: link:../vision-samonnxruntime-web:specifier: ^1.24.0version: 1.30.0