Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(framefields): load @framefields/vision-sam lazily as an optional peer

okanasldac36943c770
diff --git a/.changeset/lazy-vision-sam.md b/.changeset/lazy-vision-sam.md
new file mode 100644
index 00000000..4c7e4477
--- /dev/null
+++ b/.changeset/lazy-vision-sam.md
@@ -0,0 +1,7 @@
+---
+"framefields": minor
+"@framefields/vision": minor
+"@framefields/vision-sam": minor
+---
+
+`@framefields/vision-sam` is now an optional peer of `framefields`, loaded only when a clip uses `sam2` or `sam3`, so the SDK no longer installs or imports SAM-licensed code by default. Install it next to `framefields` to use prompted tracking, and import `Sam2Tracker`/`Sam3Tracker` from `@framefields/vision-sam`. Mask-derived object geometry moved to `@framefields/vision` as `maskObject` (formerly `sam2Object`).
diff --git a/AGENTS.md b/AGENTS.md
index d67888b1..cf63f299 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -18,7 +18,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb
- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.
- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.
- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).
-- `packages/vision-sam`: segregated package under Meta SAM License for prompted SAM 2.1 Tiny and SAM 3 / 3.1 video tracking and text selection. `comp.vision(..., { sam2 | sam3 })` shares clip mattes and exposes object/mask/pose signals. Model weights download dynamically at runtime with license confirmation.
+- `packages/vision-sam`: SAM 2.1 Tiny / SAM 3 trackers under Meta's SAM License, an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.
- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).
- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.
- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).
diff --git a/CLAUDE.md b/CLAUDE.md
index e47c63d3..da680b7b 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,6 +18,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb
- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.
- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.
- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).
+- `packages/vision-sam`: SAM 2.1 Tiny / SAM 3 trackers under Meta's SAM License, an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.
- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).
- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.
- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).
diff --git a/README.md b/README.md
index 141fb6b4..e16b93d6 100644
--- a/README.md
+++ b/README.md
@@ -194,7 +194,11 @@ Installed components import only `framefields` and the shared toolkit in `framef
## Select and track a subject
-Use SAM 2.1 Tiny to select a specific object with a point or box and track it through video. Its masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts.
+Use SAM 2.1 Tiny to select a specific object with a point or box and track it through video. Its masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. SAM ships separately under Meta's SAM License, so install it next to framefields:
+
+```bash
+pnpm add @framefields/vision-sam
+```
```ts
const subject = Layer.video("assets/source.mp4", {
diff --git a/packages/framefields/package.json b/packages/framefields/package.json
index bd331f90..473db5ce 100644
--- a/packages/framefields/package.json
+++ b/packages/framefields/package.json
@@ -118,10 +118,17 @@
"yoga-layout": "^3.2.1",
"zod": "^4.3.6"
},
+ "peerDependencies": {
+ "@framefields/vision-sam": "workspace:*"
+ },
+ "peerDependenciesMeta": {
+ "@framefields/vision-sam": {
+ "optional": true
+ }
+ },
"optionalDependencies": {
"@dimforge/rapier2d-deterministic-compat": "0.21.0",
"@dimforge/rapier3d-deterministic-compat": "0.21.0",
- "@framefields/vision-sam": "workspace:*",
"onnxruntime-web": "^1.24.0"
},
"devDependencies": {
diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.ts
index 8756c75b..0bdb05ae 100644
--- a/packages/framefields/src/clip-vision.ts
+++ b/packages/framefields/src/clip-vision.ts
@@ -9,21 +9,17 @@ import {
type ClipSubjectSource,
COCO17_KEYPOINT_NAMES,
type InstanceMask,
+ maskObject,
mergeSubjectMask,
VisionBundle,
VisionRunner,
} from "@framefields/vision";
-import {
- SAM2_TINY_REVISION,
- SAM3_REVISION,
- type Sam2FrameResult,
- type Sam2Options,
+import type {
+ Sam2FrameResult,
+ Sam2Options,
Sam2Tracker,
- type Sam3Options,
+ Sam3Options,
Sam3Tracker,
- sam2Object,
- validateSam2Prompts,
- validateSam3Prompts,
} from "@framefields/vision-sam";
import { mediaDecoderCache } from "@framefields/webgpu-renderers";
import type { VisionProps } from "./effects/index.js";
@@ -105,6 +101,25 @@ const HOLD = 24;
const CACHE_VERSION = 1;
+type VisionSam = typeof import("@framefields/vision-sam");
+
+let visionSam: Promise<VisionSam> | undefined;
+
+/**
+ * SAM lives in `@framefields/vision-sam`, an optional peer under Meta's SAM License, so it
+ * is imported only when a clip asks for `sam2` or `sam3`.
+ */
+function loadVisionSam(): Promise<VisionSam> {
+ visionSam ??= import("@framefields/vision-sam").catch((cause: unknown) => {
+ visionSam = undefined;
+ throw new Error(
+ "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package (Meta SAM License). Install it next to framefields.",
+ { cause },
+ );
+ });
+ return visionSam;
+}
+
/**
* Vision results for one clip, computed per frame on demand and shared by
* everything that reads the clip: physics, layout, signals. Unlike the
@@ -139,6 +154,7 @@ export class ClipVision implements ClipSubjectSource {
private sam2Frame = -1;
private sam3Tracker?: Sam3Tracker;
private sam3Frame = -1;
+ private sam?: Promise<VisionSam>;
private signalBundle?: VisionBundle;
private publishedSignals = new Set<number>();
private signalMaskFrame = -1;
@@ -202,14 +218,7 @@ export class ClipVision implements ClipSubjectSource {
Math.round(options.smoothing ?? (options.sam2 || options.sam3 ? 0 : 2)),
);
this.cell = Math.max(1, Math.round(options.maskCell ?? 8));
- if (options.sam2) {
- validateSam2Prompts(options.sam2.prompts);
- this.wantSubjects = true;
- }
- if (options.sam3) {
- validateSam3Prompts(options.sam3);
- this.wantSubjects = true;
- }
+ if (options.sam2 || options.sam3) this.wantSubjects = true;
}
/** @internal Explicit selections must never fall back to another detected subject. */
@@ -380,7 +389,18 @@ export class ClipVision implements ClipSubjectSource {
return this.results.has(c) && (!this.wantSubjects || this.subjects.has(c));
}
+ /** Loads `@framefields/vision-sam` and validates the prompts once, on first use. */
+ private prompted(): Promise<VisionSam> {
+ this.sam ??= loadVisionSam().then((sam) => {
+ if (this.options.sam2) sam.validateSam2Prompts(this.options.sam2.prompts);
+ if (this.options.sam3) sam.validateSam3Prompts(this.options.sam3);
+ return sam;
+ });
+ return this.sam;
+ }
+
private async compute(from: number, to: number): Promise<void> {
+ if (this.exact) await this.prompted();
this.cacheLoaded ??= this.loadCache();
await this.cacheLoaded;
if (this.wantSubjects) {
@@ -502,6 +522,7 @@ export class ClipVision implements ClipSubjectSource {
if (settings.prompts.some((prompt) => prompt.frame >= frameCount)) {
throw new Error("SAM 2 prompt frames must be inside the source video.");
}
+ const { Sam2Tracker } = await this.prompted();
this.sam2Tracker = await Sam2Tracker.create({
modelsDir: this.options.modelsDir,
baseUrl: this.options.baseUrl,
@@ -558,6 +579,7 @@ export class ClipVision implements ClipSubjectSource {
if (settings.prompts.some((prompt) => prompt.frame >= frameCount)) {
throw new Error("SAM 3 prompt frames must be inside the source video.");
}
+ const { Sam3Tracker } = await this.prompted();
this.sam3Tracker = await Sam3Tracker.create({
modelsDir: this.options.modelsDir,
baseUrl: this.options.baseUrl,
@@ -710,7 +732,7 @@ export class ClipVision implements ClipSubjectSource {
const previous = bundle
.getObjectResult(frame - 1)
.objects.find((object) => object.trackId === 1);
- const object = sam2Object(
+ const object = maskObject(
mask,
frame,
this.compositionFps,
@@ -841,6 +863,7 @@ export class ClipVision implements ClipSubjectSource {
process.env.FRAMEFIELDS_VISION_CACHE_DIR ??
path.join(os.homedir(), ".cache", "framefields", "vision");
const o = this.options;
+ const sam = this.exact ? await this.prompted() : undefined;
const key = JSON.stringify([
CACHE_VERSION,
path.resolve(this.source),
@@ -852,8 +875,8 @@ export class ClipVision implements ClipSubjectSource {
this.cell,
o.variant ?? "s",
o.confidence ?? null,
- ...(o.sam2 ? [[SAM2_TINY_REVISION, o.sam2]] : []),
- ...(o.sam3 ? [[SAM3_REVISION, o.sam3]] : []),
+ ...(o.sam2 ? [[sam?.SAM2_TINY_REVISION, o.sam2]] : []),
+ ...(o.sam3 ? [[sam?.SAM3_REVISION, o.sam3]] : []),
]);
const hash = crypto
.createHash("sha256")
diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.ts
index 2a6a24d0..d9309cd4 100644
--- a/packages/framefields/src/index.ts
+++ b/packages/framefields/src/index.ts
@@ -136,17 +136,14 @@ export {
type VisionVariant,
} from "@framefields/vision";
-export {
- type Sam2FrameResult,
- type Sam2Options,
- type Sam2Point,
- type Sam2Prompt,
- Sam2Tracker,
- type Sam2TrackerOptions,
- type Sam3Options,
- type Sam3Prompt,
- Sam3Tracker,
- type Sam3TrackerOptions,
+// Types only: the trackers ship in the optional @framefields/vision-sam peer (Meta SAM License).
+export type {
+ Sam2FrameResult,
+ Sam2Options,
+ Sam2Point,
+ Sam2Prompt,
+ Sam3Options,
+ Sam3Prompt,
} from "@framefields/vision-sam";
import fs from "node:fs/promises";
diff --git a/packages/vision-sam/src/index.ts b/packages/vision-sam/src/index.ts
index 0db26075..2dccbcca 100644
--- a/packages/vision-sam/src/index.ts
+++ b/packages/vision-sam/src/index.ts
@@ -9,7 +9,6 @@
export * from "./license.js";
export * from "./models/registry.js";
export * from "./sam2/memory.js";
-export * from "./sam2/signal-results.js";
export * from "./sam2/tracker.js";
export * from "./sam2/types.js";
export * from "./sam3/graphs.js";
diff --git a/packages/vision/src/index.ts b/packages/vision/src/index.ts
index 0241805c..7ba5d173 100644
--- a/packages/vision/src/index.ts
+++ b/packages/vision/src/index.ts
@@ -16,6 +16,7 @@ export * from "./runtime/node-webgpu-provider.js";
export * from "./runtime/session-provider.js";
export * from "./segmentation/clip-subject.js";
export * from "./segmentation/subject.js";
+export * from "./signals/mask-object.js";
export * from "./signals/vision-bundle.js";
export * from "./spatial/camera-space-transformer.js";
export * from "./spatial/spatial-pin.js";
diff --git a/packages/vision-sam/src/sam2/signal-results.ts b/packages/vision/src/signals/mask-object.ts
similarity index 78%
rename from packages/vision-sam/src/sam2/signal-results.ts
rename to packages/vision/src/signals/mask-object.ts
index e932c760..dafaae1d 100644
--- a/packages/vision-sam/src/sam2/signal-results.ts
+++ b/packages/vision/src/signals/mask-object.ts
@@ -1,16 +1,8 @@
-/**
- * Copyright (c) Meta Platforms, Inc. and affiliates.
- * All rights reserved.
- *
- * This source code is licensed under the SAM License found in the
- * LICENSE file in the root directory of this source tree.
- */
-
import type { TrackedObject } from "@framefields/core";
-import type { InstanceMask } from "@framefields/vision";
+import type { InstanceMask } from "../types.js";
-/** Derive tracked-object geometry from the selected mask, never another person's detector. */
-export function sam2Object(
+/** Tracked-object geometry of one selected mask (bounds, center, velocity), with no detector involved. */
+export function maskObject(
mask: InstanceMask,
frame: number,
fps: number,
diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.md
index 6c985f7f..602fea81 100644
--- a/plugins/framefields/skills/framefields-effects/SKILL.md
+++ b/plugins/framefields/skills/framefields-effects/SKILL.md
@@ -224,7 +224,7 @@ Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optiona
### Prompted object tracking with SAM 2.1 Tiny and SAM 3 / 3.1
-Use SAM for a specific object among similar objects, or to select an object via semantic text prompt (e.g. "performer in black suit"). SAM tracking and text selection run via `@framefields/vision-sam` (decoupled under Meta's SAM License). To use SAM at runtime, license acceptance is required via `acceptSamLicense: true`, `sam2: { acceptLicense: true, ... }`, `sam3: { acceptLicense: true, ... }`, or the environment variable `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`.
+Use SAM for a specific object among similar objects, or to select an object via semantic text prompt (e.g. "performer in black suit"). SAM tracking and text selection run via `@framefields/vision-sam`, an optional peer under Meta's SAM License: install it next to `framefields` (`pnpm add @framefields/vision-sam`); `framefields` imports it only when a clip uses `sam2` or `sam3`, and prompts are validated on that first use. Import `Sam2Tracker`/`Sam3Tracker` from `@framefields/vision-sam`. To use SAM at runtime, license acceptance is required via `acceptSamLicense: true`, `sam2: { acceptLicense: true, ... }`, `sam3: { acceptLicense: true, ... }`, or the environment variable `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`.
`comp.vision(layer, { sam2 })` runs the complete 512px SAM 2.1 Tiny video pipeline, including temporal memory. `comp.vision(layer, { sam3 })` runs SAM 3 / 3.1 quantized temporal tracking with optional CLIP/tokenizer-driven concept text selection. The pinned model assets download dynamically to the local cache (`~/.cache/framefields/models`) with size/SHA-256 integrity verification.
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 4baa159a..6ba382f7 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -3005,6 +3005,9 @@ importers:
'@framefields/vision':
specifier: workspace:*
version: link:../vision
+ '@framefields/vision-sam':
+ specifier: workspace:*
+ version: link:../vision-sam
'@framefields/webgpu-renderers':
specifier: workspace:*
version: link:../webgpu-renderers
@@ -3039,9 +3042,6 @@ importers:
'@dimforge/rapier3d-deterministic-compat':
specifier: 0.21.0
version: 0.21.0
- '@framefields/vision-sam':
- specifier: workspace:*
- version: link:../vision-sam
onnxruntime-web:
specifier: ^1.24.0
version: 1.30.0