Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(framefields): bundle SAM; publish nothing but framefields

okanasleeeae1eb6bca
diff --git a/.changeset/bundle-sam.md b/.changeset/bundle-sam.md
new file mode 100644
index 00000000..72319a4c
--- /dev/null
+++ b/.changeset/bundle-sam.md
@@ -0,0 +1,5 @@
+---
+"framefields": minor
+---
+
+SAM ships in `framefields`: `comp.vision(clip, { sam2 | sam3 })` works from a plain `npm install framefields`. The `@framefields/vision-sam` peer it asked for was never published, so prompted tracking failed for every npm user. The SAM code is a lazily loaded chunk, fetched only when a clip uses `sam2` or `sam3`; the trackers are exported as `framefields/sam`. No weights ship: they download on first use, SAM 3's only after you accept Meta's SAM License, as `NOTICE.md` describes.
diff --git a/CLAUDE.md b/CLAUDE.md
index aa83e95a..8082f79f 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -18,13 +18,14 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb
- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.
- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.
- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).
-- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.
+- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), private; `framefields` bundles it as a lazy chunk (`clip-vision.ts` imports it dynamically for `comp.vision(..., { sam2 | sam3 })`; `index.ts` re-exports types only) and exposes the trackers as `framefields/sam`. No weights ship; `packages/framefields/NOTICE.md` covers their licenses. Pins in `src/models/registry.ts`.
- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).
- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.
- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).
- `apps/renderer-service`: HTTP render service (Docker, RunPod).
- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.
- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.
+- `framefields` is the only published package: it bundles every `@framefields/*` workspace package, and its build fails if `dist` still imports one (`scripts/check-bundled-imports.mjs`).
- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.
- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.
diff --git a/README.md b/README.md
index 5a68f08c..1ee56b80 100644
--- a/README.md
+++ b/README.md
@@ -194,11 +194,7 @@ Installed components import only `framefields` and the shared toolkit in `framef
## Select and track a subject
-Select a specific object with a point or box (SAM 2.1 Tiny or Small), or by description, several at once, or a whole group as one mask (SAM 3), and track it through video. Masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. SAM ships separately under Meta's SAM License, so install it next to framefields:
-
-```bash
-pnpm add @framefields/vision-sam
-```
+Select a specific object with a point or box (SAM 2.1 Tiny or Small), or by description, several at once, or a whole group as one mask (SAM 3), and track it through video. Masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. The SAM code ships in framefields and loads only when a clip uses it; model weights download on first use, and SAM 3's are under Meta's SAM License, which you accept explicitly (see `NOTICE.md`).
```ts
const subject = Layer.video("assets/source.mp4", {
diff --git a/packages/framefields/NOTICE.md b/packages/framefields/NOTICE.md
new file mode 100644
index 00000000..eac4655e
--- /dev/null
+++ b/packages/framefields/NOTICE.md
@@ -0,0 +1,14 @@
+# Notice: framefields
+
+framefields is licensed under Apache-2.0. It contains no model weights: every vision model downloads on first use from a pinned revision into the local model cache (`~/.cache/framefields/models`) and is verified by size and SHA-256. The licenses below apply to those downloaded weights, not to framefields' code.
+
+## Prompted tracking (SAM)
+
+`comp.vision(clip, { sam2 | sam3 })` and `framefields/sam` track objects with Meta's Segment Anything models: SAM 2.1 (Tiny and Small) and SAM 3. That code is written for framefields and licensed under Apache-2.0; it contains no code or weights distributed by Meta. It loads only when a clip uses `sam2` or `sam3`.
+
+| Model | Weights license | Acceptance |
+| --- | --- | --- |
+| SAM 2.1 Tiny / Small (Diffusion Studio ONNX exports of `facebook/sam2.1-hiera-*`) | Apache-2.0 | none |
+| SAM 3 (ONNX exports of Meta's SAM 3 checkpoints) | Meta SAM License | required at runtime |
+
+Before SAM 3 weights are downloaded or run, you must accept Meta's SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE): pass `acceptSamLicense: true` in `comp.vision` options (or `acceptLicense: true` to `Sam3Tracker`), or set `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Your use of SAM 3 is then governed by that license.
diff --git a/packages/framefields/package.json b/packages/framefields/package.json
index ba0e3134..ef4497aa 100644
--- a/packages/framefields/package.json
+++ b/packages/framefields/package.json
@@ -80,13 +80,20 @@
"development": "./src/preview/index.ts",
"import": "./dist/preview/index.mjs",
"default": "./dist/preview/index.mjs"
+ },
+ "./sam": {
+ "types": "./dist/sam/index.d.mts",
+ "development": "./src/sam/index.ts",
+ "import": "./dist/sam/index.mjs",
+ "default": "./dist/sam/index.mjs"
}
},
"files": [
- "dist/**"
+ "dist/**",
+ "NOTICE.md"
],
"scripts": {
- "build": "tsdown && node ../../scripts/sync-readme.mjs && node scripts/check-dts-bundle.mjs && node scripts/check-components-treeshake.mjs && node scripts/check-registry-types.mjs",
+ "build": "tsdown && node ../../scripts/sync-readme.mjs && node scripts/check-bundled-imports.mjs && node scripts/check-components-treeshake.mjs && node scripts/check-registry-types.mjs",
"dev": "tsdown --watch",
"gallery": "tsx --conditions=development scripts/components-gallery.ts",
"clean": "rm -rf dist .turbo",
@@ -94,6 +101,7 @@
"test": "vitest run --passWithNoTests"
},
"dependencies": {
+ "@huggingface/tokenizers": "^0.2.0",
"@lottiefiles/dotlottie-web": "^0.74.0",
"@mediabunny/mp3-encoder": "^1.61.6",
"@mediabunny/server": "^1.61.6",
@@ -118,14 +126,6 @@
"yoga-layout": "^3.2.1",
"zod": "^4.3.6"
},
- "peerDependencies": {
- "@framefields/vision-sam": "*"
- },
- "peerDependenciesMeta": {
- "@framefields/vision-sam": {
- "optional": true
- }
- },
"optionalDependencies": {
"@dimforge/rapier2d-deterministic-compat": "0.21.0",
"@dimforge/rapier3d-deterministic-compat": "0.21.0",
diff --git a/packages/framefields/scripts/check-bundled-imports.mjs b/packages/framefields/scripts/check-bundled-imports.mjs
new file mode 100644
index 00000000..b3bad5de
--- /dev/null
+++ b/packages/framefields/scripts/check-bundled-imports.mjs
@@ -0,0 +1,32 @@
+// Fails the build when anything published in dist (code or types) still imports an
+// @framefields/* workspace package. framefields bundles every one of them and publishes
+// none, so such an import breaks at install or runtime (code) or silently becomes `any`
+// (types). Declaring the package as a dependency or peer does not make it installable.
+import { readdirSync, readFileSync } from "node:fs";
+import { join } from "node:path";
+
+const leaks = [];
+const walk = (dir) => {
+ for (const entry of readdirSync(dir, { withFileTypes: true })) {
+ const file = join(dir, entry.name);
+ if (entry.isDirectory()) walk(file);
+ else if (/\.(mjs|d\.mts)$/.test(entry.name)) {
+ const source = readFileSync(file, "utf8");
+ for (const m of source.matchAll(
+ /(?:from\s*|import\s*\(\s*|require\s*\(\s*)["'](@framefields\/[^"']+)["']/g,
+ ))
+ leaks.push(`${file}: ${m[1]}`);
+ }
+ }
+};
+walk("dist");
+
+if (leaks.length > 0) {
+ console.error(
+ `dist imports unpublished @framefields packages:\n${leaks.join("\n")}`,
+ );
+ console.error(
+ "Bundle them: keep them out of tsdown's `external` and in `dts.resolve`.",
+ );
+ process.exit(1);
+}
diff --git a/packages/framefields/scripts/check-dts-bundle.mjs b/packages/framefields/scripts/check-dts-bundle.mjs
deleted file mode 100644
index d7fa07eb..00000000
--- a/packages/framefields/scripts/check-dts-bundle.mjs
+++ /dev/null
@@ -1,39 +0,0 @@
-// Fails the build when a published .d.mts still imports an unpublished
-// @framefields/* package: consumers would silently get `any` for those types.
-import { readdirSync, readFileSync } from "node:fs";
-import { join } from "node:path";
-
-const pkg = JSON.parse(readFileSync("package.json", "utf8"));
-const declared = new Set([
- ...Object.keys(pkg.dependencies ?? {}),
- ...Object.keys(pkg.optionalDependencies ?? {}),
- ...Object.keys(pkg.peerDependencies ?? {}),
-]);
-
-const leaks = [];
-const walk = (dir) => {
- for (const entry of readdirSync(dir, { withFileTypes: true })) {
- const file = join(dir, entry.name);
- if (entry.isDirectory()) walk(file);
- else if (entry.name.endsWith(".d.mts")) {
- for (const m of readFileSync(file, "utf8").matchAll(
- /from "(@framefields\/[^"]+)"/g,
- )) {
- if (!declared.has(m[1])) {
- leaks.push(`${file}: ${m[1]}`);
- }
- }
- }
- }
-};
-walk("dist");
-
-if (leaks.length > 0) {
- console.error(
- `Type bundle imports unpublished packages:\n${leaks.join("\n")}`,
- );
- console.error(
- "Declare @framefields/core (or the missing package) in the upstream package.json.",
- );
- process.exit(1);
-}
diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.ts
index fb846bd4..8eb57f08 100644
--- a/packages/framefields/src/clip-vision.ts
+++ b/packages/framefields/src/clip-vision.ts
@@ -124,18 +124,9 @@ type VisionSam = typeof import("@framefields/vision-sam");
let visionSam: Promise<VisionSam> | undefined;
let clipVisionIds = 0;
-/**
- * SAM lives in `@framefields/vision-sam`, an optional peer (SAM 3's weights are under Meta's SAM License), so it
- * is imported only when a clip asks for `sam2` or `sam3`.
- */
+/** SAM ships in its own chunk, loaded only when a clip asks for `sam2` or `sam3`. */
function loadVisionSam(): Promise<VisionSam> {
- visionSam ??= import("@framefields/vision-sam").catch((cause: unknown) => {
- visionSam = undefined;
- throw new Error(
- "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package. Install it next to framefields.",
- { cause },
- );
- });
+ visionSam ??= import("@framefields/vision-sam");
return visionSam;
}
diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.ts
index 840b02d9..1d70687a 100644
--- a/packages/framefields/src/index.ts
+++ b/packages/framefields/src/index.ts
@@ -136,7 +136,7 @@ export {
type VisionVariant,
} from "@framefields/vision";
-// Types only: the trackers ship in the optional @framefields/vision-sam peer.
+// Types only, so SAM stays in its own lazily loaded chunk; the trackers are in `framefields/sam`.
export type {
Sam2FrameResult,
Sam2Model,
diff --git a/packages/framefields/src/sam/index.ts b/packages/framefields/src/sam/index.ts
new file mode 100644
index 00000000..51c9987a
--- /dev/null
+++ b/packages/framefields/src/sam/index.ts
@@ -0,0 +1,7 @@
+/**
+ * Prompted video tracking with SAM 2.1 (Tiny, Small) and SAM 3, for driving the
+ * trackers directly; `comp.vision(clip, { sam2 | sam3 })` uses them for you. The
+ * code is Apache-2.0; model weights download on first use, and SAM 3's need
+ * Meta's SAM License accepted first (see NOTICE.md).
+ */
+export * from "@framefields/vision-sam";
diff --git a/packages/framefields/tsdown.config.ts b/packages/framefields/tsdown.config.ts
index 7fccdd4d..c11c3374 100644
--- a/packages/framefields/tsdown.config.ts
+++ b/packages/framefields/tsdown.config.ts
@@ -14,6 +14,7 @@ export default defineConfig({
"src/fonts/index.ts",
"src/preview/index.ts",
"src/components/index.ts",
+ "src/sam/index.ts",
"src/cli.ts",
"src/cli/check-worker.ts",
],
@@ -59,6 +60,6 @@ export default defineConfig({
"pino",
"file-type",
"dotenv",
- "@framefields/vision-sam",
+ "@huggingface/tokenizers",
],
});
diff --git a/packages/vision-sam/NOTICE.md b/packages/vision-sam/NOTICE.md
index 7d83d900..9edb2884 100644
--- a/packages/vision-sam/NOTICE.md
+++ b/packages/vision-sam/NOTICE.md
@@ -15,4 +15,4 @@ No model weights ship in this package or in any framefields package. Each model
Before SAM 3 weights are downloaded or run, you must accept Meta's SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE): pass `acceptLicense: true` to `Sam3Tracker` (or `acceptSamLicense: true` in `comp.vision` options), or set `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Your use of SAM 3 is then governed by that license.
-`framefields` loads this package only when a clip uses `sam2` or `sam3`; it is an optional peer dependency, so projects that do not use SAM never install it.
+`framefields` bundles this package (and ships this notice as its own `NOTICE.md`), loading it only when a clip uses `sam2` or `sam3`. Use the trackers directly through `framefields/sam`.
diff --git a/packages/vision-sam/package.json b/packages/vision-sam/package.json
index 788a3d47..0423b6a9 100644
--- a/packages/vision-sam/package.json
+++ b/packages/vision-sam/package.json
@@ -2,10 +2,7 @@
"name": "@framefields/vision-sam",
"description": "Prompted video tracking for framefields: SAM 2.1 Tiny / Small and SAM 3 (text selection, several objects). SAM 3's model weights, downloaded at runtime, are under Meta's SAM License.",
"version": "2.9.2",
- "private": false,
- "publishConfig": {
- "access": "public"
- },
+ "private": true,
"type": "module",
"sideEffects": false,
"license": "Apache-2.0",
diff --git a/plugins/framefields/skills/framefields-vision/SKILL.md b/plugins/framefields/skills/framefields-vision/SKILL.md
index b16f17a5..66f0ad52 100644
--- a/plugins/framefields/skills/framefields-vision/SKILL.md
+++ b/plugins/framefields/skills/framefields-vision/SKILL.md
@@ -6,7 +6,7 @@ description: On-device computer vision in framefields — object detection and t
# framefields-vision
## Setup
-Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Prompted tracking (SAM) also needs `@framefields/vision-sam` (see [SAM](#prompted-tracking-with-sam)). Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.
+Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.
Vision runs ONNX models locally. Nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked against their exact size and SHA-256, and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`).
@@ -54,7 +54,7 @@ Video tracking at 1008², plus text ("concept") selection. Only the image encode
That is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.
-Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `@framefields/vision-sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.
+Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `framefields/sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.
## Backends
@@ -151,13 +151,9 @@ Detection picks "the largest person". Use SAM when the subject is **a specific**
- **SAM 3:** the subject has to be found from a description, or several people or objects must be tracked at once. It is the most expensive, so trim to the stretch you need.
- Unsure between Tiny and Small: prototype with Tiny on a short trimmed stretch, inspect the masks, and switch to Small if edges, small parts or identity fail.
-### Install and license
+### License
-```bash
-pnpm add @framefields/vision-sam
-```
-
-`@framefields/vision-sam` is an optional peer of `framefields`, Apache-2.0 like the rest of framefields. It is loaded only when a clip uses `sam2` or `sam3`; without it, those clips fail with an install message. SAM 2.1's weights are Apache-2.0 and need no acceptance. SAM 3's weights are under Meta's SAM License, so `sam3` needs explicit acceptance before anything downloads: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `@framefields/vision-sam` when you drive SAM directly.
+The SAM code ships in `framefields`, Apache-2.0 like the rest of it, and loads only when a clip uses `sam2` or `sam3`; nothing extra to install. SAM 2.1's weights are Apache-2.0 and need no acceptance. SAM 3's weights are under Meta's SAM License, so `sam3` needs explicit acceptance before anything downloads: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `framefields/sam` when you drive SAM directly.
### Prompts
@@ -250,7 +246,7 @@ Read the request's number before writing any vision code.
- Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.
- Setup (loading the models, then text selection on the prompt frame) takes about 2 minutes on an M1 Pro. Measured there: a 20-frame stretch with 6 tracks (hero + 5 others) renders in about 5 minutes; a 15 s clip at 25 fps is estimated at about 55 minutes. A CUDA GPU should be much faster; the CPU image encoder is about 9× slower than WebGPU.
- Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.
-- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`@framefields/vision-sam`) lists every file with its URL, size and SHA-256.
+- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`framefields/sam`) lists every file with its URL, size and SHA-256.
## Recipes
@@ -310,7 +306,6 @@ report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]
| SAM 3 text picks the wrong one | Name what is unique in **that** shot; or use a box prompt (`points` with labels 2/3) |
| Part of a fast-moving garment drops out | `variant: "m"`, or `keyBackground: true` |
| Faint halo on dark backgrounds | Lower `featherRadius`, or raise `maskThreshold` (e.g. `0.6`) |
-| "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package" | `pnpm add @framefields/vision-sam` |
| "Meta SAM License agreement required" (SAM 3) | `acceptSamLicense: true` or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`, once the user accepts the license |
| "A prompted clip matte requires the source aspect ratio" | Matte at the source aspect, crop afterwards |
| Log says `Inference on the CPU` on a GPU machine | The log's `reason` says why; on Linux NVIDIA check the CUDA 12 / cuDNN 9 libraries, or force `FRAMEFIELDS_VISION_PROVIDER` to see the error |
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 6ba382f7..50aa5431 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -350,6 +350,31 @@ importers:
specifier: ^4.1.11
+ examples/25_bakery:
+ dependencies:
+ '@framefields/compositions':
+ specifier: workspace:*
+ version: link:../../packages/compositions
+ framefields:
+ specifier: workspace:*
+ version: link:../../packages/framefields
+ mediabunny:
+ specifier: ^1.61.6
+ version: 1.61.6
+ devDependencies:
+ '@framefields/tsconfig':
+ specifier: workspace:*
+ version: link:../../packages/tsconfig
+ '@types/node':
+ specifier: ^24.1.0
+ version: 24.10.14
+ tsx:
+ specifier: ^4.20.3
+ version: 4.21.0
+ typescript:
+ specifier: ^5.8.3
+ version: 5.9.3
+
examples/26_mac_keyboard_from_3d_primitives:
dependencies:
framefields:
@@ -2767,6 +2792,9 @@ importers:
packages/framefields:
dependencies:
+ '@huggingface/tokenizers':
+ specifier: ^0.2.0
+ version: 0.2.0
'@lottiefiles/dotlottie-web':
specifier: ^0.74.0
version: 0.74.0