Code-first video & image, rendered natively on WebGPU.
feat(framefields): bundle SAM; publish nothing but framefields
okanasleeeae1eb6bca
diff --git a/.changeset/bundle-sam.md b/.changeset/bundle-sam.mdnew file mode 100644index 00000000..72319a4c--- /dev/null+++ b/.changeset/bundle-sam.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++SAM ships in `framefields`: `comp.vision(clip, { sam2 | sam3 })` works from a plain `npm install framefields`. The `@framefields/vision-sam` peer it asked for was never published, so prompted tracking failed for every npm user. The SAM code is a lazily loaded chunk, fetched only when a clip uses `sam2` or `sam3`; the trackers are exported as `framefields/sam`. No weights ship: they download on first use, SAM 3's only after you accept Meta's SAM License, as `NOTICE.md` describes.diff --git a/CLAUDE.md b/CLAUDE.mdindex aa83e95a..8082f79f 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -18,13 +18,14 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).-- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), an optional peer of `framefields`. `clip-vision.ts` imports it only for `comp.vision(..., { sam2 | sam3 })`; keep runtime imports of it out of `framefields` (types only). Pins in `src/models/registry.ts`.+- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), private; `framefields` bundles it as a lazy chunk (`clip-vision.ts` imports it dynamically for `comp.vision(..., { sam2 | sam3 })`; `index.ts` re-exports types only) and exposes the trackers as `framefields/sam`. No weights ship; `packages/framefields/NOTICE.md` covers their licenses. Pins in `src/models/registry.ts`.- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).- `apps/renderer-service`: HTTP render service (Docker, RunPod).- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.+- `framefields` is the only published package: it bundles every `@framefields/*` workspace package, and its build fails if `dist` still imports one (`scripts/check-bundled-imports.mjs`).- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.diff --git a/README.md b/README.mdindex 5a68f08c..1ee56b80 100644--- a/README.md+++ b/README.md@@ -194,11 +194,7 @@ Installed components import only `framefields` and the shared toolkit in `framef## Select and track a subject-Select a specific object with a point or box (SAM 2.1 Tiny or Small), or by description, several at once, or a whole group as one mask (SAM 3), and track it through video. Masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. SAM ships separately under Meta's SAM License, so install it next to framefields:--```bash-pnpm add @framefields/vision-sam-```+Select a specific object with a point or box (SAM 2.1 Tiny or Small), or by description, several at once, or a whole group as one mask (SAM 3), and track it through video. Masks, bounds, motion and optional matched pose feed the existing vision signals; the same clip masks drive cutouts. The SAM code ships in framefields and loads only when a clip uses it; model weights download on first use, and SAM 3's are under Meta's SAM License, which you accept explicitly (see `NOTICE.md`).```tsconst subject = Layer.video("assets/source.mp4", {diff --git a/packages/framefields/NOTICE.md b/packages/framefields/NOTICE.mdnew file mode 100644index 00000000..eac4655e--- /dev/null+++ b/packages/framefields/NOTICE.md@@ -0,0 +1,14 @@+# Notice: framefields++framefields is licensed under Apache-2.0. It contains no model weights: every vision model downloads on first use from a pinned revision into the local model cache (`~/.cache/framefields/models`) and is verified by size and SHA-256. The licenses below apply to those downloaded weights, not to framefields' code.++## Prompted tracking (SAM)++`comp.vision(clip, { sam2 | sam3 })` and `framefields/sam` track objects with Meta's Segment Anything models: SAM 2.1 (Tiny and Small) and SAM 3. That code is written for framefields and licensed under Apache-2.0; it contains no code or weights distributed by Meta. It loads only when a clip uses `sam2` or `sam3`.++| Model | Weights license | Acceptance |+| --- | --- | --- |+| SAM 2.1 Tiny / Small (Diffusion Studio ONNX exports of `facebook/sam2.1-hiera-*`) | Apache-2.0 | none |+| SAM 3 (ONNX exports of Meta's SAM 3 checkpoints) | Meta SAM License | required at runtime |++Before SAM 3 weights are downloaded or run, you must accept Meta's SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE): pass `acceptSamLicense: true` in `comp.vision` options (or `acceptLicense: true` to `Sam3Tracker`), or set `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Your use of SAM 3 is then governed by that license.diff --git a/packages/framefields/package.json b/packages/framefields/package.jsonindex ba0e3134..ef4497aa 100644--- a/packages/framefields/package.json+++ b/packages/framefields/package.json@@ -80,13 +80,20 @@"development": "./src/preview/index.ts","import": "./dist/preview/index.mjs","default": "./dist/preview/index.mjs"+ },+ "./sam": {+ "types": "./dist/sam/index.d.mts",+ "development": "./src/sam/index.ts",+ "import": "./dist/sam/index.mjs",+ "default": "./dist/sam/index.mjs"}},"files": [- "dist/**"+ "dist/**",+ "NOTICE.md"],"scripts": {- "build": "tsdown && node ../../scripts/sync-readme.mjs && node scripts/check-dts-bundle.mjs && node scripts/check-components-treeshake.mjs && node scripts/check-registry-types.mjs",+ "build": "tsdown && node ../../scripts/sync-readme.mjs && node scripts/check-bundled-imports.mjs && node scripts/check-components-treeshake.mjs && node scripts/check-registry-types.mjs","dev": "tsdown --watch","gallery": "tsx --conditions=development scripts/components-gallery.ts","clean": "rm -rf dist .turbo",@@ -94,6 +101,7 @@"test": "vitest run --passWithNoTests"},"dependencies": {+ "@huggingface/tokenizers": "^0.2.0","@lottiefiles/dotlottie-web": "^0.74.0","@mediabunny/mp3-encoder": "^1.61.6","@mediabunny/server": "^1.61.6",@@ -118,14 +126,6 @@"yoga-layout": "^3.2.1","zod": "^4.3.6"},- "peerDependencies": {- "@framefields/vision-sam": "*"- },- "peerDependenciesMeta": {- "@framefields/vision-sam": {- "optional": true- }- },"optionalDependencies": {"@dimforge/rapier2d-deterministic-compat": "0.21.0","@dimforge/rapier3d-deterministic-compat": "0.21.0",diff --git a/packages/framefields/scripts/check-bundled-imports.mjs b/packages/framefields/scripts/check-bundled-imports.mjsnew file mode 100644index 00000000..b3bad5de--- /dev/null+++ b/packages/framefields/scripts/check-bundled-imports.mjs@@ -0,0 +1,32 @@+// Fails the build when anything published in dist (code or types) still imports an+// @framefields/* workspace package. framefields bundles every one of them and publishes+// none, so such an import breaks at install or runtime (code) or silently becomes `any`+// (types). Declaring the package as a dependency or peer does not make it installable.+import { readdirSync, readFileSync } from "node:fs";+import { join } from "node:path";++const leaks = [];+const walk = (dir) => {+ for (const entry of readdirSync(dir, { withFileTypes: true })) {+ const file = join(dir, entry.name);+ if (entry.isDirectory()) walk(file);+ else if (/\.(mjs|d\.mts)$/.test(entry.name)) {+ const source = readFileSync(file, "utf8");+ for (const m of source.matchAll(+ /(?:from\s*|import\s*\(\s*|require\s*\(\s*)["'](@framefields\/[^"']+)["']/g,+ ))+ leaks.push(`${file}: ${m[1]}`);+ }+ }+};+walk("dist");++if (leaks.length > 0) {+ console.error(+ `dist imports unpublished @framefields packages:\n${leaks.join("\n")}`,+ );+ console.error(+ "Bundle them: keep them out of tsdown's `external` and in `dts.resolve`.",+ );+ process.exit(1);+}diff --git a/packages/framefields/scripts/check-dts-bundle.mjs b/packages/framefields/scripts/check-dts-bundle.mjsdeleted file mode 100644index d7fa07eb..00000000--- a/packages/framefields/scripts/check-dts-bundle.mjs+++ /dev/null@@ -1,39 +0,0 @@-// Fails the build when a published .d.mts still imports an unpublished-// @framefields/* package: consumers would silently get `any` for those types.-import { readdirSync, readFileSync } from "node:fs";-import { join } from "node:path";--const pkg = JSON.parse(readFileSync("package.json", "utf8"));-const declared = new Set([- ...Object.keys(pkg.dependencies ?? {}),- ...Object.keys(pkg.optionalDependencies ?? {}),- ...Object.keys(pkg.peerDependencies ?? {}),-]);--const leaks = [];-const walk = (dir) => {- for (const entry of readdirSync(dir, { withFileTypes: true })) {- const file = join(dir, entry.name);- if (entry.isDirectory()) walk(file);- else if (entry.name.endsWith(".d.mts")) {- for (const m of readFileSync(file, "utf8").matchAll(- /from "(@framefields\/[^"]+)"/g,- )) {- if (!declared.has(m[1])) {- leaks.push(`${file}: ${m[1]}`);- }- }- }- }-};-walk("dist");--if (leaks.length > 0) {- console.error(- `Type bundle imports unpublished packages:\n${leaks.join("\n")}`,- );- console.error(- "Declare @framefields/core (or the missing package) in the upstream package.json.",- );- process.exit(1);-}diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.tsindex fb846bd4..8eb57f08 100644--- a/packages/framefields/src/clip-vision.ts+++ b/packages/framefields/src/clip-vision.ts@@ -124,18 +124,9 @@ type VisionSam = typeof import("@framefields/vision-sam");let visionSam: Promise<VisionSam> | undefined;let clipVisionIds = 0;-/**- * SAM lives in `@framefields/vision-sam`, an optional peer (SAM 3's weights are under Meta's SAM License), so it- * is imported only when a clip asks for `sam2` or `sam3`.- */+/** SAM ships in its own chunk, loaded only when a clip asks for `sam2` or `sam3`. */function loadVisionSam(): Promise<VisionSam> {- visionSam ??= import("@framefields/vision-sam").catch((cause: unknown) => {- visionSam = undefined;- throw new Error(- "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package. Install it next to framefields.",- { cause },- );- });+ visionSam ??= import("@framefields/vision-sam");return visionSam;}diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.tsindex 840b02d9..1d70687a 100644--- a/packages/framefields/src/index.ts+++ b/packages/framefields/src/index.ts@@ -136,7 +136,7 @@ export {type VisionVariant,} from "@framefields/vision";-// Types only: the trackers ship in the optional @framefields/vision-sam peer.+// Types only, so SAM stays in its own lazily loaded chunk; the trackers are in `framefields/sam`.export type {Sam2FrameResult,Sam2Model,diff --git a/packages/framefields/src/sam/index.ts b/packages/framefields/src/sam/index.tsnew file mode 100644index 00000000..51c9987a--- /dev/null+++ b/packages/framefields/src/sam/index.ts@@ -0,0 +1,7 @@+/**+ * Prompted video tracking with SAM 2.1 (Tiny, Small) and SAM 3, for driving the+ * trackers directly; `comp.vision(clip, { sam2 | sam3 })` uses them for you. The+ * code is Apache-2.0; model weights download on first use, and SAM 3's need+ * Meta's SAM License accepted first (see NOTICE.md).+ */+export * from "@framefields/vision-sam";diff --git a/packages/framefields/tsdown.config.ts b/packages/framefields/tsdown.config.tsindex 7fccdd4d..c11c3374 100644--- a/packages/framefields/tsdown.config.ts+++ b/packages/framefields/tsdown.config.ts@@ -14,6 +14,7 @@ export default defineConfig({"src/fonts/index.ts","src/preview/index.ts","src/components/index.ts",+ "src/sam/index.ts","src/cli.ts","src/cli/check-worker.ts",],@@ -59,6 +60,6 @@ export default defineConfig({"pino","file-type","dotenv",- "@framefields/vision-sam",+ "@huggingface/tokenizers",],});diff --git a/packages/vision-sam/NOTICE.md b/packages/vision-sam/NOTICE.mdindex 7d83d900..9edb2884 100644--- a/packages/vision-sam/NOTICE.md+++ b/packages/vision-sam/NOTICE.md@@ -15,4 +15,4 @@ No model weights ship in this package or in any framefields package. Each modelBefore SAM 3 weights are downloaded or run, you must accept Meta's SAM License (https://github.com/facebookresearch/sam3/blob/main/LICENSE): pass `acceptLicense: true` to `Sam3Tracker` (or `acceptSamLicense: true` in `comp.vision` options), or set `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Your use of SAM 3 is then governed by that license.-`framefields` loads this package only when a clip uses `sam2` or `sam3`; it is an optional peer dependency, so projects that do not use SAM never install it.+`framefields` bundles this package (and ships this notice as its own `NOTICE.md`), loading it only when a clip uses `sam2` or `sam3`. Use the trackers directly through `framefields/sam`.diff --git a/packages/vision-sam/package.json b/packages/vision-sam/package.jsonindex 788a3d47..0423b6a9 100644--- a/packages/vision-sam/package.json+++ b/packages/vision-sam/package.json@@ -2,10 +2,7 @@"name": "@framefields/vision-sam","description": "Prompted video tracking for framefields: SAM 2.1 Tiny / Small and SAM 3 (text selection, several objects). SAM 3's model weights, downloaded at runtime, are under Meta's SAM License.","version": "2.9.2",- "private": false,- "publishConfig": {- "access": "public"- },+ "private": true,"type": "module","sideEffects": false,"license": "Apache-2.0",diff --git a/plugins/framefields/skills/framefields-vision/SKILL.md b/plugins/framefields/skills/framefields-vision/SKILL.mdindex b16f17a5..66f0ad52 100644--- a/plugins/framefields/skills/framefields-vision/SKILL.md+++ b/plugins/framefields/skills/framefields-vision/SKILL.md@@ -6,7 +6,7 @@ description: On-device computer vision in framefields — object detection and t# framefields-vision## Setup-Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Prompted tracking (SAM) also needs `@framefields/vision-sam` (see [SAM](#prompted-tracking-with-sam)). Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.+Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.Vision runs ONNX models locally. Nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked against their exact size and SHA-256, and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`).@@ -54,7 +54,7 @@ Video tracking at 1008², plus text ("concept") selection. Only the image encodeThat is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.-Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `@framefields/vision-sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.+Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `framefields/sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.## Backends@@ -151,13 +151,9 @@ Detection picks "the largest person". Use SAM when the subject is **a specific**- **SAM 3:** the subject has to be found from a description, or several people or objects must be tracked at once. It is the most expensive, so trim to the stretch you need.- Unsure between Tiny and Small: prototype with Tiny on a short trimmed stretch, inspect the masks, and switch to Small if edges, small parts or identity fail.-### Install and license+### License-```bash-pnpm add @framefields/vision-sam-```--`@framefields/vision-sam` is an optional peer of `framefields`, Apache-2.0 like the rest of framefields. It is loaded only when a clip uses `sam2` or `sam3`; without it, those clips fail with an install message. SAM 2.1's weights are Apache-2.0 and need no acceptance. SAM 3's weights are under Meta's SAM License, so `sam3` needs explicit acceptance before anything downloads: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `@framefields/vision-sam` when you drive SAM directly.+The SAM code ships in `framefields`, Apache-2.0 like the rest of it, and loads only when a clip uses `sam2` or `sam3`; nothing extra to install. SAM 2.1's weights are Apache-2.0 and need no acceptance. SAM 3's weights are under Meta's SAM License, so `sam3` needs explicit acceptance before anything downloads: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `framefields/sam` when you drive SAM directly.### Prompts@@ -250,7 +246,7 @@ Read the request's number before writing any vision code.- Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.- Setup (loading the models, then text selection on the prompt frame) takes about 2 minutes on an M1 Pro. Measured there: a 20-frame stretch with 6 tracks (hero + 5 others) renders in about 5 minutes; a 15 s clip at 25 fps is estimated at about 55 minutes. A CUDA GPU should be much faster; the CPU image encoder is about 9× slower than WebGPU.- Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.-- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`@framefields/vision-sam`) lists every file with its URL, size and SHA-256.+- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`framefields/sam`) lists every file with its URL, size and SHA-256.## Recipes@@ -310,7 +306,6 @@ report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]| SAM 3 text picks the wrong one | Name what is unique in **that** shot; or use a box prompt (`points` with labels 2/3) || Part of a fast-moving garment drops out | `variant: "m"`, or `keyBackground: true` || Faint halo on dark backgrounds | Lower `featherRadius`, or raise `maskThreshold` (e.g. `0.6`) |-| "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package" | `pnpm add @framefields/vision-sam` || "Meta SAM License agreement required" (SAM 3) | `acceptSamLicense: true` or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`, once the user accepts the license || "A prompted clip matte requires the source aspect ratio" | Matte at the source aspect, crop afterwards || Log says `Inference on the CPU` on a GPU machine | The log's `reason` says why; on Linux NVIDIA check the CUDA 12 / cuDNN 9 libraries, or force `FRAMEFIELDS_VISION_PROVIDER` to see the error |diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 6ba382f7..50aa5431 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -350,6 +350,31 @@ importers:specifier: ^4.1.11version: 4.1.11(@opentelemetry/[email protected])(@types/[email protected])(@vitest/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))+ examples/25_bakery:+ dependencies:+ '@framefields/compositions':+ specifier: workspace:*+ version: link:../../packages/compositions+ framefields:+ specifier: workspace:*+ version: link:../../packages/framefields+ mediabunny:+ specifier: ^1.61.6+ version: 1.61.6+ devDependencies:+ '@framefields/tsconfig':+ specifier: workspace:*+ version: link:../../packages/tsconfig+ '@types/node':+ specifier: ^24.1.0+ version: 24.10.14+ tsx:+ specifier: ^4.20.3+ version: 4.21.0+ typescript:+ specifier: ^5.8.3+ version: 5.9.3+examples/26_mac_keyboard_from_3d_primitives:dependencies:framefields:@@ -2767,6 +2792,9 @@ importers:packages/framefields:dependencies:+ '@huggingface/tokenizers':+ specifier: ^0.2.0+ version: 0.2.0'@lottiefiles/dotlottie-web':specifier: ^0.74.0version: 0.74.0