Code-first video & image, rendered natively on WebGPU.
feat(plugins): add a framefields-vision skill
okanasl91f7a34d9c3f
diff --git a/.agents/skills/framefields-vision b/.agents/skills/framefields-visionnew file mode 120000index 00000000..b538761d--- /dev/null+++ b/.agents/skills/framefields-vision@@ -0,0 +1 @@+../../plugins/framefields/skills/framefields-vision\ No newline at end of filediff --git a/.claude/skills/framefields-vision b/.claude/skills/framefields-visionnew file mode 120000index 00000000..b538761d--- /dev/null+++ b/.claude/skills/framefields-vision@@ -0,0 +1 @@+../../plugins/framefields/skills/framefields-vision\ No newline at end of filediff --git a/AGENTS.md b/AGENTS.mdindex cf63f299..d007ec36 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -24,7 +24,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).- `apps/renderer-service`: HTTP render service (Docker, RunPod).- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.-- `plugins/framefields/skills/framefields{,-compose,-effects,-render}`: agent skills, symlinked from `.agents/skills` and `.Codex/skills`.+- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.Codex/skills`.- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.diff --git a/CLAUDE.md b/CLAUDE.mdindex da680b7b..1a8a8ef8 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -24,7 +24,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).- `apps/renderer-service`: HTTP render service (Docker, RunPod).- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.-- `plugins/framefields/skills/framefields{,-compose,-effects,-render}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.+- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.diff --git a/plugins/framefields/README.md b/plugins/framefields/README.mdindex fd922b1b..b22f09d3 100644--- a/plugins/framefields/README.md+++ b/plugins/framefields/README.md@@ -6,7 +6,8 @@ Agent skills for [framefields](https://github.com/gatewai-dev/framefields), a co| --- | --- || `framefields` | Starting a project: install from npm, scaffold a composition and render script, first verified render; generating images, video and audio with AI models (`ai`) || `framefields-compose` | Compositions, layer trees, layout, animation and easing, beat grids, film structure |-| `framefields-effects` | Post-processing effects, color grading, VFX, and on-device vision (tracking, segmentation, pose) |+| `framefields-effects` | Post-processing effects, color grading, VFX |+| `framefields-vision` | On-device vision: detection, segmentation, pose, cutouts, and prompted tracking with SAM 2.1 / SAM 3 || `framefields-render` | Headless rendering, frame-grid inspection, pixel probes, MP4 delivery checks |Skills load automatically when a task matches, for example *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*.diff --git a/plugins/framefields/plugin.json b/plugins/framefields/plugin.jsonindex 8107a1ba..e659ba8f 100644--- a/plugins/framefields/plugin.json+++ b/plugins/framefields/plugin.json@@ -29,6 +29,7 @@"Set up a framefields project","Compose and animate video","Apply effects and color grading",+ "Track people and cut out subjects","Render frames, grids, and MP4"],"websiteURL": "https://framefields.dev",diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.mdindex 2796ef46..819e3187 100644--- a/plugins/framefields/skills/framefields-compose/SKILL.md+++ b/plugins/framefields/skills/framefields-compose/SKILL.md@@ -487,7 +487,7 @@ stage.animate(LayerAnimation.create().signal("y", world.impact, { multiplier: 14- **Joints:** `world.pin(body, [x, y])` (swing around a point), `world.hinge(a, b, [x, y])`, `world.weld(a, b)`, `world.spring(a, bOrPoint, { frequency: 2, dampingRatio: 0.2, length })`, `world.rope(a, bOrPoint, { length })`. Points are world px.- **Driving the simulation:** `world.force(body, { x, y })` (px/s², mass-independent), `world.impulse(body, frame, [vx, vy])` (a kick), kinematic `follow: { x, y, rotation }`. Each of these takes a number, a signal (audio, `Signal.fromArray`) or `(frame) => value`: e.g. a floor that bounces on the kick.- **Force fields:** `world.field((s) => [ax, ay], { bodies })` is called every solver step with a body's center `s.x`, `s.y` (px), velocity `s.vx`, `s.vy` and `s.frame`; return px/s² or nothing. Wind, vortices, attractors, flow fields — and performers (below).-- **Performers in the simulation:** `const dancer = comp.vision("dancer.mp4")` gives the clip's pose and silhouette per frame, analysed on demand (see `framefields-effects`). Make the world **live** so it steps as frames render instead of all at once: `world.live(comp)` in place of `simulate()`. Before each frame draws, the clip's vision for that frame is computed (or read from the cache) and the world advances to it — nothing is analysed up front, the preview plays at once and fills in, and export gives the identical result (same simulation, same order). Then, with `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:+- **Performers in the simulation:** `const dancer = comp.vision("dancer.mp4")` gives the clip's pose and silhouette per frame, analysed on demand (see `framefields-vision`). Make the world **live** so it steps as frames render instead of all at once: `world.live(comp)` in place of `simulate()`. Before each frame draws, the clip's vision for that frame is computed (or read from the cache) and the world advances to it — nothing is analysed up front, the preview plays at once and fills in, and export gives the identical result (same simulation, same order). Then, with `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:- limbs as kinematic colliders that catch and bat things: `world.body({ x, y, width: 2 * r, height: 2 * r }, { type: "kinematic", shape: "circle", follow: { x: (f) => person.point("leftWrist", f).x - r, y: (f) => person.point("leftWrist", f).y - r } })`, a few balls along each shoulder → elbow → wrist;- the silhouette (dress included) as a force field: `person.sample(frame, x, y)` returns `alpha` (inside > 0.5) and a gradient `gx`, `gy` pointing inward, so `world.field((s) => { const m = person.sample(s.frame, s.x, s.y); … return [-m.gx * k, -m.gy * k]; })` makes particles and props flow around a moving body.- Reads are synchronous and need the frame analysed: inside a live world, while rendering, or after `await dancer.ready(frame)` (e.g. `ready(0)` to lay things out around the first pose). A live world can't look ahead, so place things from early frames.diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 8723055c..0c685f06 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -1,6 +1,6 @@---name: framefields-effects-description: Apply, configure, and modulate WebGPU post-processing shaders, stylized looks (pixel art, ASCII, VHS, comic, watercolor, risograph, holographic — and which look fits which request), cinematic color grading, tone mapping, procedural VFX, and on-device vision (object tracking, instance segmentation, pose, person mattes) in framefields. Use when adding blur, grain, LUTs, relighting, curves, greenscreen color keying, subject cutouts, or tracking-driven effects in framefields compositions.+description: Apply, configure, and modulate WebGPU post-processing shaders, stylized looks (pixel art, ASCII, VHS, comic, watercolor, risograph, holographic — and which look fits which request), cinematic color grading, tone mapping, and procedural VFX in framefields. Use when adding blur, grain, LUTs, relighting, curves, greenscreen color keying, or effects driven by vision signals in framefields compositions; detection, cutouts, pose and SAM tracking themselves are in framefields-vision.---# framefields-effects@@ -157,204 +157,9 @@ The effects behind them, for finer control:----## On-Device Vision+## Vision-driven effects-`framefields` runs vision models on the rendered frame and exposes the results as reactive signals. Models download lazily on first use and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`).--| Option | Model | Gives you |-| --- | --- | --- |-| `enableDetection` (default) | RTMDet-Ins | COCO-80 boxes, tracked over time → `vision.objects` |-| `enableSegmentation` | RTMDet-Ins (same pass) | Soft per-instance masks → `vision.masks`, `vision.segmentation` |-| `enablePose` | RTMO | 17 COCO keypoints per person → `vision.poseLandmarks`, `track.pose` |-| `enableMatte` | Selfie Segmenter | Fast person alpha → `vision.segmentation.matte` |--`variant: "t" | "s" | "m"` trades speed for accuracy (default `"s"`). `confidence` (default 0.3) and `classes: ["person", ...]` filter detections.--```typescript-// Whole composition: returns the reactive bundle-const vision = comp.withVision({ enableSegmentation: true, enablePose: true });--// One layer: renders through a node mode-Layer.video("assets/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });-```--Node modes: `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`. For `mask` / `matte` / `crop`, `matteSource: "instance"` (default; any COCO class, overlapping parts such as a dress merged into the subject) or `"selfie"` (people only, fastest).--### Choosing settings--Measured per 1280–2048 px frame on CPU (`onnxruntime-node`):--| Task | `t` | `s` (default) | `m` |-| --- | --- | --- | --- |-| detect / segment (one shared pass) | 150–300 ms | 200–340 ms | 420–700 ms |-| pose | ~50 ms | ~120 ms | ~280 ms |-| matte (Selfie Segmenter) | ~20 ms | ~20 ms | ~20 ms |--- Start with `"s"`. Use `"m"` for hero shots with fast motion blur or busy backgrounds; `"t"` mainly saves time on pose.-- Enabling both `enableDetection` and `enableSegmentation` costs one inference, not two.-- **Selfie matte only for close framing.** It is tuned for a person filling much of the frame: it misses distant figures and can report a "person" on close-ups with nobody in them. Keep `matteSource: "instance"` for anything else.-- Raise `confidence` (e.g. `0.5`) on abstract or stylized footage — at the default `0.3` the model will put loose labels ("teddy bear", "donut") on smoke, eyes and planets.-- Restrict `classes` when you only care about one thing; it also stops the subject from switching to another object when the person leaves frame.--### GPU or CPU--Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optional dependency of `framefields`, installed by default) on a hardware adapter, and on the CPU through `onnxruntime-node` otherwise: when `onnxruntime-web` is missing, the only adapter is a software one (lavapipe, SwiftShader), or WebGPU fails. On an Apple Silicon Mac the GPU roughly halves render time against the CPU. Layers that use the same `variant` share their models, and layers showing the same pixels share one detect/segment pass.--| Variable | Effect |-| --- | --- |-| `FRAMEFIELDS_VISION_PROVIDER` | `auto` (default: CUDA first on Linux with the NVIDIA driver loaded, then WebGPU, then CPU), `webgpu`, `cpu`, or `cuda` (onnxruntime-node's CUDA EP on Linux x64; needs the CUDA 12 and cuDNN 9 libraries). A forced backend that fails throws instead of falling back. |-| `FRAMEFIELDS_VISION_THREADS` | CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default. |--### How it behaves at render time--- **One-frame delay.** Vision reads the layer's *previous* rendered frame, so the very first frame has no results (a cutout renders transparent) and masks trail the plate by one frame. This is invisible at normal playback speed. A matte of a clip the composition also reads with `comp.vision` has neither (see Clip Vision below).-- **Render vision frames in order.** `renderVideo` does this for you. For stills, render at least two consecutive frames with the same renderer and keep the last one. Jumping straight to a later frame (or using a frame grid) cuts out pixels from whatever frame was rendered before it, which shows up as a ghosted double of the subject.-- **Subject choice.** `mask` / `matte` / `crop` use the largest person; with no person, the most confident instance. Instances overlapping the subject and no more than twice its size are merged in (a dress, a held instrument) — large containers around it (a tunnel, a window frame) are not. `classes` limits what the subject can be, not what merges into it: COCO has no clothing class, so a flowing dress often comes back as a "kite" or "umbrella" and still joins a `classes: ["person"]` cutout.-- Each vision layer tracks objects independently; track ids from two layers are unrelated.--### Troubleshooting--| Symptom | Fix |-| --- | --- |-| Part of a fast-moving garment drops out of the cutout | `variant: "m"`, or `keyBackground: true` to grow the subject into connected foreground |-| Cutout is the wrong object | Set `classes: ["person"]` (or the class you want) |-| Faint halo around the cutout on dark backgrounds | Lower `featherRadius`, or raise `maskThreshold` (e.g. `0.6`) |-| Renders offline / in CI | Pre-download with `runner.preload([...])` into `$FRAMEFIELDS_MODELS_DIR`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror |-| Check the models themselves | `await runner.preload([...])` on a fresh models directory; it downloads each model and verifies its SHA-256 (about 380 MB for all of them) |--### Prompted object tracking with SAM 2.1 Tiny and SAM 3 / 3.1--Use SAM for a specific object among similar objects, or to select an object via semantic text prompt (e.g. "performer in black suit"). SAM tracking and text selection run via `@framefields/vision-sam`, an optional peer under Meta's SAM License: install it next to `framefields` (`pnpm add @framefields/vision-sam`); `framefields` imports it only when a clip uses `sam2` or `sam3`, and prompts are validated on that first use. Import `Sam2Tracker`/`Sam3Tracker` from `@framefields/vision-sam`. To use SAM at runtime, license acceptance is required via `acceptSamLicense: true`, `sam2: { acceptLicense: true, ... }`, `sam3: { acceptLicense: true, ... }`, or the environment variable `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`.--`comp.vision(layer, { sam2 })` runs the complete 512px SAM 2.1 Tiny video pipeline, including temporal memory. `comp.vision(layer, { sam3 })` runs SAM 3 / 3.1 temporal tracking with optional CLIP/tokenizer-driven concept text selection. The pinned model assets download dynamically to the local cache (`~/.cache/framefields/models`) with size/SHA-256 integrity verification, and only the image encoder the backend runs fastest is fetched: fp16 on WebGPU (Macs, the preview) and CUDA, q4 on the CPU (`SAM3_ENCODER`; override with `sam3.precision: "fp16" | "q4f16" | "q4"`). Every SAM 3 clip in a process shares that encoder and its result per frame, so tracking several clips of one source costs one encoder pass per frame (about 3.6 s at 1008² on an M1 Pro; CPU-only tracking is roughly 10× slower).--```ts-const subject = Layer.video("assets/source.mp4", {- width: 1920, height: 1080, fit: "contain", muted: false,-}).withVision({ mode: "matte" });-comp.add(subject);--// Option A: SAM 3 with semantic text selection-const tracked = comp.vision(subject, {- acceptSamLicense: true,- sam3: {- text: "performer in black suit",- category: "person",- },- pose: true, // optional RTMO pass matched to the selected cutout-});--// Option B: SAM 2.1 or SAM 3 with point/box prompts-const trackedBox = comp.vision(subject, {- acceptSamLicense: true,- sam2: {- category: "person",- prompts: [- { frame: 0, points: [- { x: 0.25, y: 0.10, label: 2 }, // box top-left- { x: 0.70, y: 0.95, label: 3 }, // box bottom-right- { x: 0.48, y: 0.50, label: 1 }, // foreground point- ] },- // Additional prompt frames correct tracking or re-seed memory- ],- },-});--// Option C: several instances at once — one SAM 3 pipeline, tracks 1…N by score-const crowd = comp.vision("assets/source.mp4", {- acceptSamLicense: true,- sam3: { text: "student in blazer", category: "person", maxObjects: 6 },-});-crowd.signals.objects.get(2); // second-best match; points prompts can add or correct one: { frame, object: 3, points }--const signals = tracked.signals; // stable VisionBundle; request before rendering-const target = signals.objects.get(1);-// target.bounds, anchors, confidence, active, kinematics and pose use the existing signals API.-await tracked.ready(0);-const mask = signals.segmentation.matte.at(0); // source-sized selected-object alpha-```--Coordinates are normalized to the original source image, before layer placement. Labels are foreground `1`, background `0`, and box corners `2`/`3`. Prompt frames are nonnegative increasing integers on the source analysis clock (`options.fps`, default composition FPS), before trimming/time remapping. SAM propagates forward from the earliest prompt; frames before it are empty. Corrections reset the conditioning memory. Reset/re-prompt across cuts or when identity is lost; appearance tracking cannot guarantee re-identification after occlusion. `sam2` and `sam3` are mutually exclusive on a single clip.--Text prompts are concepts: describe what sets the target apart ("man with shaved head", not "man in white shirt" in a crowd of white shirts). With `maxObjects`, one encoder pass per frame serves every track; each track keeps its own memory, overlaps go to the more confident track, and the clip's matte/subject is all tracks merged. A later text prompt re-selects every track; a points prompt with `object: k` seeds or corrects only track k (k = next unused ID adds one). SAM masks do not merge other detected people. Default smoothing is zero and a missing object stays empty. Pose is disabled unless explicitly requested; SAM itself provides no skeleton. `tracked.person` continues to expose silhouette sampling and optional matched pose; `.place(...)` maps those into a layer's layout. The `tracked.signals` bundle uses source geometry on the composition clock; object and mask signals refer to track ID `1`. Full masks are reconstructed lazily from compressed history, avoiding an uncompressed mask per frame.--Frame requests analyze intervening source frames in order, so a seek forward can take time; a backward seek, hold, reverse, and time-remapped playback reuse source-frame results. When analysis FPS is lower than playback FPS, intermediate frames reuse the preceding analysis mask. Source masks are persisted separately from low-resolution silhouette/pose data; cache identity includes all prompts and the model revision. `pnpm --filter @framefields/vision-sam test:sam2` and `test:sam3` run real-model tests on native Dawn (requires GPU access; downloads are cached). A matte/mask/crop layer of the same clip reuses the selected masks through the existing clip-matte link. Use the default matte settings and matching source aspect ratio for that link; crop after source-space matting. An explicit selection fails clearly instead of falling back to another detected person when the mask cannot align. `tracked.release()` frees SAM sessions, keeps results readable, and replays the source memory if further uncached frames are requested.--### 1. Subject Sandwich ("Text Behind Subject")-Cuts out the foreground subject from footage and sandwiches typography or graphics directly behind them:--```typescript-comp.addSubjectSandwich({- source: "assets/dancer.mp4",- behind: [- Layer.text("HEADLINE BEHIND", {- fontSize: 120,- fill: "#FF5A1F",- fontWeight: 900- })- ],- feather: 4,- fit: "cover"-});-```--### 2. Smart Re-Framing (16:9 to 9:16 Auto-Crop)-Smoothly reframes landscape video into vertical shorts by following a tracked subject with virtual camera damping:-```typescript-const vision = comp.withVision({ enableDetection: true });-comp.addSmartFraming({- source: "assets/action.mp4",- target: vision.objects.primary,- targetAspect: 9 / 16,- damping: 0.15,- leadHeadroom: 0.1-});-```--### 3. Subject Outline Glow & Neon Pulse-Strokes the segmented subject boundary with an audio-reactive contour glow:-```typescript-const vision = comp.withVision({ enableSegmentation: true });-comp.addSubjectOutline(vision.segmentation.subject, {- source: "assets/character.mp4",- color: "#FF5A1F",- width: 6,- blur: 12-});-```--### 4. Tracked Region Blur-Blurs faces, license plates, or any detected class by following a live track:-```typescript-const vision = comp.withVision({ classes: ["person"] });-layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });-```--### 5. Clip Vision: Results Before the Frame Renders-`withVision` analyses a layer's *previous* rendered frame, so its results trail by a frame and exist only while rendering. When something must know the motion *at* the frame — physics, layout, keyframes — use `comp.vision`, which analyses the clip's own frames on demand:-```typescript-const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)-const person = dancer.person.place({ width: W, height: H, fit: "cover" });-person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed over ±2 frames-person.signals("nose").x; // a signal for .signal("x", …)-person.sample(48, x, y); // silhouette alpha + inward gradient (dress and held things included)-await dancer.ready(0); // analyse a frame (or a range: ready(0, 120)) to read it at build time-```-- The composition analyses each frame's clips just before drawing it, in every render path and the preview, and a live physics world (`world.live(comp)`) before stepping. Nothing runs up front; a preview starts at once.-- Results persist in Node (`~/.cache/framefields/vision`, or `$FRAMEFIELDS_VISION_CACHE_DIR`; `cacheDir: false` to opt out), keyed by the file and settings, so later runs and renders after a preview are instant.-- Options: `startFrame` / `trimStartSec` (where the clip plays), `pose`, `mask`, `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. When the person is lost the last pose is held for up to a second.-- A `withVision({ mode: "matte" })` cutout (or `mask` / `crop`) of the same clip reuses these results: the composition hands it the clip's subject for the frame being drawn, so it is segmented once, not again at render, and the cutout lines up with the plate on the same frame. Layers with `classes`, `keyBackground`, `matteSource: "selfie"` or other model settings than the clip keep their own pass.-- Analysis costs about 200 ms per frame on CPU (segmentation dominates); on WebGPU it is faster. In a live preview the playhead waits for frames not analysed yet.--### 6. Analyze Before Authoring-One-shot, ffmpeg-free report of what is in a clip (tracks, classes, mask coverage):-```typescript-const report = await comp.analyzeVisionSequence("assets/street.mp4", {- tasks: ["detect", "pose"],- categories: ["person"]-});-report.tracks; // [{ trackId, category, frames: [start, end], centerPath, ... }]-```+Cutouts, subject mattes, tracked boxes, pose, smart re-framing, subject outlines, tracked region blur, and prompted tracking of a specific person or several people (SAM 2.1 / SAM 3) live in the `framefields-vision` skill. Effects here consume its signals: for example `layer.blurRegion(vision.objects.byCategory("person"))`, or an effect uniform driven by `clip.signals.objects.get(1).bounds`.---diff --git a/plugins/framefields/skills/framefields-vision/SKILL.md b/plugins/framefields/skills/framefields-vision/SKILL.mdnew file mode 100644index 00000000..30703446--- /dev/null+++ b/plugins/framefields/skills/framefields-vision/SKILL.md@@ -0,0 +1,282 @@+---+name: framefields-vision+description: On-device computer vision in framefields — object detection and tracking, instance segmentation, pose, person mattes and cutouts (RTMDet-Ins, RTMO, Selfie Segmenter), and prompted video tracking of a specific object or several people with SAM 2.1 Tiny (point/box prompts) and SAM 3 (text prompts such as "man in white shirt", multiple objects at once). Covers which API to use (withVision, comp.vision, analyzeVisionSequence), every pinned ONNX model with sizes and licenses, GPU/CUDA/CPU backends, model downloads and caching, performance, and troubleshooting. Use when a framefields composition needs to know where people or objects are, cut a subject out, follow someone through a clip, or select one person in a crowd.+---++# framefields-vision++## Setup+Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Prompted tracking (SAM) also needs `@framefields/vision-sam` (see [SAM](#prompted-tracking-with-sam)). Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.++Vision runs ONNX models locally. Nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked against their exact size and SHA-256, and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`).++## Which API++| Need | API | Timing |+| --- | --- | --- |+| Cut out, box, or draw a skeleton on a layer while rendering | `layer.withVision({ mode })` | Reads the layer's **previous** rendered frame: results trail by one frame and exist only while rendering |+| Signals for the whole composition (boxes, masks, pose) | `comp.withVision({ … })` | Same, on the composited frame |+| Know the motion **at** a frame (physics, layout, keyframes), or reuse results across renders | `comp.vision(clipOrLayer, options)` | Analyses the clip's own frames on demand, before they draw; persisted on disk |+| Follow **one specific** object or person, or select by description | `comp.vision(…, { sam2 })` / `{ sam3 }` | As above, with SAM's temporal memory |+| Find out what is in a clip before authoring | `comp.analyzeVisionSequence(path, …)` | One-shot report |++## Models++### Core models (Apache-2.0, bundled with `framefields`)++| Task | Model | `t` | `s` (default) | `m` | Input |+| --- | --- | --- | --- | --- | --- |+| Detect + segment (one shared pass), COCO-80 | RTMDet-Ins | 24 MB | 43 MB | 116 MB | 640² |+| Pose, 17 COCO keypoints | RTMO | 27 MB (416²) | 40 MB | 89 MB | 640² |+| Person matte | MediaPipe Selfie Segmenter | square 0.5 MB (256²), landscape 0.5 MB (256×144) for wide frames | | | |++About 380 MB for all of them. `variant: "t" | "s" | "m"` picks the size.++### SAM 2.1 Tiny (`sam2`, Apache-2.0)+Meta SAM 2.1 Hiera Tiny, Diffusion Studio's 512 px fp16 video export: vision encoder 58 MB, mask decoder 9 MB, memory attention 13 MB, memory encoder 3 MB, pointer encoding and constants (under 0.1 MB). About 83 MB in total. Point and box prompts only.++### SAM 3 (`sam3`, Meta SAM License)+Video tracking at 1008², plus text ("concept") selection. Only the image encoder the backend runs fastest is downloaded:++| Graph | Size | Source |+| --- | --- | --- |+| Image encoder, **fp16** (WebGPU, CUDA) | 935 MB | `onnx-community/sam3-tracker-ONNX` |+| Image encoder, **q4** (CPU) | 369 MB | same |+| Image encoder, q4f16 (opt-in) | 296 MB | same |+| Mask decoder, memory encoder, memory attention | 18 + 6 + 35 MB | `jax-image-tools/sam3-video-onnx` |+| Dense-prompt and pointer heads | 67 MB | `RamRom/sam3-onnx` |+| Text selection: image encoder (int4, fp16 attention), text encoder (int8), decoder (int8), tokenizer | 293 + 357 + 27 + 4 MB | `danilobukvic/sam3-text-onnx` |++That is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.++Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `@framefields/vision-sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.++## Backends++Inference picks its backend once per process, before any model downloads:++| Machine | Backend |+| --- | --- |+| Mac (Apple Silicon) | WebGPU through `onnxruntime-web` on the Metal device |+| Linux with an NVIDIA GPU | CUDA through `onnxruntime-node` when the NVIDIA driver is loaded and the CUDA 12 / cuDNN 9 libraries are present, else WebGPU (Vulkan), else CPU |+| No usable GPU (CI, lavapipe, SwiftShader) | CPU through `onnxruntime-node` |+| The preview (`startPreview`) | WebGPU in the browser; models are served from the local cache by the preview server |++| Variable | Effect |+| --- | --- |+| `FRAMEFIELDS_VISION_PROVIDER` | `auto` (default; the order above), `webgpu`, `cuda` or `cpu`. A forced backend that fails throws instead of falling back. |+| `FRAMEFIELDS_VISION_THREADS` | CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default. |+| `FRAMEFIELDS_MODELS_DIR` / `FRAMEFIELDS_MODELS_BASE_URL` | Model cache directory, or a mirror to download from (offline and CI renders) |+| `FRAMEFIELDS_VISION_CACHE_DIR` | Where `comp.vision` persists results (default `~/.cache/framefields/vision`) |++The renderer logs `[vision] Inference on the GPU (WebGPU)`, or the backend it fell back to.++## Detection, segmentation, pose, mattes++| Option | Model | Gives you |+| --- | --- | --- |+| `enableDetection` (default) | RTMDet-Ins | COCO-80 boxes, tracked over time → `vision.objects` |+| `enableSegmentation` | RTMDet-Ins (same pass) | Soft per-instance masks → `vision.masks`, `vision.segmentation` |+| `enablePose` | RTMO | 17 COCO keypoints per person → `vision.poseLandmarks`, `track.pose` |+| `enableMatte` | Selfie Segmenter | Fast person alpha → `vision.segmentation.matte` |++```typescript+// Whole composition: returns the reactive bundle+const vision = comp.withVision({ enableSegmentation: true, enablePose: true });++// One layer: renders through a node mode+Layer.video("assets/dancer.mp4").withVision({ mode: "matte", enableSegmentation: true });+```++Node modes: `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`. For `mask` / `matte` / `crop`, `matteSource: "instance"` (default; any COCO class, overlapping parts such as a dress merged into the subject) or `"selfie"` (people only, fastest). `confidence` (default 0.3) and `classes: ["person", …]` filter detections.++Measured per 1280–2048 px frame on CPU (`onnxruntime-node`); the GPU is faster:++| Task | `t` | `s` | `m` |+| --- | --- | --- | --- |+| detect / segment (one shared pass) | 150–300 ms | 200–340 ms | 420–700 ms |+| pose | ~50 ms | ~120 ms | ~280 ms |+| matte (Selfie Segmenter) | ~20 ms | ~20 ms | ~20 ms |++- Start with `"s"`. Use `"m"` for hero shots with fast motion blur or busy backgrounds; `"t"` mainly saves time on pose.+- **Use the selfie matte only for close framing.** It misses distant figures and can report a "person" in close-ups with nobody in them.+- Raise `confidence` (e.g. `0.5`) on abstract or stylized footage. At 0.3 the model puts loose labels ("teddy bear", "donut") on smoke, eyes and planets.+- Restrict `classes` when you only care about one thing. It also stops the subject switching to another object when the person leaves frame.++At render time:+- **One-frame delay.** `withVision` reads the previous rendered frame, so the first frame has no results (a cutout renders transparent) and masks trail the plate by a frame. A matte of a clip the composition also reads with `comp.vision` has neither problem.+- **Render vision frames in order.** `renderVideo` does. For stills, render at least two consecutive frames with the same renderer and keep the last; jumping to a later frame shows a ghosted double of the subject.+- **Subject choice** for `mask` / `matte` / `crop`: the largest person, else the most confident instance. Instances overlapping it and no more than twice its size are merged in (a dress, a held instrument); large containers (a tunnel, a window frame) are not. `classes` limits what the subject can be, not what merges into it.+- Each vision layer tracks independently; track IDs from two layers are unrelated. Layers with the same `variant` share models, and layers showing the same pixels share one detect/segment pass.++## Clip vision (`comp.vision`)++Use `comp.vision` when something must know the motion **at** a frame:++```typescript+const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)+const person = dancer.person.place({ width: W, height: H, fit: "cover" });+person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed over ±2 frames+person.signals("nose").x; // a signal for .signal("x", …)+person.sample(48, x, y); // silhouette alpha + inward gradient+await dancer.ready(0); // analyse a frame (or ready(0, 120)) to read it at build time+```++- The composition analyses each frame's clips just before drawing it, in every render path and the preview. Nothing runs up front.+- Results persist in Node, keyed by the file and settings (`cacheDir: false` opts out). Later renders and renders after a preview are instant.+- Options: `startFrame` / `trimStartSec` (where the clip plays; pass `trimStartSec` yourself when you give a path instead of the layer), `fps` (analysis rate), `pose`, `mask`, `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. A lost person's last pose is held for up to a second.+- A `withVision({ mode: "matte" })` cutout (or `mask` / `crop`) of the same clip reuses the clip's subject for the frame being drawn: segmented once, aligned with the plate. Layers with `classes`, `keyBackground`, `matteSource: "selfie"` or different model settings keep their own pass. With several `comp.vision` clips of one source, the matte links to the **first one created**.++## Prompted tracking with SAM++Detection picks "the largest person". Use SAM when the subject is **a specific** person or object, especially in a crowd, or when you can only describe it.++| | SAM 2.1 Tiny (`sam2`) | SAM 3 (`sam3`) |+| --- | --- | --- |+| Select by | points and boxes | points, boxes, or **text** |+| Objects per clip | 1 | 1 to `maxObjects` |+| Resolution | 512² | 1008² |+| Cost per frame, M1 Pro | ~0.3 s | ~3.6 s image encoder (shared by every SAM 3 clip of the source) + ~0.7 s per tracked object |+| Download | ~83 MB | ~1.75 GB (GPU) / ~1.2 GB (CPU) |+| License | Apache-2.0 | Meta SAM License |++### Install and license++```bash+pnpm add @framefields/vision-sam+```++`@framefields/vision-sam` is an optional peer of `framefields`. It is loaded only when a clip uses `sam2` or `sam3`; without it, those clips fail with an install message. The package's code is under Meta's SAM License (the SAM 2.1 weights themselves are Apache-2.0), so both `sam2` and `sam3` need explicit acceptance: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam2` / `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Import `Sam2Tracker` / `Sam3Tracker` from `@framefields/vision-sam` when you drive SAM directly.++### Prompts++```typescript+const plate = Layer.video("assets/source.mp4", { width: 1920, height: 1080, fit: "contain" })+ .withVision({ mode: "matte" });+comp.add(plate);++// SAM 3: one person by description+const hero = comp.vision(plate, {+ acceptSamLicense: true,+ sam3: { category: "person", text: "man in white shirt" },+ pose: true, // optional RTMO pass matched to the selection+});++// SAM 3: everyone else at once, tracks 1…N by score+const crowd = comp.vision("assets/source.mp4", {+ acceptSamLicense: true,+ sam3: { category: "person", text: "student in blazer", maxObjects: 5 },+});++// SAM 2.1: a box plus a point, and a correction later+const ball = comp.vision(plate, {+ acceptSamLicense: true,+ sam2: {+ category: "ball",+ prompts: [+ { frame: 0, points: [+ { x: 0.40, y: 0.55, label: 2 }, // box top-left+ { x: 0.48, y: 0.66, label: 3 }, // box bottom-right+ { x: 0.44, y: 0.60, label: 1 }, // foreground point+ ] },+ { frame: 120, points: [{ x: 0.61, y: 0.52, label: 1 }] },+ ],+ },+});++const target = hero.signals.objects.get(1); // request signals before rendering+Layer.box({+ position: "absolute", x: 0, y: 0, width: 1, height: 1,+ background: "transparent", borderWidth: 6, borderColor: "#00ffc3",+}).animate(+ LayerAnimation.create()+ .signal("x", target.bounds.screenX)+ .signal("y", target.bounds.screenY)+ .signal("width", target.bounds.screenWidth)+ .signal("height", target.bounds.screenHeight)+ .signal("opacity", target.active),+);+```++- **Coordinates** are normalized to the source image, before layout. Labels: `1` foreground, `0` background, `2` / `3` box corners.+- **Prompt frames** are increasing integers on the source analysis clock, before trimming. SAM propagates forward from the first prompt; earlier frames are empty and are not analysed. To track only a stretch of a long clip, trim the layer and prompt on the stretch's first source frame.+- **Text prompts are concepts.** Name what sets the target apart in that shot. In a crowd where everyone shows a white collar, "man in white shirt" picks a student. In the same film's wide shot, where only he has no blazer, it picks him. Check the first frame of every prompt.+- **Several objects** (`maxObjects`): one encoder pass per frame serves every track, and each track keeps its own memory. Where tracks overlap, the more confident one keeps the pixel. `signals.objects.get(k)` and `signals.masks` carry tracks `1…N`; the clip's matte and subject are all tracks merged. A later text prompt re-selects every track (IDs renumber); a points prompt with `object: k` seeds or corrects track `k` only (the next unused ID adds one).+- Two text-prompted clips of one shot can claim the same person (a "student" track on the hero). Prefer prompts that exclude each other.+- **Corrections and cuts.** A prompt on a later frame resets that object's memory. Re-prompt after hard cuts or long occlusions; appearance tracking cannot guarantee re-identification.+- An explicit selection never falls back to another detected person. If its mask cannot align with a matte layer (a different aspect ratio), the render fails. Matte at the source aspect ratio and crop afterwards.+- SAM gives masks, not skeletons: `pose: true` matches an RTMO pose to the selection. Default smoothing is 0, and an absent object stays empty.+- `clip.release()` frees the SAM sessions. Analysed frames stay readable.++### Cost and caching++- Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.+- Setup (loading the models, then text selection on the prompt frame) takes about 2 minutes on an M1 Pro. Measured there: a 20-frame stretch with 6 tracks (hero + 5 others) renders in about 5 minutes; a 15 s clip at 25 fps is estimated at about 55 minutes. A CUDA GPU should be much faster; the CPU image encoder is about 9× slower than WebGPU.+- Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.+- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`@framefields/vision-sam`) lists every file with its URL, size and SHA-256.++## Recipes++### Subject sandwich (text behind the subject)+```typescript+comp.addSubjectSandwich({+ source: "assets/dancer.mp4",+ behind: [Layer.text("HEADLINE BEHIND", { fontSize: 120, fill: "#FF5A1F", fontWeight: 900 })],+ feather: 4,+ fit: "cover",+});+```++### Smart re-framing (16:9 to 9:16)+```typescript+const vision = comp.withVision({ enableDetection: true });+comp.addSmartFraming({+ source: "assets/action.mp4",+ target: vision.objects.primary,+ targetAspect: 9 / 16,+ damping: 0.15,+ leadHeadroom: 0.1,+});+```++### Subject outline glow+```typescript+const vision = comp.withVision({ enableSegmentation: true });+comp.addSubjectOutline(vision.segmentation.subject, {+ source: "assets/character.mp4",+ color: "#FF5A1F",+ width: 6,+ blur: 12,+});+```++### Tracked region blur (faces, plates)+```typescript+const vision = comp.withVision({ classes: ["person"] });+layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });+```++### Analyse before authoring+```typescript+const report = await comp.analyzeVisionSequence("assets/street.mp4", {+ tasks: ["detect", "pose"],+ categories: ["person"],+});+report.tracks; // [{ trackId, category, frames: [start, end], centerPath, … }]+```++## Troubleshooting++| Symptom | Fix |+| --- | --- |+| Cutout or box is on the wrong person | `classes: ["person"]`; for a specific person use SAM |+| SAM 3 text picks the wrong one | Name what is unique in **that** shot; or use a box prompt (`points` with labels 2/3) |+| Part of a fast-moving garment drops out | `variant: "m"`, or `keyBackground: true` |+| Faint halo on dark backgrounds | Lower `featherRadius`, or raise `maskThreshold` (e.g. `0.6`) |+| "Prompted sam2/sam3 tracking needs the optional @framefields/vision-sam package" | `pnpm add @framefields/vision-sam` |+| "Meta SAM License agreement required" | `acceptSamLicense: true` or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1` |+| "A prompted clip matte requires the source aspect ratio" | Matte at the source aspect, crop afterwards |+| Log says `Inference on the CPU` on a GPU machine | The log's `reason` says why; on Linux NVIDIA check the CUDA 12 / cuDNN 9 libraries, or force `FRAMEFIELDS_VISION_PROVIDER` to see the error |+| A preview of a clip outside the project shows nothing | Add its folder to `startPreview(…, { root: [process.cwd(), dir] })` |+| Renders offline or in CI | Copy the model cache, or set `FRAMEFIELDS_MODELS_BASE_URL` |+| Check the models themselves | `await runner.preload([...])` on a fresh `FRAMEFIELDS_MODELS_DIR`; SAM: `pnpm --filter @framefields/vision-sam test:sam2` / `test:sam3` on a GPU machine |diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex d4c9bcd0..231c16ba 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -1,6 +1,6 @@---name: framefields-description: Start a new framefields video project, or add framefields to an existing Node.js project. Use when the user wants to make a video with framefields and the project has no framefields setup yet, or asks how to install framefields, scaffold a composition and render script, get from an empty folder to a first MP4, generate images, video, music or voice for a film with AI models (`ai` from framefields, the framefields AI gateway), or add ready-made components (pointers, captions, title cards, device frames …) with `npx framefields add`. Hands off to framefields-compose, framefields-effects, and framefields-render for the details.+description: Start a new framefields video project, or add framefields to an existing Node.js project. Use when the user wants to make a video with framefields and the project has no framefields setup yet, or asks how to install framefields, scaffold a composition and render script, get from an empty folder to a first MP4, generate images, video, music or voice for a film with AI models (`ai` from framefields, the framefields AI gateway), or add ready-made components (pointers, captions, title cards, device frames …) with `npx framefields add`. Hands off to framefields-compose, framefields-effects, framefields-vision, and framefields-render for the details.---# framefields@@ -13,7 +13,8 @@ This skill takes a project from nothing to a verified first render. For anything| Task | Skill || --- | --- || Scenes, layout, typography, animation, beat grids, film structure | `framefields-compose` |-| Post-processing, color grading, VFX, on-device vision (tracking, cutouts, pose) | `framefields-effects` |+| Post-processing, color grading, VFX | `framefields-effects` |+| On-device vision: detection, cutouts, pose, tracking a specific person or several people (SAM 2.1 / SAM 3) | `framefields-vision` || Frame inspection, frame grids, pixel tests, video export and QA | `framefields-render` || Ready-made components (`npx framefields add`) | this skill, section 8 || Generating images, video, music, voice with AI models (`ai`) | this skill, section 6 |@@ -384,7 +385,7 @@ Coding agents interacting with the AI gateway can discover endpoints, inspect cr## 8. Grow the film-Once the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading, grain, and vision-driven effects, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.+Once the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading and grain, `framefields-vision` for cutouts, pose and tracking people, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.### Ready-made componentsdiff --git a/scripts/validate-agent-plugin.mjs b/scripts/validate-agent-plugin.mjsindex e444a90f..128dae41 100644--- a/scripts/validate-agent-plugin.mjs+++ b/scripts/validate-agent-plugin.mjs@@ -36,6 +36,7 @@ const SKILL_NAMES = ["framefields-compose","framefields-effects","framefields-render",+ "framefields-vision",];const PORTABLE_FIELDS = new Set(["$schema",