Code-first video & image, rendered natively on WebGPU.
docs(skills): concise skills; cloud rendering gets its own skill
okanasl70741114da33
diff --git a/.agent/rules/general-information.md b/.agent/rules/general-information.mdindex 30d8489a..e7af8616 100644--- a/.agent/rules/general-information.md+++ b/.agent/rules/general-information.md@@ -32,7 +32,7 @@ Framefields is a code-first video engine on native WebGPU (Dawn) in Node, with n- `nodes/node-*` (59): one VFX/audio/layout node each. Layout: `src/shared/` (config and schema), `src/renderers/` (WebGPU), `src/index.ts`. Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (`packages/compositions/src/program/skillmd-check.test.ts` guards it).- `apps/renderer-service`: HTTP render service (Docker, RunPod worker).- `examples/NN_*`: reference films/ads that import `framefields` via workspace. Study `19_framefields_film` (best structure), `22_framefields_launch` (lyric sync), and `28_physics_dancer` (physics, live vision, GPU particles) before writing a new film.-- `plugins/framefields/skills/framefields{,-compose,-effects,-render}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.+- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision,-cloud}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.- `examples/22_framefields_launch`: reference for a lyric-driven launch film.- Package `exports` have a `development` condition pointing at `src/`. Otherwise consumers resolve `dist/`, so run `pnpm build` after changing a dependency.diff --git a/.agents/skills/framefields-cloud b/.agents/skills/framefields-cloudnew file mode 120000index 00000000..6b32b087--- /dev/null+++ b/.agents/skills/framefields-cloud@@ -0,0 +1 @@+../../plugins/framefields/skills/framefields-cloud\ No newline at end of filediff --git a/.claude/skills/framefields-cloud b/.claude/skills/framefields-cloudnew file mode 120000index 00000000..6b32b087--- /dev/null+++ b/.claude/skills/framefields-cloud@@ -0,0 +1 @@+../../plugins/framefields/skills/framefields-cloud\ No newline at end of filediff --git a/AGENTS.md b/AGENTS.mdindex 49f75ea9..792aa60e 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -24,7 +24,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`).- `apps/renderer-service`: HTTP render service (Docker, RunPod).- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.-- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.Codex/skills`.+- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision,-cloud}`: agent skills, symlinked from `.agents/skills` and `.Codex/skills`.- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.diff --git a/CLAUDE.md b/CLAUDE.mdindex 44b1614b..9e2ab493 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -26,7 +26,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`). Effect renderers declare `reach` and `maxPixelScale` (node-sdk) when they can: without `maxPixelScale` a scaled-up leaf using the effect stays at layout resolution.- `apps/renderer-service`: HTTP render service (Docker, RunPod). It renders single jobs, including the pieces of a split render (`probe`, `range` chunks, `stitch`); the API repo (`/Users/okanaslankan/framefields-dev/api`, `src/render/`) plans, dispatches and bills them.- `examples/NN_*`: reference films. Study `19_framefields_film` (structure), `22_framefields_launch` (lyric sync), `28_physics_dancer` (physics, vision, GPU particles) before writing one.-- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.+- `plugins/framefields/skills/framefields{,-compose,-effects,-render,-vision,-cloud}`: agent skills, symlinked from `.agents/skills` and `.claude/skills`.- `framefields` is the only published package: it bundles every `@framefields/*` workspace package, and its build fails if `dist` still imports one (`scripts/check-bundled-imports.mjs`).- Package `exports` have a `development` condition pointing at `src/`; otherwise `dist/`, so `pnpm build` after changing a dependency.- Docs site is a separate repo: `/Users/okanaslankan/framefields-dev/docs` (Astro Starlight). `docs/lab` pins the published `framefields` version and builds the component demos; deploy with `pnpm run deploy`.diff --git a/plugins/framefields/README.md b/plugins/framefields/README.mdindex b22f09d3..2fc1d934 100644--- a/plugins/framefields/README.md+++ b/plugins/framefields/README.md@@ -8,7 +8,8 @@ Agent skills for [framefields](https://github.com/gatewai-dev/framefields), a co| `framefields-compose` | Compositions, layer trees, layout, animation and easing, beat grids, film structure || `framefields-effects` | Post-processing effects, color grading, VFX || `framefields-vision` | On-device vision: detection, segmentation, pose, cutouts, and prompted tracking with SAM 2.1 / SAM 3 |-| `framefields-render` | Headless rendering, frame-grid inspection, pixel probes, MP4 delivery checks |+| `framefields-render` | Headless rendering, frame-grid inspection, pixel probes, live preview, MP4 delivery checks |+| `framefields-cloud` | Rendering on framefields cloud GPUs (`framefields render --cloud`, `renderCloud`), cost plans, split renders, cloud error codes |Skills load automatically when a task matches, for example *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*.diff --git a/plugins/framefields/skills/framefields-cloud/SKILL.md b/plugins/framefields/skills/framefields-cloud/SKILL.mdnew file mode 100644index 00000000..0b955d58--- /dev/null+++ b/plugins/framefields/skills/framefields-cloud/SKILL.md@@ -0,0 +1,95 @@+---+name: framefields-cloud+description: Render framefields projects on framefields cloud GPUs (api.framefields.dev) or a self-hosted render worker. Use when running `framefields render --cloud`, calling `renderCloud` from `framefields/cloud`, planning cost and workers, splitting a long film across several GPUs, starting renders from the framefields dashboard, getting a project render-ready (push, lockfile, pinned assets, private repos), or fixing a cloud render error code such as ASSET_NOT_FOUND or LOCKFILE_MISSING.+---++# framefields-cloud++The cloud renders the **pushed commit** of a project on framefields' GPUs and can split a long film across several. Local rendering, checks and named renders in `framefields.json` are in `framefields-render`; project setup is in `framefields`.++## Make the project render-ready++The worker clones the repo, installs from the lockfile, fetches assets and calls the render's export. So:++- **Commit and push first.** Uncommitted changes aren't rendered (the CLI warns); a commit not on the remote, a folder outside git, or a repo without a remote is refused.+- **Commit the lockfile** and pin `packageManager`. The worker renders with the project's own framefields version (2.11+ for split renders, 2.12+ for named renders). bun lockfiles can't be installed by the worker.+- **Every input must reach the worker:** pinned with `framefields assets add` (preferred), tracked in Git LFS and pushed, or committed. Other `.gitignore`d files don't exist there.+- Build asset paths from `import.meta.url`, never `process.cwd()`. Nothing may write into `node_modules` while rendering.+- **Builders stay pure:** no rendering, generation or writes while building. `npx framefields check <name>` runs the same import locally; run it before every cloud render.+- **Private repos:** the user connects GitHub once at https://framefields.dev/dashboard/settings#connections (the framefields GitHub App, ideally on all repositories). Public repos need nothing.+- **Key:** it bills the account behind `FRAMEFIELDS_API_KEY` (https://framefields.dev/dashboard/keys), the same key as the AI gateway. Ask the user to set it in the environment; never print, write or embed it.++Pinned assets: the CLI and `renderCloud` upload any the bucket lacks and sign reads for remotes without a `publicUrl` (24 h; keys stay local). Workers download and cache them by hash; R2 downloads are free, unlike LFS bandwidth. Dashboard renders have nobody to sign, so they only read remotes with a `publicUrl`.++## CLI++Run from the project folder (where `framefields.json` and `package.json` are; in a monorepo that folder is sent as the workdir).++```bash+npx framefields render film --cloud --plan # workers, time, estimated cost; renders nothing+npx framefields render film --cloud # render, then download to the render's output+npx framefields render --cloud # the only named render+npx framefields render src/film.ts --cloud --export buildFilm --workers auto # a module, no manifest+```++- Flags override the named render's fields: `--workers <n|auto>`, `--max-workers <n>`, `--codec h264|vp9`, `--quality`, `--select`, `--output`, `--export`, `--json` (final status as JSON).+- `--workers` defaults to the render's `workers`, else 1. `auto` measures frame cost on one worker, then adds workers only while they stay efficient: a 5-minute film stays on one, a 3-hour film spreads to `--max-workers` (API cap 10).+- **Show the user `--plan` before a long or expensive render.** It prints e.g. `Plan: 4 workers, 1 h 50 min of render work, ready in about 35 min, estimated $1.20.`+- Output is one line per phase (`probing`, rendering N chunks and the audio, `stitching`, `Done in …: <file>, <GPU seconds>, <cost>.`), with progress and cost so far at most every 10 s. Run it in the background and let it finish; Ctrl-C cancels (used GPU time is charged).++## From code++No GPU needed locally:++```ts+import { CloudRenderError, renderCloud } from "framefields/cloud";++try {+ const result = await renderCloud({+ render: "film", // or { entry: "src/film.ts", export: "buildFilm", workers: "auto" }+ outputPath: "output/film.mp4",+ onProgress: (s) => console.log(s.status, s.progress.frames, s.progress.totalFrames, s.costUsd),+ // dryRun: true // plan only: result.plan, result.estimateUsd+ });+ console.log(result.filePath, result.gpuSeconds, result.costUsd);+} catch (err) {+ if (err instanceof CloudRenderError) console.error(err.code, err.message);+ else throw err;+}+```++Options override the named render's; `signal` cancels. `cloudClient({ apiKey })` exposes `start`, `status`, `cancel`. HTTP: `POST /v1/render` (`source`, `target`, `render`, `workers`, `max_workers`, `dry_run`, `name`, `title`), `GET` / `DELETE /v1/render/:id`, `Authorization: Bearer <key>`.++## Billing and the dashboard++- A credit hold is placed at start (raised to the estimate once planned); the charge is the GPU seconds used, the rest is released. Failed and cancelled renders charge what they used.+- `insufficient_credits`: the user adds funds at https://framefields.dev/dashboard/billing. Don't retry.+- https://framefields.dev/dashboard/renders shows every render with live progress, cost, Cancel and the video. **New render** lists the named renders of a connected repo's branch (greyed out when `entry` isn't on that branch), with Plan and Render.++## Errors++The error carries the worker's code. Fix the project, commit, push, render again.++| Code | Fix |+|---|---|+| `REPO_NOT_FOUND`, `GIT_AUTH_FAILED` | Connect GitHub (Settings → Connections) with access to this repo. |+| `REF_NOT_FOUND`, `COMMIT_MISMATCH` | `git push`. |+| `LOCKFILE_MISSING`, `INSTALL_FAILED` | Commit a lockfile that installs cleanly. |+| `ENTRY_NOT_FOUND`, `EXPORT_NOT_FOUND`, `EXPORT_INVALID` | Fix `entry`/`export`; `framefields check <name>`. |+| `COMPOSITION_BUILD_FAILED`, `EXPORT_SIDE_EFFECT` | The builder throws or renders/writes; `framefields check` reproduces it. |+| `ASSET_NOT_FOUND`, `LFS_OBJECT_MISSING`, `LFS_POINTER_UNRESOLVED` | `git lfs ls-files`, `git lfs push --all origin`; un-ignore needed assets. |+| `ASSET_NOT_UPLOADED`, `ASSET_HASH_MISMATCH` | `npx framefields assets push` from the machine with the right file. |+| `ASSET_PRIVATE`, `ASSET_FORBIDDEN` | Dashboard render of a remote without a working `publicUrl`: render from the CLI or fix `publicUrl`. |+| `ASSET_MANIFEST_INVALID` | `npx framefields assets status` shows the error. |+| `ENGINE_UNSUPPORTED` | Upgrade framefields (2.11+) or use `--workers 1`. |+| `RENDER_TIMEOUT`, `RENDER_OOM` | `--workers auto` or more workers. |++Infrastructure errors (`GIT_NETWORK`, `RENDER_STALLED`, a lost worker) are retried by the API, and a straggling chunk gets a backup worker.++A stitched video with a jump at a chunk seam means a stateful effect reads wall-clock time; keep effects deterministic in the frame number.++## Self-hosted render worker++The same worker runs on your own GPU (Docker, RunPod pod or serverless): it renders a pushed commit with the rules above and uploads the result. A failed job returns `{ code, hint, retryable }`; resubmit unchanged only when `retryable` is true. Job format: https://framefields.dev/docs/guides/render-service/++To split a render yourself, see **Splitting a long render** in `framefields-render`.diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.mdindex 71c2523c..37e7499d 100644--- a/plugins/framefields/skills/framefields-compose/SKILL.md+++ b/plugins/framefields/skills/framefields-compose/SKILL.md@@ -5,20 +5,11 @@ description: Design, structure, and animate code-first video compositions using# framefields-compose-## Setup-Import everything from the [`framefields`](https://www.npmjs.com/package/framefields) npm package. If the project doesn't have framefields set up yet, follow the `framefields` skill first (install, project scaffold, first render).+Import everything from [`framefields`](https://www.npmjs.com/package/framefields) (no project yet: the `framefields` skill). A composition is a tree of layout nodes (Yoga flexbox, GPU vector text, 3D, physics, particles, signals) compiled into a deterministic `(composition, frame) → pixels`.-## Overview-`framefields` is a code-first, deterministic video SDK on native WebGPU (Dawn) in Node.js: Yoga flexbox layout, GPU vector text, 3D, physics, particles, and signal-driven animation.+## Structure-A composition is a tree of layout nodes compiled into a deterministic program: `(composition, frame) → pixels`.-------## Architecture & Core Concepts--### 1. Composition Lifecycle-A `Composition` instance defines canvas dimensions, frame rate, background, registered fonts, global audio tracks, and the layer hierarchy:+### Composition```typescriptimport { Composition, Layer, FontManager } from "framefields";@@ -38,14 +29,13 @@ comp.add(Layer.box({ /* ... */ }));comp.add(Layer.audio("assets/score.mp3", { volume: 1, durationFrames: 300 }));```-### 2. Scene Architecture (Production Pattern)-For commercials, films, and multi-shot videos, break the project into discrete, modular scene functions using scene-local frames:+### Scenes (films, ads, multi-shot)-- **Tokens first (`theme.ts`):** Centralize color palette, typography, easing curves, and a tempo-locked beat grid (`BEAT = Math.round(fps * 60 / BPM)`). Never use arbitrary margins or ad-hoc hex codes.-- **Scene modules (`scenes/hero.ts`):** Each scene returns a full-frame root box with `startFrame` and `durationFrames`. Children position themselves relative to scene start (`frame: 0` = scene onset).-- **Master assembly (`film.ts`):** Imports scenes, mounts them to `comp.add(scene())`, attaches the master soundtrack and composition-level grading/grain.+- **`theme.ts`:** palette, type, eases and a tempo-locked beat grid (`BEAT = Math.round(fps * 60 / BPM)`). No ad-hoc margins or hex codes elsewhere.+- **`scenes/<name>.ts`:** each returns a full-frame root box with `startFrame` / `durationFrames`; children key from the scene's frame 0.+- **`film.ts`:** mounts the scenes (`comp.add(scene())` or `Layer.sequence`), the soundtrack and composition-level grade/grain.-For the exported `Scene` / `Sequence` builder, scene windows are half-open: `[startFrame, startFrame + durationFrames)`. A root layer starting at local frame 30 in a 60-frame scene defaults to 30 frames of duration. Explicit root-layer windows must fit their scene; invalid starts or overflow throw with the scene and layer IDs. Children and their keyframes remain local to their parent. Outgoing overlaps place the next scene and cannot exceed either adjacent scene's duration. The final scene's outgoing overlap has no destination and does not shorten the timeline. `calculateTimeline().totalFrames` is the maximum exclusive scene end.+`Scene` / `Sequence` windows are half-open `[startFrame, startFrame + durationFrames)`. A root layer starting at local frame 30 of a 60-frame scene defaults to 30 frames; explicit windows must fit the scene (overflow throws with the IDs). Overlaps can't exceed either neighbour; the last scene's outgoing overlap doesn't shorten the timeline. `calculateTimeline().totalFrames` is the last exclusive scene end.```typescript// theme.ts@@ -104,8 +94,8 @@ All `Layer.*` builders return `AnimatableNode<T>` instances with chainable `.ani| `Layer.model(options)` | `"model3d"` | 3D mesh loader (`.gltf`, `.glb`, `.obj`, `.fbx`, `.stl`, `.ply`). Supports skinning and audio deformation compute shaders. || `Layer3D.roundedBox / sphere / cylinder / capsule / torus / ring / prism / pyramid / polyhedron / icosahedron / icosphere / shard(options)` | `"model3d"` | Procedural solid meshes, centered on `x`/`y`/`z` (y is down, so "top" is -y). `roundedBox`: `width`, `height`, `depth`, `radius`, `segments`, `taper`. `sphere`: `radius`. `cylinder`: `radius` or `radiusTop`/`radiusBottom`, `height` (required), `openEnded`, `flatShaded`. `capsule`: `radius`, `length` (straight shaft; total height is `length + 2 * radius`). `torus`: `radius`, `tube` (tube radius), `arc`. `ring`: flat annulus (`innerRadius`, `outerRadius`). `prism` / `pyramid`: solid capped flat-shaded N-gon prism or pyramid (`sides`, `radius`, `height`). `polyhedron` (`type: "tetrahedron" \| "octahedron" \| "dodecahedron" \| "icosahedron"`), `icosahedron`, `icosphere` (`radius`, `detail`, `flatShaded`). `shard`: procedural flat-faceted polyhedron rock/crystal (`radius` or `size: [x, y, z]`, `facets`, `irregularity`, `seed`). Surface: `color`, `tint`, `roughness`, `metallic`, `emissive`, `emissiveIntensity`, `material` (`"lit" \| "glass" \| "acrylic" \| "frosted"`), `castShadow`, `wireframe`. Shared GPU buffers cache identical shapes. || `Layer3D.extrude / lathe / tube / line / points(options)` | `"model3d"` | Custom & path-driven 3D geometry. `extrude`: extrude a 2D SVG path `d` (with holes) or `contour`/`holes` (`depth`, `bevelThickness`, `bevelSize`, `bevelSegments`, `bevelColor`). `lathe`: revolve a 2D `points` profile `[r, y][]` around the Y axis (`segments`, `phiStart`, `phiLength`, `closed`, `flatShaded`). `tube`: sweep a solid or thick-walled hollow pipe (`radius`, `innerRadius`, `radiusTaper`, `closed`, `capped`) along a `Path3D` or 3D control `points` using rotation-minimizing frames. `line`: 3D polyline (`points` or `Path3D`, `dashSize`, `gapSize`, per-vertex `colors`). `points`: 3D point cloud (`points`, `pointSize` for lit sphere dots or `0` for `point-list`, per-point `colors`). |-| `Layer3D.instances(options)` | `"model3d"` | Hardware-instanced single or multi-variant mesh rendering in 1 GPU draw call. Options: `count`, `mesh` / `meshes` (array of Layer3D primitives or meshes), `instanceAt(index, frame): { position, rotation, scale, color, emissive }`, or bind directly to Rapier `Physics.world3d` bodies via `bodies: Body3D[]` with `emissiveBySpeed` and `colorRamp`. |-| `Layer3D.swarm(options)` | `"model3d"` | Continuous 3D GPU particle swarm with compute-driven physics, 3D curl turbulence, vortex/cyclone field (`swirl`, `pull`, `axial`), ground plane collisions with Coulomb sliding friction and rotational roll torque, kinetic heat accumulation/cooling, and zero-copy packing directly into GPU instance buffers. |+| `Layer3D.instances(options)` | `"model3d"` | One draw call for many copies: `count`, `mesh` / `meshes` (variants), `instanceAt(index, frame): { position, rotation, scale, color, emissive }`, or `bodies: Body3D[]` from `Physics.world3d` with `emissiveBySpeed`, `colorRamp`. |+| `Layer3D.swarm(options)` | `"model3d"` | GPU 3D shard swarm: curl turbulence, vortex (`swirl`, `pull`, `axial`), ground collisions with friction and roll, heat glow. || `Layer3D.terrain(options)` | `"model3d"` | Procedural landscape, closed underneath. `width`, `depth`, `height` (relief px above the node's y, the valley floor), `segments`, `noise: { seed, scale, octaves, persistence, lacunarity, ridgeMode: "fbm" \| "ridged" \| "terrace", terraceSteps }`, or `heightFn(u, v) => 0..1` / `heights`, `overhangWarp`, `colorByElevation: { lowColor, midColor, highColor, cliffColor, cliffThreshold, bands }`, `flatShaded`. Returns a node with `heightAt(x, z)`, `normalAt(x, z)`, `elevationAt(x, z)` to stand cameras, lights, video screens and models on the ground. Static `Physics.world3d` bodies collide with its exact triangles. || `Layer3D.sdf(options)` | `"model3d"` | Boolean sculpture: `primitives: [{ shape: "sphere" \| "box" \| "cylinder" \| "capsule" \| "torus" \| "gyroid", op: "union" \| "subtract" \| "intersect", smoothness (px blend), center, rotation, color, …size }]` meshed watertight inside `bounds: [w, h, d]` at `resolution` (16..160). Colors blend across smooth joins; a colored cutter paints its cut. || `Layer3D.scatter(options)` | `"model3d"` | Scatter `mesh` (one or an array of variants, any Layer3D mesh or `Layer.model`) over `surface` (terrain, sdf, primitive or model file): `count`, `minDistance` (Poisson spacing), `seed`, `slopeRange` (deg from level), `elevationRange` (0..1), `alignToNormal`, `scaleRange`, `randomYaw`, `exclude: [{ x, z, radius, falloff }]`, `sink`, `wind`. Deterministic, one draw call per variant; `.points()` returns where they stand. |@@ -141,9 +131,9 @@ comp.add(Layer.video("assets/clip.mp4", {}));```-Without frame hooks (`onRequestFrame`, `onPrepareFrame`, `comp.vision`, physics) the signal is sampled up front. With them it is live: each boundary is recorded once, after that frame's `onRequestFrame` hooks, and never ahead of the frame being rendered. A seek forward runs the skipped frames' hooks in order; a seek back reuses the recorded history. Source bounds are checked as each frame records, and a layer that outlives the composition holds its last value. A signal that reads its own layer's current source time is reported as a circular dependency.+Without frame hooks (`onRequestFrame`, `onPrepareFrame`, `comp.vision`, physics) the signal is sampled up front; with them it is recorded live, once per frame after its hooks, never ahead (seeks forward run skipped hooks, seeks back reuse history). A signal reading its own layer's source time is a circular dependency.-Keyframes must start at frame 0, end at `durationFrames`, and increase in frame. `interpolation` belongs to the starting key of a segment: `"linear"` (default), `"ease-in"`, `"ease-out"`, or `"smooth"`. These curves are monotonic and do not overshoot; `smooth` starts and ends at zero speed. Adjacent segments may have different speeds at their join. `comp.vision(layer)` requires the layer to have already been added; clip analysis runs in source order and is sampled on the mapped clock. Effects stay on the composition clock. Source ranges are checked before rendering. Audio uses resampling rather than pitch-preserving time stretch. Reverse video can render more slowly due to source seeking; headless decoding retains at most 48 MiB of cached frames per decoder.+Keyframes start at 0, end at `durationFrames` and increase. `interpolation` (on a segment's first key): `"linear"` (default), `"ease-in"`, `"ease-out"`, `"smooth"`; all monotonic, no overshoot. `comp.vision(layer)` needs the layer added first and samples on the mapped clock; effects stay on the composition clock. Audio resamples (no pitch-preserving stretch). Reverse video seeks, so renders slower.## Animation System (`LayerAnimation`)@@ -243,9 +233,9 @@ const camAnim = CameraAnimation.camera()});```-**Cube cameras.** `lens.projection` picks the picture: `"perspective"` (default), `"equirectangular"` (the whole sphere around the eye, 360 × 180, for VR/360 video; `fov` is ignored), or `"fisheye"` (equidistant; `fov` is the full angle across the frame's height, up to 360°, so it can see behind the camera). Anything but perspective draws the scene as six 90° cube faces around the eye and remaps them, with mipmaps, so it costs about six renders. `lens.cubeSize` overrides the face size (default: matches the output's sharpness at the centre of the view, up to 4096).+**Cube cameras.** `lens.projection`: `"perspective"` (default), `"equirectangular"` (360 × 180 for VR; `fov` ignored) or `"fisheye"` (equidistant; `fov` = full angle across the height, up to 360°). Non-perspective renders six 90° faces and remaps them (~6× cost); `lens.cubeSize` overrides the face size (default matches output sharpness, ≤ 4096).-**Relativity.** `relativity: { speed, direction?, doppler?, beaming?, exposure? }` films the scene as seen by a camera moving at `speed` × c (0 to 0.9999) along `direction` (world space, default where the camera looks). It is physics, not grading: rays crowd towards the direction of motion (aberration; objects at the side swing forward and look turned, not squashed), colours shift blue ahead and red behind (spectral Doppler, each colour modelled as a reflectance lit by daylight) and ahead brightens with beaming. `exposure: "auto"` (default) meters the centre of the view so what the camera looks at keeps its brightness; a number is fixed stops, where `0` is the true, often blinding, light. Animate the speed with the `relativitySpeed` prop, e.g. `CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.99, { start: 0, end: 90, ease: "expo.in" })`. A camera with `relativity` is always a cube camera, even at speed 0, and at rest matches the ordinary camera. Works with every projection.+**Relativity.** `relativity: { speed, direction?, doppler?, beaming?, exposure? }` films the scene from a camera moving at `speed` × c (0–0.9999) along `direction` (default: the view). Physical: aberration crowds the view forward, spectral Doppler shifts blue ahead and red behind, beaming brightens ahead. `exposure: "auto"` (default) meters the centre; a number is fixed stops (`0` = true, often blinding light). Animate `relativitySpeed`, e.g. `CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.99, { start: 0, end: 90, ease: "expo.in" })`. A relativity camera is always a cube camera.```typescriptLayer.camera({@@ -255,7 +245,7 @@ Layer.camera({});```-Cube cameras draw outlines, shadows, floor reflections, volumetric fog and bloom, but not depth of field, motion blur or screen-space reflections and GI, which need one perspective view (`comp.lint()` reports `camera_cube_dof`, `camera_cube_motion_blur`, `camera_cube_screen_space`; rendering throws). Glass refracts only the 3D scene, not the 2D layers below.+Cube cameras draw outlines, shadows, floor reflections, volumetric fog and bloom, but not depth of field, motion blur, SSR or GI (`lint` reports `camera_cube_dof`, `camera_cube_motion_blur`, `camera_cube_screen_space`; rendering throws). Their glass refracts only the 3D scene.### 4. Procedural & Audio-Reactive DriversProperties can be bound to Perlin wiggles, harmonic springs, or audio-extracted signals:@@ -280,20 +270,9 @@ const beatAnim2 = LayerAnimation.create().signal("opacity", "drum_stem_signal", { multiplier: 0.5, offset: 0.5 });```-`.signal()` / `.colorSignal()` drive the prop with `value × multiplier + offset` every frame. A signal-object is carried on its track and registered with the program automatically (under a generated-`inline_signal_N` id, so the spec stays plain JSON); a string is looked up in `comp.addSignal` names.-Anything else (a number, `undefined`) throws. Signals are sampled at the **composition** frame-(`ctx.frame`) by default, which shared and audio signals need. Motion authored for one scene-(`Signal.fromArray` of per-frame poses) passes `{ clock: "layer" }` to sample at the layer's own-frame, counted from its start like keyframes, so it plays right inside `Layer.sequence`.--To pull signals out of audio, use `AudioSignal`, which runs on the GPU and accepts audio or video files.-`AudioSignal.extract(src, { fps })` gives `primary`, `beat`, `bass` and `energy` envelopes.-`AudioSignal.pitch(src, { fps, minHz, maxHz })` tracks the fundamental frequency, such as an engine note or a melody line.-It returns `hz`, `confidence` and `voiced` signals plus raw `samples`. `hz` is interpolated across unvoiced gaps,-so a gauge never drops to 0. Use `voiced` to find where the pitch was actually measured: short gaps mark gear-shifts and note onsets. For a 4-stroke engine, the full-cycle pitch is RPM / 120.+`.signal()` / `.colorSignal()` set the prop to `value × multiplier + offset` each frame. A signal object is registered automatically (as `inline_signal_N`, keeping the spec plain JSON); a string names a `comp.addSignal` signal; anything else throws. Signals sample the **composition** frame by default (needed for shared and audio signals); `{ clock: "layer" }` samples the layer's own frame, so per-scene motion plays right inside `Layer.sequence`.++`AudioSignal` (GPU; audio or video files): `extract(src, { fps })` gives `primary`, `beat`, `bass`, `energy`; `pitch(src, { fps, minHz, maxHz })` gives `hz` (interpolated over gaps), `confidence`, `voiced` and raw `samples`. Gaps in `voiced` mark gear shifts and note onsets; a 4-stroke engine's cycle pitch is RPM / 120.```typescriptconst engine = await AudioSignal.pitch("assets/onboard.mp4", { fps, minHz: 50, maxHz: 400 });const rpm = engine.samples.hz.map((hz) => hz * 120);@@ -420,7 +399,7 @@ Where depth jumps (an object's outline) the surface stretches across with the ed### Motion blur (3D)-`Layer.camera({ ..., motionBlur: true })` (or `{ shutter: 180, samples: 20, maxBlur: 48 }`) streaks every mesh by its own motion over the shutter, along with the camera's: moving layers, skinned joints, morph targets, audio-deformed meshes, crowds, swarms, wind and fractured pieces. 3D layer planes (cards, video screens) move by their transform, and text in the 3D scene streaks glyph by glyph. Still things stay sharp, and a frame renders the same in order or after a seek (the frame before is rendered first, recording only). `shutter` is in degrees (180 = film, 360 = the whole frame interval). `maxBlur` caps the streak in px. A layer's own `motionBlurShutter` still streaks a whole flat layer by its position; use the camera's for 3D.+`Layer.camera({ ..., motionBlur: true })` (or `{ shutter: 180, samples: 20, maxBlur: 48 }`) streaks every mesh by its own motion plus the camera's: skinning, morphs, audio deformation, crowds, swarms, wind, fracture pieces, 3D cards and glyphs of 3D text. Seeks render identically. `shutter` in degrees (180 = film), `maxBlur` in px. A layer's `motionBlurShutter` streaks a flat layer by its position; use the camera's for 3D.### Characters: blending, IK, look-at, springs, crowds@@ -473,9 +452,9 @@ Layer.glb("robot.glb", {Order of evaluation: clips, `pose`, `aim`, `lookAt`, `ik` (so IK can still plant the feet), then springs. Retargeting is 2D: limbs swing in the plane facing the camera (no depth from one view); unsure keypoints (below `minVisibility`, default 0.3) hand their bone back to the animated pose. Use `mirror: true` for a character seen from behind. Functions and signals are in-process only; JSON keeps numbers and keys.-Rules: a crowd mesh keeps its own `rotateX` / `scale` under each instance (y-up glTF: `rotateX: 180`). Spring bones follow the clip's motion and the model's current orientation, not the layer's keyframed travel. Per-instance `clip`, `animOffset`, `animSpeed` in `transforms` / `instanceAt` override the crowd's seeded choices.+Rules: a crowd mesh keeps its own `rotateX` / `scale` under each instance (y-up glTF: `rotateX: 180`). Spring bones follow the clip and the model's orientation, not the layer's keyframed travel. Per-instance `clip`, `animOffset`, `animSpeed` in `transforms` / `instanceAt` override the crowd's seeded choices.-**Morph targets by name** (`morphs` on `Layer.glb`): set any of the file's morph targets by name, matched case-insensitively, each a number, keys or a signal; unnamed targets keep the clip's weights. Sparse morph accessors stay sparse on the GPU, so files with hundreds of corrective shapes fit. Skins take up to eight influences per vertex (`JOINTS_1` / `WEIGHTS_1`). Texture slots honor `KHR_texture_transform`, and each map samples with its own glTF sampler.+**Morph targets by name** (`morphs`, case-insensitive): a number, keys or a signal each; unnamed targets keep the clip's weights.```tsLayer.glb("head.glb", { morphs: { jawOpen: [[0, 0], [12, 0.6]], mouthSmileLeft: 0.8, eyeBlinkLeft: blink } });@@ -575,31 +554,28 @@ Layer3D.cylinder({ ..., height: 400, color: "#111", emissive: "#4da3ff", emissiv- **Tone mapping `"neutral"` (default) keeps brand colors as authored**: orange stays orange. Use `"agx"` for a filmic roll-off.- **Bloom only takes light well past white** (emissive ≳ 3, hot speculars), so ordinary bright surfaces don't glow. Raise `emissiveIntensity` to make something glow.- **Shadows:** `directional`, `spot`, `rect` (`Layer.rectLight` / `Light.rect`), and omnidirectional `point` (`Layer.pointLight` / `Light.point`, 6-face cube shadow map) lights can each cast shadows, subject to device texture-array limits. Meshes cast by default (`castShadow: false` opts out); flat layers (planes, cards) cast only with `castShadow: true`. Everything receives.-- **Rect Area Lights (`Layer.rectLight` / `Light.rect`):** studio softbox / LED panel lights with `width`, `height`, `twoSided`, `rotation`, `x/y/z`, `targetX/Y/Z`, `intensity`, `radius`, `decay`, and `castShadow`. Uses solid-angle diffuse irradiance and representative-point GGX specular highlights that stretch across rough and glossy surfaces. Numeric light props accept `Signal`s directly (e.g. `Layer.rectLight({ intensity: bassSignal, width: 300, height: 120 })`).-- **Volumetric Fog & Light Shafts (`volumetricFog`):** `Layer.environment("night", { volumetricFog: true | { density, anisotropy, heightFalloff, baseHeight, color, lightIntensity, ambientIntensity, maxDistance, steps, noise: { scale, strength, speed, octaves } } })` raymarches participating media at half resolution with depth-aware bilateral upsampling. Any shadow-casting `directional`, `spot`, `rect`, or `point` light automatically carves volumetric light shafts (god rays) through the fog behind occluders. `density`, `anisotropy`, and `noise.strength` accept numbers or live `Signal` objects directly.-- **Glass refracts the 3D scene:** glass, acrylic and frosted meshes show the 3D objects behind them as well as the layers under the batch (the batch is drawn once without its glass to make that picture). glTF materials with `KHR_materials_transmission` render as glass with their `KHR_materials_ior`; a `KHR_materials_volume` thickness bends by the refracted ray through that depth, so the bend scales with the object on screen (corneas, potions, ice).-- **Model files bring their own lobes:** glTF materials with `KHR_materials_clearcoat`, `_sheen`, `_anisotropy` or `_iridescence`, and `subsurface` (in the material's `extras.framefields.subsurface`), render with those lobes; a layer's own lobe props override them. Subsurface scattering conserves energy: each channel's light wraps past the terminator by how far it scatters (`subsurfaceColor`; red farthest for skin, e.g. `"#ff5a3c"`), so form keeps its shading and only shadow edges warm; thin parts glow when back-lit.-- **Substrate lobes (physical shading, needs an environment):** every `Layer3D.*` mesh, `Layer.model` and 3D card accepts `clearcoat` / `clearcoatRoughness` (car paint, lacquer), `anisotropy` (-1..1) / `anisotropyRotation` (deg; brushed metal), `subsurface` / `subsurfaceColor` (wax, jade, skin), `sheen` / `sheenColor` / `sheenRoughness` (velvet, cloth), `iridescence` (0..1 or `{ intensity, thickness, ior }`; soap, oil, beetle shell) with `iridescenceThickness` (nm) / `iridescenceIor`, and `triplanar: true | { scale, sharpness }` for world-space textures on UV-less meshes. Numeric lobes take numbers or live `Signal`s (`clearcoat: bass`). Example: `Layer3D.sphere({ radius: 90, color: "#8a0f1c", metallic: 0.6, roughness: 0.35, clearcoat: 1, clearcoatRoughness: 0.05 })`.+- **Rect lights** (`Layer.rectLight` / `Light.rect`): softboxes and LED panels; `width`, `height`, `twoSided`, `rotation`, position/target, `intensity`, `radius`, `decay`, `castShadow`. Numeric light props take `Signal`s (`intensity: bass`).+- **Volumetric fog** (`volumetricFog: true | { density, anisotropy, heightFalloff, baseHeight, color, lightIntensity, ambientIntensity, maxDistance, steps, noise: { scale, strength, speed, octaves } }`): half-res raymarched medium; every shadow-casting light carves god rays. `density`, `anisotropy`, `noise.strength` take signals.+- **Model files bring their own lobes:** glTF `KHR_materials_clearcoat`, `_sheen`, `_anisotropy`, `_iridescence`, `_transmission` (glass, with `_ior`; `_volume` thickness scales the bend) and `extras.framefields.subsurface`; layer props override them. Subsurface wraps light past the terminator per channel (`subsurfaceColor`, e.g. `"#ff5a3c"` for skin); thin parts glow back-lit.+- **Substrate lobes (need an environment):** every `Layer3D.*` mesh, `Layer.model` and 3D card accepts `clearcoat` / `clearcoatRoughness` (car paint, lacquer), `anisotropy` (-1..1) / `anisotropyRotation` (deg; brushed metal), `subsurface` / `subsurfaceColor` (wax, jade, skin), `sheen` / `sheenColor` / `sheenRoughness` (velvet, cloth), `iridescence` (0..1 or `{ intensity, thickness, ior }`; soap, oil, beetle shell) with `iridescenceThickness` (nm) / `iridescenceIor`, and `triplanar: true | { scale, sharpness }` for world-space textures on UV-less meshes. Numeric lobes take numbers or live `Signal`s (`clearcoat: bass`). Example: `Layer3D.sphere({ radius: 90, color: "#8a0f1c", metallic: 0.6, roughness: 0.35, clearcoat: 1, clearcoatRoughness: 0.05 })`.- **HDRI environments:** pass a `.hdr` or `.exr` path instead of a preset: `Layer.environment("assets/hdri/studio.exr", { intensity: 1.2, rotation: 90 })` (or `{ hdri: path }`). The panorama lights and reflects every physical surface; preset names keep working; relative paths resolve from the working directory. EXR files must use lossless compression (NONE, RLE, ZIPS, ZIP, PIZ or PXR24); re-save B44/DWA files as ZIP.- **Screen-space reflections & GI (`ssr`, `gi`):** `Layer.environment("night", { ssr: true, gi: true })`. `ssr` (`{ intensity, maxRoughness, thickness, steps }`) reflects nearby geometry and emissives in glossy surfaces: chrome, wet floors, lacquer, walls. `gi` (`{ intensity, radius, samples }`) adds contact occlusion and color bleeding from visible surfaces, e.g. a neon sign tinting the floor. Each costs a second 3D pass; only what is on screen can be reflected or bounce, so keep key reflected objects in frame. Use `floor` for a planar mirror that also shows off-screen objects.-- **Art-directed materials:** `material: "toon"` works on meshes, flat 3D cards and native 3D text. Its `toon` options are `shadowColor` (shadow-side albedo multiplier; model shade color or `#80808c`), `lightColor` (lit-side multiplier; white), `threshold` (terminator cosine; 0.075), `softness` (transition width; 0.25) and `diffuse` (0 flat through 1 Lambert; 0). Toon surfaces use all scene lights, their colors, point/spot attenuation and real cast shadows; environment exposure, tone mapping, fog and emissive bloom still apply. Lower `threshold` to wrap light around a surface, or raise `shadowColor` for a bright shadow floor on illustrated cards. Physical and unlit materials are independent of these options.-- **Geometry outlines (ink):** `Layer.environment("neutral", { outlines: true })`, or `outlines: { color: "#191721", width: 2, referenceHeight: 1080, creaseAngle: 45, depthThreshold: 0.01, opacity: 1 }`. `width` is px at `referenceHeight`, scaled with the output height; start near 1 px at export height. Ink comes from depth and normals, so textures, shadows and coplanar seams get none; `creaseAngle: 180` keeps only silhouettes. `outline: false` on a layer/model excludes it but keeps it occluding ink behind it (for already-inked artwork). Ink follows fog and runs before depth blur and bloom; heavy lens blur spreads it into wide bands. Separate from the image-space `Toon` effect. `Layer3D.cube({ faces: { front: Layer.box(...) } })` faces inherit the cube's `toon`, `outline`, emissive and shadow settings unless set per face. Example: `examples/34_spatial_styling` in the engine repo.+- **Toon:** `material: "toon"` on meshes, 3D cards and 3D text; `toon: { shadowColor (#80808c), lightColor (white), threshold (0.075), softness (0.25), diffuse (0 flat … 1 Lambert) }`. Uses every scene light and real shadows; exposure, tone mapping, fog and bloom still apply. Lower `threshold` wraps light further.+- **Ink outlines:** `outlines: true | { color, width, referenceHeight: 1080, creaseAngle: 45, depthThreshold: 0.01, opacity }` on the environment. `width` is px at `referenceHeight` (start near 1). Ink comes from depth and normals (textures and shadows get none; `creaseAngle: 180` = silhouettes only). `outline: false` on a layer excludes it but keeps it occluding. Ink runs before depth blur and bloom. `Layer3D.cube` faces inherit the cube's `toon`, `outline`, emissive and shadow settings. Distinct from the 2D `Toon` effect.- **3D pivots:** `anchorX/Y` choose the point held fixed during scale/rotation; changing an anchor at identity transform leaves the layer in place. Use `anchorY: 1` to keep a character's feet planted while scaling.- **Fog** fades to `fog.color` exactly, so set it to the composition background for a seamless horizon.-- **Models:** glTF materials bring base color, normal, metallic-roughness, occlusion and emissive maps; OBJ (`map_Kd`, `norm` in the `.mtl`) and FBX bring diffuse and normal maps, looked up next to the model. Textures can be PNG, JPEG, WebP, GIF, TIFF, AVIF or Basis KTX2 (ETC1S/UASTC, transcoded offline to RGBA8); TGA, PSD and DDS are skipped with a warning. Draco and meshopt-compressed glTF geometry is decoded by the asynchronous compositor loader; individual maps retain their own UV sets and texture transforms. glTF morph weights animate on the GPU before skinning; sparse morph targets stay sparse from the file to the GPU (memory follows the deltas that exist, so faces with hundreds of correctives fit), and skins take up to eight influences per vertex (`JOINTS_1` / `WEIGHTS_1`). Unknown required glTF extensions fail clearly. Vertex colors (VOX, PLY, OFF, glTF `COLOR_0`) multiply the base color.+- **Models:** glTF base color, normal, metallic-roughness, occlusion and emissive maps (own UV sets, samplers, `KHR_texture_transform`); OBJ (`map_Kd`, `norm`) and FBX diffuse and normal maps. PNG, JPEG, WebP, GIF, TIFF, AVIF, KTX2 (Basis); TGA, PSD, DDS are skipped with a warning. Draco and meshopt decode. Morphs run on the GPU before skinning and sparse targets stay sparse (hundreds of correctives fit); skins take up to eight influences (`JOINTS_1` / `WEIGHTS_1`). Unknown required extensions fail clearly. Vertex colors multiply the base color.- **Model orientation:** the world is y-down, so a y-up model (most glTF, OBJ and FBX files) needs `rotateX: 180` to stand upright.- **Reflective floor:** `floor: { y, color, reflectivity, roughness, fadeDistance, opacity }` mirrors the scene; `roughness` blurs the reflection, which also fades with distance and strengthens at grazing angles.- `emissive` / `emissiveIntensity` / `exposure` animate like any prop (`.keys("emissiveIntensity", …)`).-- **Transparent 3D meshes:** glTF `BLEND` materials and meshes with reduced opacity draw after opaque scene items, sorted back to front by each mesh's bounds center without writing depth. This is an object-level approximation: intersecting or self-overlapping meshes cannot always sort correctly; translucent flat layers and glass use separate rendering paths.-- **Glass:** `material: "glass" | "acrylic" | "frosted"` on any Layer3D primitive, extruded text (`Layer3D.extrudedText`) or model refracts whatever is behind it on screen — the video, 2D layers, particles — with a rainbow fringe (`dispersion`, default 0.04), bend strength `thickness` (px, default 80) and `ior` (1.5). `color` tints the glass (deeper toward the edges); `frosted` blurs by `roughness`. Glass needs no environment: it draws its own studio reflections, and against a dark grade it reads best — skip `Lighting.*` rigs for glass-only scenes (their environment washes glass out). Glass shows only 2D content and opaque 3D drawn before it, not other glass.+- **Transparent meshes** (glTF `BLEND`, reduced opacity) draw after opaque ones, sorted back to front by bounds center; intersecting meshes may mis-sort.+- **Glass:** `material: "glass" | "acrylic" | "frosted"` on any Layer3D solid, `Layer3D.extrudedText` or model refracts what's behind it (3D objects, video, 2D layers, particles; not other glass) with `dispersion` (0.04), `thickness` (px, 80), `ior` (1.5); `color` tints toward the edges, `frosted` blurs by `roughness`. It needs no environment and reads best on a dark grade; skip `Lighting.*` rigs for glass-only scenes.---## Physics (`Physics.world2d` / `Physics.world3d`)-Use for anything that should *fall, bounce, stack, swing or collide*: gravity drops of words or cards, UI pieces tumbling into a pile, a "jiggle" on a spring, balls rattling in a box, dominoes, physics-driven 3D products. Use keyframes instead for motion that should hit an exact designed pose at an exact frame.--A world runs **once** while the composition is built (Rapier, deterministic build: identical on every machine) and replays as per-frame signals, so frames render in any order. `simulate()` binds each body's layer (`x`, `y`, `rotation`; in 3D `x/y/z`, `rotateX/Y/Z`):+For things that *fall, bounce, stack, swing or collide* (dropped words, tumbling UI, springs, dominoes, 3D products); keyframes for exact designed poses. A world runs **once** at build time (Rapier, deterministic on every machine) and replays as per-frame signals, so frames render in any order. `simulate()` binds each body's layer (`x`, `y`, `rotation`; 3D `x/y/z`, `rotateX/Y/Z`):```typescriptimport { Layer, LayerAnimation, Physics } from "framefields";@@ -626,7 +602,7 @@ stage.animate(LayerAnimation.create().signal("y", world.impact, { multiplier: 14- **Joints:** `world.pin(body, [x, y])` (swing around a point), `world.hinge(a, b, [x, y])`, `world.weld(a, b)`, `world.spring(a, bOrPoint, { frequency: 2, dampingRatio: 0.2, length })`, `world.rope(a, bOrPoint, { length })`. Points are world px.- **Driving the simulation:** `world.force(body, { x, y })` (px/s², mass-independent), `world.impulse(body, frame, [vx, vy])` (a kick), kinematic `follow: { x, y, rotation }`. Each of these takes a number, a signal (audio, `Signal.fromArray`) or `(frame) => value`: e.g. a floor that bounces on the kick.- **Force fields:** `world.field((s) => [ax, ay], { bodies })` is called every solver step with a body's center `s.x`, `s.y` (px), velocity `s.vx`, `s.vy` and `s.frame`; return px/s² or nothing. Wind, vortices, attractors, flow fields — and performers (below).-- **Performers in the simulation:** `const dancer = comp.vision("dancer.mp4")` gives the clip's pose and silhouette per frame, analysed on demand (see `framefields-vision`). Make the world **live** so it steps as frames render instead of all at once: `world.live(comp)` in place of `simulate()`. Before each frame draws, the clip's vision for that frame is computed (or read from the cache) and the world advances to it — nothing is analysed up front, the preview plays at once and fills in, and export gives the identical result (same simulation, same order). Then, with `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:+- **Performers in the simulation:** with `const dancer = comp.vision("dancer.mp4")` (`framefields-vision`), call `world.live(comp)` instead of `simulate()`: the world steps as frames render, each after that frame's vision, so the preview starts at once and the export is identical. With `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:- limbs as kinematic colliders that catch and bat things: `world.body({ x, y, width: 2 * r, height: 2 * r }, { type: "kinematic", shape: "circle", follow: { x: (f) => person.point("leftWrist", f).x - r, y: (f) => person.point("leftWrist", f).y - r } })`, a few balls along each shoulder → elbow → wrist;- the silhouette (dress included) as a force field: `person.sample(frame, x, y)` returns `alpha` (inside > 0.5) and a gradient `gx`, `gy` pointing inward, so `world.field((s) => { const m = person.sample(s.frame, s.x, s.y); … return [-m.gx * k, -m.gy * k]; })` makes particles and props flow around a moving body.- Reads are synchronous and need the frame analysed: inside a live world, while rendering, or after `await dancer.ready(frame)` (e.g. `ready(0)` to lay things out around the first pose). A live world can't look ahead, so place things from early frames.@@ -688,11 +664,11 @@ Layer.vfx({ effectType: "Aurora", width: W, height: H, colorStart: "#2bff88", co1. **Local Keyframe Coordinates:** Keyframes are strictly relative to the node's own timeline window (`0` = node appearance). Never offset keyframe frames by the parent's `startFrame`.2. **Coordinate Overrides:** Animating `y` overrides the node's layout `y`. If a node is placed at `y: 400`, animate `fromTo("y", 440, 400)`, NOT `fromTo("y", 40, 0)`.3. **No String Styles for Borders:** Use `borderColor` and `borderWidth`. CSS `border: "1px solid red"` is silently ignored.-4. **Font Registration is Mandatory:** In headless WebGPU, system fonts do not exist. Register every font with `FontManager.register()` before building text layers. Emoji need a color emoji font too: framefields ships none, so register one with `FontManager.registerEmojiFont("assets/fonts/NotoColorEmoji.ttf")` (a CBDT/CBLC bitmap font such as Noto Color Emoji). Without it exports draw nothing for emoji.+4. **Register every font** with `FontManager.register()` before building text: there are no system fonts. Emoji need `FontManager.registerEmojiFont("assets/fonts/NotoColorEmoji.ttf")` (CBDT/CBLC) or draw nothing.5. **Integer Frames Only:** Keyframe frames, start frames, and durations must be integers. Decimal frames cause schema validation errors.6. **`x`/`y` mean absolute:** a node with `x` or `y` and no `position` is absolutely positioned (out of flex flow), relative to its parent's content box.7. **Audio Layer Duration:** Every `Layer.audio()` must be assigned `durationFrames` to prevent audio truncation in headless rendering.8. **Lint before rendering:** `comp.lint()` lists colors/gradients that would draw transparent, low-contrast text, off-canvas layers, shadow settings that do nothing and `roughness`/`metallic` without an environment. `renderVideo({ qa: true })` includes these findings.9. **Lottie vs. Image Layers:** Use `Layer.lottie("file.json" | "file.lottie", { width, height, fit })` for Lottie vector animations. Never pass `.json` or `.lottie` to `Layer.image()`, which only decodes raster formats and throws/warns if given a vector animation.-10. **AI-generated media are assets, not build steps:** Images, clips, music or voice made with `ai` (from `framefields`) are generated once by a separate script into `assets/generated/` and loaded like any other file. Never call `ai` from `buildFilm()` or a scene function: it would spend money and change the film on every render, preview and test. The `framefields` skill (section 6) covers the key, the script and the model catalog.-11. **Zooming into flat layers stays sharp:** text, shapes, images and SVGs rasterize at their scale on the canvas (up to 4×), so a camera push or a parent at `scale: 2` keeps glyphs and edges crisp. A leaf keeps its layout-size raster, and softens when magnified, if it is 3D or deflickered, is media with `effects`/`crop`, or carries an effect other than `LayerStyle` or a Gaussian/box `Blur`.+10. **AI media are assets, not build steps:** generate with `ai` once in a separate script into `assets/generated/` (`framefields` skill); never call it from `buildFilm()` or a scene.+11. **Zoomed flat layers stay sharp:** text, shapes, images and SVGs rasterize at their on-canvas scale (≤ 4×). A leaf stays at layout resolution (softens when magnified) if it is 3D, deflickered, media with `effects`/`crop`, or carries an effect other than `LayerStyle` or Gaussian/box `Blur`.diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 7deaba05..63572d57 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -5,130 +5,75 @@ description: Apply, configure, and modulate WebGPU post-processing shaders, styl# framefields-effects-## Setup-Import everything from the [`framefields`](https://www.npmjs.com/package/framefields) npm package. If the project doesn't have framefields set up yet, follow the `framefields` skill first (install, project scaffold, first render).+Effects are typed `Effect` classes imported from [`framefields`](https://www.npmjs.com/package/framefields) (grading, lens, texture, stylize, keying, temporal). Every prop takes a value or a `Signal`. No project yet: see the `framefields` skill.-## Overview-Effects are typed `Effect` classes (grading, lens, texture, stylize, keying, temporal). Every prop takes a static value or a `Signal`.+**Look every prop up in [references/effects-catalog.md](references/effects-catalog.md) before setting it** (generated: every class, prop, default and range). Scales differ per effect, and a value on the wrong scale renders as nothing, without an error.-**Before setting a prop, look it up in [references/effects-catalog.md](references/effects-catalog.md)**: the generated list of every class with its props, defaults and ranges. Scales differ between effects, and a value on the wrong scale renders as nothing.-------## Attaching Effects--Four ways to attach effects:--### 1. A layer (`withEffect` / `withEffects`)-Any layer (`box`, `flex`, `shape`, `text`, `media`):+## Attaching effects```typescript-import { Layer, Curves, Vignette, Blur, Pixelate } from "framefields";--// Media layer with color grade and lens effects-const videoLayer = Layer.video("assets/footage.mp4", { fit: "cover" })- .withEffect(new Curves({- master: [{ x: 0, y: 0.05 }, { x: 0.5, y: 0.52 }, { x: 1, y: 0.95 }]- }))- .withEffect(new Vignette({ strength: 40, radius: 0.85, softness: 0.6 }))- .withEffect(new Blur({ strength: 8, blurType: "Gaussian" }));--// Styled container box with blur-const cardLayer = Layer.box({ width: 400, height: 300, background: "#1e1e24" })- .withEffect(new Blur({ strength: 12 }));--// Kinetic text with pixelation-const titleLayer = Layer.text("CYBERPUNK", { fontSize: 72 })- .withEffect(new Pixelate({ size: 8 }));-```+import { ApplyLUT, Blur, Crop, Curves, FilmGrain, Layer, Media, Vignette } from "framefields";-### 2. The whole composition (`comp.apply`)-A grade or grain over everything rendered:-```typescript-import { Composition, FilmGrain, ColorBalance } from "framefields";+// 1. A layer (box, flex, shape, text, media): withEffect / withEffects+Layer.video("assets/footage.mp4", { fit: "cover" })+ .withEffect(new Curves({ master: [{ x: 0, y: 0.05 }, { x: 0.5, y: 0.52 }, { x: 1, y: 0.95 }] }))+ .withEffect(new Vignette({ strength: 40, radius: 0.85, softness: 0.6 }));-const comp = new Composition({ width: 1920, height: 1080, fps: 30, durationMs: 10_000 });--// Shared organic 35mm grain applied across titles, background video, and vector graphics+// 2. The whole compositioncomp.apply(new FilmGrain({ strength: 18, size: 1.5, monochrome: true }));-comp.apply(new ColorBalance({- shadows: { cyanRed: 4, yellowBlue: -6 },- highlights: { cyanRed: 2, yellowBlue: -4 },- preserveLuminosity: true,-}));-```--### 3. A region (`comp.section` / `Layer.section`)-Effects inside a rectangle (or a tracked object) of a plate:-```typescript-// Box blur on a tracked object or explicit bounding rectangle-const blurredPlate = comp.section({- source: "assets/interview.mp4",- x: 400,- y: 200,- width: 300,- height: 300,- borderRadius: 150,- effects: [new Blur({ strength: 25, blurType: "Gaussian" })],-});-```--### 4. Media chains (`Media`)-Process a clip outside the layout tree:-```typescript-import { Media, ApplyLUT, Crop } from "framefields";+// 3. A region (rectangle or tracked object) of a plate+comp.section({ source: "assets/interview.mp4", x: 400, y: 200, width: 300, height: 300, borderRadius: 150,+ effects: [new Blur({ strength: 25, blurType: "Gaussian" })] });+// 4. A clip outside the layout treeconst clip = Media.video("assets/raw_log.mp4")- .apply(new ApplyLUT({ lutUrl: "assets/luts/cinematic.cube", intensity: 1.0 }))+ .apply(new ApplyLUT({ lutUrl: "assets/luts/cinematic.cube", intensity: 1 })).apply(new Crop({ cropType: "rect", topPercentage: 10, heightPercentage: 80 }));--const pngBuffer = await clip.renderFrame({ atMs: 1500 });+await clip.renderFrame({ atMs: 1500 });```------## Effect Categories--### 1. Color Grading & Tonal Dynamics-- **`Curves`**: Spline-interpolated tone curves for master, red, green, blue, plus hue/saturation curves (`hueVsHue`, `hueVsSat`, `lumVsSat`, `satVsSat`).-- **`ColorBalance`**: Shadows, midtones, and highlights split color adjustments with `preserveLuminosity`.-- **`Levels`**: Master and per-channel input/output black/white points and gamma.-- **`SelectiveColor`**: CMYK-style photographic selective adjustments targeting reds, yellows, greens, cyans, blues, magentas, whites, neutrals, and blacks.-- **`GradientMap`**: Multi-stop gradient remapping (`stops: [{ position: 0, color: "#000" }, { position: 1, color: "#FFF" }]`).-- **`ShadowsHighlights`**: Separate shadow lifting and highlight recovery with radius and tonal width controls.-- **`ApplyLUT`**: 1D and 3D `.cube` color lookup table applicator.-- **`Modulate`**: Real-time HSL/contrast controls: `hue`, `brightness`, `contrast`, `exposure`, `saturation`, `sepia`.--### 2. Optical, Texture & Cinematic Emulsions-- **`FilmGrain`**: GPU-synthesized film grain with grain sizing, shadow/highlight masking, and temporal animation speed.-- **`Vignette`**: Elliptical or circular lens falloff with customizable center, roundness, and softness.-- **`Blur`**: High-performance multi-mode blur: `"Gaussian"`, `"Box"`, `"Median"`, `"Motion"`, `"Bilateral"`, `"Edge-preserving"`, `"Radial"`, `"Zoom"`. Supports partial blur regions and tracked object bounding boxes.-- **`UnsharpMask`**: High-frequency edge sharpener with threshold suppression.-- **`HighPass`**: Edge and frequency isolation with contrast boost.-- **`HalftoneScreen`**: Procedural print screening with `"Circle"`, `"Diamond"`, `"Line"`, or `"Square"` dots in `"Monochrome"` or `"CMYK"` angles.-- **`TileOffset`**: Seamless UV wrapping with `"wrap"`, `"clamp"`, `"mirror"`, or `"transparent"` boundaries.-- **`MotionBlur`**: Directional shutter simulation with custom shutter angles and velocity clamps.-- **`TemporalDeflicker`**: Multi-frame optical flow blending to eliminate flickering in generative AI clips or high-speed footage.--### Temporal (effects with memory)+## Effects++### Grading+- **`Curves`**: master/RGB tone curves plus hue and saturation curves (`hueVsHue`, `hueVsSat`, `lumVsSat`, `satVsSat`).+- **`ColorBalance`**: shadows/midtones/highlights shifts, `preserveLuminosity`.+- **`Levels`**: black/white points and gamma, master or per channel.+- **`SelectiveColor`**: CMYK-style adjustments per color range (reds … blacks).+- **`GradientMap`**: gradient remap (`stops: [{ position: 0, color: "#000" }, { position: 1, color: "#FFF" }]`).+- **`ShadowsHighlights`**: shadow lift and highlight recovery.+- **`ApplyLUT`**: 1D/3D `.cube` LUTs.+- **`Modulate`**: `hue`, `brightness`, `contrast`, `exposure`, `saturation`, `sepia`.++### Optics and texture+- **`FilmGrain`**: grain with size, tonal masking, animation speed.+- **`Vignette`**: lens falloff: center, roundness, softness.+- **`Blur`**: `"Gaussian"`, `"Box"`, `"Median"`, `"Motion"`, `"Bilateral"`, `"Edge-preserving"`, `"Radial"`, `"Zoom"`; partial regions and tracked boxes.+- **`UnsharpMask`**: sharpen with threshold.+- **`HighPass`**: frequency isolation.+- **`HalftoneScreen`**: print screen: `"Circle"`, `"Diamond"`, `"Line"`, or `"Square"` dots in `"Monochrome"` or `"CMYK"` angles.+- **`TileOffset`**: UV wrap: `"wrap"`, `"clamp"`, `"mirror"`, or `"transparent"`.+- **`MotionBlur`**: directional shutter blur.+- **`TemporalDeflicker`**: optical-flow blend that removes flicker (AI clips, high-speed footage).++### Temporal (with memory)- **`Trails`**: the layer over a decaying copy of its own past. `mode: "blend"` (default; echoes on full-frame footage), `"over"` (cutouts and text), `"add"` (light painting on dark scenes), `"max"` (light trails). `decay` 0.8 short echo → 0.97 long trails; per-frame `zoom` (1.02 = tunnel), `rotation`, `offsetX/Y`, `hueShift` (psychedelic cycling).- **`Datamosh`**: the previous picture dragged along the new frame's optical flow instead of being refreshed, until a `keyframes` frame (I-frame) shows the clean picture. Put keyframes where you want it clean; leave a cut out to smear one shot with the next one's motion. `strength`, `bleed` (clean picture mixed back per frame).- Both step as frames render; after a jump (scrubbing, a single `renderFrame`) they replay the last frames (`warmup`) to rebuild the history.-### 3. 3D Spatial Lighting & Materials-- **`Relight3D`**: Screen-space normal map relighting with point, spot, and directional lights, specular highlights, roughness, and metallic uniforms.-- **`SSAO`**: Screen-space ambient occlusion generating contact shadows from depth buffers.-- **`PBRGlass`**: Physically-based transmission, index of refraction (IOR), optical dispersion, and Fresnel reflections.-- **`DepthOfField`**: Camera lens simulation with aperture size, focal length, focus plane distance, and circle-of-confusion (CoC) blur.+### Screen-space 3D+- **`Relight3D`**: screen-space normal relighting (point, spot, directional).+- **`SSAO`**: contact shadows from depth.+- **`PBRGlass`**: refraction, IOR, dispersion, Fresnel.+- **`DepthOfField`**: aperture, focal length, focus distance.-### 4. Matte, Keying & Edge Operations-- **`ColorKey`**: Studio greenscreen/bluescreen keyer with spill suppression and smoothness controls.+### Keying and crop+- **`ColorKey`**: green/bluescreen keyer with spill suppression.- **`Crop`**: `cropType: "rect"` (percent edges, `roundness` for rounded or circular corners) or `"path"` (`pathPoints` polygon); `mode: "rest"` keeps the outside instead.-### 5. Stylize Looks (`Styles.*`)+### Stylized looks (`Styles.*`)-**Which look for which request:** see [references/style-guide.md](references/style-guide.md). It maps phrases like "VHS", "8-bit", "comic" or "liquid logo" to presets, effects and techniques. Read it whenever someone names a style.+**When someone names a style** ("VHS", "8-bit", "comic", "liquid logo"), read [references/style-guide.md](references/style-guide.md): it maps requests to presets, effects and techniques.`Styles.<name>(options?)` returns an effect stack: `layer.withEffects(Styles.vhs())`. Presets: `pixelArt`, `gameBoy`, `teletext`, `ledWall`, `ascii`, `oneBit`, `risograph`, `comic`, `cartoon`, `thermal`, `vhs`, `crt`, `glitch`, `oilPainting`, `watercolor`, `topographic`, `kaleidoscope`, `opArt`, `holographic`, `noir`, `film`.@@ -150,31 +95,13 @@ The effects behind them, for finer control:----## Vision-driven effects--Cutouts, subject mattes, tracked boxes, pose, smart re-framing, subject outlines, tracked region blur, and prompted tracking of a specific person or several people (SAM 2.1 / SAM 3) live in the `framefields-vision` skill. Effects here consume its signals: for example `layer.blurRegion(vision.objects.byCategory("person"))`, or an effect uniform driven by `clip.signals.objects.get(1).bounds`.-------## Reactive Signal Modulation--Any numeric prop takes a `Signal` (audio envelopes, physics impacts, vision bounds, `sineSignal`), sampled every frame:+## Signals and vision-```typescript-import { Layer, Blur, sineSignal } from "framefields";--// Oscillating breath blur-const blurSignal = sineSignal({ frequency: 0.5, min: 0, max: 20 });+Any numeric prop takes a `Signal` sampled every frame: audio envelopes, physics impacts, vision bounds, `sineSignal({ frequency: 0.5, min: 0, max: 20 })`. Detection, cutouts, pose and SAM tracking live in `framefields-vision`; effects consume their signals, e.g. `layer.blurRegion(vision.objects.byCategory("person"))` or a uniform driven by `clip.signals.objects.get(1).bounds`.-const layer = Layer.image("assets/bg.jpg")- .withEffect(new Blur({- strength: blurSignal,- blurType: "Gaussian"- }));-```+## Rules of thumb-### Best Practices:-1. **Check the prop's scale in the catalog before picking a number.** Scales differ per effect: `FilmGrain` and `Vignette` `strength` run 0–100 (defaults 15 and 50), `Blur` `strength` 0–100, while `Blur` `radius` is the 0–1 size of a *partial* blur region, not the blur amount. A value on the wrong scale renders as nothing, without an error.-2. **Emulsion restraint:** keep `FilmGrain` around 10–25 and `Vignette` around 25–45. Over-filtering looks amateurish.-3. **Order matters:** Place tonal grading (`Curves`, `ColorBalance`) before lens optics (`Blur`, `DepthOfField`), and apply grain (`FilmGrain`) as the outermost layer.-4. **Stills:** set `animated: false` on `FilmGrain` so the grain does not change between frames of a held image.+1. **Check the scale in the catalog.** `FilmGrain` and `Vignette` `strength` run 0–100 (defaults 15 and 50), `Blur` `strength` 0–100, while `Blur` `radius` is the 0–1 size of a *partial* blur region.+2. **Restraint:** `FilmGrain` around 10–25, `Vignette` 25–45.+3. **Order:** tonal grading (`Curves`, `ColorBalance`) before optics (`Blur`, `DepthOfField`); grain outermost.+4. **Stills:** `animated: false` on `FilmGrain` keeps grain fixed on a held image.diff --git a/plugins/framefields/skills/framefields-effects/references/style-guide.md b/plugins/framefields/skills/framefields-effects/references/style-guide.mdindex 6ce313c8..f96d07c7 100644--- a/plugins/framefields/skills/framefields-effects/references/style-guide.md+++ b/plugins/framefields/skills/framefields-effects/references/style-guide.md@@ -1,6 +1,6 @@# Style guide: which look for which request-Use this guide to turn what someone asks for ("make it look like a VHS tape", "8-bit", "liquid logo") into the framefields features that produce it. Start from a `Styles.*` preset when one fits, then tune it or add layers. Every preset returns an effect stack: `layer.withEffects(Styles.vhs())`. Effects attach to media layers (video, image, SVG, particles, VFX). To stylize a whole scene, apply the effects to the composition's master pass (see SKILL.md "Whole-Composition Master Emulsion").+Use this guide to turn what someone asks for ("make it look like a VHS tape", "8-bit", "liquid logo") into the framefields features that produce it. Start from a `Styles.*` preset when one fits, then tune it or add layers. Every preset returns an effect stack: `layer.withEffects(Styles.vhs())`. Effects attach to media layers (video, image, SVG, particles, VFX). To stylize a whole scene, apply the effects to the whole composition with `comp.apply` (SKILL.md, "Attaching effects").## Visual stylesdiff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 6f4be976..df3a791b 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -1,465 +1,216 @@---name: framefields-render-description: Render, inspect, test, and export high-performance WebGPU video and frame grids with the framefields headless engine, locally or on framefields cloud GPUs. Use when verifying compositions, diagnosing blank frames, profiling GPU pipelines, running visual frame grids, exporting final MP4 video, naming renders in framefields.json, rendering in the cloud (`framefields render --cloud`, `renderCloud`, the dashboard), or splitting a long film across several GPUs in framefields.+description: Render, inspect, test, preview and export WebGPU video and frame grids with the framefields headless engine. Use when verifying compositions, diagnosing blank frames, running frame grids or pixel tests, showing the user a live preview, exporting MP4/WebM, reading render progress, running video QA, naming renders in framefields.json, rendering without a GPU, or splitting a render into chunks in framefields. Cloud GPU rendering is in framefields-cloud.---# framefields-render-## Setup-Import everything from the [`framefields`](https://www.npmjs.com/package/framefields) npm package. If the project doesn't have framefields set up yet, follow the `framefields` skill first (install, project scaffold, first render).+`framefields` renders headlessly in Node.js on native WebGPU (Dawn), encoding through FFmpeg with a hardware encoder when one works (NVENC, VideoToolbox), else software. No browser. Project setup is in `framefields`; rendering on framefields' GPUs is in `framefields-cloud`.-## Overview-`framefields` renders headlessly in Node.js on top of native WebGPU (Dawn), encoding through FFmpeg with a hardware encoder when one works (NVENC, VideoToolbox) and a software one otherwise. It does not run a browser, Puppeteer, or Chromium.+## The verification ladder-Because video rendering is resource-intensive and native GPU pipelines run asynchronously in worker pools, framefields workflows enforce a strict **verification ladder**: inspect inexpensive intermediate artifacts first before committing to a full video render.+Go in order: a layout or timing bug costs seconds at step 1–3 and minutes after a full export.------## The Verification Ladder--Always proceed through the ladder in order. Catching a layout or timing bug at step 1 or 2 takes seconds; catching it after a 30-second 4K video render wastes minutes.--### Step 1: Static Typecheck & Schema Validation-Before touching the GPU, verify TypeScript types and parse the composition against the strict runtime schema:+### 1. Typecheck, schema, lint```typescriptimport { CompositorProgramSchema } from "framefields";-// 1. Compile spec-const spec = comp.toSpec();--// 2. Validate against strict schema-const parsed = CompositorProgramSchema.safeParse(spec);-if (!parsed.success) {- console.error("Schema validation failed:", parsed.error.format());- process.exit(1);-}--// 3. Lint: things that render without error but not as intended-for (const d of comp.lint()) console.log(`${d.severity} ${d.code}: ${d.message}`);+const parsed = CompositorProgramSchema.safeParse(comp.toSpec());+if (!parsed.success) { console.error(parsed.error.format()); process.exit(1); }+for (const d of comp.lint()) console.log(`${d.severity} ${d.code}: ${d.message}`); // renders, but not as intended```-Typecheck the project:-```bash-npx tsc --noEmit -p .-```--In projects initialized with the current CLI, `pnpm check` runs typechecking and document preflight. For a named render, run `pnpm exec framefields check teaser --json`; with no argument it checks every render in `framefields.json`. For another module, `pnpm exec framefields check src/film.ts --export buildFilm --json`. The export is a `Composition` or a builder returning one; importing it executes project code, so keep rendering, generation and writes out of the builder. The CLI runs it in an isolated process, routes builder logs to stderr and writes one versioned JSON result to stdout. Errors fail the command; warnings fail with `--warnings-as-errors`. `--timeout 60000` bounds the worker lifetime.--`pnpm check:render` adds GPU verification, a frame and a three-point grid under `output/check`; `--output <dir>` changes the location. This smoke check does not replace targeted pixel assertions or video QA. A failed or unavailable GPU check is reported as a failure. Use `pnpm exec framefields api` for timing, grid and preflight type declarations generated from this SDK version.+- `npx tsc --noEmit -p .` typechecks.+- `npx framefields check [name] --json` imports the render's export in an isolated process and writes one versioned JSON result (no name: every render; a module: `check src/film.ts --export buildFilm`). Errors fail it; warnings with `--warnings-as-errors`; `--timeout 60000` bounds it. The export must only build: importing it runs project code.+- `--render` (or `pnpm check:render` in scaffolded projects) adds a GPU smoke check: a frame and a three-point grid under `output/check` (`--output <dir>`). Not a substitute for pixel assertions or video QA.+- `npx framefields api` prints timing, grid and preflight type declarations for the installed version.------### Step 2: Single Frame Inspection (`renderFrame`)-Render isolated keyframe points (e.g. entry, peak action, exit) directly to PNG files on disk to inspect alignment, typography, colors, and opacity:+### 2. Single frames (`renderFrame`)```typescript-import fs from "node:fs/promises";-import path from "node:path";import { HeadlessMediaRenderer } from "framefields";const renderer = new HeadlessMediaRenderer();--// Render frame 45 (1.5s at 30fps)-const pngBuffer = await comp.renderFrame({- frame: 45,- renderer,-});--await fs.writeFile("output/frame-45.png", pngBuffer);+await fs.writeFile("output/f45.png", await comp.renderFrame({ frame: 45, renderer }));```-`atMs` takes precedence when both `frame` and `atMs` are supplied, including for frame hooks. Frame indices must be nonnegative safe integers; fractional timestamps and FPS are supported. A composition's FPS is authoritative: a different render/grid FPS override throws. Media pipelines can still select their rendering FPS.+`atMs` wins over `frame`. Frames are nonnegative safe integers; fractional timestamps and FPS are fine. The composition's FPS is authoritative: a different render/grid FPS throws.------### Step 3: Motion & Layout Stability Contact Sheets (`renderFrameGrid`)-To verify easing curves, cut points, kinetic sweeps, or text wrapping without encoding a full video, generate a sequential frame grid image:+### 3. Contact sheets (`renderFrameGrid`)```typescript-import fs from "node:fs/promises";--// Samples frames across the intro sweep (frames 0 to 60)-const gridPng = await comp.renderFrameGrid({+const png = await comp.renderFrameGrid({frames: [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60],- columns: 4,- cellWidth: 480, // Downscaled thumbnail size for rapid visual scanning- showLabels: true, // frame number and timestamp under each cell+ columns: 4, cellWidth: 480, showLabels: true,});--await fs.writeFile("output/intro-grid.png", gridPng);```-Contact sheets immediately reveal:-- Motion hitches and non-smooth easing deceleration.-- Text clipping or wrapping jumps across keyframe boundaries.-- Bad layer z-indexing or missing cut transitions.+Shows hitches, easing, text wrap jumps, z-order and missing cuts at a glance. Automatic sampling stops before the exported frame count (`max(1, round(durationMs * fps / 1000))`); explicit frames may go past it. Order and duplicates are kept. At most 64 samples by default, 120 max.-Grid frame ranges are inclusive, but the composition's duration is exclusive. Automatic sampling stops before the exported frame count (`max(1, round(durationMs * fps / 1000))`). Explicit points may sample beyond the export window for diagnostics. Sample order and duplicates are preserved. Frame steps must be positive safe integers, timestamp steps finite and positive. The sample limit is checked while generating points (64 by default, at most 120); invalid or excessive requests fail before creating a large list. Labels describe the actual selected frame, with timestamps formatted for display only.-------### Step 4: Programmatic Conformance Probes in Vitest-Write automated headless vitest tests that assert pixel colors, transparency, or layout bounding boxes:+### 4. Pixel tests (vitest)```typescript-import { describe, it, expect } from "vitest";-import { HeadlessMediaRenderer } from "framefields";-import { buildFilm } from "../film.js";--describe("Hero Scene Conformance", () => {- it("renders non-black title text on frame 30", async () => {- const comp = await buildFilm();- const renderer = new HeadlessMediaRenderer();- const png = await comp.renderFrame({ frame: 30, renderer });-- expect(png.length).toBeGreaterThan(1000);- // Use skia-canvas or pixel probes to assert color at target coordinates- });+it("renders the title on frame 30", async () => {+ const comp = await buildFilm();+ const png = await comp.renderFrame({ frame: 30, renderer: new HeadlessMediaRenderer() });+ expect(png.length).toBeGreaterThan(1000); // decode and probe pixels at known coordinates});```----+### 5. Show the user (`preview`)-### Step 5: Show the User Before Exporting (`preview`)-When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code and the project's files. The soundtrack is the export's own mix, made in the page around the playhead, so sound starts at once. Point it at the module that builds the composition:+When a person is in the loop, let them watch before the export. `npx framefields preview <name>` or `startPreview` serves a localhost player (sound, waveform timeline, frame stepping). The page runs the composition's own code with WebGPU in the browser in real time; a slow device skips frames to keep the sound in time. The soundtrack is the export's own mix.```typescriptimport { startPreview } from "framefields";-const session = await startPreview(- { entry: new URL("./film.ts", import.meta.url), export: "buildFilm" },- { title: "Launch film" },-);+const session = await startPreview({ entry: new URL("./film.ts", import.meta.url), export: "buildFilm" }, { title: "Launch film" });console.log(`Preview at ${session.url}`);-await session.closed; // keep the process alive while the user watches+await session.closed; // "idle" | "replaced" | "closed"```-- **The export is a function returning the `Composition`** (it may be async), or a `Composition` itself. The browser calls it, so build-time Node code in it must stay simple: `node:path`, `fileURLToPath(import.meta.url)` and synchronous `fs` reads (JSON, text) work through shims; writes are skipped. Assets referenced by absolute path load from the server, limited to the project that holds the entry (its nearest `.git` or `package.json`); pass `root` with one or more paths for assets kept outside it (e.g. a shared fonts folder).-- **Run it in the background, and read its output.** Start the script as a background command with the longest timeout your harness allows, tell the user the URL, and ask for feedback in the chat. The command's output (stderr, prefixed `[framefields preview]`) says when the page opened, reports each distinct error from the page once (a build error, a frame that threw, a font or renderer that failed to load, no WebGPU), and says why it stopped. Check it after the page opens instead of assuming the film drew. `log` redirects or silences these lines, and `session.closed` resolves with the reason: `"idle"`, `"replaced"` or `"closed"`.-- **One tab for the whole loop.** The port is fixed per project directory. Running the preview again (after you've made changes) takes over from the one already running, and a tab left open reconnects and reloads into the new version by itself, at the moment it was showing. Don't ask the user to reopen anything. When the user closes the tab, the preview stops by itself about 5 s later (a reload doesn't stop it; `idleCloseMs` changes the delay). If the preview was stopped (tab closed, time limit, or you killed it), just start it again.-- **Notes pinned on the picture are the user's precise feedback.** Paused, the user clicks the frame (or drags an area) and writes what should change; the note keeps the frame, the spot as fractions of the frame (`x`, `y`, and `w`, `h` for an area) and the text. Each note prints in the output (`note 3 at 0:12.50 (frame 750), spot 42%,31%: …`) with the path of `note-<id>.jpg`, the frame with the spot marked, and all of them are in `.framefields/preview-notes/notes.json` (`.framefields` ignores itself in git; `notesDir` moves it, `false` turns notes off). To act on them: read `notes.json`, open each open note's picture to see exactly what it points at, find the layer in the code from the moment and the spot, make the change, and run the preview again. The tab comes back with the notes, so the user can check them and tick them off. You can also set `"done": true` in `notes.json` yourself. A chat that can't read files gets the same list from the **Copy** button in the Notes panel.-- **Point at a moment.** `#t=12.5` in the URL opens the player at 12.5 s (`#frame=750` by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.-- **Drive it from a script, not with clicks.** If you can run JavaScript in the page (an agent browser), use `window.framefieldsPreview`: `await seek(seconds)` or `await seekFrame(n)` resolve once that frame is drawn (take the screenshot then), `play()` and `pause()` control playback, `notes()` lists the notes, and `state()` returns `{ state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }`. `<html data-preview="ready">` marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.-- **Show it where the user is.** `startPreview` only serves the page and returns the URL. If your app has a built-in browser, open the URL there. Otherwise give the user the link, or pass `open: true` to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).-- **Playback runs in real time.** A slow device skips frames to keep pace with the sound; `state()` reports the reached `playbackFps` and the last frame's `renderMs`. Paused, the page draws the exact frame. Vision effects analyse each frame from the one drawn before it, so check cutouts and tracking paused or in the export.+- **The export runs in the browser.** It may be a `Composition` or a (async) function returning one. `node:path`, `fileURLToPath(import.meta.url)` and synchronous `fs` reads work through shims; writes are skipped. Build asset paths from `import.meta.url`, not `process.cwd()`. Files inside the project (nearest `.git` or `package.json`) are served; pass `root: [...]` for others (e.g. a shared fonts folder).+- **Run it in the background and read its output.** Stderr lines (`[framefields preview]`) say when the page opened, each distinct page error once (build error, frame threw, missing font, no WebGPU), and why it stopped. Check them instead of assuming the film drew.+- **One tab for the whole loop.** The port is fixed per project. Running it again replaces the running one and the open tab reloads at the same moment. It stops ~5 s after its last tab closes (`idleCloseMs`); just start it again.+- **Show it where the user is.** It opens nothing: use the app's built-in browser if there is one, else give the link, or pass `open: true`. Needs current Chrome, Edge or Safari.+- **Notes.** Paused, the user clicks (or drags an area) and writes a note. Each prints (`note 3 at 0:12.50 (frame 750), spot 42%,31%: …`) and is saved in `.framefields/preview-notes/notes.json` with `note-<id>.jpg` (the frame, spot marked; `notesDir` moves it, `false` disables). To act: read `notes.json`, open each open note's picture, find the layer from the moment and spot, change it, rerun the preview. You may set `"done": true` yourself.+- **Links and scripting.** `#t=12.5` or `#frame=750` opens at a moment. `window.framefieldsPreview` has `seek(seconds)` / `seekFrame(n)` (resolve once drawn: screenshot then), `play()`, `pause()`, `notes()`, `state()` (`{ state, error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }`). `<html data-preview="ready">` marks the first frame. Hidden tabs pause.+- Vision effects analyse each frame from the one drawn before: check cutouts paused or in the export.------### Step 6: Full Video Export (`renderVideo`)-Once static frames and motion grids are verified, encode the final MP4/WebM video:+### 6. Export (`renderVideo`)```typescriptconst result = await comp.renderVideo({- outputPath: "output/final-video.mp4",- quality: "very_high", // "very_low" | "low" | "high" | "very_high" (default)- codec: "h264", // "h264" (default, MP4) | "vp9" | "vp8" (WebM) | "gif"; "aac" | "mp3" | "opus" render audio only+ outputPath: "output/film.mp4",+ quality: "very_high", // very_low | low | high | very_high (default)+ codec: "h264", // h264 (default) | vp9 | vp8 | gif; aac | mp3 | opus = audio only+ qa: true, // see Video QA});-await result.cleanup?.(); // releases temporary files+await result.cleanup?.();```-There is no HEVC or AV1 output; an unknown `codec` falls back to H.264.+No HEVC or AV1; an unknown codec falls back to H.264. Or from the CLI: `npx framefields render <name>` uses the project's own tsx and framefields, calls the export with the render's `args`/`select`, downloads missing pinned assets first, and writes the render's `output`.-```typescript-```+Always end render scripts with `process.exit(0)`: native GPU and encoder threads keep Node alive.-Pass `qa: true` to have the engine check the output while it renders (see **Built-in Video QA** below) and return the findings as `result.qa`.+## Named renders (`framefields.json`)-### Splitting a long render into chunks-To render on several GPUs at once, have each process render a range of frames without audio, render the audio once, then join the files without re-encoding:--```typescript-import { stitchChunks } from "framefields/renderer";--// On each worker (its own process and composition): frames start..end-1.-await comp.renderVideo({ outputPath: "out/chunk-0.mp4", frames: { start: 0, end: 900 }, audio: false });-// Once, anywhere: the whole timeline's audio.-await comp.renderVideo({ outputPath: "out/audio.mp4", codec: "aac" });-// On any machine (no GPU needed): chunks in timeline order.-await stitchChunks({ chunks: ["out/chunk-0.mp4", "out/chunk-1.mp4"], audio: "out/audio.mp4", outputPath: "out/final.mp4" });-```--- Trails, datamosh and particles rebuild their state before `start` by replaying earlier frames, so a chunk starting late costs extra time but renders the same picture.-- Render every chunk with the same framefields version, codec and quality on the same GPU type. `stitchChunks` refuses chunks with a different codec, size or encoder parameters.-- SAM 2 / SAM 3 tracks run in order from their first prompted frame. Run the vision analysis once and point every worker at the shared cache (`FRAMEFIELDS_VISION_CACHE_DIR`), so no chunk re-tracks the frames before it.-- `qa` checks each chunk on its own. Check the stitched file with ffmpeg (below).-- `planChunks({ totalFrames, samples, workers: "auto" })` picks the chunk count and cuts: it adds workers only while each chunk spends at least 85% of its time rendering (about 5.7× the per-worker start-up), and splits at equal render cost. `renderer.probeRender(comp)` measures the `samples`.--## Named Renders (`framefields.json`)--`framefields.json` at the project root names what the project renders. `init` writes a `film` render; add one per cut (teaser, vertical edit, language) instead of parameterizing `render.ts`. The cloud CLI, `renderCloud`, `framefields check` and the dashboard's New render dialog all pick renders by name, so nobody chooses files and exports.+`framefields.json` names what the project renders. Add one per cut (teaser, vertical, language) rather than flags in `render.ts`; the CLI, `framefields check`, the cloud and the dashboard pick them by name.```json{- "components": "src/components","renders": {"film": { "title": "Launch film", "entry": "src/film.ts", "export": "buildFilm", "workers": "auto", "output": "output/film.mp4" },- "teaser": { "title": "15 s teaser", "description": "Vertical cut for socials", "entry": "src/film.ts", "export": "buildTeaser", "codec": "vp9" }+ "teaser": { "title": "15 s teaser", "description": "Vertical cut", "entry": "src/film.ts", "export": "buildTeaser", "codec": "vp9" }}}```| Field | Meaning ||---|---|-| `entry` | Module that builds the composition, relative to the project root. Required. |-| `export` | A `Composition`, or a function returning one. Default `buildFilm`. |-| `title`, `description` | Shown in the dashboard and `framefields renders`. Write them for the user, not for code. |-| `args` | Passed to the export when it is a function. |-| `select` | When the export returns several compositions (a `Record`), the one to render. |-| `workers` | Cloud GPUs: `1` (default), a number, or `"auto"`. |-| `maxWorkers` | Upper bound for `"auto"`. |-| `codec` | `h264` (default, MP4) or `vp9` (WebM). |-| `quality` | `very_low`, `low`, `high` or `very_high` (default). |-| `output` | Where `framefields render <name> --cloud` downloads the video. |--- Names use letters, digits, `-`, `_` and `.` (at most 64). Errors name the render and field: `framefields.json: renders.teaser needs "entry"`.-- `npx framefields renders` lists them (`--json` for tools). `npx framefields check` with no argument checks every render; `npx framefields check teaser` checks one (artifacts under `output/check/<name>` when checking several).-- After adding a render, run `framefields check <name>`, then commit and push: the cloud and the dashboard read `framefields.json` from the pushed branch, and the dashboard greys out a render whose `entry` isn't on that branch.-- Named renders need framefields 2.12 or later. With older versions, render a module path (below).--## Rendering in the Cloud (`framefields render --cloud`)--api.framefields.dev renders the project on framefields' GPUs and splits a long film across several. It fetches the project from git, so it renders the **pushed commit**:--- Commit and push first. Uncommitted changes are not rendered (the CLI warns), a commit not on the remote is refused, and so is a folder outside a git repo or without a remote.-- Run it from the project folder (where `framefields.json` and `package.json` are). In a monorepo, the CLI sends that folder as the workdir.-- The worker installs the project from its lockfile and renders with **the project's own framefields version**, so commit the lockfile. Footage, audio and fonts must be pinned in `framefields.json` with `framefields assets add`, in Git LFS, or committed; other `.gitignore`d files don't exist on the worker.-- Pinned assets: before starting, the CLI (and `renderCloud`) uploads any the bucket lacks and signs reads of assets on remotes without a `publicUrl` (valid 24 h; the keys stay on the machine). Each worker downloads them (cached by hash for its next job); on R2 those downloads are free, unlike LFS bandwidth. Dashboard renders have nobody to sign, so they read only remotes with a `publicUrl`.-- Private repos: the user connects GitHub once at https://framefields.dev/dashboard/settings#connections (the framefields GitHub App, ideally on all repositories, so new repos render without another step). Public repos need nothing.-- It bills the account behind `FRAMEFIELDS_API_KEY` (keys at https://framefields.dev/dashboard/keys). If authentication is required, ask the user to set it in the process environment; never print, write or embed it.--```bash-npx framefields render film --cloud --plan # workers, time and estimated cost; renders nothing-npx framefields render film --cloud # render, then download to the render's output-npx framefields render --cloud # the only named render-npx framefields render src/film.ts --cloud --export buildFilm --workers auto # a module, without a manifest-```--- Flags override the named render's fields: `--workers <n|auto>`, `--max-workers <n>`, `--codec h264|vp9`, `--quality`, `--select <name>`, `--output <file>`, `--export`. `--json` prints the final status as JSON.-- `--workers` defaults to the render's `workers`, else 1 (one GPU, no planning step). `auto` measures frame costs on one worker, then uses as many workers as stay efficient: a 5-minute film stays on one, a 3-hour film spreads to `--max-workers` (API cap 10). A number splits into that many.-- Show the user `--plan` before a long or expensive render: it prints `Plan: 4 workers, 1 h 50 min of render work, ready in about 35 min, estimated $1.20.` and the command that renders it.-- The CLI prints one line per phase: measuring (`probing`), rendering N chunks and the audio in parallel, joining chunks (`stitching`), then `Done in …: <file>, <GPU seconds>, <cost>.` While rendering it prints progress at most every 10 s with the cost so far. Let it run; a split render of a long film takes minutes.-- Billing: a credit hold is placed when the render starts (raised to the estimate once planned); the final charge is the GPU seconds actually used, and the rest is released. Failed and cancelled renders charge the GPU time they used. `insufficient_credits` means the user must add funds at https://framefields.dev/dashboard/billing.-- Ctrl-C cancels the render; GPU time already used is charged.-- The dashboard (https://framefields.dev/dashboard/renders) shows every render, API or dashboard started, with live progress, cost so far, Cancel and the finished video. Its **New render** dialog lists the named renders of a connected repo's branch, with Plan and Render.--From code (no GPU needed locally):--```ts-import { CloudRenderError, renderCloud } from "framefields/cloud";--try {- const result = await renderCloud({- render: "film", // or { entry: "src/film.ts", export: "buildFilm", workers: "auto" }- outputPath: "output/film.mp4", // downloads the finished video- onProgress: (s) => console.log(s.status, s.progress.frames, s.progress.totalFrames, s.costUsd),- // dryRun: true // plan only: result.plan and result.estimateUsd- });- console.log(result.filePath, result.gpuSeconds, result.costUsd, result.output?.url);-} catch (err) {- if (err instanceof CloudRenderError) console.error(err.code, err.message); // the worker's code, e.g. ASSET_NOT_FOUND- else throw err;-}-```--Options passed to `renderCloud` override the named render's; `signal` cancels. `cloudClient({ apiKey })` (key defaults to `FRAMEFIELDS_API_KEY`) exposes `start`, `status` and `cancel` for callers that track renders themselves. The HTTP API is `POST /v1/render` (`source`, `target`, `render`, `workers`, `max_workers`, `dry_run`, optional `name` and `title`), `GET` and `DELETE /v1/render/:id`, with `Authorization: Bearer <key>`.--### Cloud render failures--The error carries the worker's code. Fix the project, commit, push and render again.--| Code | Cause | Fix |-|---|---|---|-| `REPO_NOT_FOUND`, `GIT_AUTH_FAILED` | Private repo the framefields GitHub App can't reach | Connect GitHub (Settings → Connections) with access to this repo. |-| `REF_NOT_FOUND`, `COMMIT_MISMATCH` | Branch or commit not on the remote | `git push`. |-| `LOCKFILE_MISSING`, `INSTALL_FAILED` | No committed lockfile, or it doesn't install | Commit `pnpm-lock.yaml` / `package-lock.json`; check it installs cleanly. |-| `ENTRY_NOT_FOUND`, `EXPORT_NOT_FOUND`, `EXPORT_INVALID` | `entry`/`export` in framefields.json or flags are wrong | `framefields check <name>` locally. |-| `COMPOSITION_BUILD_FAILED`, `EXPORT_SIDE_EFFECT` | The builder throws, or renders/writes while building | Keep builders pure; `framefields check` reproduces it. |-| `ASSET_NOT_FOUND`, `LFS_OBJECT_MISSING`, `LFS_POINTER_UNRESOLVED` | An asset isn't in the repo or its LFS object wasn't pushed | `git lfs ls-files`, `git lfs push --all origin`; un-ignore assets the film needs. |-| `ASSET_NOT_UPLOADED`, `ASSET_HASH_MISMATCH` | A pinned asset isn't on the bucket, or the bucket holds other bytes | `npx framefields assets push` from the machine with the right file. |-| `ASSET_PRIVATE`, `ASSET_FORBIDDEN` | The worker can't read the bucket: no `publicUrl` and no signature (a dashboard render), or the custom domain doesn't serve it | Render from the CLI, or give the remote a working `publicUrl`. |-| `ASSET_MANIFEST_INVALID` | `storage` or `assets` in framefields.json is malformed | `npx framefields assets status` shows the error locally. |-| `ENGINE_UNSUPPORTED` | The project's framefields is too old to render chunks | Upgrade framefields in the project (2.11+), or render with `--workers 1`. |-| `RENDER_TIMEOUT`, `RENDER_OOM` | One worker can't finish its part | Use `--workers auto` (or more workers) to split it. |--Retryable infrastructure errors (`GIT_NETWORK`, `RENDER_STALLED`, a lost worker) are retried by the API, and a straggling chunk gets a backup worker; they rarely reach the user.-------## Rendering Without a GPU--Agents often run in sandboxes or CI runners with no GPU. framefields then renders on the CPU, with the same output for every feature (text, media, Lottie, effects, 3D, vision; vision inference moves to the CPU too).+| `entry` | Module that builds the composition (required) |+| `export` | A `Composition` or a function returning one (default `buildFilm`) |+| `title`, `description` | Shown to the user in the dashboard and `framefields renders` |+| `args` / `select` | Arguments for a function export / which composition when it returns a `Record` |+| `workers`, `maxWorkers` | Cloud GPUs: `1` (default), a number, or `"auto"` |+| `codec`, `quality` | `h264` / `vp9`; `very_low` … `very_high` |+| `output` | Where the video is written |-- **Linux:** install Mesa lavapipe, `apt-get install -y libvulkan1 mesa-vulkan-drivers`. Without a Vulkan driver at all, nothing can render.-- **Windows:** Direct3D 12's WARP renderer is built in; lavapipe (e.g. from mesa-dist-win) works through Vulkan too.-- **macOS:** always has a Metal GPU, VMs included; framefields' WebGPU build has no Vulkan, so lavapipe isn't used there.-- **Confirm it:** `await describeGpu()` returns `software: true`.-- **Budget for speed:** about 12× slower than an Apple M1 Pro's GPU in our Linux test (the smoke project's three videos: 9.7 min against 49 s); it scales with CPU cores. Frames, grids and short exports are fine. Vision layers are the slowest part.-- **Long films:** push the project and render it in the cloud (`npx framefields render film --cloud`, above), or on a GPU machine of your own. To run the render worker yourself (Docker or RunPod, renders a git commit on a GPU), see its project checklist and job format: https://framefields.dev/docs/guides/render-service/+Names: letters, digits, `-`, `_`, `.` (≤ 64). `npx framefields renders [--json]` lists them. After adding one, run `framefields check <name>`. Needs framefields 2.12+.-## Production Render Script Pattern--Production projects structure their `src/render.ts` to support both quick frame probes and full exports from the CLI:--```typescript-// src/render.ts-import fs from "node:fs/promises";-import path from "node:path";-import { HeadlessMediaRenderer, startPreview } from "framefields";-import { buildFilm } from "./film.js";--const OUT = path.resolve(import.meta.dirname, "../output");-const [mode = "video", ...args] = process.argv.slice(2);--if (mode === "preview") {- // The browser runs film.ts itself; no need to build the film here.- const session = await startPreview({ entry: new URL("./film.ts", import.meta.url), export: "buildFilm" });- console.log(`Preview at ${session.url}`);- await session.closed; // serves until its tab closes or the next preview replaces it- process.exit(0);-}--const comp = await buildFilm();-await fs.mkdir(OUT, { recursive: true });--if (mode === "frames") {- const renderer = new HeadlessMediaRenderer();- await fs.mkdir(path.join(OUT, "frames"), { recursive: true });- for (const f of args.map(Number)) {- const png = await comp.renderFrame({ frame: f, renderer });- const file = path.join(OUT, "frames", `f${String(f).padStart(4, "0")}.png`);- await fs.writeFile(file, png);- console.log(`Rendered frame: ${file}`);- }-} else {- const started = Date.now();- const result = await comp.renderVideo({- outputPath: path.join(OUT, "output.mp4"),- quality: "high",- });- await result.cleanup?.();- console.log(`Exported video: ${result.filePath} in ${((Date.now() - started) / 1000).toFixed(1)}s`);-}--// CRITICAL: Always explicitly exit process to prevent native WebGPU/mediabunny thread hangs-process.exit(0);-```--CLI Usage:-```bash-# Render specific keyframes for instant review:-tsx src/render.ts frames 0 18 36 72+## Reading render progress-# Live player for the user (run in the background; re-run after changes and the open tab reloads):-npx framefields preview film--# Render the full video on this machine, to the render's output in framefields.json:-npx framefields render film--# The same render on framefields cloud GPUs, to the same file:-npx framefields render film --cloud-```--`framefields render <name>` and `framefields preview <name>` load the module with the project's own tsx and framefields, like the cloud worker, and call the export with the render's `args` (and `select`); the preview calls it without them. A local render downloads pinned assets that are missing first; the preview streams them from their bucket.--### Reading render progress--`renderVideo` logs its own progress to stderr, one short `[framefields]` line per event. The log stays short however long the film is, about 15 lines for an export:+`renderVideo` logs about 15 short `[framefields]` lines to stderr however long the film:```text[framefields] Preparing 1800 frames (60.0s) at 1920x1080, 30 fps, h264-[framefields] Rendering 1800 frames[framefields] Rendered 10% (180/1800 frames) · 41.2 fps · eta 39.3s-…-[framefields] Rendered 100% (1800/1800 frames) · 41.8 fps · in 43.1s[framefields] Mixing audio-[framefields] Encoding 60.0s of audio (aac)[framefields] Muxing mp4[framefields] Done in 51.4s (14.2 MB): prepare 1.2s · render 43.1s · audio 3.2s · audio-encode 1.1s · mux 0.8s```-- **Work continues after the last frame.** "Mixing audio" decodes every audio and video clip in the film, then the soundtrack is encoded and "Muxing" writes the file. With many or long clips this takes a while. It is not a hang; wait for `Done in …`.-- **A quiet phase prints `Still …` lines**, e.g. `Still mixing audio (42%, 3/7 clips: music.mp3), 1m 10s in this phase`: after 10 s, 30 s, 70 s, then backing off to one every 5 min. Each says which phase is running and for how long. A render is stuck only when no line arrives for well over the last gap. Killing it at any point leaves no video.-- **A failure names its phase:** `Failed while mixing audio after 12.3s`, with the error.-- **Run long exports in the background** with the longest timeout your harness allows, and read the output when it finishes rather than polling it.-- `LOG_LEVEL=warn` silences these lines; `LOG_LEVEL=debug` adds per-stage detail. Warnings and errors read `[framefields] warn: …` / `error: …`.-- With `NODE_ENV=production` (or `FRAMEFIELDS_LOG_FORMAT=json`) the lines are pino JSON on stdout instead, with the text in `msg`; `FRAMEFIELDS_LOG_FORMAT=text` forces the short lines.-- For your own progress output, `onProgress` gets every event as `{ phase, frame, totalFrames, elapsedMs, phaseProgress?, fps?, etaSec?, detail? }`. `phase` runs `prepare`, `render` (audio-only output skips it), `audio`, `audio-encode`, `mux`, `convert` (GIF only), `done`.+- **Work continues after `Rendered 100%`:** mixing decodes every clip's audio, then encode and mux. Not a hang; wait for `Done in …`.+- Quiet phases print `Still …` lines after 10 s, 30 s, 70 s, then every 5 min. It's stuck only when no line arrives well past the last gap. Killing it leaves no video. Failures name their phase.+- Run long exports in the background with the longest timeout available and read the output when done.+- `LOG_LEVEL=warn|debug`; `NODE_ENV=production` or `FRAMEFIELDS_LOG_FORMAT=json` gives pino JSON on stdout.+- `onProgress` receives `{ phase, frame, totalFrames, elapsedMs, phaseProgress?, fps?, etaSec?, detail? }`; phases `prepare`, `render`, `audio`, `audio-encode`, `mux`, `convert` (GIF), `done`.------## Built-in Video QA (`renderVideo({ qa })`)+## Video QA (`renderVideo({ qa })`)-The engine measures the video while it renders: every frame as it is read back for the encoder, and the audio as it is mixed. No second decode pass, and the numbers describe exactly what was encoded.+The engine measures every frame as it's read back and the audio as it's mixed: no second decode.```typescriptimport { formatQaReport } from "framefields";-const result = await comp.renderVideo({- outputPath: "output/final-video.mp4",- qa: { targetLufs: -14 }, // or `qa: true` for defaults (no loudness target)-});+const result = await comp.renderVideo({ outputPath: "output/film.mp4", qa: { targetLufs: -14 } });console.log(formatQaReport(result.qa!));-if (!result.qa!.passed) process.exitCode = 1; // fail CI / the agent loop+if (!result.qa!.passed) process.exitCode = 1;```-`result.qa` is structured, so branch on it instead of parsing logs:-| `issues[].code` | Severity | Meaning ||---|---|---|| `frames_missing` | error | Fewer frames encoded than the composition has |-| `black_frames` | warning | ≥ `minBlackSec` (0.5 s) of black frames; has `startSec`/`endSec` |+| `black_frames` | warning | ≥ `minBlackSec` (0.5 s) of black; `startSec`/`endSec` || `audio_clipping` | warning | Samples at or beyond full scale |-| `loudness_off_target` | warning | Integrated loudness more than `lufsTolerance` (1 LU) from `targetLufs` |-| `animation_truncated` | warning | Keyframes past a layer's clip window never play; has `layerId` |-| `frozen_video` | info | Picture unchanged for ≥ `minFrozenSec` (2 s); fine for held title cards |-| `audio_silence` | info | ≥ `minSilenceSec` (2 s) below `silenceDbfs` (-60) |-| `audio_silent` | info | The whole track is silent |-| `invalid_gradient` | error | A `background` / `fill` gradient string that does not parse: it draws nothing |-| `unsupported_image_format` | error | An image layer receiving a `.json` / `.lottie` vector animation, video, or audio file |-| `unknown_color` | warning | A color string that is not a color: it draws transparent |-| `low_contrast_text` | warning | Text below 3:1 contrast against the solid color behind it (suggests a readable one) |-| `offscreen_layer` | warning | A static absolute layer entirely outside the canvas |-| `duplicate_id` | error | Two layers share an id: the renderer keeps one and drops the other |-| `shadow_unsupported_light` | warning | `castShadow` on a light that casts none (ambient, environment) |-| `pbr_without_environment` | info | `roughness` / `metallic` / `clearcoat` / `anisotropy` set but no `Layer.environment()` (no physical shading or coat reflections) |-| `static_track` | info | Every keyframe on a track holds the same value |+| `loudness_off_target` | warning | More than `lufsTolerance` (1 LU) from `targetLufs` |+| `animation_truncated` | warning | Keyframes past a layer's window never play |+| `frozen_video` | info | Unchanged for ≥ `minFrozenSec` (2 s) |+| `audio_silence` / `audio_silent` | info | ≥ 2 s below `silenceDbfs` (-60) / whole track silent |-The document checks also run on their own, before any render: `comp.lint()` returns the same `{ severity, code, message, layerId }` entries. Run it in Step 1, right after typechecking.+`result.qa` also includes every `comp.lint()` finding (`invalid_gradient`, `unknown_color`, `low_contrast_text`, `offscreen_layer`, `duplicate_id`, `unsupported_image_format`, `shadow_unsupported_light`, `pbr_without_environment`, `static_track`, …). `passed` is false on any error or warning. `qa.audio.integratedLufs` is BS.1770 (platforms normalize to about -14); `qa.audio.peakDbfs` is the sample peak.-`passed` is false when any error or warning is present. `qa.audio.integratedLufs` is ITU-R BS.1770 loudness (streaming platforms normalize to about -14 LUFS); `qa.audio.peakDbfs` is the sample peak.+For files rendered elsewhere:-## Inspecting a Finished File with ffmpeg+```bash+ffprobe -v error -show_entries stream=codec_name,width,height,duration,r_frame_rate out.mp4+ffmpeg -i out.mp4 -vf "blackdetect=d=0.5:pix_th=0.10" -f null -+ffmpeg -i out.mp4 -af volumedetect -f null -+```-For files rendered elsewhere, or to compare against a previous render, inspect with `ffmpeg`/`ffprobe`:+## Splitting a long render-```bash-# 1. Check video and audio stream properties-ffprobe -v error -show_entries stream=codec_name,width,height,duration,r_frame_rate output/final-video.mp4+Each process renders a frame range without audio, the audio renders once, then the files join without re-encoding:-# 2. Detect unexpected black frames or dead transitions-ffmpeg -i output/final-video.mp4 -vf "blackdetect=d=0.5:pix_th=0.10" -f null -+```typescript+import { stitchChunks } from "framefields/renderer";-# 3. Verify audio loudness levels (prevent clipping / silence)-ffmpeg -i output/final-video.mp4 -af "volumedetect" -f null -+await comp.renderVideo({ outputPath: "out/chunk-0.mp4", frames: { start: 0, end: 900 }, audio: false }); // per worker+await comp.renderVideo({ outputPath: "out/audio.mp4", codec: "aac" }); // once+await stitchChunks({ chunks: ["out/chunk-0.mp4", "out/chunk-1.mp4"], audio: "out/audio.mp4", outputPath: "out/final.mp4" }); // no GPU```----+- Trails, datamosh and particles replay earlier frames before `start`: same picture, extra time.+- Same framefields version, codec, quality and GPU type for every chunk; `stitchChunks` refuses mismatched codec, size or encoder parameters.+- Run SAM tracking once and share `FRAMEFIELDS_VISION_CACHE_DIR` so no chunk re-tracks earlier frames.+- `qa` checks each chunk alone; check the stitched file with ffmpeg.+- `planChunks({ totalFrames, samples, workers: "auto" })` picks count and cuts (adds workers while each spends ≥ 85% of its time rendering; splits at equal cost). `renderer.probeRender(comp)` measures `samples`.++The cloud does all of this for you: `framefields-cloud`.++## Rendering without a GPU++framefields falls back to the CPU with identical output for every feature (vision inference too).++- **Linux:** `apt-get install -y libvulkan1 mesa-vulkan-drivers` (lavapipe). Without any Vulkan driver nothing renders.+- **Windows:** WARP (built in) or lavapipe. **macOS:** always has Metal.+- `await describeGpu()` reports `software: true`.+- About 12× slower than an M1 Pro GPU, scaling with cores. Fine for frames, grids and short exports; render long films in the cloud (`framefields-cloud`) or on a GPU machine.-## Common Rendering Failure Modes+## Failure modes-| Symptom | Root Cause | Solution |+| Symptom | Cause | Fix ||---|---|---|-| Entire frame renders black | Missing font registration or `NaN` in a signal | Register every font `Layer.text` uses with `FontManager.register()` before building layers. Check signal outputs are finite numbers. |-| Media layer is invisible / blank | Wrong path, an LFS pointer file, or zero opacity | Check the path exists and is real media (`git lfs pull`, `framefields assets pull`), and `opacity > 0`. |-| Audio cuts out early | Missing `durationFrames` on audio layer | Explicitly assign `durationFrames: totalFrames` on `Layer.audio()`. |-| Vision cutout / mask is empty on a single `renderFrame` | Vision reads the previous rendered frame; frame 0 has none | Render two or more consecutive frames with the same `HeadlessMediaRenderer` and keep the last. `renderVideo` handles this automatically. |-| Ghosted double of the subject in a vision cutout | Frames rendered out of order (frame grid, or jumping from frame 3 to 48), so the cutout uses pixels from another frame | Verify vision layers with sequential frames or the exported MP4, not `renderFrameGrid`. |-| Output stops after `Rendered 100%` | Audio is being mixed, encoded or muxed | Not a hang: wait for `Still …` lines and `Done in …` (see **Reading render progress**). |-| Script hangs after `Done in …` | Background worker thread pool held open | Always include `process.exit(0)` at the end of render scripts. |-| Render very slow on a Linux server | Frames drawn on a CPU adapter (lavapipe) | `await describeGpu()` shows `software: true`; fix the GPU's Vulkan driver (`vulkaninfo --summary`). |-| Encoder fails to open (`h264_rkmpp`, NVENC) | framefields < 2.2 uses the hardware encoder untested | Use framefields 2.2+, which falls back to libx264 etc., or set `FRAMEFIELDS_HW_CODECS=off`. |-| Render worker or cloud render fails | Project not render-ready (lockfile, export, LFS) | Read the `code` and `hint`; see **Cloud render failures** above and https://framefields.dev/docs/guides/render-service/ |-| Stitched video has a jump at a chunk seam | Chunks rendered with different framefields versions, settings or GPU types, or a stateful effect that reads wall-clock time | Render all chunks from one commit and one job; keep effects deterministic in the frame number. |-| An effect shows no change | Value on the wrong scale (e.g. `Vignette` `strength: 0.4` on its 0–100 scale) | Look the prop up in the `framefields-effects` catalog. |-| Text looks misaligned or truncated | Layout node box smaller than text string | Set explicit wrapping `width` (e.g. `width: 1400`) or use `width: "fill"`. |+| Whole frame black | Unregistered font or `NaN` in a signal | `FontManager.register()` every font before building; keep signals finite. |+| Media layer blank | Wrong path, LFS pointer, zero opacity | Check the file is real (`git lfs pull`, `framefields assets pull`). |+| Audio cuts out early | No `durationFrames` on `Layer.audio()` | Set it to the film's length. |+| Vision cutout empty on a single frame | Vision reads the previous frame | Render ≥ 2 consecutive frames with one renderer; `renderVideo` handles it. |+| Ghosted double in a cutout | Frames rendered out of order (grids, jumps) | Check vision with sequential frames or the export. |+| Pause after `Rendered 100%` | Audio mix / encode / mux | Wait for `Still …` and `Done in …`. |+| Script hangs after `Done in …` | Worker threads hold Node | `process.exit(0)`. |+| Very slow on a Linux server | CPU adapter (lavapipe) | `describeGpu()`; fix the Vulkan driver (`vulkaninfo --summary`). |+| Encoder fails to open (NVENC, `h264_rkmpp`) | framefields < 2.2 | Upgrade, or `FRAMEFIELDS_HW_CODECS=off`. |+| Jump at a chunk seam | Mismatched chunk settings, or an effect reading wall-clock time | One commit and settings for all chunks; deterministic effects. |+| Effect shows no change | Value on the wrong scale | Look it up in the `framefields-effects` catalog. |+| Text truncated | Box narrower than the text | Set `width` or `width: "fill"`. |diff --git a/plugins/framefields/skills/framefields-vision/SKILL.md b/plugins/framefields/skills/framefields-vision/SKILL.mdindex 221c4559..e265b940 100644--- a/plugins/framefields/skills/framefields-vision/SKILL.md+++ b/plugins/framefields/skills/framefields-vision/SKILL.md@@ -5,10 +5,7 @@ description: On-device computer vision in framefields — object detection and t# framefields-vision-## Setup-Import from [`framefields`](https://www.npmjs.com/package/framefields). If the project has no framefields setup yet, follow the `framefields` skill first. Effects that consume vision (blur, grading, outlines) are in `framefields-effects`; rendering and frame checks in `framefields-render`.--Vision runs ONNX models locally. Nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked against their exact size and SHA-256, and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`).+Vision runs ONNX models locally from [`framefields`](https://www.npmjs.com/package/framefields); nothing is sent anywhere. Models download on first use from pinned Hugging Face revisions, are checked by size and SHA-256, and are cached in `$FRAMEFIELDS_MODELS_DIR` (default `~/.cache/framefields/models`). Effects that consume vision are in `framefields-effects`; frame checks in `framefields-render`.## Which API@@ -34,7 +31,7 @@ Vision runs ONNX models locally. Nothing is sent anywhere. Models download on fiAbout 480 MB for all of them. `variant: "t" | "s" | "m"` picks the size.### SAM 2.1 Tiny and Small (`sam2`, Apache-2.0)-Meta SAM 2.1 Hiera checkpoints in Diffusion Studio's fp16 video exports, with the same five graphs and memory bank (the prompted frame, the 6 latest frames, 16 object pointers). Point and box prompts only. Pick one with `sam2: { model }`:+Meta SAM 2.1 Hiera checkpoints (Diffusion Studio fp16 video exports; memory of the prompted frame, the 6 latest frames, 16 object pointers). Point and box prompts only. `sam2: { model }`:| `model` | Input | Image encoder | Memory attention | Total | Per frame, M1 Pro WebGPU || --- | --- | --- | --- | --- | --- |@@ -55,7 +52,7 @@ Video tracking at 1008², plus text ("concept") selection. Only the image encodeThat is about 1.75 GB on a GPU and 1.2 GB on the CPU. The text graphs load only for text prompts.-Why fp16 on the GPU, measured for one 1008² frame on an M1 Pro: WebGPU fp16 took 3.65 s, q4f16 4.43 s, q4 6.63 s, and int8 140 s (its ops fall back to the CPU). fp16 is also the closest to full precision. On WebGPU, 4-bit weights are dequantized before every matmul, so they save disk and memory, not time. The backend-to-precision table is `SAM3_ENCODER` in `framefields/sam`; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.+fp16 is the fastest and most precise on WebGPU (one 1008² frame on an M1 Pro: fp16 3.65 s, q4f16 4.43 s, q4 6.63 s; 4-bit saves memory, not time). `SAM3_ENCODER` in `framefields/sam` maps backend to precision; `sam3.precision: "fp16" | "q4f16" | "q4"` overrides it.## Backends@@ -117,8 +114,6 @@ At render time:## Clip vision (`comp.vision`)-Use `comp.vision` when something must know the motion **at** a frame:-```typescriptconst dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)const person = dancer.person.place({ width: W, height: H, fit: "cover" });@@ -128,8 +123,7 @@ person.sample(48, x, y); // silhouette alpha + inward gradientawait dancer.ready(0); // analyse a frame (or ready(0, 120)) to read it at build time```-- The composition analyses each frame's clips just before drawing it, in every render path and the preview. Nothing runs up front.-- Results persist in Node, keyed by the file and settings (`cacheDir: false` opts out). Later renders and renders after a preview are instant.+- Each frame is analysed just before it draws, in every render path and the preview; nothing runs up front. Results persist, keyed by file and settings (`cacheDir: false` opts out), so later renders are instant.- Options: `startFrame` / `trimStartSec` (where the clip plays; pass `trimStartSec` yourself when you give a path instead of the layer), `fps` (analysis rate), `pose`, `mask`, `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. A lost person's last pose is held for up to a second.- `depth: true` adds each frame's relative depth (Depth Anything V2 Small; 0 far … 1 near, per-frame range, not metric) at about 924×518 for 16:9. A depth clip tracks no pose or silhouette unless `pose` / `mask` ask. Read it with `clip.depthMap(frame)` (`{ data, width, height }`), or draw it with `withVision({ mode: "depth" })` and lift a 3D layer with it via `withRelief` (framefields-compose). Depth frames persist one file each, keyed by the file only, so other options never invalidate them. About 0.4 s per frame on an M1 Pro GPU.- A `withVision({ mode: "matte" })` cutout (or `mask` / `crop`) of the same clip reuses the clip's subject for the frame being drawn: segmented once, aligned with the plate. Layers with `classes`, `keyBackground`, `matteSource: "selfie"` or different model settings keep their own pass. With several `comp.vision` clips of one source, the matte links to the **first one created**.@@ -147,15 +141,15 @@ Detection picks "the largest person". Use SAM when the subject is **a specific**| Download | ~83 MB | ~112 MB | ~1.75 GB (GPU) / ~1.2 GB (CPU) || Weights license | Apache-2.0 | Apache-2.0 | Meta SAM License |-**Choosing.** Decide from the shot and the request, and say which model you chose and why:-- **SAM 2.1 Tiny:** one subject you can point at, which is large in frame (a close-up, a dancer, a speaker). It is the fastest by far; on such subjects its masks match Small's closely (IoU 0.98 on a close-up).-- **SAM 2.1 Small:** still one subject from points or a box, but small in frame, thin or detailed (a ball, a hand, a bike, hair, a distant runner), or a shot with cuts or look-alikes where Tiny latches onto a similar object. About 4× Tiny's cost.-- **SAM 3:** the subject has to be found from a description, or several people or objects must be tracked at once. It is the most expensive, so trim to the stretch you need.-- Unsure between Tiny and Small: prototype with Tiny on a short trimmed stretch, inspect the masks, and switch to Small if edges, small parts or identity fail.+**Choosing** (tell the user which and why):+- **Tiny:** one subject you can point at, large in frame (close-up, dancer, speaker). Fastest; masks match Small's on such shots (IoU 0.98).+- **Small:** one subject that is small, thin or detailed (ball, hand, hair, distant runner), or cuts and look-alikes that fool Tiny. ~4× Tiny's cost.+- **SAM 3:** the subject must be found from a description, or several must be tracked. Most expensive: trim to the stretch you need.+- Unsure: prototype with Tiny on a short stretch, switch to Small if edges, small parts or identity fail.### License-The SAM code ships in `framefields`, Apache-2.0 like the rest of it, and loads only when a clip uses `sam2` or `sam3`; nothing extra to install. SAM 2.1's weights are Apache-2.0 and need no acceptance. SAM 3's weights are under Meta's SAM License, so `sam3` needs explicit acceptance before anything downloads: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `framefields/sam` when you drive SAM directly.+SAM code ships in `framefields` (Apache-2.0) and loads only when a clip uses it. SAM 2.1 weights are Apache-2.0. SAM 3 weights are under Meta's SAM License and need explicit acceptance before download: `acceptSamLicense: true` on the clip (or `acceptLicense: true` inside `sam3`), or `FRAMEFIELDS_ACCEPT_SAM_LICENSE=1`. Ask the user before accepting it on their behalf. Import `Sam2Tracker` / `Sam3Tracker` from `framefields/sam` when you drive SAM directly.### Prompts@@ -218,10 +212,10 @@ Layer.box({- **Coordinates** are normalized to the source image, before layout. Labels: `1` foreground, `0` background, `2` / `3` box corners.- **Prompt frames** are increasing integers on the source analysis clock, before trimming. SAM propagates forward from the first prompt; earlier frames are empty and are not analysed. To track only a stretch of a long clip, trim the layer and prompt on the stretch's first source frame.-- **Text prompts are concepts.** Name what sets the target apart in that shot. In a crowd where everyone shows a white collar, "man in white shirt" picks a student. In the same film's wide shot, where only he has no blazer, it picks him. Check the first frame of every prompt.+- **Text prompts are concepts.** Name what sets the target apart *in that shot* ("man in white shirt" fails where every student shows a white collar). Check the first frame of every prompt.- **Several objects** (`maxObjects`): one encoder pass per frame serves every track, and each track keeps its own memory. Where tracks overlap, the more confident one keeps the pixel. `signals.objects.get(k)` and `signals.masks` carry tracks `1…N`; the clip's matte and subject are all tracks merged. A later text prompt re-selects every track (IDs renumber); a points prompt with `object: k` seeds or corrects track `k` only (the next unused ID adds one).-- **One entity for a whole group** (`merge: true`): every instance the text matches (up to `maxObjects`, default all) becomes one seed mask, tracked as one object with one memory. Use it when you need the group's mask or matte (cut out a crowd, grade everyone but the hero), not each member's box; it costs the same as one track however many match (4.1 s per frame for a whole crowd of students on an M1 Pro). Use `maxObjects` without `merge` when each member needs its own track, at about 0.7 s per member per frame.-- **People who appear later** (`redetect: N`): a text selection is made once, on the prompt frame; anyone missed there, uncovered later, or lost by the tracker never joins on their own. `redetect: N` runs the text again every N frames and adds what no track covers: into the one object with `merge`, as new track IDs up to `maxObjects` otherwise. On a 4 s close-up where the man in front bends away, a merged "students in black blazers" group kept 46–61% of the frame through the end instead of falling to 22–33%. Each pass costs about as much as the prompt frame's selection (about 50 s on an M1 Pro), so pick N for how fast people enter the shot (10–25 frames), not 1.+- **One entity for a whole group** (`merge: true`): every match (up to `maxObjects`, default all) becomes one object with one memory, at the cost of one track. Use it for a group's mask or matte (cut out a crowd); use `maxObjects` without `merge` when each member needs its own track (~0.7 s per member per frame).+- **People who appear later** (`redetect: N`): text selects once, on the prompt frame. `redetect: N` re-runs it every N frames and adds what no track covers (into the merged object, or as new IDs up to `maxObjects`). Each pass costs about a prompt-frame selection (~50 s on an M1 Pro): pick N by how fast people enter (10–25), not 1.- Two text-prompted clips of one shot can claim the same person (a "student" track on the hero). Prefer prompts that exclude each other.- **Corrections and cuts.** A prompt on a later frame resets that object's memory. Re-prompt after hard cuts or long occlusions; appearance tracking cannot guarantee re-identification.- An explicit selection never falls back to another detected person. If its mask cannot align with a matte layer (a different aspect ratio), the render fails. Matte at the source aspect ratio and crop afterwards.@@ -238,15 +232,14 @@ Read the request's number before writing any vision code.| **A group** of look-alikes as a whole: "the students", "all dancers", "everyone in black jackets", "the crowd", "the cars" | one entity | **one** `comp.vision` clip, **one** text prompt, `merge: true` || **Each member** separately: "box every student", "label each car", "count them" | N objects | one clip, one text prompt, `maxObjects: N` |-- **Never select a group one member at a time.** One clip or detection per student, or several clips with the same or overlapping prompts, re-detect the same people. That gives duplicate and overlapping masks, members claimed twice or missed, and N times the cost. A group is always one prompt on one clip.-- Plural wording is fine in the prompt itself: "students in black jackets" and "student in black jacket" selected the same group (87% of the frame either way). Grammatical number does not change which instances SAM 3 matches; `merge` decides whether they are one entity, and `maxObjects` whether they are separate tracks.-- Name the group by what its members share and others lack ("in black jackets" excluded the main character in a white shirt). If the group still includes someone it shouldn't, describe the group more precisely; don't add a second clip to subtract them.+- **Never select a group one member at a time.** Several clips with the same or overlapping prompts re-detect the same people: duplicate masks, members claimed twice or missed, N× the cost. A group is one prompt on one clip.+- Grammatical number in the prompt doesn't matter; `merge` and `maxObjects` decide. Name what members share and others lack ("in black jackets"); if someone wrong is included, describe the group more precisely instead of adding a clip to subtract them.- A group plus a different individual (the crowd and the main character) is two clips with prompts that exclude each other: `merge` on the group, a single object for the individual.### Cost and caching- Every SAM 3 clip in a process shares one image encoder and reuses its result per frame. Tracking the hero and the crowd of one source costs one encoder pass, not two.-- Setup (loading the models, then text selection on the prompt frame) takes about 2 minutes on an M1 Pro. Measured there: a 20-frame stretch with 6 tracks (hero + 5 others) renders in about 5 minutes; a 15 s clip at 25 fps is estimated at about 55 minutes. A CUDA GPU should be much faster; the CPU image encoder is about 9× slower than WebGPU.+- On an M1 Pro: ~2 min setup (models, text selection), then a 20-frame stretch with 6 tracks takes ~5 min; a 15 s clip at 25 fps ~55 min. CUDA is much faster; the CPU encoder ~9× slower than WebGPU.- Results persist like other clip vision, and the cache key includes every prompt, the model revisions and the encoder precision. Re-rendering after changing layout, colours or effects skips SAM entirely, so prototype prompts on a short trimmed stretch first.- Offline or CI: download the weights on a connected machine once, then copy `~/.cache/framefields/models`, or point `FRAMEFIELDS_MODELS_BASE_URL` at a mirror. `SAM_MODELS` (`framefields/sam`) lists every file with its URL, size and SHA-256.diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 550a0842..bc0f1343 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -1,64 +1,39 @@---name: framefields-description: Start a new framefields video project, or add framefields to an existing Node.js project. Use when the user wants to make a video with framefields and the project has no framefields setup yet, or asks how to install framefields, scaffold a composition and render script, get from an empty folder to a first MP4, generate images, video, music or voice for a film with AI models (`ai` from framefields, the framefields AI gateway), or add ready-made components (pointers, captions, title cards, device frames …) with `npx framefields add`. Hands off to framefields-compose, framefields-effects, framefields-vision, and framefields-render for the details.+description: Start a new framefields video project, or add framefields to an existing Node.js project. Use when the user wants to make a video with framefields and the project has no framefields setup yet, or asks how to install framefields, scaffold a composition and render script, get from an empty folder to a first MP4, store footage (asset buckets, Git LFS), generate images, video, music or voice for a film with AI models (`ai` from framefields, the framefields AI gateway), or add ready-made components (pointers, captions, title cards, device frames …) with `npx framefields add`. Hands off to framefields-compose, framefields-effects, framefields-vision, framefields-render and framefields-cloud for the details.---# framefields-## Overview-`framefields` is a code-first video SDK. A video is a TypeScript `Composition` (a tree of layers with keyframed animation) that renders natively on WebGPU (Dawn) in Node.js, with no browser, Puppeteer, or Chromium. The engine is the [`framefields`](https://www.npmjs.com/package/framefields) npm package; everything below runs in the user's own project.--This skill takes a project from nothing to a verified first render. For anything past that, load the specialist skill:+A video is a TypeScript `Composition` (a layer tree with keyframed animation) rendered natively on WebGPU (Dawn) in Node.js, no browser. The engine is the [`framefields`](https://www.npmjs.com/package/framefields) npm package. This skill takes a project to a verified first render; then load the specialist:| Task | Skill || --- | --- |-| Scenes, layout, typography, animation, beat grids, film structure | `framefields-compose` |-| Post-processing, color grading, VFX | `framefields-effects` |-| On-device vision: detection, cutouts, pose, tracking a specific person or several people (SAM 2.1 / SAM 3) | `framefields-vision` |-| Frame inspection, frame grids, pixel tests, video export and QA | `framefields-render` |-| Ready-made components (`npx framefields add`) | this skill, section 8 |-| Generating images, video, music, voice with AI models (`ai`) | this skill, section 6 |-| Autonomous agent payments & runway checks | this skill, section 7 |-| Reporting bugs or engine defects to GitHub | this skill, section 9 |-----+| Scenes, layout, typography, animation, 3D, physics, film structure | `framefields-compose` |+| Post-processing, grading, stylized looks | `framefields-effects` |+| Detection, cutouts, pose, tracking a specific person (SAM 2.1 / SAM 3) | `framefields-vision` |+| Frames, grids, pixel tests, live preview, export, QA | `framefields-render` |+| Rendering on framefields' cloud GPUs or a self-hosted worker | `framefields-cloud` |-## 1. Check the environment+## 1. Environment-- **Node.js 22 or later.** Run `node --version`. Older versions fail at import.-- **A GPU, or a CPU renderer.** macOS uses Metal (every Mac and macOS VM has a Metal GPU). Linux uses Vulkan, Windows Direct3D 12 or Vulkan. Without a GPU, framefields renders on the CPU: on Linux (a sandbox, CI runner or container) through Mesa lavapipe, installed with `apt-get install -y libvulkan1 mesa-vulkan-drivers`; on Windows through WARP (built into Windows) or lavapipe. Every feature renders the same there (text, media, Lottie, effects, 3D, vision), only slower: about 12× slower than an M1 Pro's GPU in our Linux test. `describeGpu()` reports `software: true`. Fine for frames, grids and short exports; for long films, use a GPU machine or the render worker (`framefields-render`).-- **An ES module project.** The project's `package.json` needs `"type": "module"`.+- **Node.js 22+** and an ES module project (`"type": "module"`).+- **A GPU or a CPU renderer.** macOS: Metal. Linux: Vulkan, or Mesa lavapipe on the CPU (`apt-get install -y libvulkan1 mesa-vulkan-drivers`). Windows: D3D12/Vulkan, or WARP. On the CPU everything renders the same, ~12× slower (`describeGpu()` reports `software: true`).-## 2. Install--**Fastest path: `npx framefields init`.** It writes everything in sections 2 and 3 for you: `package.json` (with the install-script approvals below, scripts, dependencies, `packageManager`), `tsconfig.json`, `src/theme.ts` / `film.ts` / `render.ts`, `assets/`, the LFS `.gitattributes` and `.gitignore`, and `framefields.json` (the components folder, installed fonts, and a `film` render naming `buildFilm` in `src/film.ts`). It also downloads Inter, JetBrains Mono, and Noto Color Emoji (pinned, with their OFL licenses), registers them in the generated `src/components/fonts.ts` that `theme.ts` calls, and runs `git init`.+## 2. Install: `npx framefields init````bash-npx framefields init my-film --install # a new project; or `init` inside an existing one+npx framefields init my-film --install # or `init` inside an existing project```-In an existing project it never overwrites: existing files are left alone, and `package.json`, `.gitignore` and `.gitattributes` only gain what they lack (a `"type": "commonjs"` package is refused). Options: `--pm npm|pnpm|yarn|bun` (default: the one running it), `--no-fonts`, `--no-emoji`, `--no-git`, `--name`. It asks no questions, so it runs unattended in CI. The AI gateway key is optional: `init` writes `.env.example` and git-ignores `.env`, and `render.ts` loads `.env` when it exists; the user sets `FRAMEFIELDS_API_KEY` there themselves (never write a key for them). Then go to section 4. The rest of this section and section 3 describe what it writes, for doing it by hand or merging into an unusual setup.--Use the package manager the project already uses (look for `pnpm-lock.yaml`, `yarn.lock`, `bun.lock`, or `package-lock.json`).+It writes `package.json` (install-script approvals, scripts, dependencies, `packageManager`), `tsconfig.json`, `src/theme.ts` / `film.ts` / `render.ts`, `assets/`, LFS `.gitattributes`, `.gitignore`, `.env.example`, and `framefields.json` (components folder, fonts, a `film` render). It downloads Inter, JetBrains Mono and Noto Color Emoji (pinned, OFL), registers them in `src/components/fonts.ts`, and runs `git init`. It never prompts or overwrites: existing files are kept and `package.json`, `.gitignore`, `.gitattributes` only gain what they lack (a `"type": "commonjs"` package is refused). Options: `--pm npm|pnpm|yarn|bun`, `--no-fonts`, `--no-emoji`, `--no-git`, `--name`.-**Write `package.json` before installing.** npm 11, pnpm 10, and bun block dependency install scripts unless the project approves them, and framefields needs exactly one: `skia-canvas` downloads its native binary in its install script. Without it, `import "framefields"` fails with `Cannot find module '.../skia.node'`. The other packages that ship install scripts work without them, so deny them to keep installs quiet and skip downloads framefields never uses.--For a new project, create `package.json` with all three blocks. In an existing project, merge them into its `package.json`, keeping any entries it already has:+**By hand**, use the project's package manager and write `package.json` *before* installing. npm 11, pnpm 10 and bun block install scripts; framefields needs exactly one (`skia-canvas`, else `Cannot find module '.../skia.node'`). Deny the rest:```json{- "name": "my-film",- "private": true,"type": "module",- "allowScripts": {- "skia-canvas": true,- "sharp": false,- "webgpu": false,- "onnxruntime-node": false,- "node-av": false,- "esbuild": false,- "protobufjs": false- },+ "allowScripts": { "skia-canvas": true, "sharp": false, "webgpu": false, "onnxruntime-node": false, "node-av": false, "esbuild": false, "protobufjs": false },"pnpm": {"onlyBuiltDependencies": ["skia-canvas"],"ignoredBuiltDependencies": ["sharp", "webgpu", "onnxruntime-node", "node-av", "esbuild", "protobufjs"]@@ -67,61 +42,27 @@ For a new project, create `package.json` with all three blocks. In an existing p}```-`allowScripts` is read by npm, `pnpm` by pnpm, and `trustedDependencies` by bun; each tool ignores the others' fields. Yarn runs install scripts by default and needs none of them. Also add `"packageManager"` with the exact manager and version you install with (e.g. `"[email protected]"`), and commit the lockfile the install writes (pnpm, npm or yarn; bun lockfiles can't be rendered by the worker). Then install:--```bash-npm install framefields-npm install --save-dev tsx typescript @types/node-```--The install must finish without an `install-scripts` warning. If framefields was installed before the approval was in place, add it and rerun the skipped script with `npm rebuild skia-canvas` (`pnpm rebuild skia-canvas`). Never approve every script wholesale (`npm install-scripts approve --all`).--A minimal `tsconfig.json`:--```json-{- "compilerOptions": {- "module": "NodeNext",- "moduleResolution": "NodeNext",- "target": "ES2022",- "strict": true,- "skipLibCheck": true,- "noEmit": true,- "types": ["node"]- },- "include": ["src/**/*"]-}-```--## 3. Scaffold the project+`allowScripts` is npm's, `pnpm` pnpm's, `trustedDependencies` bun's; yarn needs none. Pin `"packageManager"` and commit the lockfile. Then `npm install framefields` and `npm install -D tsx typescript @types/node`. The install must finish without an `install-scripts` warning; if it didn't, add the approval and `npm rebuild skia-canvas`. Never approve every script wholesale. `tsconfig.json`: `module`/`moduleResolution` `NodeNext`, `target` `ES2022`, `strict`, `skipLibCheck`, `noEmit`, `types: ["node"]`, `include: ["src/**/*"]`.-Keep the composition separate from the script that renders it, so tests, the preview, the render script and the render worker all build the same film. `film.ts` must only build: no rendering, no `process.exit()`, no writes at import time.+## 3. Project shape```-src/- theme.ts # canvas size, fps, colors, fonts: the only place for magic numbers- film.ts # buildFilm(): assembles the Composition- render.ts # renders frames for review, a live preview, or the full video-assets/- fonts/ # .ttf / .otf files the film uses- footage/ # source videos (.mp4, .mov) and audio (.mp3, .wav) tracked in Git LFS-output/ # renders (add to .gitignore)+src/theme.ts # size, fps, colors, fonts: the only place for magic numbers+src/film.ts # buildFilm(): builds the Composition, nothing else+src/render.ts # frames and contact sheets for quick looks+assets/ # fonts/, footage/ (bucket or Git LFS)+output/ # renders (git-ignored)+framefields.json```-Text needs a real font file. Ask the user which font to use.+`film.ts` must only build (no rendering, `process.exit()` or writes at import): tests, preview, CLI and the cloud all import it. Text needs a real font file; ask the user which.```typescript// src/theme.tsimport { FontManager } from "framefields";--export const W = 1920;-export const H = 1080;-export const FPS = 30;-export const INK = "#0E0D0C";-export const PAPER = "#EEE9E0";-export const SANS = "Inter";--export async function registerFonts(): Promise<void> {+export const W = 1920, H = 1080, FPS = 30;+export const INK = "#0E0D0C", PAPER = "#EEE9E0", SANS = "Inter";+export async function registerFonts() {await FontManager.register({ family: SANS, source: "assets/fonts/Inter.ttf" });}```@@ -131,46 +72,21 @@ export async function registerFonts(): Promise<void> {import { Composition, Layer, LayerAnimation } from "framefields";import { FPS, H, INK, PAPER, registerFonts, SANS, W } from "./theme.js";-const DURATION = 3 * FPS; // frames-export async function buildFilm(): Promise<Composition> {- // Fonts must be registered before any text layer is built.- await registerFonts();-- const film = new Composition({- width: W,- height: H,- fps: FPS,- durationMs: (DURATION / FPS) * 1000,- backgroundColor: INK,- });-+ await registerFonts(); // before any text layer+ const film = new Composition({ width: W, height: H, fps: FPS, durationMs: 3000, backgroundColor: INK });film.add(Layer.text("Hello, framefields", {- id: "title",- position: "absolute",- x: 0,- y: H / 2 - 60,- width: W,- height: 120,- fontFamily: SANS,- fontSize: 96,- fontWeight: 700,- fill: PAPER,- align: "center",- }).animate(- LayerAnimation.create()- .fadeIn(0, 20, "power2.out")- .fromTo("y", H / 2 - 20, H / 2 - 60, { start: 0, end: 30, ease: "expo.out" }),- ),+ position: "absolute", x: 0, y: H / 2 - 60, width: W, height: 120,+ fontFamily: SANS, fontSize: 96, fontWeight: 700, fill: PAPER, align: "center",+ }).animate(LayerAnimation.create().fadeIn(0, 20, "power2.out").fromTo("y", H / 2 - 20, H / 2 - 60, { start: 0, end: 30, ease: "expo.out" })),);-return film;}``````typescript-// src/render.ts: quick looks (frames, a contact sheet). Full renders and the preview use the CLI.+// src/render.ts: `tsx src/render.ts frames 0 30 60` or `tsx src/render.ts grid`import fs from "node:fs/promises";import path from "node:path";import { HeadlessMediaRenderer } from "framefields";@@ -178,150 +94,58 @@ import { buildFilm } from "./film.js";const OUT = path.resolve(import.meta.dirname, "../output");const [mode = "frames", ...args] = process.argv.slice(2);-const film = await buildFilm();await fs.mkdir(OUT, { recursive: true });-if (mode === "grid") {- const png = await film.renderFrameGrid({- frames: args.length ? args.map(Number) : [0, 10, 20, 30, 45, 60, 75, 89],- columns: 4,- cellWidth: 480,- showLabels: true,- });- await fs.writeFile(path.join(OUT, "grid.png"), png);+ const frames = args.length ? args.map(Number) : [0, 10, 20, 30, 45, 60, 75, 89];+ await fs.writeFile(path.join(OUT, "grid.png"), await film.renderFrameGrid({ frames, columns: 4, cellWidth: 480, showLabels: true }));} else {const renderer = new HeadlessMediaRenderer();- for (const frame of (args.length ? args : ["0"]).map(Number)) {- const png = await film.renderFrame({ frame, renderer });- await fs.writeFile(path.join(OUT, `f${String(frame).padStart(4, "0")}.png`), png);- }+ for (const frame of (args.length ? args : ["0"]).map(Number))+ await fs.writeFile(path.join(OUT, `f${String(frame).padStart(4, "0")}.png`), await film.renderFrame({ frame, renderer }));}--// Native GPU and encoder threads keep Node alive; always exit explicitly.-process.exit(0);+process.exit(0); // native threads keep Node alive```-Add scripts to `package.json`:+Scripts: `typecheck` (`tsc --noEmit -p .`), `frames`, `grid`, `check` (`framefields check`), `check:render`, `preview` (`framefields preview film`), `render` (`framefields render film`), `render:cloud` (`framefields render film --cloud`). All three CLI commands read the `film` render in `framefields.json`, so local and cloud renders write the same file. For another cut (teaser, vertical, language) add an export and a named render (`framefields-render`), not a flag.-```json-"scripts": {- "typecheck": "tsc --noEmit -p .",- "frames": "tsx src/render.ts frames",- "grid": "tsx src/render.ts grid",- "preview": "framefields preview film",- "render": "framefields render film",- "render:cloud": "framefields render film --cloud"-}-```+### Assets: a bucket, or Git LFS-`framefields render film`, `framefields render film --cloud` and `framefields preview film` all read the `film` render below, so a render writes the same file wherever it runs: switching between this machine and the cloud is the `--cloud` flag, nothing else. Flags (`--export`, `--select`, `--codec`, `--quality`, `--output`) override the render's fields either way.+Footage, audio, Lottie, LUTs and generated media don't belong in plain git; small files (logos, a LUT) can be committed.-`framefields.json` names what the project renders; `film` is the scaffold's. Add an entry per cut (teaser, vertical edit) so the CLI, `framefields check` and the dashboard pick it by name:+**Preferred: `framefields assets` on an S3-compatible bucket** (suggest Cloudflare R2: no egress fees). Files stay out of git; `framefields.json` pins each by sha256.```json-{- "components": "src/components",- "renders": {- "film": { "title": "Launch film", "entry": "src/film.ts", "export": "buildFilm", "workers": "auto", "output": "output/film.mp4" }- }-}+"storage": { "remotes": { "media": {+ "endpoint": "https://<account-id>.r2.cloudflarestorage.com", "bucket": "films", "prefix": "launch/",+ "publicUrl": "https://media.example.com"+} } }```-`render:cloud` renders on framefields' GPUs (api.framefields.dev) from the pushed commit, with the same `FRAMEFIELDS_API_KEY` as the AI gateway, and splits long films across several GPUs; `framefields-render` covers it. When the user wants another version of the film (a teaser, a vertical cut, a language), add an export and a named render for it rather than a flag in `render.ts`.--### Video assets: a bucket, or Git LFS--Footage, audio, Lottie files, LUTs and generated media are too big for plain git. Render outputs in `output/` stay in `.gitignore`. Small files (logos, short sound effects, a LUT) can simply be committed.--**Preferred: `framefields assets` on the project's own bucket.** Any S3-compatible bucket works; Cloudflare R2 is the one to suggest, since it charges no egress. The files stay out of git, `framefields.json` pins each one by sha256, and anyone (or any worker) gets exactly those bytes back.+- `publicUrl` (optional) lets reads skip keys; objects upload `immutable`. AWS needs `region`. Several remotes: `default` names where new assets go, `--remote` picks another.+- Keys come from the environment only: `FRAMEFIELDS_STORAGE_MEDIA_ACCESS_KEY_ID` / `_SECRET_ACCESS_KEY` (remote name upper-cased) or `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. Ask the user to set them; never print or write them.+- `npx framefields assets add assets/footage` hashes, uploads what's missing, records under `assets`, git-ignores them. Commit `framefields.json` and `.gitignore`. `assets pull` (verified; `--overwrite` replaces local edits), `assets status` (`ok` / `missing` / `changed`), `assets push`.+- Code uses ordinary paths. Where files are missing, `framefields preview` streams them from the bucket (needs a CORS rule: https://framefields.dev/docs/guides/assets/#preview-from-the-bucket), `framefields render` pulls them first, `framefields check` says to pull.-1. Add a remote to `framefields.json`. `publicUrl` is optional: a custom domain (or CDN, or the `r2.dev` URL) in front of the bucket, so reads need no keys. Content-addressed objects are uploaded `immutable`, so a CDN caches them forever.- ```json- "storage": {- "remotes": {- "media": {- "endpoint": "https://<account-id>.r2.cloudflarestorage.com",- "bucket": "films",- "prefix": "launch/",- "publicUrl": "https://media.example.com"- }- }- }- ```- AWS needs `"region"`; R2 uses `auto`. Several remotes are fine (a public one for media you may share, a private one for raw footage): `"default"` names the one new assets go to, `--remote` picks another.-2. Keys come from the environment, never the file: `FRAMEFIELDS_STORAGE_MEDIA_ACCESS_KEY_ID` and `FRAMEFIELDS_STORAGE_MEDIA_SECRET_ACCESS_KEY` (the remote's name upper-cased), or `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. Ask the user to set them; never print or write them.-3. `npx framefields assets add assets/footage` hashes the files (a folder means every file in it), uploads what the bucket lacks, records them under `assets` and adds them to `.gitignore`. Commit `framefields.json` and `.gitignore`. Adding a changed file again records the new version.-4. In a fresh clone or container, `npx framefields assets pull` downloads what's missing and checks every byte; it leaves files changed locally alone unless `--overwrite`. `assets status` lists each asset as `ok`, `missing` or `changed`; `assets push` re-uploads what the bucket lacks.+**Or Git LFS** (init writes `.gitattributes`). Fresh clones hold ~130-byte pointers that render blank: `git lfs pull` first, and never git-ignore LFS-tracked media. LFS bandwidth is billed to the repo owner on every clone and cloud render.-Code references assets by path as usual (`Layer.video("assets/footage/intro.mp4")`); nothing in the film changes. Where the files aren't on disk (a fresh clone, a cloud agent's sandbox):+## 4. Verify, cheapest first-- `framefields preview` streams them from the bucket: the page is redirected to the remote's `publicUrl` (or a URL signed with the remote's keys), so nothing is downloaded into the sandbox. The bucket needs a CORS rule allowing the preview's origin; see the assets guide (https://framefields.dev/docs/guides/assets/#preview-from-the-bucket).-- `framefields render` downloads the missing ones first (`assets pull`), then renders.-- `framefields check` says to run `assets pull`.+1. `npm run typecheck`, then `npm run check`.+2. `npm run frames -- 0 30 60` and look at the PNGs. Black usually means a missing font, a layer off canvas, or opacity stuck at 0.+3. `npm run grid` for motion.+4. `npm run preview` when a person is watching (background command; give them the URL; read their pinned notes). Details in `framefields-render`.+5. `npm run render` in the background with a long timeout. A pause after the last frame is audio mix/mux, not a hang.-**Or Git LFS**, when the project already uses it or there's no bucket. `npx framefields init` writes the LFS `.gitattributes`. In fresh clones LFS media are ~130-byte pointer files that decode as blank frames, so run `git lfs pull` first, and never add `*.mp4` to `.gitignore` for LFS-tracked footage. Every clone and every cloud render worker downloads LFS objects from the git host, which bills that bandwidth to the repo owner; a bucket with no egress fees avoids it.+Show the user frames or the grid before the full render. For long films or no GPU, `npm run render:cloud` (`framefields-cloud`).-## 4. Verify, cheapest step first+## 5. Generate assets with AI-Never go straight to a full video render. On a machine without a GPU every render runs on the CPU (lavapipe or WARP), so the cheap steps matter even more. Each step catches problems in seconds that a video render takes minutes to show:+The Framefields MCP server (when connected) answers `list_models`, `get_model_schema` and `estimate_generation_cost`; in Claude Code they're `mcp__plugin_framefields_framefields__*`. It does not run jobs: generation runs through `ai` from `framefields`, authenticated by a gateway key (`FRAMEFIELDS_API_KEY`, `ff_live_…` from https://framefields.dev/dashboard/keys), not the MCP connection.-1. `npm run typecheck`-2. `npm run frames -- 0 30 60` and look at the PNGs in `output/`. A black frame usually means a missing font, a layer outside the canvas, or opacity stuck at 0.-3. `npm run grid` and check the motion reads smoothly across the contact sheet.-4. `npm run preview` when a person is watching: they play the film with sound before you export (see below).-5. `npm run render` once frames and motion look right. Run it in the background with a long timeout. It logs each phase (`Rendered 40% …`, `Mixing audio`, `Muxing mp4`, `Done in …`) and a `Still …` line when one runs long, so a pause after the last frame is the audio mix or mux, not a hang.+**The key:** ask the user to set it in the environment themselves (init writes `.env.example`; `render.ts` loads `.env`). Never ask for it in chat, print it, inspect env or credential files, embed or commit it.-Show the user the frame PNGs or the grid before the full render. `framefields-render` covers pixel-probe tests, render QA (`renderVideo({ qa })`), and diagnosing blank frames. If you suspect an issue or defect in the engine itself or ai gateway, see [section 9](#9-report-engine-issues).--### Live preview--`npm run preview` serves a localhost player (picture, sound, timeline) and prints its URL. The page runs `film.ts` itself and renders with WebGPU in real time, so the user sees the real film; a slow device skips frames to keep up with the sound.--- Run it as a background command and read its output: it says when the page opened, prints errors from the page (a failed build, a frame that threw, a missing font), and says why it stopped. It stops 5 s after its last tab closes; just run it again.-- It doesn't open anything. If your app has a built-in browser, open the URL there; otherwise give the user the link. `startPreview(..., { open: true })` opens their default browser.-- **Reference assets from the file's URL.** The page runs `film.ts` itself, so asset paths must resolve in the browser too: build them from the module's own URL, `path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../assets/fonts")`, rather than `process.cwd()` or a bare relative path. The preview serves any file inside the project (the folder with `package.json`); pass `root` (one or more paths) to `startPreview` for fonts or media kept elsewhere.-- After changing the film, run it again. It takes over the same URL, and the open tab reloads by itself at the same moment.-- Link to a moment with `#t=12.5` (seconds). In a browser you can script, `window.framefieldsPreview` has `seek(seconds)`, `play()`, `pause()` and `state()`; `seek` resolves once the frame is drawn, so screenshot after it.-- **The user can pin notes on the picture.** Pausing and clicking (or dragging an area) adds a note at that spot and moment. Each one prints in the preview's output and is saved in `.framefields/preview-notes/`: `notes.json`, plus `note-<id>.jpg`, the frame with the spot marked. When the user says they left notes, read `notes.json` and look at each open note's picture before you edit. Then run the preview again so they can check the changes and tick them off. `framefields-render` has the details.--## 5. Render on a server--The framefields render worker (Docker, RunPod pod or serverless) renders a pushed commit: it pulls LFS assets, downloads the assets `framefields.json` pins, installs from the lockfile, imports `src/film.ts`, calls `buildFilm`, and uploads the result. A project built as above works unchanged if:--- `framefields` is 2.2+, the lockfile is committed and matches `package.json`, and `packageManager` is pinned;-- every input is under `assets/`, and either pinned with `framefields assets add` (and on the bucket) or tracked by Git LFS and pushed (`git lfs ls-files` lists it);-- asset paths come from `import.meta.url`, never `process.cwd()`;-- nothing writes into `node_modules` while rendering (files are shared read-only).--A failed job returns `{ code, hint, retryable }`; resubmit unchanged only when `retryable` is true. Job format and error codes: https://framefields.dev/docs/guides/render-service/--## 6. Generate assets with AI--Use the connected Framefields MCP server for `list_models`, `get_model_schema`, and `estimate_generation_cost` when available (in Claude Code sessions, these are exposed as `mcp__plugin_framefields_framefields__list_models`, `mcp__plugin_framefields_framefields__get_model_schema`, and `mcp__plugin_framefields_framefields__estimate_generation_cost`). Use `import { ai } from "framefields"` in a local script to generate, edit, or process media; MCP does not submit or retrieve generation jobs. The SDK authenticates with a separate gateway API key (`FRAMEFIELDS_API_KEY`), not the MCP OAuth connection. Use only the currently supported catalog models and the input fields from the selected model's schema.--- `list_models`: inspect catalog models and category tasks (`text-to-image`, `text-to-video`, `speech-to-text`, etc.).-- `get_model_schema`: check required and optional inputs for the model before writing the script.-- `estimate_generation_cost`: check published rates or request cost estimates before running paid generation.--`framefields` exports `ai` for image, video and audio operations through the Framefields gateway. Each request uses a model ID and account credits. `run`, `subscribe`, `stream`, `queue`, and `storage` support the gateway workflow.--```typescript-import { ai } from "framefields";--const apiKey = process.env.FRAMEFIELDS_API_KEY;-if (!apiKey) throw new Error("Set FRAMEFIELDS_API_KEY before generating assets.");-ai.config({ apiKey });--const { data, requestId } = await ai.subscribe<{ images: { url: string }[] }>("seedream/4.5", {- input: { prompt: "Neon street at night, rain, 35mm", image_size: "landscape_16_9" },-});-console.log(data.images[0].url);-```--**The key.** Requests authenticate with an `ff_live_…` key from https://framefields.dev/dashboard/keys. For local execution, ask the user to set `FRAMEFIELDS_API_KEY` in the process environment themselves; never ask them to paste a key into chat. The script reads `process.env.FRAMEFIELDS_API_KEY` and fails with an actionable message when it is missing. Do not inspect or print environment variables or credential files, embed the key in source, or commit it. AI generation is optional and does not affect local composition and rendering. Without a key, gateway calls cannot run. Failures throw `ApiError` with the gateway's `type`: on `insufficient_credits` (status `402`) tell the user to top up at https://framefields.dev/dashboard/billing rather than retrying; `key_limit_reached` means the key's monthly cap; `rate_limited` and `too_many_inflight_requests` mean wait and retry. A `ValidationError` (`422`) lists the bad input fields in `fieldErrors`.--**Generate once, into `assets/`, in its own script.** Generation costs money and is not deterministic, so it never runs inside `buildFilm()`, the preview, tests or the render worker. Write `src/generate.ts`, run it on purpose, download every output into `assets/generated/`, pin it with `npx framefields assets add assets/generated` (or track it in Git LFS), and reference the local file from the film. Generated output URLs expire, and the worker renders from the repository.+**Generate once, into `assets/`, in its own script.** It costs money and isn't deterministic, so never call `ai` from `buildFilm()`, the preview, tests or the worker. Download outputs (their URLs expire) into `assets/generated/`, pin them (`framefields assets add`), reference the local files.```typescript// src/generate.ts@@ -332,114 +156,57 @@ import { ai } from "framefields";const apiKey = process.env.FRAMEFIELDS_API_KEY;if (!apiKey) throw new Error("Set FRAMEFIELDS_API_KEY before generating assets.");ai.config({ apiKey });-const OUT = path.resolve(import.meta.dirname, "../assets/generated");await fs.mkdir(OUT, { recursive: true });--async function save(url: string, name: string): Promise<string> {- const file = path.join(OUT, name);- await fs.writeFile(file, Buffer.from(await (await fetch(url)).arrayBuffer()));- return file;-}+const save = async (url: string, name: string) =>+ fs.writeFile(path.join(OUT, name), Buffer.from(await (await fetch(url)).arrayBuffer()));const still = await ai.subscribe<{ images: { url: string }[] }>("seedream/4.5", {input: { prompt: "Neon street at night, rain, 35mm", image_size: "landscape_16_9", seed: 7 },});await save(still.data.images[0].url, "street.jpg");-// Feed a local file to a model: put a Blob in the input and it is uploaded for you-// (or call ai.storage.upload(blob) yourself and pass the URL).+// A Blob in the input is uploaded for you (or ai.storage.upload(blob) and pass the URL).const image_url = new Blob([await fs.readFile(path.join(OUT, "street.jpg"))], { type: "image/jpeg" });const clip = await ai.subscribe<{ video: { url: string } }>("seedance/2.0/image-to-video", {input: { prompt: "slow push in, rain falling", image_url, duration: "5" },- logs: true,- onQueueUpdate: (u) => u.status === "IN_PROGRESS" && u.logs.forEach((l) => console.log(l.message)),});await save(clip.data.video.url, "street.mp4");process.exit(0);```-Then `Layer.image("assets/generated/street.jpg")` / `Layer.video("assets/generated/street.mp4")` in the film.--- **Pick the model and its input from the catalog.** https://framefields.dev/models lists every model with its id, category, price per unit and input schema. Each model has its own input and output fields (`images[0].url`, `video.url`, `audio.url`, `audio_file.url` …): read the schema and pass the output shape as the type argument, or log `data` once, instead of guessing.-- **Long jobs** (video, music): `subscribe` waits and polls. For several at once, `ai.queue.submit(model, { input })` each, keep every `request_id`, then `ai.queue.result(model, { requestId })`; an interrupted script can collect the results instead of paying again. Reading a status or result is free and retried on transient errors; submitting is never retried, so it can't bill twice.-- **Costs:** Generation consumes credits. When useful, use the MCP price-estimate tool or the model catalog to report an estimate or unit rate; actual charges can vary. Do not invent a total when only a unit rate is known.-- Pass a fixed `seed` where the model accepts one, and keep the prompts in `generate.ts`, so a result can be regenerated.-- Frame-rate and size: generated clips rarely match the composition's `fps` or aspect. Use `fit: "cover"` on the layer, and check the clip's real duration before cutting to it.--## 7. Autonomous agent payments+- **Models and inputs come from the catalog** (https://framefields.dev/models or `get_model_schema`). Output shapes differ (`images[0].url`, `video.url`, `audio.url` …): read the schema or log `data` once.+- **Many long jobs:** `ai.queue.submit` each, keep the `request_id`s, collect with `ai.queue.result`. Status and result reads are free and retried; submits never retry, so nothing bills twice.+- **Costs:** quote the estimate tool or catalog rate; don't invent a total from a unit rate. Use a fixed `seed` and keep prompts in `generate.ts`.+- Generated clips rarely match the film's fps or aspect: `fit: "cover"`, check real durations.+- **Errors** throw `ApiError` with `type`: `insufficient_credits` (402: user tops up at https://framefields.dev/dashboard/billing; don't retry), `key_limit_reached` (monthly cap), `rate_limited` / `too_many_inflight_requests` (wait, retry). `ValidationError` (422) lists `fieldErrors`.-Coding agents interacting with the AI gateway can discover endpoints, inspect credit runway, and handle top-ups programmatically:+**Agent payments.** `GET https://framefields.dev/.well-known/agent-payment.json` describes auth, packs and endpoints. `GET https://framefields.dev/api/v1/agent/balance` (bearer `FRAMEFIELDS_API_KEY`) returns `{ balance_usd, held_usd, available_usd }`: check it before big batches. A 402 carries `action_required.url`, a checkout link: pass it to the user. `POST https://framefields.dev/api/v1/agent/top-up` with `{"amount": 10 | 25 | 50 | 100}` creates one on demand.-- **Manifest discovery:** Read `GET https://framefields.dev/.well-known/agent-payment.json` for live auth schemes, pack amounts, and endpoint paths.-- **Runway check:** `GET https://framefields.dev/api/v1/agent/balance` using the `FRAMEFIELDS_API_KEY` process environment variable as the bearer credential (never print it) returns `{ balance_usd, held_usd, available_usd }`. Query this before expensive generation batches.-- **Autonomous 402 flow:** When credits are exhausted, the gateway returns HTTP `402 Payment Required` with a dynamic Dodo checkout link:- ```json- {- "status": "payment_required",- "action_required": {- "type": "redirect_to_checkout",- "url": "https://checkout.dodopayments.com/buy/...",- "instructions": "Pass this URL to your human user to complete authorization."- }- }- ```- Pass the URL to the user in chat/terminal to authorize via Dodo Payments.-- **On-demand top-up:** Alternatively, call `POST https://framefields.dev/api/v1/agent/top-up` with `{"amount": 25}` (or 10, 50, 100) to generate a checkout session dynamically.+## 6. Ready-made components-## 8. Grow the film--Once the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading and grain, `framefields-vision` for cutouts, pose and tracking people, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.--### Ready-made components--framefields has a registry of parametric motion components: a mouse pointer with clicks and drags, a camera that zooms to clicks, window and device frames, word-timed captions, text reveals, counters, annotations, a terminal, code blocks, title cards, lower thirds, meters, chat threads, toasts, callouts, spotlights, confetti, logo reveals, backdrops, audio waveforms, word cycles, touches, before/after splits, selections and card stacks. They are not in the npm package: `framefields add` copies their source into the project, where it is ordinary project code to read and change.+A registry of parametric motion components (mouse pointer with clicks, click zoom, window and device frames, word-timed captions, text reveals, counters, annotations, terminal, code blocks, title cards, lower thirds, chat threads, toasts, callouts, confetti, logo reveals, waveforms, before/after splits …). `framefields add` copies their source into the project, where it's ordinary code to read and change. **Run `npx framefields list` before hand-writing a common motion piece.**```bash-npx framefields list # every component, one line each-npx framefields list --json # generated props, dependencies, fonts and examples-npx framefields fonts list --json # pinned font catalog and configured paths-npx framefields api # generated timing/grid/preflight reference+npx framefields list [--json] # one line each, or props, deps, fonts, examplesnpx framefields add mouse-pointer focus-zoom # writes src/components/<name>/-npx framefields fonts add emoji # color emoji (chat-thread and captions add it themselves)+npx framefields fonts add emoji # color emoji for plain Layer.text```-The generated project also has `check` (TypeScript plus schema/lint/timeline preflight) and `check:render` (adds a GPU frame/grid smoke check) scripts. They produce versioned JSON for agents and terminate with a failure code on errors. For another named render, use `framefields check teaser --json` (no name checks them all); for a module without one, `framefields check src/film.ts --export buildFilm --json`; add `--render` to write inspection artifacts under `output/check`. Keep the builder free of generation, rendering and writes, because preflight imports and executes it.-```typescriptimport { MousePointer } from "./components/mouse-pointer/index.js";--const pointer = MousePointer({- path: [- { to: [400, 300], at: 0 },- { to: [960, 540], at: 30, click: true },- ],-});-film.add(pointer);-// pointer.cues.clicks[0].frame: key a button press or a sound to the click.+const pointer = MousePointer({ path: [{ to: [400, 300], at: 0 }, { to: [960, 540], at: 30, click: true }] });+film.add(pointer); // pointer.cues.clicks[0].frame: key a sound or press to the click```-- **Before writing a common motion piece by hand, run `npx framefields list`.** A component already handles timing, theme, cues and edge cases, and the copy is the project's to adjust.-- Files go to `src/components` (or `components` when there is no `src`); the folder is saved in `framefields.json`, and `--dir` changes it on first use. Components that build on others bring them along (`terminal` adds `app-frame` and `counter`).-- Installed code imports only `framefields` and `framefields/components` (the shared toolkit), so it always matches the installed framefields version. After upgrading framefields, `npx framefields add <name> --overwrite` refreshes a component; without the flag, files you changed are left alone and reported as `skipped`.-- **Fonts come with the components that need them.** `chat-thread` and `captions` ask for a color emoji font. `add` downloads Noto Color Emoji (pinned by SHA-256, ~10.7 MB, saved with its OFL license) into `assets/fonts/`, records it under `fonts` in `framefields.json`, and writes `src/components/fonts.ts`. Call its `registerFonts()` once before building or rendering, next to the `FontManager.register` calls. `--no-fonts` skips the download, `npx framefields fonts add emoji` installs it later (or for plain `Layer.text`), and pointing `fonts.emoji` in `framefields.json` at another font keeps yours. Track `assets/fonts/**` with Git LFS.-- **Emoji without an emoji font draw nothing.** `comp.lint()` reports `missing_emoji_font`, and the first render prints the same warning. Fix it with the command above, not by removing the emoji.-- Every component takes the base props (`x`, `y`, `width`, `height`, `at`, `duration`, `hold`, `theme`, `enter`, `exit`, `seed`, `fps`, `canvas`) and returns a normal box layer with `cues` (frames of its events), `span` and `parts` (inner layers to animate or style). Time is in frames.-- Each installed `index.ts` documents what the component does at the top and defines its props in its schema; `npx framefields list` gives one-line summaries. Props are validated when the component is built, and a wrong one throws with the reason.--## 9. Report engine issues--When you encounter an unexpected crash, native WebGPU / Dawn failure, WGSL shader compilation error, compositor schema validation defect, or behavior bug in `framefields` itself (distinguishable from project-level configuration or asset errors):--1. **Ask the user first:** Never open an issue autonomously without explicit confirmation. Explain what went wrong, summarize the reproduction, and ask if they would like an issue reported to `gatewai-dev/framefields`.-2. **Collect diagnostic details:**- - Operating system and architecture (`process.platform`, `process.arch`).- - Node.js version (`node --version`) and `framefields` version.- - GPU backend and device (run `describeGpu()` or note Metal, Vulkan, DX12, or CPU lavapipe/WARP).- - Minimal reproduction code (the smallest composition or script that triggers it).- - Exact error message, full stack trace, or validation failure log.-3. **Open the issue:** Once the user confirms, submit the issue via GitHub CLI:- ```bash- gh issue create --repo gatewai-dev/framefields --title "<concise summary>" --body "<diagnostic details and repro>"- ```- If `gh` is unavailable or not authenticated in the environment, output the title, formatted body, and a direct link for the user: https://github.com/gatewai-dev/framefields/issues/new.+- Every component takes base props (`x`, `y`, `width`, `height`, `at`, `duration`, `hold`, `theme`, `enter`, `exit`, `seed`, `fps`, `canvas`; time in frames) and returns a box layer with `cues`, `span` and `parts`. Props are validated at build; each `index.ts` documents itself at the top.+- Files go to the folder saved in `framefields.json` (`--dir` on first use); dependencies come along. Installed code imports only `framefields` and `framefields/components`. After upgrading framefields, `add <name> --overwrite` refreshes; without it, edited files are `skipped`.+- Components needing emoji (`chat-thread`, `captions`) download Noto Color Emoji (pinned, ~10.7 MB) into `assets/fonts/` and write `src/components/fonts.ts`: call its `registerFonts()` before building. Emoji without an emoji font draw nothing (`lint` reports `missing_emoji_font`): install the font, don't remove the emoji.++## 7. Report engine issues++For a crash, Dawn/WebGPU failure, WGSL compile error, schema defect or behavior bug in framefields itself (not project config or assets):++1. **Ask the user first.** Summarize the problem and repro; never file autonomously.+2. Collect OS/arch, Node and framefields versions, GPU backend (`describeGpu()`), a minimal repro, the exact error and stack.+3. On confirmation: `gh issue create --repo gatewai-dev/framefields --title "…" --body "…"`. Without `gh`, give the user the title, body and https://github.com/gatewai-dev/framefields/issues/new.diff --git a/scripts/validate-agent-plugin.mjs b/scripts/validate-agent-plugin.mjsindex 128dae41..02e3dc25 100644--- a/scripts/validate-agent-plugin.mjs+++ b/scripts/validate-agent-plugin.mjs@@ -33,6 +33,7 @@ const AGENT_PLUGINS_SCHEMA ="https://agent-plugins.org/schemas/1.0.0/plugin.schema.json";const SKILL_NAMES = ["framefields",+ "framefields-cloud","framefields-compose","framefields-effects","framefields-render",