Code-first video & image, rendered natively on WebGPU.
docs(skills): render-ready projects, framefields 2.2 GPU and encoder notes
okanaslfc7f78692be1
diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex d8df0995..9dee605f 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -9,7 +9,7 @@ description: Render, inspect, test, and export high-performance WebGPU video andImport everything from the [`framefields`](https://www.npmjs.com/package/framefields) npm package. If the project doesn't have framefields set up yet, follow the `framefields` skill first (install, project scaffold, first render).## Overview-`framefields` renders headlessly in Node.js on top of native WebGPU (Dawn) and mediabunny WebCodecs hardware encoders. It does not run a browser, Puppeteer, or Chromium.+`framefields` renders headlessly in Node.js on top of native WebGPU (Dawn), encoding through FFmpeg with a hardware encoder when one works (NVENC, VideoToolbox) and a software one otherwise. It does not run a browser, Puppeteer, or Chromium.Because video rendering is resource-intensive and native GPU pipelines run asynchronously in worker pools, framefields workflows enforce a strict **verification ladder**: inspect inexpensive intermediate artifacts first before committing to a full video render.@@ -292,5 +292,8 @@ ffmpeg -i output/final-video.mp4 -af "volumedetect" -f null -| Vision cutout / mask is empty on a single `renderFrame` | Vision reads the previous rendered frame; frame 0 has none | Render two or more consecutive frames with the same `HeadlessMediaRenderer` and keep the last. `renderVideo` handles this automatically. || Ghosted double of the subject in a vision cutout | Frames rendered out of order (frame grid, or jumping from frame 3 to 48), so the cutout uses pixels from another frame | Verify vision layers with sequential frames or the exported MP4, not `renderFrameGrid`. || Script hangs after rendering | Background worker thread pool held open | Always include `process.exit(0)` at the end of render scripts. |+| Render very slow on a Linux server | Frames drawn on a CPU adapter (lavapipe) | `await describeGpu()` shows `software: true`; fix the GPU's Vulkan driver (`vulkaninfo --summary`). |+| Encoder fails to open (`h264_rkmpp`, NVENC) | framefields < 2.2 uses the hardware encoder untested | Use framefields 2.2+, which falls back to libx264 etc., or set `FRAMEFIELDS_HW_CODECS=off`. |+| Render worker job fails | Project not render-ready (lockfile, export, LFS) | Read the job's `code` and `hint`; checklist at https://framefields.dev/docs/guides/render-service/ || Text looks misaligned or truncated | Layout node box smaller than text string | Set explicit wrapping `width` (e.g. `width: 1400`) or use `width: "fill"`. || Colors appear washed out / dark edges | Unpremultiplied alpha blending | The engine operates with premultiplied alpha. Ensure custom shaders use `srcFactor: "one"`. |diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex bbc33609..15fdac3e 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -21,7 +21,8 @@ This skill takes a project from nothing to a verified first render. For anything## 1. Check the environment- **Node.js 22 or later.** Run `node --version`. Older versions fail at import.-- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. Rendering strictly requires a GPU; headless CI environments or virtual machines without a GPU cannot render (there is no CPU software rasterization fallback).+- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. From framefields 2.2, Linux without a GPU falls back to a CPU Vulkan driver (Mesa lavapipe) when one is installed, 10–100× slower; `describeGpu()` reports `software: true` then. Machines with neither cannot render.+- **framefields version.** `framefields` installs the stable 2.1. GPU vision, render progress/cancel and the render worker need 2.2, on `framefields@beta` until 2.2.0 is released. Use the beta when the user wants those.- **An ES module project.** The project's `package.json` needs `"type": "module"`.## 2. Install@@ -53,7 +54,7 @@ For a new project, create `package.json` with all three blocks. In an existing p}```-`allowScripts` is read by npm, `pnpm` by pnpm, and `trustedDependencies` by bun; each tool ignores the others' fields. Yarn runs install scripts by default and needs none of them. Then install:+`allowScripts` is read by npm, `pnpm` by pnpm, and `trustedDependencies` by bun; each tool ignores the others' fields. Yarn runs install scripts by default and needs none of them. Also add `"packageManager"` with the exact manager and version you install with (e.g. `"[email protected]"`), and commit the lockfile the install writes (pnpm, npm or yarn; bun lockfiles can't be rendered by the worker). Then install:```bashnpm install framefields@@ -81,7 +82,7 @@ A minimal `tsconfig.json`:## 3. Scaffold the project-Keep the composition separate from the script that renders it, so tests and the render script build the same film:+Keep the composition separate from the script that renders it, so tests, the preview, the render script and the render worker all build the same film. `film.ts` must only build: no rendering, no `process.exit()`, no writes at import time.```src/@@ -246,7 +247,7 @@ Video inputs (`Layer.video("assets/footage/clip.mp4")`), Lottie animations (`Lay## 4. Verify, cheapest step first-Never go straight to a full video render. Rendering strictly requires a WebGPU-capable GPU (Metal on macOS, Vulkan on Linux/Windows; CPU software rendering is not supported). Each step catches problems in seconds that a video render takes minutes to show:+Never go straight to a full video render. Rendering needs a WebGPU-capable GPU (Metal on macOS, Vulkan on Linux/Windows); CPU rendering through lavapipe works on 2.2 but is far slower. Each step catches problems in seconds that a video render takes minutes to show:1. `npm run typecheck`2. `npm run frames -- 0 30 60` and look at the PNGs in `output/`. A black frame usually means a missing font, a layer outside the canvas, or opacity stuck at 0.@@ -267,6 +268,17 @@ Show the user the frame PNGs or the grid before the full render. `framefields-re- Link to a moment with `#t=12.5` (seconds). In a browser you can script, `window.framefieldsPreview` has `seek(seconds)`, `play()`, `pause()` and `state()`; `seek` resolves once the frame is drawn, so screenshot after it.- **The user can pin notes on the picture.** Pausing and clicking (or dragging an area) adds a note at that spot and moment. Each one prints in the preview's output and is saved in `.framefields/preview-notes/`: `notes.json`, plus `note-<id>.jpg`, the frame with the spot marked. When the user says they left notes, read `notes.json` and look at each open note's picture before you edit. Then run the preview again so they can check the changes and tick them off. `framefields-render` has the details.-## 5. Grow the film+## 5. Render on a server++The framefields render worker (Docker, RunPod pod or serverless) renders a pushed commit: it pulls LFS assets, installs from the lockfile, imports `src/film.ts`, calls `buildFilm`, and uploads the result. A project built as above works unchanged if:++- `framefields` is 2.2+, the lockfile is committed and matches `package.json`, and `packageManager` is pinned;+- every input is under `assets/`, tracked by Git LFS and pushed (`git lfs ls-files` lists it);+- asset paths come from `import.meta.url`, never `process.cwd()`;+- nothing writes into `node_modules` while rendering (files are shared read-only).++A failed job returns `{ code, hint, retryable }`; resubmit unchanged only when `retryable` is true. Job format and error codes: https://framefields.dev/docs/guides/render-service/++## 6. Grow the filmOnce the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading, grain, and vision-driven effects, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.