Code-first video & image, rendered natively on WebGPU.
cfg
okanasl51fa9fe6a0b2
diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.mdindex 52072561..26aefd5f 100644--- a/apps/renderer-service/README.md+++ b/apps/renderer-service/README.md@@ -34,13 +34,8 @@ job ─▶ resolve ref → fetch (mirror cache) → checkout → git LFS pull- **Your repo picks the framefields version.** The worker image contains no engine. Each repo depends on `framefields` in its `package.json` (with a committed lockfile), and the worker renders with exactly that install.-> **Use framefields 2.2 (beta for now).** The worker features (progress, cancel, staying warm between jobs, working video encoders on every GPU) need framefields 2.2. It is published under npm's `beta` tag:->-> ```bash-> npm i framefields@beta # → "framefields": "^2.2.0-beta.0"; 2.2.0 satisfies it once released-> ```->-> A plain `npm i framefields` still installs the stable 2.1.x. Projects on 2.1.x render too, but without those features.+> **Use framefields 2.2 or newer.** The worker features (progress, cancel, staying warm between jobs, working video encoders on every GPU) need it. Projects on 2.1.x render too, but without those features.+- **Your code runs inside the worker process.** The first job loads its project's framefields and initialises the GPU. Later jobs on the same version reuse that warm engine, even from other repos. A job that needs a different version fails with retryable `ENGINE_SWITCH`, and the worker restarts so the retry loads it. Keep a worker (or endpoint) per framefields version if your projects pin different ones.- **Everything is pinned to one commit.** A branch is resolved once per job, and the result reports the SHA it rendered.- **Caches survive between jobs:** git mirrors, LFS objects and package stores live on the cache volume, and checkouts and `node_modules` are reused. A warm job spends almost all its time rendering.@@ -94,7 +89,7 @@ CI ([`renderer-image.yml`](../../.github/workflows/renderer-image.yml)) publishePull requests only build and smoke-test the image. A manual run (**Actions → renderer-image → Run workflow**) publishes the selected branch.-Until the first stable `renderer-v*` release, the RunPod templates and docker-compose use `:beta`. Switch them to `:latest` (or a fixed version) afterwards.+The RunPod templates and docker-compose use `:latest`. Pin a version (`:2.2.0`) to control when workers upgrade.Building the image yourself, from the repo root:@@ -158,7 +153,7 @@ At startup the worker looks for the GPU's device files (`/dev/nvidia*`, or `/devFor NVIDIA GPUs in Docker, the NVIDIA Container Toolkit must mount the Vulkan driver. The image sets `NVIDIA_DRIVER_CAPABILITIES=graphics,utility,video,compute`, which does that. Keep `graphics` in the list if you override it, or the worker reports that the GPU has no Vulkan driver.-**Encoding video.** framefields 2.2 (`framefields@beta` until 2.2.0 is released) tests the hardware encoder once per codec, and uses it only if it works:+**Encoding video.** framefields 2.2 tests the hardware encoder once per codec, and uses it only if it works:- NVENC on NVIDIA, VideoToolbox on macOS.- Otherwise software: libx264 (H.264), libx265 (HEVC), libvpx (VP9), SVT-AV1.diff --git a/apps/renderer-service/deploy/docker-compose.yml b/apps/renderer-service/deploy/docker-compose.ymlindex 8c988004..9de2eb35 100644--- a/apps/renderer-service/deploy/docker-compose.yml+++ b/apps/renderer-service/deploy/docker-compose.yml@@ -5,7 +5,7 @@# Needs the NVIDIA Container Toolkit. API on :8082, MinIO console on :9001.services:renderer:- image: ${FF_IMAGE:-ghcr.io/gatewai-dev/framefields-renderer:beta}+ image: ${FF_IMAGE:-ghcr.io/gatewai-dev/framefields-renderer:latest}restart: unless-stopped # recycling exits the process on purposeenv_file: .envenvironment:diff --git a/apps/renderer-service/deploy/runpod/pod.template.json b/apps/renderer-service/deploy/runpod/pod.template.jsonindex be9ea9cb..c900f2b7 100644--- a/apps/renderer-service/deploy/runpod/pod.template.json+++ b/apps/renderer-service/deploy/runpod/pod.template.json@@ -1,6 +1,6 @@{"name": "framefields-renderer (pod, HTTP API)",- "imageName": "ghcr.io/gatewai-dev/framefields-renderer:beta",+ "imageName": "ghcr.io/gatewai-dev/framefields-renderer:latest","isServerless": false,"containerDiskInGb": 40,"volumeInGb": 100,diff --git a/apps/renderer-service/deploy/runpod/serverless.template.json b/apps/renderer-service/deploy/runpod/serverless.template.jsonindex 5732faf0..b8ba6d1b 100644--- a/apps/renderer-service/deploy/runpod/serverless.template.json+++ b/apps/renderer-service/deploy/runpod/serverless.template.json@@ -1,6 +1,6 @@{"name": "framefields-renderer (serverless)",- "imageName": "ghcr.io/gatewai-dev/framefields-renderer:beta",+ "imageName": "ghcr.io/gatewai-dev/framefields-renderer:latest","isServerless": true,"containerDiskInGb": 40,"volumeInGb": 0,diff --git a/mediabunny-issue-draft.md b/mediabunny-issue-draft.mdnew file mode 100644index 00000000..437da6c9--- /dev/null+++ b/mediabunny-issue-draft.md@@ -0,0 +1,51 @@+# Server: the software encoder fallback can pick a hardware encoder (h264_rkmpp on linux-arm64)++`NodeAvVideoEncoder` uses `NodeAv.Codec.findEncoder(codecId)` as its software encoder (`getSoftwareCodec()` in `video-encoder.ts`). FFmpeg returns whichever encoder it registered first for that codec ID, and that isn't always a software one. In node-av's linux-arm64 build, the first H.264 and HEVC encoders are `h264_rkmpp` and `hevc_rkmpp` (Rockchip MPP), listed before libx264 and libx265. So on any arm64 Linux machine that isn't a Rockchip board (AWS Graviton, Ampere, Docker on Apple silicon), the encoder that should be the safe fallback is a hardware one that can't run there.++## What happens++On @mediabunny/server 1.61.6 with node-av 6.x in a linux-arm64 container (Debian, no Rockchip hardware):++- `bitrateMode: 'quantizer'` with `hardwareAcceleration: 'prefer-software'` (or with no usable hardware encoder) throws: `Encoder 'h264_rkmpp' cannot be used for quantizer-based encoding.`+- In the other bitrate modes, `h264_rkmpp` is still the encoder that gets chosen, on a machine with no MPP device.++`Codec.findEncoderByName('libx264')` returns the encoder fine on the same machine, so libx264 is in the build; it just isn't the first one registered.++## Suggested fix++`getSoftwareCodec()` already prefers `prores_ks` by name for ProRes, and the alpha path prefers `libvpx-vp9`. The same approach would work for the other codecs: try known software encoders by name first, then fall back to `findEncoder`:++| Codec | Preferred software encoders |+|---|---|+| avc | `libx264`, `libopenh264` |+| hevc | `libx265` |+| vp8 | `libvpx` |+| vp9 | `libvpx-vp9` |+| av1 | `libsvtav1`, `libaom-av1` |++Something like:++```ts+const SOFTWARE_ENCODERS: Partial<Record<VideoCodec, string[]>> = {+ avc: ['libx264', 'libopenh264'],+ hevc: ['libx265'],+ vp8: ['libvpx'],+ vp9: ['libvpx-vp9'],+ av1: ['libsvtav1', 'libaom-av1'],+ prores: ['prores_ks'],+};++const getSoftwareCodec = () => {+ for (const name of SOFTWARE_ENCODERS[this.codec] ?? []) {+ const codec = NodeAv.Codec.findEncoderByName(name as NodeAv.FFEncoderCodec);+ if (codec) return codec;+ }+ return NodeAv.Codec.findEncoder(codecId);+};+```++A side benefit: the quantizer mapping that follows already knows these encoder names, so a quantizer encode would no longer reach the `cannot be used for quantizer-based encoding` branch on builds that include a hardware encoder.++## Workaround++We currently wrap `Codec.findEncoder` on the node-av instance that @mediabunny/server loads, so it returns the encoders above when they exist. It works, but it relies on loading the exact same node-av module instance as the server, which is fragile.diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 15fdac3e..d1c8f7cb 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -21,8 +21,7 @@ This skill takes a project from nothing to a verified first render. For anything## 1. Check the environment- **Node.js 22 or later.** Run `node --version`. Older versions fail at import.-- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. From framefields 2.2, Linux without a GPU falls back to a CPU Vulkan driver (Mesa lavapipe) when one is installed, 10–100× slower; `describeGpu()` reports `software: true` then. Machines with neither cannot render.-- **framefields version.** `framefields` installs the stable 2.1. GPU vision, render progress/cancel and the render worker need 2.2, on `framefields@beta` until 2.2.0 is released. Use the beta when the user wants those.+- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. Linux without a GPU falls back to a CPU Vulkan driver (Mesa lavapipe) when one is installed, 10–100× slower; `describeGpu()` reports `software: true` then. Machines with neither cannot render.- **An ES module project.** The project's `package.json` needs `"type": "module"`.## 2. Install