Skip to content

Code-first video & image, rendered natively on WebGPU.

cfg

okanasl51fa9fe6a0b2
diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.md
index 52072561..26aefd5f 100644
--- a/apps/renderer-service/README.md
+++ b/apps/renderer-service/README.md
@@ -34,13 +34,8 @@ job ─▶ resolve ref → fetch (mirror cache) → checkout → git LFS pull
- **Your repo picks the framefields version.** The worker image contains no engine. Each repo depends on `framefields` in its `package.json` (with a committed lockfile), and the worker renders with exactly that install.
-> **Use framefields 2.2 (beta for now).** The worker features (progress, cancel, staying warm between jobs, working video encoders on every GPU) need framefields 2.2. It is published under npm's `beta` tag:
->
-> ```bash
-> npm i framefields@beta # → "framefields": "^2.2.0-beta.0"; 2.2.0 satisfies it once released
-> ```
->
-> A plain `npm i framefields` still installs the stable 2.1.x. Projects on 2.1.x render too, but without those features.
+> **Use framefields 2.2 or newer.** The worker features (progress, cancel, staying warm between jobs, working video encoders on every GPU) need it. Projects on 2.1.x render too, but without those features.
+
- **Your code runs inside the worker process.** The first job loads its project's framefields and initialises the GPU. Later jobs on the same version reuse that warm engine, even from other repos. A job that needs a different version fails with retryable `ENGINE_SWITCH`, and the worker restarts so the retry loads it. Keep a worker (or endpoint) per framefields version if your projects pin different ones.
- **Everything is pinned to one commit.** A branch is resolved once per job, and the result reports the SHA it rendered.
- **Caches survive between jobs:** git mirrors, LFS objects and package stores live on the cache volume, and checkouts and `node_modules` are reused. A warm job spends almost all its time rendering.
@@ -94,7 +89,7 @@ CI ([`renderer-image.yml`](../../.github/workflows/renderer-image.yml)) publishe
Pull requests only build and smoke-test the image. A manual run (**Actions → renderer-image → Run workflow**) publishes the selected branch.
-Until the first stable `renderer-v*` release, the RunPod templates and docker-compose use `:beta`. Switch them to `:latest` (or a fixed version) afterwards.
+The RunPod templates and docker-compose use `:latest`. Pin a version (`:2.2.0`) to control when workers upgrade.
Building the image yourself, from the repo root:
@@ -158,7 +153,7 @@ At startup the worker looks for the GPU's device files (`/dev/nvidia*`, or `/dev
For NVIDIA GPUs in Docker, the NVIDIA Container Toolkit must mount the Vulkan driver. The image sets `NVIDIA_DRIVER_CAPABILITIES=graphics,utility,video,compute`, which does that. Keep `graphics` in the list if you override it, or the worker reports that the GPU has no Vulkan driver.
-**Encoding video.** framefields 2.2 (`framefields@beta` until 2.2.0 is released) tests the hardware encoder once per codec, and uses it only if it works:
+**Encoding video.** framefields 2.2 tests the hardware encoder once per codec, and uses it only if it works:
- NVENC on NVIDIA, VideoToolbox on macOS.
- Otherwise software: libx264 (H.264), libx265 (HEVC), libvpx (VP9), SVT-AV1.
diff --git a/apps/renderer-service/deploy/docker-compose.yml b/apps/renderer-service/deploy/docker-compose.yml
index 8c988004..9de2eb35 100644
--- a/apps/renderer-service/deploy/docker-compose.yml
+++ b/apps/renderer-service/deploy/docker-compose.yml
@@ -5,7 +5,7 @@
# Needs the NVIDIA Container Toolkit. API on :8082, MinIO console on :9001.
services:
renderer:
- image: ${FF_IMAGE:-ghcr.io/gatewai-dev/framefields-renderer:beta}
+ image: ${FF_IMAGE:-ghcr.io/gatewai-dev/framefields-renderer:latest}
restart: unless-stopped # recycling exits the process on purpose
env_file: .env
environment:
diff --git a/apps/renderer-service/deploy/runpod/pod.template.json b/apps/renderer-service/deploy/runpod/pod.template.json
index be9ea9cb..c900f2b7 100644
--- a/apps/renderer-service/deploy/runpod/pod.template.json
+++ b/apps/renderer-service/deploy/runpod/pod.template.json
@@ -1,6 +1,6 @@
{
"name": "framefields-renderer (pod, HTTP API)",
- "imageName": "ghcr.io/gatewai-dev/framefields-renderer:beta",
+ "imageName": "ghcr.io/gatewai-dev/framefields-renderer:latest",
"isServerless": false,
"containerDiskInGb": 40,
"volumeInGb": 100,
diff --git a/apps/renderer-service/deploy/runpod/serverless.template.json b/apps/renderer-service/deploy/runpod/serverless.template.json
index 5732faf0..b8ba6d1b 100644
--- a/apps/renderer-service/deploy/runpod/serverless.template.json
+++ b/apps/renderer-service/deploy/runpod/serverless.template.json
@@ -1,6 +1,6 @@
{
"name": "framefields-renderer (serverless)",
- "imageName": "ghcr.io/gatewai-dev/framefields-renderer:beta",
+ "imageName": "ghcr.io/gatewai-dev/framefields-renderer:latest",
"isServerless": true,
"containerDiskInGb": 40,
"volumeInGb": 0,
diff --git a/mediabunny-issue-draft.md b/mediabunny-issue-draft.md
new file mode 100644
index 00000000..437da6c9
--- /dev/null
+++ b/mediabunny-issue-draft.md
@@ -0,0 +1,51 @@
+# Server: the software encoder fallback can pick a hardware encoder (h264_rkmpp on linux-arm64)
+
+`NodeAvVideoEncoder` uses `NodeAv.Codec.findEncoder(codecId)` as its software encoder (`getSoftwareCodec()` in `video-encoder.ts`). FFmpeg returns whichever encoder it registered first for that codec ID, and that isn't always a software one. In node-av's linux-arm64 build, the first H.264 and HEVC encoders are `h264_rkmpp` and `hevc_rkmpp` (Rockchip MPP), listed before libx264 and libx265. So on any arm64 Linux machine that isn't a Rockchip board (AWS Graviton, Ampere, Docker on Apple silicon), the encoder that should be the safe fallback is a hardware one that can't run there.
+
+## What happens
+
+On @mediabunny/server 1.61.6 with node-av 6.x in a linux-arm64 container (Debian, no Rockchip hardware):
+
+- `bitrateMode: 'quantizer'` with `hardwareAcceleration: 'prefer-software'` (or with no usable hardware encoder) throws: `Encoder 'h264_rkmpp' cannot be used for quantizer-based encoding.`
+- In the other bitrate modes, `h264_rkmpp` is still the encoder that gets chosen, on a machine with no MPP device.
+
+`Codec.findEncoderByName('libx264')` returns the encoder fine on the same machine, so libx264 is in the build; it just isn't the first one registered.
+
+## Suggested fix
+
+`getSoftwareCodec()` already prefers `prores_ks` by name for ProRes, and the alpha path prefers `libvpx-vp9`. The same approach would work for the other codecs: try known software encoders by name first, then fall back to `findEncoder`:
+
+| Codec | Preferred software encoders |
+|---|---|
+| avc | `libx264`, `libopenh264` |
+| hevc | `libx265` |
+| vp8 | `libvpx` |
+| vp9 | `libvpx-vp9` |
+| av1 | `libsvtav1`, `libaom-av1` |
+
+Something like:
+
+```ts
+const SOFTWARE_ENCODERS: Partial<Record<VideoCodec, string[]>> = {
+ avc: ['libx264', 'libopenh264'],
+ hevc: ['libx265'],
+ vp8: ['libvpx'],
+ vp9: ['libvpx-vp9'],
+ av1: ['libsvtav1', 'libaom-av1'],
+ prores: ['prores_ks'],
+};
+
+const getSoftwareCodec = () => {
+ for (const name of SOFTWARE_ENCODERS[this.codec] ?? []) {
+ const codec = NodeAv.Codec.findEncoderByName(name as NodeAv.FFEncoderCodec);
+ if (codec) return codec;
+ }
+ return NodeAv.Codec.findEncoder(codecId);
+};
+```
+
+A side benefit: the quantizer mapping that follows already knows these encoder names, so a quantizer encode would no longer reach the `cannot be used for quantizer-based encoding` branch on builds that include a hardware encoder.
+
+## Workaround
+
+We currently wrap `Codec.findEncoder` on the node-av instance that @mediabunny/server loads, so it returns the encoders above when they exist. It works, but it relies on loading the exact same node-av module instance as the server, which is fragile.
diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.md
index 15fdac3e..d1c8f7cb 100644
--- a/plugins/framefields/skills/framefields/SKILL.md
+++ b/plugins/framefields/skills/framefields/SKILL.md
@@ -21,8 +21,7 @@ This skill takes a project from nothing to a verified first render. For anything
## 1. Check the environment
- **Node.js 22 or later.** Run `node --version`. Older versions fail at import.
-- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. From framefields 2.2, Linux without a GPU falls back to a CPU Vulkan driver (Mesa lavapipe) when one is installed, 10–100× slower; `describeGpu()` reports `software: true` then. Machines with neither cannot render.
-- **framefields version.** `framefields` installs the stable 2.1. GPU vision, render progress/cancel and the render worker need 2.2, on `framefields@beta` until 2.2.0 is released. Use the beta when the user wants those.
+- **A WebGPU-capable GPU.** macOS uses Metal; Linux and Windows use Vulkan. Linux without a GPU falls back to a CPU Vulkan driver (Mesa lavapipe) when one is installed, 10–100× slower; `describeGpu()` reports `software: true` then. Machines with neither cannot render.
- **An ES module project.** The project's `package.json` needs `"type": "module"`.
## 2. Install