Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(renderer-service): GPU selection with CPU fallback (FF_GPU)

okanasl04710992f657
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.md
index 1f96e354..3044a327 100644
--- a/apps/renderer-service/PLAN.md
+++ b/apps/renderer-service/PLAN.md
@@ -406,6 +406,28 @@ On the next attempt, the worker reads the manifest right after checkout, once th
Without a key, outputs are overwritten (last writer wins). Output keys can't be checked before rendering, because `{name}`/`{ext}` are only known afterwards; that is why a separate manifest is used.
+### 8.7 GPU selection (`src/worker/gpu.ts`)
+
+WebGPU (Dawn) runs on Vulkan on Linux. The image ships Mesa's CPU driver (lavapipe) so machines without a GPU can render. A GPU machine whose Vulkan driver isn't mounted would otherwise fall back to the CPU silently, and run 10–100× slower.
+
+- **At boot, before any engine loads,** the worker:
+ - looks for a usable GPU: `/dev/nvidia*`, or a `/dev/dri/renderD*` node plus an NVIDIA/AMD/Intel PCI vendor in `/sys/class/drm`. Listing in `/sys` alone isn't enough, since containers see host GPUs there;
+ - splits the Vulkan ICD manifests in `/etc/vulkan/icd.d`, `/usr/share/vulkan/icd.d` and `/usr/local/share/vulkan/icd.d` into GPU and CPU drivers.
+- **`FF_GPU` modes:**
+ - `auto` (default): a GPU machine sets `VK_DRIVER_FILES`/`VK_ICD_FILENAMES` to the GPU drivers only; a machine without a GPU renders on lavapipe.
+ - `require` (RunPod templates): like `auto`, but a machine without a GPU fails.
+ - `cpu`: limits Vulkan to lavapipe.
+ - If the operator already set `VK_DRIVER_FILES`/`VK_ICD_FILENAMES`, the worker leaves them alone.
+- **Boot failures:** a GPU with no Vulkan driver (e.g. `NVIDIA_DRIVER_CAPABILITIES` without `graphics`), or `require` with no GPU. Either one fails readiness with `GPU_UNAVAILABLE`. On RunPod the worker exits and gets replaced.
+- **After the first engine binds,** the worker checks the actual adapter. It uses framefields' `describeGpu()` when the project's version has it; otherwise it calls `navigator.gpu.requestAdapter()` with framefields' options. A GPU machine whose adapter is a CPU one (llvmpipe, SwiftShader, `isFallbackAdapter`) fails the job with `GPU_UNAVAILABLE` and marks the worker unhealthy. The RunPod shim then sets `refresh_worker`.
+- **Reporting:** `/health` → `components.gpu` and each result's `engine.gpu` report the mode, the GPU model and the adapter.
+
+**Video encoding** is framefields' job (`packages/renderer/src/video-encoders.ts`, framefields 2.2.0+):
+
+- @mediabunny/server uses FFmpeg's first registered encoder as its software encoder. In node-av's linux-arm64 build that is `h264_rkmpp`/`hevc_rkmpp`, which fail without Rockchip hardware. framefields makes libx264/libx265/libvpx/SVT-AV1 the defaults.
+- framefields tests the platform's hardware encoder once per codec (open, then encode one frame). It uses NVENC or VideoToolbox only when that works. A100/H100 have no NVENC, so they fall back to software.
+- `FRAMEFIELDS_HW_CODECS=off` forces software.
+
## 9. Trust & security model
The worker **executes code from the repo**. Anyone who can submit a job and point it at a repo they control gets code execution on the GPU host, and that host can reach whatever its network can. The defaults are therefore closed:
@@ -424,7 +446,7 @@ Every failure maps to a stable `code`. `retryable` is what RunPod or the caller
| Phase | Code | Retryable | Typical cause / hint |
|---|---|---|---|
| boot | `CONFIG_INVALID` | – | Bad or missing env. The message lists every invalid key. |
-| boot | `GPU_UNAVAILABLE` | – (worker recycled) | No Vulkan device, or the driver doesn't match the image's CUDA/Vulkan. |
+| boot / render | `GPU_UNAVAILABLE` | – (worker replaced) | `FF_GPU=require` on a machine without a GPU; a GPU without a Vulkan driver; or a GPU machine whose WebGPU adapter is a CPU one (§8.7). |
| boot | `TOOL_MISSING` | – | `git` or `git-lfs` is not on PATH (custom image). |
| validate | `JOB_INVALID` | no | Schema error, with zod paths. |
| validate | `REPO_NOT_ALLOWED` / `BUCKET_NOT_ALLOWED` / `PATH_ESCAPE` | no | |
@@ -473,6 +495,7 @@ On boot the service validates **all** keys and fails with the complete list of p
| Paths & cache | `FF_CACHE_DIR`, `FF_WORK_DIR`, `FF_CACHE_MAX_GB` (50), `FF_WORK_MAX_GB` (40), `FRAMEFIELDS_MODELS_DIR` (`$FF_CACHE_DIR/models`), `FRAMEFIELDS_FONTS_DIR` |
| Limits | `FF_QUEUE_MAX` (8), `FF_JOB_TIMEOUT_MAX_SEC` (7200), `FF_TIMEOUT_{GIT,LFS,INSTALL,LOAD,RENDER}_SEC`, `FF_STALL_TIMEOUT_SEC` (300), `FF_KILL_GRACE_SEC` (20), `FF_JOB_TTL_SEC` (3600), `FF_JOB_HISTORY_MAX` (500) |
| Recycling | `FF_MAX_JOBS_PER_WORKER` (25; 0 = unlimited), `FF_RECYCLE_HEAP_MB` (4096), `FF_RECYCLE_RSS_MB` (0 = off), `FF_EVENT_LOOP_BLOCK_SEC` (120), `FF_STRAY_ERROR_LIMIT` (3), `FF_LEAKED_HANDLE_LIMIT` (50). `NODE_OPTIONS=--max-old-space-size` is set by the image. |
+| GPU | `FF_GPU` (`auto`\|`require`\|`cpu`; default `auto`, RunPod templates `require`). The engine reads `FRAMEFIELDS_HW_CODECS=off` to force software video encoding. |
| Job env | `FF_JOB_ENV_ALLOW` (`*` minus a reserved denylist), `FF_KEEP_FAILED_JOBS` (false), `FF_QA_BLOCK_UPLOAD` (false) |
`BASE_URL`, `R2_*` and `R2_CUSTOM_DOMAIN` are removed together with the inline path. Repo assets are local files in the checkout, so `configureAssetUrls` is no longer called. A project that loads assets over HTTP uses absolute URLs.
@@ -545,7 +568,7 @@ Each phase is its own PR-sized step with tests.
8. **Image + template + compose + docs.** Built and run locally (arm64, Docker on macOS, without a GPU):
- **Inside the container:** a file:// repo pinned to the published `[email protected]` went through checkout, then `npm ci` from the registry (40 s cold), then engine binding with GPU init on Mesa's software Vulkan, then a VP9 render, verification and delivery. The output was a valid 2 s VP9/Opus file.
- **Findings fixed on the way:** glibc 2.38 (moved to trixie); EPIPE from the `exec` helper writing stdin that git never reads; corepack's npm differing from the lockfile's npm; install errors keeping only npm's usage text.
- - **Engine issue found (not the worker):** on linux-arm64, `@napi-rs/webcodecs` picks the Rockchip `h264_rkmpp` encoder, which fails with "cannot be used for quantizer-based encoding". H.264 renders fail on ARM Linux; VP9 works. amd64/NVIDIA (RunPod) takes a different path, but this should be fixed in framefields.
+ - **Engine issue found and fixed in framefields (§8.7):** on linux-arm64, H.264 renders failed with "Encoder 'h264_rkmpp' cannot be used for quantizer-based encoding". The cause was @mediabunny/server taking FFmpeg's first registered H.264 encoder (node-av's build lists `h264_rkmpp` before libx264), not `@napi-rs/webcodecs`. With the fixed framefields (packed locally), an H.264 job in the arm64 container rendered 48/48 frames with libx264 on lavapipe. `FF_GPU=require` on the same container fails readiness with `GPU_UNAVAILABLE`.
- Pinned SSH host keys, the RunPod templates and `create-template.sh`, `docker-compose.yml` with MinIO, `env.example`, the release workflow, and the README guide as listed above.
- **Still to do:** an amd64 build (CI), a GPU smoke test on a RunPod pod and a serverless endpoint, the nightly host smoke tests (6.5), and a page on the docs website.
9. **(Later)**
diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.md
index 829b337f..9f1f9403 100644
--- a/apps/renderer-service/README.md
+++ b/apps/renderer-service/README.md
@@ -15,6 +15,7 @@ It runs as a **RunPod serverless endpoint**, a **RunPod pod**, or a **Docker con
- [How it works](#how-it-works)
- [Quick start (Docker + MinIO)](#quick-start-docker--minio)
- [RunPod](#runpod)
+- [GPU and CPU rendering](#gpu-and-cpu-rendering)
- [Jobs](#jobs)
- [Configuration](#configuration)
- [Errors](#errors)
@@ -37,7 +38,7 @@ job ─▶ resolve ref → fetch (mirror cache) → checkout → git LFS pull
## Quick start (Docker + MinIO)
-You need an NVIDIA GPU with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/).
+Use an NVIDIA GPU with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/). Without a GPU the worker still runs, but renders on the CPU, which is much slower (see [GPU and CPU rendering](#gpu-and-cpu-rendering)).
```bash
cd apps/renderer-service/deploy
@@ -92,7 +93,8 @@ Checklist for serverless endpoints:
- **Set the endpoint's execution timeout** above your longest render. RunPod's default is much shorter than a long video.
- **Attach a network volume** in the same data centre as the GPUs. Without one, every worker starts cold: a full clone, LFS download and install.
- **FlashBoot** keeps warm workers around between jobs, which is what makes the caches pay off.
-- Choose GPUs with Vulkan support (any recent NVIDIA card). VRAM needs grow with resolution and effects.
+- **Choose GPUs with a hardware video encoder (NVENC):** L4, L40S, A40, RTX A4000/A5000/A6000, RTX 4090/5090. A100 and H100 are built for AI training and have no NVENC: they render, but encode on the CPU, and cost more. VRAM needs grow with resolution and effects.
+- The templates set `FF_GPU=require`, so a RunPod worker never renders on the CPU. A worker whose GPU is unusable reports `GPU_UNAVAILABLE` and asks RunPod for a fresh worker.
A serverless job is the same JSON as above, wrapped as `{"input": …}`:
@@ -108,6 +110,38 @@ How results come back:
- Progress events appear in `/status` and `/stream`.
- The worker asks RunPod for a fresh worker (`refresh_worker`) when it recycles.
+## GPU and CPU rendering
+
+framefields draws frames with WebGPU, which runs on Vulkan on Linux. Video is then encoded with FFmpeg. The two steps choose their hardware separately.
+
+**Drawing frames.** `FF_GPU` decides where frames are drawn:
+
+| `FF_GPU` | Machine with a GPU | Machine without a GPU |
+|---|---|---|
+| `auto` (default) | Renders on the GPU. Fails with `GPU_UNAVAILABLE` if the GPU is unusable, instead of falling back to the CPU. | Renders on the CPU (Mesa lavapipe, included in the image). |
+| `require` (RunPod templates) | Renders on the GPU. | The worker fails its health check with `GPU_UNAVAILABLE`. |
+| `cpu` | Renders on the CPU. | Renders on the CPU. |
+
+At startup the worker looks for the GPU's device files (`/dev/nvidia*`, or `/dev/dri` render nodes for AMD and Intel) and limits Vulkan to that GPU's driver, so it can't fall back to the CPU driver silently. After the first job loads framefields, the worker checks which adapter WebGPU actually picked.
+
+For NVIDIA GPUs in Docker, the NVIDIA Container Toolkit must mount the Vulkan driver. The image sets `NVIDIA_DRIVER_CAPABILITIES=graphics,utility,video,compute`, which does that. Keep `graphics` in the list if you override it, or the worker reports that the GPU has no Vulkan driver.
+
+**Encoding video.** framefields (2.2.0 and later) tests the hardware encoder once per codec, and uses it only if it works:
+
+- NVENC on NVIDIA, VideoToolbox on macOS.
+- Otherwise software: libx264 (H.264), libx265 (HEVC), libvpx (VP9), SVT-AV1.
+
+Older framefields versions always try the hardware encoder. That fails on GPUs without NVENC and on Linux ARM (`h264_rkmpp`). Set `FRAMEFIELDS_HW_CODECS=off` to always encode in software.
+
+**Where to see it.** `/health` reports the GPU under `components.gpu`, and each result reports it under `engine.gpu`:
+
+```json
+"gpu": {"mode": "auto", "hardware": "NVIDIA L4", "renderer": "gpu",
+ "adapter": {"name": "NVIDIA L4", "software": false}}
+```
+
+Here `renderer` is `gpu` or `cpu`. `hardware` is `null` when the machine has no GPU.
+
## Jobs
```jsonc
@@ -217,6 +251,8 @@ Everything is set by environment variables. The worker checks them all at startu
| `FF_CACHE_DIR`, `FF_CACHE_MAX_GB`, `FF_WORK_MAX_GB` | Cache location and size limits. Eviction runs automatically. |
| `FF_JOB_TIMEOUT_MAX_SEC`, `FF_STALL_TIMEOUT_SEC` | Time limits. |
| `FF_MAX_JOBS_PER_WORKER` | Recycle after N jobs (default 25). |
+| `FF_GPU` | `auto` (default), `require` or `cpu`. See [GPU and CPU rendering](#gpu-and-cpu-rendering). |
+| `FRAMEFIELDS_HW_CODECS` | `off` encodes video in software even when a hardware encoder works. |
| `FF_NPMRC` | `.npmrc` contents, for private registries. |
## Errors
@@ -242,7 +278,8 @@ A failed job returns:
| `RENDER_TIMEOUT` / `RENDER_STALLED` | The render took too long, or stopped making progress. |
| `ENGINE_NOT_FOUND` | The project doesn't depend on `framefields`, so there is nothing to render with. |
| `ENGINE_SWITCH` | Retry. The project needs another framefields version, or new dependencies; the worker restarts to load them. |
-| `GPU_DEVICE_LOST` / `RENDER_OOM` / `GPU_UNAVAILABLE` | Retry. The worker replaces itself. |
+| `GPU_DEVICE_LOST` / `RENDER_OOM` | Retry. The worker replaces itself. |
+| `GPU_UNAVAILABLE` | The machine has no usable GPU (`FF_GPU=require`), or has one that WebGPU couldn't use. Retry on another worker; on RunPod this happens automatically. Check the GPU's Vulkan driver with `vulkaninfo --summary`. |
| `QA_FAILED` | `render.qa.failOn` matched. The outputs are still uploaded and listed. |
## Security
diff --git a/apps/renderer-service/deploy/runpod/pod.template.json b/apps/renderer-service/deploy/runpod/pod.template.json
index 8fda4fc4..2daa23ea 100644
--- a/apps/renderer-service/deploy/runpod/pod.template.json
+++ b/apps/renderer-service/deploy/runpod/pod.template.json
@@ -9,6 +9,7 @@
"ports": "8082/http",
"env": [
{ "key": "FF_MODE", "value": "http" },
+ { "key": "FF_GPU", "value": "require" },
{ "key": "RENDERER_API_KEY", "value": "{{ RUNPOD_SECRET_ff_api_key }}" },
{ "key": "FF_REPO_ALLOWLIST", "value": "github.com/your-org/*" },
{ "key": "GIT_TOKEN", "value": "{{ RUNPOD_SECRET_ff_git_token }}" },
diff --git a/apps/renderer-service/deploy/runpod/serverless.template.json b/apps/renderer-service/deploy/runpod/serverless.template.json
index bb2c1c98..0cfa4e9d 100644
--- a/apps/renderer-service/deploy/runpod/serverless.template.json
+++ b/apps/renderer-service/deploy/runpod/serverless.template.json
@@ -8,6 +8,7 @@
"ports": "",
"env": [
{ "key": "FF_MODE", "value": "runpod-serverless" },
+ { "key": "FF_GPU", "value": "require" },
{ "key": "FF_REPO_ALLOWLIST", "value": "github.com/your-org/*" },
{ "key": "FF_DEFAULT_REF", "value": "main" },
{ "key": "GIT_TOKEN", "value": "{{ RUNPOD_SECRET_ff_git_token }}" },
diff --git a/apps/renderer-service/docker/entrypoint.sh b/apps/renderer-service/docker/entrypoint.sh
index a51a1c13..ae144104 100755
--- a/apps/renderer-service/docker/entrypoint.sh
+++ b/apps/renderer-service/docker/entrypoint.sh
@@ -22,7 +22,7 @@ fi
if command -v nvidia-smi >/dev/null 2>&1; then
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv,noheader || true
else
- echo "warning: nvidia-smi not found; rendering needs a Vulkan-capable GPU" >&2
+ echo "no NVIDIA GPU visible (nvidia-smi not found); FF_GPU=${FF_GPU:-auto} decides whether to render on the CPU" >&2
fi
case "$MODE" in
diff --git a/apps/renderer-service/runpod_worker.py b/apps/renderer-service/runpod_worker.py
index a5d316af..28c3ddcb 100644
--- a/apps/renderer-service/runpod_worker.py
+++ b/apps/renderer-service/runpod_worker.py
@@ -92,8 +92,8 @@ def handler(job):
return result
if record["status"] in ("FAILED", "CANCELLED"):
error = record.get("error") or {"code": "INTERNAL", "message": "unknown failure"}
- # Device loss / crashes leave the worker unhealthy: ask for a fresh one.
- refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED") or draining()
+ # No usable GPU, device loss, crashes: ask RunPod for a fresh worker.
+ refresh = error.get("code") in ("GPU_UNAVAILABLE", "GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED") or draining()
return failure(error, refresh=refresh)
diff --git a/apps/renderer-service/src/config/schema.ts b/apps/renderer-service/src/config/schema.ts
index 7fadb393..8b1478cf 100644
--- a/apps/renderer-service/src/config/schema.ts
+++ b/apps/renderer-service/src/config/schema.ts
@@ -125,6 +125,9 @@ export const envSchema = z.object({
FF_JOB_TTL_SEC: int(3600),
FF_JOB_HISTORY_MAX: int(500),
+ // GPU: auto = the GPU when the machine has one, else the CPU.
+ FF_GPU: z.enum(["auto", "require", "cpu"]).default("auto"),
+
// Recycling
FF_MAX_JOBS_PER_WORKER: int(25),
FF_RECYCLE_HEAP_MB: int(4096),
@@ -347,6 +350,7 @@ export function buildConfig(
jobTtlSec: e.FF_JOB_TTL_SEC,
jobHistoryMax: e.FF_JOB_HISTORY_MAX,
},
+ gpu: e.FF_GPU,
recycle: {
maxJobs: e.FF_MAX_JOBS_PER_WORKER,
heapMb: e.FF_RECYCLE_HEAP_MB,
diff --git a/apps/renderer-service/src/readiness.test.ts b/apps/renderer-service/src/readiness.test.ts
index fdcdad78..0d6b9083 100644
--- a/apps/renderer-service/src/readiness.test.ts
+++ b/apps/renderer-service/src/readiness.test.ts
@@ -6,9 +6,9 @@ describe("ReadinessTracker", () => {
const tracker = new ReadinessTracker();
expect(tracker.isReady).toBe(false);
tracker.setToolsReady(true);
- expect(tracker.status).toEqual({
+ expect(tracker.status).toMatchObject({
ready: true,
- components: { tools: true, engine: null },
+ components: { tools: true, engine: null, gpu: { adapter: null } },
});
tracker.setEngine("2.1.3");
expect(tracker.status.components.engine).toBe("2.1.3");
diff --git a/apps/renderer-service/src/readiness.ts b/apps/renderer-service/src/readiness.ts
index 1901be99..be913026 100644
--- a/apps/renderer-service/src/readiness.ts
+++ b/apps/renderer-service/src/readiness.ts
@@ -1,3 +1,5 @@
+import { gpuFailure, gpuStatus } from "./worker/gpu.js";
+
/**
* Worker readiness. The worker is ready once the tools it shells out to are
* present; the GPU is initialised by the first job, with that project's
@@ -9,14 +11,25 @@ export class ReadinessTracker {
private _failureReason: string | null = null;
get isReady(): boolean {
- return this._toolsReady && !this._failureReason;
+ return this._toolsReady && !this.failure;
+ }
+
+ /** Boot failure, or a GPU this worker can't render on (worker/gpu.ts). */
+ private get failure(): string | null {
+ const gpu = gpuFailure();
+ return this._failureReason ?? (gpu ? `GPU_UNAVAILABLE: ${gpu}` : null);
}
get status() {
+ const error = this.failure;
return {
ready: this.isReady,
- components: { tools: this._toolsReady, engine: this._engine },
- ...(this._failureReason && { error: this._failureReason }),
+ components: {
+ tools: this._toolsReady,
+ engine: this._engine,
+ gpu: gpuStatus(),
+ },
+ ...(error && { error }),
};
}
diff --git a/apps/renderer-service/src/render/engine.ts b/apps/renderer-service/src/render/engine.ts
index 7a915304..cea764f1 100644
--- a/apps/renderer-service/src/render/engine.ts
+++ b/apps/renderer-service/src/render/engine.ts
@@ -4,6 +4,12 @@ import path from "node:path";
import { pathToFileURL } from "node:url";
import { RenderError } from "../errors.js";
import { rendererLogger } from "../logger.js";
+import {
+ describeAdapter,
+ type GpuAdapter,
+ gpuStatus,
+ verifyAdapter,
+} from "../worker/gpu.js";
import { workerHealth } from "../worker/health.js";
import { setEngineEntry } from "./engine-alias.js";
@@ -18,6 +24,8 @@ export interface EngineModule {
FontManager?: { clear(): void };
/** Present in framefields versions built for long-lived workers. */
resetRenderCaches?: () => void;
+ /** The adapter the engine renders on (newer framefields versions). */
+ describeGpu?: () => Promise<GpuAdapter>;
}
export interface EngineRenderer {
@@ -155,6 +163,8 @@ export async function bindEngine(
},
);
}
+ // A GPU machine must not end up rendering on the CPU driver (gpu.ts).
+ verifyAdapter(await describeAdapter(mod).catch(() => undefined));
bound = {
...found,
workspace,
@@ -162,7 +172,11 @@ export async function bindEngine(
workerHooks: typeof mod.resetRenderCaches === "function",
};
rendererLogger.info(
- { version: found.version, workerHooks: bound.workerHooks },
+ {
+ version: found.version,
+ workerHooks: bound.workerHooks,
+ gpu: gpuStatus(),
+ },
"Bound framefields engine and initialised the GPU",
);
return bound;
diff --git a/apps/renderer-service/src/render/render-step.ts b/apps/renderer-service/src/render/render-step.ts
index 5507354a..acec12f6 100644
--- a/apps/renderer-service/src/render/render-step.ts
+++ b/apps/renderer-service/src/render/render-step.ts
@@ -8,6 +8,7 @@ import type {
StepContext,
} from "../jobs/pipeline.js";
import type { JobInput, RenderSettings } from "../jobs/schema.js";
+import { gpuStatus } from "../worker/gpu.js";
import { workerHealth } from "../worker/health.js";
import { type BoundEngine, bindEngine, markDepsLoaded } from "./engine.js";
import { clearWorkspaceRequireCache } from "./engine-alias.js";
@@ -112,6 +113,7 @@ export const render: Pipeline["render"] = async (checkout, input, ctx) => {
framefields: engine.version,
source: "project",
workerHooks: engine.workerHooks,
+ gpu: gpuStatus().adapter,
},
};
} catch (err) {
diff --git a/apps/renderer-service/src/worker/boot.ts b/apps/renderer-service/src/worker/boot.ts
index 88865f70..9f70abdd 100644
--- a/apps/renderer-service/src/worker/boot.ts
+++ b/apps/renderer-service/src/worker/boot.ts
@@ -22,6 +22,7 @@ import { boundEngine } from "../render/engine.js";
import { installEngineAlias } from "../render/engine-alias.js";
import { render } from "../render/render-step.js";
import { installProcessGuards, strayErrorCount } from "../render/sandbox.js";
+import { applyGpuPlan, gpuStatus, planGpu, probeGpu } from "./gpu.js";
import { workerHealth } from "./health.js";
import { CacheJanitor } from "./janitor.js";
import { currentStats, recycleReason } from "./recycle.js";
@@ -38,7 +39,7 @@ export interface Worker {
isDraining(): boolean;
/** Stops taking jobs and waits for the current one. */
drain(reason: RenderError, abortActive: boolean): Promise<void>;
- /** Checks git, git-lfs and ffprobe; rejects with TOOL_MISSING. */
+ /** Checks git, git-lfs, ffprobe and the GPU; rejects with TOOL_MISSING or GPU_UNAVAILABLE. */
checkTools(): Promise<void>;
stop(): void;
}
@@ -57,6 +58,16 @@ export function bootWorker(
"Renderer config loaded",
);
+ // Before any engine loads: Vulkan reads its driver list once (§8.7).
+ const gpu = planGpu(config.gpu, probeGpu());
+ applyGpuPlan(gpu);
+ rendererLogger[gpu.problem ? "error" : "info"](
+ { gpu: { ...gpuStatus(), icds: gpu.icds }, problem: gpu.problem },
+ gpu.expect === "software"
+ ? "No GPU in use: rendering on the CPU"
+ : "Rendering on the GPU",
+ );
+
// Workspaces live here; repo code under it gets the engine alias (§7).
const workspaces = path.join(config.paths.workDir, "ws");
mkdirSync(workspaces, { recursive: true });
@@ -176,6 +187,12 @@ export function bootWorker(
`Missing: ${required.join(", ")}`,
);
}
+ if (gpu.problem) {
+ readiness.setFailure(`GPU_UNAVAILABLE: ${gpu.problem}`);
+ throw new RenderError("GPU_UNAVAILABLE", gpu.problem, {
+ hint: gpu.hint,
+ });
+ }
readiness.setToolsReady(true);
baselineHandles = process.getActiveResourcesInfo().length;
},
diff --git a/apps/renderer-service/src/worker/gpu.test.ts b/apps/renderer-service/src/worker/gpu.test.ts
new file mode 100644
index 00000000..ace43ae7
--- /dev/null
+++ b/apps/renderer-service/src/worker/gpu.test.ts
@@ -0,0 +1,143 @@
+import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import path from "node:path";
+import { describe, expect, it } from "vitest";
+import {
+ applyGpuPlan,
+ classifyAdapter,
+ findVulkanIcds,
+ type GpuProbe,
+ gpuFailure,
+ planGpu,
+ verifyAdapter,
+} from "./gpu.js";
+
+const NVIDIA = { vendor: "nvidia" as const, model: "NVIDIA L4" };
+const ICDS = {
+ hardware: ["/etc/vulkan/icd.d/nvidia_icd.json"],
+ software: ["/usr/share/vulkan/icd.d/lvp_icd.x86_64.json"],
+};
+const probe = (over: Partial<GpuProbe> = {}): GpuProbe => ({
+ platform: "linux",
+ hardware: NVIDIA,
+ icds: ICDS,
+ ...over,
+});
+
+describe("planGpu", () => {
+ it("limits Vulkan to the GPU drivers on a GPU machine", () => {
+ const plan = planGpu("auto", probe(), {});
+ expect(plan).toMatchObject({ expect: "hardware", icds: ICDS.hardware });
+ expect(plan.problem).toBeUndefined();
+ });
+
+ it("renders on the CPU when the machine has no GPU", () => {
+ const plan = planGpu("auto", probe({ hardware: null }), {});
+ expect(plan).toMatchObject({ expect: "software" });
+ expect(plan.icds).toBeUndefined();
+ expect(plan.problem).toBeUndefined();
+ });
+
+ it("fails without a GPU when one is required", () => {
+ const plan = planGpu("require", probe({ hardware: null }), {});
+ expect(plan.problem).toMatch(/no GPU/);
+ });
+
+ it("fails when the GPU's Vulkan driver is missing", () => {
+ const plan = planGpu(
+ "auto",
+ probe({ icds: { hardware: [], software: ICDS.software } }),
+ {},
+ );
+ expect(plan.problem).toMatch(/NVIDIA L4 found, but no Vulkan driver/);
+ expect(plan.hint).toMatch(/NVIDIA_DRIVER_CAPABILITIES/);
+ });
+
+ it("uses only the CPU driver in cpu mode", () => {
+ const plan = planGpu("cpu", probe(), {});
+ expect(plan).toMatchObject({ expect: "software", icds: ICDS.software });
+ });
+
+ it("respects Vulkan driver overrides set by the operator", () => {
+ const plan = planGpu(
+ "auto",
+ probe({ icds: { hardware: [], software: [] } }),
+ {
+ VK_DRIVER_FILES: "/custom.json",
+ },
+ );
+ expect(plan.problem).toBeUndefined();
+ expect(plan.icds).toBeUndefined();
+ });
+
+ it("needs no driver list on macOS", () => {
+ const plan = planGpu(
+ "require",
+ {
+ platform: "darwin",
+ hardware: { vendor: "apple", model: "Metal" },
+ icds: { hardware: [], software: [] },
+ },
+ {},
+ );
+ expect(plan).toMatchObject({ expect: "hardware" });
+ expect(plan.icds).toBeUndefined();
+ });
+});
+
+describe("findVulkanIcds", () => {
+ it("splits GPU and CPU drivers", () => {
+ const dir = mkdtempSync(path.join(tmpdir(), "icd-"));
+ mkdirSync(dir, { recursive: true });
+ const icd = (name: string, lib: string) =>
+ writeFileSync(
+ path.join(dir, name),
+ JSON.stringify({ ICD: { library_path: lib } }),
+ );
+ icd("nvidia_icd.json", "libGLX_nvidia.so.0");
+ icd("lvp_icd.x86_64.json", "/usr/lib/libvulkan_lvp.so");
+ icd("radeon_icd.x86_64.json", "/usr/lib/libvulkan_radeon.so");
+ writeFileSync(path.join(dir, "README"), "");
+ const found = findVulkanIcds([dir]);
+ expect(found.hardware.map((f) => path.basename(f))).toEqual([
+ "nvidia_icd.json",
+ "radeon_icd.x86_64.json",
+ ]);
+ expect(found.software.map((f) => path.basename(f))).toEqual([
+ "lvp_icd.x86_64.json",
+ ]);
+ });
+});
+
+describe("verifyAdapter", () => {
+ const llvmpipe = classifyAdapter({
+ vendor: "mesa",
+ architecture: "",
+ device: "llvmpipe (LLVM 19.1.7, 256 bits)",
+ description: "",
+ });
+
+ it("recognises CPU adapters", () => {
+ expect(llvmpipe.software).toBe(true);
+ expect(
+ classifyAdapter({ vendor: "nvidia", description: "NVIDIA L4" }).software,
+ ).toBe(false);
+ expect(classifyAdapter({ isFallbackAdapter: true }).software).toBe(true);
+ });
+
+ it("fails a GPU machine that ended up on the CPU driver", () => {
+ applyGpuPlan({ ...planGpu("auto", probe(), {}), icds: undefined });
+ expect(() => verifyAdapter(llvmpipe)).toThrow(
+ expect.objectContaining({ code: "GPU_UNAVAILABLE" }),
+ );
+ expect(gpuFailure()).toMatch(
+ /NVIDIA L4 is present but WebGPU picked the CPU/,
+ );
+ });
+
+ it("accepts the CPU driver on a machine without a GPU", () => {
+ applyGpuPlan(planGpu("auto", probe({ hardware: null }), {}));
+ expect(() => verifyAdapter(llvmpipe)).not.toThrow();
+ expect(gpuFailure()).toBeUndefined();
+ });
+});
diff --git a/apps/renderer-service/src/worker/gpu.ts b/apps/renderer-service/src/worker/gpu.ts
new file mode 100644
index 00000000..6b47ff30
--- /dev/null
+++ b/apps/renderer-service/src/worker/gpu.ts
@@ -0,0 +1,314 @@
+import { existsSync, readdirSync, readFileSync } from "node:fs";
+import path from "node:path";
+import { RenderError } from "../errors.js";
+
+/**
+ * GPU selection (PLAN.md §8.7). framefields renders through WebGPU (Dawn),
+ * which on Linux runs on Vulkan. The image also ships Mesa's CPU Vulkan
+ * driver (lavapipe) so a machine without a GPU can still render. Without
+ * care, a GPU machine whose Vulkan driver isn't mounted would silently render
+ * on the CPU too, so:
+ *
+ * - auto (default): a machine with a GPU must render on it (Vulkan is limited
+ * to the GPU drivers); a machine without one renders on the CPU.
+ * - require: always the GPU; a machine without one fails.
+ * - cpu: always the CPU driver.
+ */
+type GpuMode = "auto" | "require" | "cpu";
+
+export interface GpuHardware {
+ vendor: "nvidia" | "amd" | "intel" | "apple";
+ /** Model name when the driver reports one. */
+ model: string;
+}
+
+interface GpuPlan {
+ mode: GpuMode;
+ hardware: GpuHardware | null;
+ /** What the rendering adapter must be. */
+ expect: "hardware" | "software";
+ /** Vulkan drivers (ICD manifests) to limit the loader to, if any. */
+ icds?: string[];
+ /** Why this machine can't render as configured. */
+ problem?: string;
+ hint?: string;
+}
+
+/** The adapter framefields renders on (framefields' describeGpu shape). */
+export interface GpuAdapter {
+ vendor: string;
+ architecture: string;
+ device: string;
+ description: string;
+ software: boolean;
+}
+
+export interface GpuProbe {
+ platform: NodeJS.Platform;
+ hardware: GpuHardware | null;
+ icds: { hardware: string[]; software: string[] };
+}
+
+const ICD_DIRS = [
+ "/etc/vulkan/icd.d",
+ "/usr/share/vulkan/icd.d",
+ "/usr/local/share/vulkan/icd.d",
+];
+const SOFTWARE_ICD = /lvp|lavapipe|swiftshader/i;
+const SOFTWARE_ADAPTER =
+ /llvmpipe|lavapipe|softpipe|swiftshader|\bwarp\b|software/i;
+const PCI_VENDORS: Record<string, GpuHardware["vendor"]> = {
+ "0x10de": "nvidia",
+ "0x1002": "amd",
+ "0x8086": "intel",
+};
+
+function safeList(dir: string): string[] {
+ try {
+ return readdirSync(dir);
+ } catch {
+ return [];
+ }
+}
+
+function nvidiaModel(): string {
+ for (const gpu of safeList("/proc/driver/nvidia/gpus")) {
+ try {
+ const info = readFileSync(
+ path.join("/proc/driver/nvidia/gpus", gpu, "information"),
+ "utf8",
+ );
+ const model = /^Model:\s*(.+)$/m.exec(info)?.[1]?.trim();
+ if (model) return model;
+ } catch {}
+ }
+ return "NVIDIA GPU";
+}
+
+/** Finds a GPU this process can use (device nodes present, not just listed in /sys). */
+function detectGpuHardware(
+ platform: NodeJS.Platform = process.platform,
+): GpuHardware | null {
+ if (platform === "darwin") return { vendor: "apple", model: "Metal" };
+ if (platform !== "linux") return null;
+ if (existsSync("/dev/nvidiactl") || existsSync("/dev/nvidia0")) {
+ return { vendor: "nvidia", model: nvidiaModel() };
+ }
+ // AMD/Intel: needs a render node; /sys lists host GPUs even when the
+ // container wasn't given one.
+ if (!safeList("/dev/dri").some((n) => n.startsWith("renderD"))) return null;
+ for (const card of safeList("/sys/class/drm")) {
+ if (!/^card\d+$/.test(card)) continue;
+ try {
+ const id = readFileSync(
+ path.join("/sys/class/drm", card, "device/vendor"),
+ "utf8",
+ ).trim();
+ const vendor = PCI_VENDORS[id];
+ if (vendor) return { vendor, model: `${vendor.toUpperCase()} GPU` };
+ } catch {}
+ }
+ return null;
+}
+
+/** Vulkan driver manifests, split into GPU and CPU drivers. */
+export function findVulkanIcds(dirs = ICD_DIRS): GpuProbe["icds"] {
+ const hardware: string[] = [];
+ const software: string[] = [];
+ for (const dir of dirs) {
+ for (const name of safeList(dir).sort()) {
+ if (!name.endsWith(".json")) continue;
+ const file = path.join(dir, name);
+ let library = "";
+ try {
+ library = String(
+ JSON.parse(readFileSync(file, "utf8"))?.ICD?.library_path ?? "",
+ );
+ } catch {}
+ (SOFTWARE_ICD.test(name) || SOFTWARE_ICD.test(library)
+ ? software
+ : hardware
+ ).push(file);
+ }
+ }
+ return { hardware, software };
+}
+
+export function probeGpu(): GpuProbe {
+ return {
+ platform: process.platform,
+ hardware: detectGpuHardware(),
+ icds:
+ process.platform === "linux"
+ ? findVulkanIcds()
+ : { hardware: [], software: [] },
+ };
+}
+
+/**
+ * Decides where to render. `env` is checked for Vulkan driver overrides an
+ * operator already set; those are respected as they are.
+ */
+export function planGpu(
+ mode: GpuMode,
+ probe: GpuProbe,
+ env: NodeJS.ProcessEnv = process.env,
+): GpuPlan {
+ const { hardware, icds } = probe;
+ const linux = probe.platform === "linux";
+ const overridden = Boolean(env.VK_DRIVER_FILES || env.VK_ICD_FILENAMES);
+ const limit = (files: string[]) =>
+ linux && !overridden && files.length ? files : undefined;
+
+ if (mode === "cpu") {
+ if (linux && !overridden && icds.software.length === 0) {
+ return {
+ mode,
+ hardware,
+ expect: "software",
+ problem: "FF_GPU=cpu but no CPU Vulkan driver is installed.",
+ hint: "Install mesa-vulkan-drivers (lavapipe).",
+ };
+ }
+ return { mode, hardware, expect: "software", icds: limit(icds.software) };
+ }
+
+ if (hardware) {
+ if (linux && !overridden && icds.hardware.length === 0) {
+ return {
+ mode,
+ hardware,
+ expect: "hardware",
+ problem: `${hardware.model} found, but no Vulkan driver for it.`,
+ hint:
+ hardware.vendor === "nvidia"
+ ? "Run with the NVIDIA container runtime and NVIDIA_DRIVER_CAPABILITIES including graphics (the image sets graphics,utility,video,compute)."
+ : "Install the GPU's Vulkan driver (mesa-vulkan-drivers covers AMD and Intel).",
+ };
+ }
+ return {
+ mode,
+ hardware,
+ expect: "hardware",
+ icds: limit(icds.hardware),
+ };
+ }
+
+ if (mode === "require") {
+ return {
+ mode,
+ hardware,
+ expect: "hardware",
+ problem: "FF_GPU=require but this machine has no GPU.",
+ hint: "Run on a GPU machine, or set FF_GPU=auto to allow CPU rendering.",
+ };
+ }
+ return { mode, hardware, expect: "software" };
+}
+
+let active: GpuPlan | undefined;
+let adapter: GpuAdapter | undefined;
+let adapterProblem: string | undefined;
+
+/** Limits Vulkan to the planned drivers. Must run before the engine loads. */
+export function applyGpuPlan(plan: GpuPlan): void {
+ active = plan;
+ adapter = undefined;
+ adapterProblem = undefined;
+ if (plan.icds) {
+ const files = plan.icds.join(path.delimiter);
+ process.env.VK_DRIVER_FILES = files;
+ process.env.VK_ICD_FILENAMES = files; // Vulkan loaders before 1.3.207
+ }
+}
+
+export function classifyAdapter(info: {
+ vendor?: unknown;
+ architecture?: unknown;
+ device?: unknown;
+ description?: unknown;
+ isFallbackAdapter?: unknown;
+}): GpuAdapter {
+ const out = {
+ vendor: String(info.vendor ?? ""),
+ architecture: String(info.architecture ?? ""),
+ device: String(info.device ?? ""),
+ description: String(info.description ?? ""),
+ };
+ return {
+ ...out,
+ software:
+ info.isFallbackAdapter === true ||
+ SOFTWARE_ADAPTER.test(
+ `${out.vendor} ${out.architecture} ${out.device} ${out.description}`,
+ ),
+ };
+}
+
+/**
+ * The adapter the engine renders on: framefields' describeGpu() where the
+ * project's version has it, else the adapter WebGPU picks with the same
+ * options framefields uses.
+ */
+export async function describeAdapter(mod: {
+ describeGpu?: () => Promise<GpuAdapter>;
+}): Promise<GpuAdapter | undefined> {
+ if (typeof mod.describeGpu === "function") return mod.describeGpu();
+ const gpu = (
+ globalThis as {
+ navigator?: { gpu?: { requestAdapter(o?: object): Promise<unknown> } };
+ }
+ ).navigator?.gpu;
+ if (!gpu) return undefined;
+ const found = (await gpu.requestAdapter({
+ powerPreference: "high-performance",
+ })) as { info?: Record<string, unknown>; isFallbackAdapter?: unknown } | null;
+ if (!found) return undefined;
+ return classifyAdapter({
+ isFallbackAdapter: found.isFallbackAdapter,
+ ...found.info,
+ });
+}
+
+function adapterName(a: GpuAdapter): string {
+ return a.description || a.device || a.architecture || a.vendor || "unknown";
+}
+
+/**
+ * Checks the engine's adapter against the plan. A GPU machine that ended up
+ * on the CPU driver fails with GPU_UNAVAILABLE and the worker reports itself
+ * unhealthy, so a scheduler (RunPod) replaces it.
+ */
+export function verifyAdapter(found: GpuAdapter | undefined): void {
+ adapter = found;
+ const plan = active;
+ if (!plan || !found) return;
+ if (plan.expect === "hardware" && found.software) {
+ adapterProblem = `${plan.hardware?.model ?? "A GPU"} is present but WebGPU picked the CPU driver (${adapterName(found)}).`;
+ throw new RenderError("GPU_UNAVAILABLE", adapterProblem, {
+ phase: "render",
+ hint: "Check the GPU's Vulkan driver (vulkaninfo --summary). FF_GPU=cpu renders on the CPU on purpose.",
+ });
+ }
+}
+
+/** Boot or adapter problem that makes this worker unable to render. */
+export function gpuFailure(): string | undefined {
+ return active?.problem ?? adapterProblem;
+}
+
+/** GPU state for /health. */
+export function gpuStatus() {
+ return {
+ mode: active?.mode ?? null,
+ hardware: active?.hardware?.model ?? null,
+ renderer: gpuFailure()
+ ? null
+ : (adapter ? adapter.software : active?.expect === "software")
+ ? "cpu"
+ : "gpu",
+ adapter: adapter
+ ? { name: adapterName(adapter), software: adapter.software }
+ : null,
+ };
+}