Skip to content

Code-first video & image, rendered natively on WebGPU.

fix(renderer-service): RunPod result URLs without a query string; cap the network-volume cache

okanasl75be57e725c2
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.md
index e624733c..c79b0831 100644
--- a/apps/renderer-service/PLAN.md
+++ b/apps/renderer-service/PLAN.md
@@ -623,13 +623,19 @@ Each phase is its own PR-sized step with tests.
- **vision:** 60 frames in 388 s. The ONNX models run on the CPU, and this includes their first download.
- **Encoding:** NVENC isn't usable on driver 580: node-av 6.1.1's FFmpeg needs NVENC API 13.1, i.e. driver ≥ 610. The encoder selection fell back to libx264 as designed.
- **tini:** RunPod doesn't start the container with tini as PID 1, so it now runs with `-s` (subreaper).
+ - **RunPod Serverless (RTX A4500, EU-RO-1, `AMPERE_16`, driver 595.91, 10 GB network volume, R2 output), `:2.2.0-beta.1`:**
+ - **all:** checkout 13.8 s, install 17.0 s, render 195.6 s (film, threeD, vision), upload to R2 3.3 s. Vision is twice as fast as on the RTX 3070 Pod, on a host with more CPU.
+ - **threeD on a second worker:** 34 s total, render 6.5 s. The network-volume cache barely sped up the checkout (12.4 s vs 13.8 s).
+ - **Results never reached RunPod:** every POST to the SDK's `job-done` URL returned 500, so RunPod re-queued finished jobs. The URL in the log was `…/job-done/<worker>/<job>&isStream=false`: the platform's URL had no query string, and the SDK appends `&isStream=…`. The shim now adds the missing `?` before importing the SDK and logs the URL's shape.
+ - **The network volume reports 2.3 PB,** so free-space eviction is off for it; the serverless template sets `FF_CACHE_MAX_GB`.
+ - RunPod started three workers on an endpoint with `max: 1`.
- **LFS end to end:**
- `src/git/lfs.test.ts` runs real git-lfs (3.8) against file:// origins. It covers a cold download into the shared store, a warm hit, objects shared across repos, include patterns with the `LFS_POINTERS_LEFT` warning, LFS off, and a missing object (`LFS_OBJECT_MISSING`).
- In the arm64 container (git-lfs 3.6.1), a repo whose framefields tarball and logo PNG are both LFS-tracked installed from the LFS tarball and rendered the logo (centre pixel red). The job reported `cache.lfs: "miss"`.
- Two file://-only bugs were found and fixed. With `-c lfs.storage`, git-lfs's local transfer agent also looked for the remote's objects in our store, so the setting now goes in the repo config. The agent drops `GIT_CONFIG_GLOBAL` and `-c` but reads `$HOME/.gitconfig`, so the session's isolated config now lives there, with `safe.directory` for local repos (otherwise "dubious ownership" on a mounted repo).
- **Engine issue found and fixed in framefields (§8.7):** on linux-arm64, H.264 renders failed with "Encoder 'h264_rkmpp' cannot be used for quantizer-based encoding". The cause was @mediabunny/server taking FFmpeg's first registered H.264 encoder (node-av's build lists `h264_rkmpp` before libx264), not `@napi-rs/webcodecs`. With the fixed framefields (packed locally), an H.264 job in the arm64 container rendered 48/48 frames with libx264 on lavapipe. `FF_GPU=require` on the same container fails readiness with `GPU_UNAVAILABLE`.
- Pinned SSH host keys, the RunPod templates and `create-template.sh`, `docker-compose.yml` with MinIO, `env.example`, the release workflow, and the README guide as listed above.
- - **Still to do:** an amd64 build (CI), a GPU smoke test on a RunPod pod and a serverless endpoint, the nightly host smoke tests (6.5), and a page on the docs website.
+ - **Still to do:** an amd64 build (CI), a serverless run that returns its result (after the shim fix), the nightly host smoke tests (6.5), and a page on the docs website.
9. **(Later)**
- Chunked distributed rendering: split a frame range across N jobs and concat with a stream copy.
- Webhooks on completion (`output.webhook` with an HMAC signature).
diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.md
index 53b9a570..fac4856d 100644
--- a/apps/renderer-service/README.md
+++ b/apps/renderer-service/README.md
@@ -123,6 +123,7 @@ Checklist for serverless endpoints:
- **Set the endpoint's execution timeout** above your longest render. RunPod's default is much shorter than a long video.
- **Attach a network volume** in the same data centre as the GPUs. Without one, every worker starts cold: a full clone, LFS download and install.
+- **Set `FF_CACHE_MAX_GB`** to about 80% of the volume's size. A network volume reports the size of the whole storage cluster (petabytes), so the worker can't evict by free space there. The template sets 40, for a 50 GB volume.
- **FlashBoot** keeps warm workers around between jobs, which is what makes the caches pay off.
- **Choose GPUs with a hardware video encoder (NVENC):** L4, L40S, A40, RTX A4000/A5000/A6000, RTX 4090/5090. A100 and H100 are built for AI training and have no NVENC: they render, but encode on the CPU, and cost more. VRAM needs grow with resolution and effects.
- The templates set `FF_GPU=require`, so a RunPod worker never renders on the CPU. A worker whose GPU is unusable reports `GPU_UNAVAILABLE` and asks RunPod for a fresh worker.
@@ -210,7 +211,7 @@ Shared files are read-only, because changing one would change it for every proje
Some filesystems report sizes that can't be true, such as Docker Desktop host mounts (around 118 TB) or a network volume reporting a whole storage cluster. Above 64 TiB a disk is marked `measurable: false` and the worker doesn't evict by free space there. Set `FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` if such a disk needs a limit.
-`FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` add hard caps if you want them. They're only needed where the container reports the host's disk size instead of its own allowance. Some Docker storage drivers do that, and the RunPod smoke test will show whether RunPod does.
+`FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` add hard caps if you want them. On RunPod the container disk reports its real size (no cap needed), but a network volume reports 2.3 PB, so set `FF_CACHE_MAX_GB` for it.
## Jobs
diff --git a/apps/renderer-service/deploy/runpod/serverless.template.json b/apps/renderer-service/deploy/runpod/serverless.template.json
index 87c33441..5732faf0 100644
--- a/apps/renderer-service/deploy/runpod/serverless.template.json
+++ b/apps/renderer-service/deploy/runpod/serverless.template.json
@@ -26,6 +26,7 @@
"value": "{{ RUNPOD_SECRET_ff_storage_secret }}"
},
{ "key": "STORAGE_PUBLIC_BASE_URL", "value": "" },
+ { "key": "FF_CACHE_MAX_GB", "value": "40" },
{ "key": "FF_JOB_TIMEOUT_MAX_SEC", "value": "3600" },
{ "key": "LOG_LEVEL", "value": "info" }
]
diff --git a/apps/renderer-service/runpod_worker.py b/apps/renderer-service/runpod_worker.py
index 28c3ddcb..64c3723b 100644
--- a/apps/renderer-service/runpod_worker.py
+++ b/apps/renderer-service/runpod_worker.py
@@ -7,11 +7,33 @@ relays progress, and returns the result or a structured error.
import json
import os
+import re
+import sys
import time
import urllib.error
import urllib.request
-import runpod
+
+def fix_result_urls():
+ """The SDK appends `&isStream=...` to RunPod's result URLs, assuming they
+ already carry a query string. Some endpoints get URLs without one, and
+ every result POST then fails with 500 and the job is re-run. Must run
+ before `import runpod`, which reads these variables at import time."""
+ for key in ("RUNPOD_WEBHOOK_POST_OUTPUT", "RUNPOD_WEBHOOK_POST_STREAM"):
+ url = os.environ.get(key)
+ if not url:
+ continue
+ # Path and parameter names only: no parameter values in the log.
+ shape = re.sub(r"=[^&]*", "=…", url.split("://", 1)[-1].split("/", 1)[-1])
+ fixed = "?" not in url
+ if fixed:
+ os.environ[key] = url + "?"
+ print(f"[runpod-shim] {key}: {shape}{' (added ?)' if fixed else ''}", file=sys.stderr, flush=True)
+
+
+fix_result_urls()
+
+import runpod # noqa: E402
BASE = f"http://127.0.0.1:{os.environ.get('RENDERER_PORT', '8082')}"
API_KEY = os.environ.get("RENDERER_API_KEY", "")