Code-first video & image, rendered natively on WebGPU.
fix(renderer-service): LFS with local repos; disk space in /health
okanaslaacc8b10b376
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.mdindex ec2af525..09beb892 100644--- a/apps/renderer-service/PLAN.md+++ b/apps/renderer-service/PLAN.md@@ -614,6 +614,10 @@ Each phase is its own PR-sized step with tests.8. **Image + template + compose + docs.** Built and run locally (arm64, Docker on macOS, without a GPU):- **Inside the container:** a file:// repo pinned to the published `[email protected]` went through checkout, then `npm ci` from the registry (40 s cold), then engine binding with GPU init on Mesa's software Vulkan, then a VP9 render, verification and delivery. The output was a valid 2 s VP9/Opus file.- **Findings fixed on the way:** glibc 2.38 (moved to trixie); EPIPE from the `exec` helper writing stdin that git never reads; corepack's npm differing from the lockfile's npm; install errors keeping only npm's usage text.+ - **LFS end to end:**+ - `src/git/lfs.test.ts` runs real git-lfs (3.8) against file:// origins. It covers a cold download into the shared store, a warm hit, objects shared across repos, include patterns with the `LFS_POINTERS_LEFT` warning, LFS off, and a missing object (`LFS_OBJECT_MISSING`).+ - In the arm64 container (git-lfs 3.6.1), a repo whose framefields tarball and logo PNG are both LFS-tracked installed from the LFS tarball and rendered the logo (centre pixel red). The job reported `cache.lfs: "miss"`.+ - Two file://-only bugs were found and fixed. With `-c lfs.storage`, git-lfs's local transfer agent also looked for the remote's objects in our store, so the setting now goes in the repo config. The agent drops `GIT_CONFIG_GLOBAL` and `-c` but reads `$HOME/.gitconfig`, so the session's isolated config now lives there, with `safe.directory` for local repos (otherwise "dubious ownership" on a mounted repo).- **Engine issue found and fixed in framefields (§8.7):** on linux-arm64, H.264 renders failed with "Encoder 'h264_rkmpp' cannot be used for quantizer-based encoding". The cause was @mediabunny/server taking FFmpeg's first registered H.264 encoder (node-av's build lists `h264_rkmpp` before libx264), not `@napi-rs/webcodecs`. With the fixed framefields (packed locally), an H.264 job in the arm64 container rendered 48/48 frames with libx264 on lavapipe. `FF_GPU=require` on the same container fails readiness with `GPU_UNAVAILABLE`.- Pinned SSH host keys, the RunPod templates and `create-template.sh`, `docker-compose.yml` with MinIO, `env.example`, the release workflow, and the README guide as listed above.- **Still to do:** an amd64 build (CI), a GPU smoke test on a RunPod pod and a serverless endpoint, the nightly host smoke tests (6.5), and a page on the docs website.diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.mdindex 8d713c8d..05a4d161 100644--- a/apps/renderer-service/README.md+++ b/apps/renderer-service/README.md@@ -179,6 +179,17 @@ Shared files are read-only, because changing one would change it for every proje- **Never evicted:** the current repo, the workspace the loaded engine came from, vision models, and other workers' recently used mirrors.- **Logging:** every eviction is logged with the disks' free space.+`/health` reports both disks under `disks`. The figures are refreshed at most every 10 seconds:++```json+"disks": {+ "work": {"path": "/tmp/framefields", "freeGb": 31.2, "totalGb": 40, "evictBelowGb": 10, "lowOnSpace": false},+ "cache": {"path": "/runpod-volume/framefields", "freeGb": 88.4, "totalGb": 100, "evictBelowGb": 15, "lowOnSpace": false}+}+```++`sameDiskAs: "work"` marks a cache dir on the workspace disk. `lowOnSpace` means the next job starts by evicting.+`FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` add hard caps if you want them. They're only needed where the container reports the host's disk size instead of its own allowance. Some Docker storage drivers do that, and the RunPod smoke test will show whether RunPod does.## Jobsdiff --git a/apps/renderer-service/src/git/auth.ts b/apps/renderer-service/src/git/auth.tsindex 176b6ac4..de8bf073 100644--- a/apps/renderer-service/src/git/auth.ts+++ b/apps/renderer-service/src/git/auth.ts@@ -110,15 +110,23 @@ export async function openGitSession(LC_ALL: "C",TZ: "UTC",// Isolate from the host's git config (credential stores, insteadOf, hooks).+ // The file is $HOME/.gitconfig too: git-lfs's file:// agent drops+ // GIT_CONFIG_GLOBAL (and -c) but still reads the home config.GIT_CONFIG_NOSYSTEM: "1",- GIT_CONFIG_GLOBAL: path.join(dir, "gitconfig"),+ GIT_CONFIG_GLOBAL: path.join(dir, ".gitconfig"),GIT_TERMINAL_PROMPT: "0",GIT_ASKPASS: "/bin/false",SSH_ASKPASS: "/bin/false",GIT_LFS_SKIP_SMUDGE: "1",GCM_INTERACTIVE: "never",};- await fs.writeFile(env.GIT_CONFIG_GLOBAL as string, "", { mode: 0o600 });+ // A local repo (FF_ALLOW_LOCAL_REPOS) usually belongs to another user;+ // git, and git-lfs's agent running git inside it, must treat it as safe.+ const safe =+ repo.protocol === "file"+ ? `[safe]\n\tdirectory = ${new URL(repo.url).pathname}\n`+ : "";+ await fs.writeFile(env.GIT_CONFIG_GLOBAL as string, safe, { mode: 0o600 });for (const [host, cred] of credentialMap(repo, config, job)) {env[`FF_CRED_${credEnvKey(host)}_USER`] = cred.username;diff --git a/apps/renderer-service/src/git/lfs.test.ts b/apps/renderer-service/src/git/lfs.test.tsnew file mode 100644index 00000000..eaeb244b--- /dev/null+++ b/apps/renderer-service/src/git/lfs.test.ts@@ -0,0 +1,198 @@+import { execFileSync } from "node:child_process";+import { randomBytes } from "node:crypto";+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { afterAll, beforeAll, describe, expect, it } from "vitest";+import { RenderError } from "../errors.js";+import type { StepContext } from "../jobs/pipeline.js";+import { validateJob } from "../jobs/validate.js";+import { testConfig, testJob } from "../test-helpers.js";+import { checkout } from "./checkout.js";++/**+ * End to end with real git-lfs: origin repos with LFS-tracked assets, served+ * over file:// (git-lfs's standalone transfer), checked out the way a job is.+ */+const hasLfs = (() => {+ try {+ execFileSync("git", ["lfs", "version"], { stdio: "ignore" });+ return true;+ } catch {+ return false;+ }+})();++let tmp: string;+const assets: Record<string, Buffer> = {+ "video/assets/hero.mp4": randomBytes(300 * 1024),+ "video/assets/music.bin": randomBytes(120 * 1024),+ "other/huge.bin": randomBytes(200 * 1024),+};++const gitEnv = {+ ...process.env,+ GIT_CONFIG_NOSYSTEM: "1",+ GIT_CONFIG_GLOBAL: "/dev/null",+ GIT_AUTHOR_NAME: "t",+ GIT_AUTHOR_EMAIL: "[email protected]",+ GIT_COMMITTER_NAME: "t",+ GIT_COMMITTER_EMAIL: "[email protected]",+};+const sh = (cwd: string, args: string[]) =>+ execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8" }).trim();++/** A repo whose assets are committed as LFS pointers, objects in its .git/lfs. */+async function lfsOrigin(name: string): Promise<string> {+ const dir = path.join(tmp, name);+ await fs.mkdir(dir, { recursive: true });+ sh(dir, ["init", "--quiet", "--initial-branch=main"]);+ sh(dir, ["lfs", "install", "--local"]);+ await fs.writeFile(+ path.join(dir, ".gitattributes"),+ "*.mp4 filter=lfs diff=lfs merge=lfs -text\n*.bin filter=lfs diff=lfs merge=lfs -text\n",+ );+ await fs.mkdir(path.join(dir, "video/src"), { recursive: true });+ await fs.writeFile(path.join(dir, "video/package.json"), '{"name":"v"}');+ await fs.writeFile(path.join(dir, "video/src/film.ts"), "export {}\n");+ for (const [rel, content] of Object.entries(assets)) {+ await fs.mkdir(path.dirname(path.join(dir, rel)), { recursive: true });+ await fs.writeFile(path.join(dir, rel), content);+ }+ sh(dir, ["add", "-A"]);+ sh(dir, ["commit", "--quiet", "-m", "assets"]);+ // Committed as pointers, not the bytes.+ const blob = sh(dir, ["show", "HEAD:video/assets/hero.mp4"]);+ if (!blob.startsWith("version https://git-lfs")) {+ throw new Error("fixture: LFS clean filter did not run");+ }+ return dir;+}++function setup(repo: string, lfs: Record<string, unknown> = {}) {+ const config = testConfig({+ FF_ALLOW_LOCAL_REPOS: "true",+ FF_CACHE_DIR: path.join(tmp, "cache"),+ FF_WORK_DIR: path.join(tmp, "work"),+ });+ const job = validateJob(+ testJob({+ source: { repo: `file://${repo}`, ref: "main", workdir: "video", lfs },+ }),+ config,+ );+ const warnings: string[] = [];+ const ctx: StepContext = {+ jobId: "lfs-test",+ config,+ signal: new AbortController().signal,+ jobDir: path.join(tmp, "job"),+ progress: () => {},+ warn: (code) => warnings.push(code),+ };+ return { job, ctx, warnings };+}++const isPointer = async (file: string) =>+ (await fs.readFile(file, "utf8").catch(() => "")).startsWith(+ "version https://git-lfs.github.com/spec/v1",+ );++describe.skipIf(!hasLfs)("LFS (real git-lfs, file:// remotes)", () => {+ let origin: string;++ beforeAll(async () => {+ tmp = await fs.realpath(+ await fs.mkdtemp(path.join(os.tmpdir(), "ff-lfs-test-")),+ );+ origin = await lfsOrigin("origin");+ });+ afterAll(async () => {+ await fs.rm(tmp, { recursive: true, force: true });+ });++ it("downloads the assets into the checkout (cold)", async () => {+ const { job, ctx, warnings } = setup(origin);+ const result = await checkout(job, ctx);+ expect(result.cache.lfs).toBe("miss");+ for (const [rel, content] of Object.entries(assets)) {+ expect(await fs.readFile(path.join(result.root, rel))).toEqual(content);+ }+ expect(warnings).toEqual([]);+ // Objects land in the shared store on the cache volume.+ const objects = await fs.readdir(path.join(tmp, "cache/lfs/objects"), {+ recursive: true,+ });+ expect(objects.filter((o) => o.length === 64 + 6)).toHaveLength(3);+ });++ it("serves a warm checkout from the object store", async () => {+ const { job, ctx } = setup(origin);+ const result = await checkout(job, ctx);+ expect(result.cache).toMatchObject({ workspace: "warm", lfs: "hit" });+ expect(+ await fs.readFile(path.join(result.root, "video/assets/hero.mp4")),+ ).toEqual(assets["video/assets/hero.mp4"]);+ });++ it("shares objects across repos (content-addressed)", async () => {+ // Another repo with the same assets: nothing to download.+ const fork = path.join(tmp, "fork");+ await fs.cp(origin, fork, { recursive: true });+ const { job, ctx } = setup(fork);+ const result = await checkout(job, ctx);+ expect(result.cache).toMatchObject({ workspace: "cold", lfs: "hit" });+ expect(+ await fs.readFile(path.join(result.root, "video/assets/music.bin")),+ ).toEqual(assets["video/assets/music.bin"]);+ });++ it("downloads only the include patterns, warning about the rest of the workdir", async () => {+ // A cold workspace: a warm one keeps assets an earlier job downloaded.+ const fresh = path.join(tmp, "include-only");+ await fs.cp(origin, fresh, { recursive: true });+ await fs.rm(path.join(tmp, "cache/lfs"), { recursive: true, force: true });+ const { job, ctx, warnings } = setup(fresh, {+ include: ["video/assets/hero.mp4"],+ });+ const result = await checkout(job, ctx);+ expect(result.cache.lfs).toBe("miss");+ expect(+ await isPointer(path.join(result.root, "video/assets/hero.mp4")),+ ).toBe(false);+ expect(+ await isPointer(path.join(result.root, "video/assets/music.bin")),+ ).toBe(true);+ expect(warnings).toContain("LFS_POINTERS_LEFT");+ });++ it("can skip LFS entirely", async () => {+ const { job, ctx } = setup(origin, { enabled: false });+ const result = await checkout(job, ctx);+ expect(result.cache.lfs).toBe("off");+ });++ it("fails with LFS_OBJECT_MISSING when the server lacks an object", async () => {+ const broken = await lfsOrigin("broken");+ // Lose the object behind music.bin, as if it was never pushed.+ const pointer = sh(broken, ["show", "HEAD:video/assets/music.bin"]);+ const oid = /oid sha256:([0-9a-f]{64})/.exec(pointer)?.[1] ?? "";+ await fs.rm(+ path.join(+ broken,+ ".git/lfs/objects",+ oid.slice(0, 2),+ oid.slice(2, 4),+ oid,+ ),+ );+ await fs.rm(path.join(tmp, "cache/lfs"), { recursive: true, force: true });+ const { job, ctx } = setup(broken);+ const err = await checkout(job, ctx).then(+ () => undefined,+ (e: unknown) => e,+ );+ expect(err).toBeInstanceOf(RenderError);+ expect((err as RenderError).code).toBe("LFS_OBJECT_MISSING");+ });+});diff --git a/apps/renderer-service/src/git/lfs.ts b/apps/renderer-service/src/git/lfs.tsindex e1b4bf86..349770de 100644--- a/apps/renderer-service/src/git/lfs.ts+++ b/apps/renderer-service/src/git/lfs.ts@@ -65,12 +65,14 @@ export async function pullLfs(const storage = path.join(ctx.config.paths.cacheDir, "lfs");await fs.mkdir(storage, { recursive: true });+ // The shared object store goes in the repo's own config, not `-c`: with+ // `-c`, git-lfs's file:// transfer agent (which reads the remote repo)+ // also used it, and looked for the remote's objects in our store.+ await git(ctx, ["-C", root, "config", "lfs.storage", storage]);const lfsArgs = ["-C",root,"-c",- `lfs.storage=${storage}`,- "-c",`lfs.concurrenttransfers=${Math.max(1, ctx.config.source.lfsConcurrency)}`,"-c","lfs.fetchrecentalways=false",diff --git a/apps/renderer-service/src/index.ts b/apps/renderer-service/src/index.tsindex bd842d52..9f5a535d 100644--- a/apps/renderer-service/src/index.ts+++ b/apps/renderer-service/src/index.ts@@ -41,6 +41,7 @@ const app = createApp(config, runner, {active: runner.queue.activeId ?? null,waiting: runner.queue.depth,},+ disks: worker.disks() ?? null,...(status.error && { error: status.error }),...(workerHealth.recycleReason && {recycle: workerHealth.recycleReason,diff --git a/apps/renderer-service/src/worker/boot.ts b/apps/renderer-service/src/worker/boot.tsindex 98e23cc3..cb7084de 100644--- a/apps/renderer-service/src/worker/boot.ts+++ b/apps/renderer-service/src/worker/boot.ts@@ -24,7 +24,7 @@ import { render } from "../render/render-step.js";import { installProcessGuards, strayErrorCount } from "../render/sandbox.js";import { applyGpuPlan, gpuStatus, planGpu, probeGpu } from "./gpu.js";import { workerHealth } from "./health.js";-import { CacheJanitor } from "./janitor.js";+import { CacheJanitor, type DiskSummary } from "./janitor.js";import { currentStats, recycleReason } from "./recycle.js";import { startWatchdog } from "./watchdog.js";@@ -39,6 +39,8 @@ export interface Worker {isDraining(): boolean;/** Stops taking jobs and waits for the current one. */drain(reason: RenderError, abortActive: boolean): Promise<void>;+ /** Disk space of the work and cache dirs (cached; see CacheJanitor.disks). */+ disks(): Record<string, DiskSummary> | undefined;/** Checks git, git-lfs, ffprobe and the GPU; rejects with TOOL_MISSING or GPU_UNAVAILABLE. */checkTools(): Promise<void>;stop(): void;@@ -162,6 +164,7 @@ export function bootWorker(runner,readiness,isDraining: () => draining,+ disks: () => janitor.disks(),async drain(reason, abortActive) {draining = true;runner.queue.close(reason, abortActive);diff --git a/apps/renderer-service/src/worker/janitor.test.ts b/apps/renderer-service/src/worker/janitor.test.tsindex 8f2350d6..90e4bac9 100644--- a/apps/renderer-service/src/worker/janitor.test.ts+++ b/apps/renderer-service/src/worker/janitor.test.ts@@ -91,49 +91,38 @@ describe("CacheJanitor (free space)", () => {expect(names(await j.enforce(["protected"]))).toEqual(["old"]);});- it(- "frees shared module files only when their last workspace goes",- {- timeout: 30_000,- },- async () => {- const root = path.join(tmp, "shared-case");- const store = moduleStoreDir(path.join(root, "work"));- const slot = async (repo: string, ageMin: number) => {- const dir = path.join(root, "work/ws", repo);- await blob(- path.join(dir, "0/node_modules/webgpu/dawn.node"),- 400,- 0,- 7,- );- await blob(path.join(dir, "0/node_modules", `${repo}.js`), 100);- await shareModules(path.join(dir, "0"), store);- await blob(path.join(dir, LAST_USED), 0, ageMin);- };- await slot("a", 60);- await slot("b", 30);- await slot("c", 1);- // Stored once: 400 shared + 3 × 100 own = 700 MB.- const used = async () =>- Math.round((await usage([path.join(root, "work")])) / MB);- expect(await used()).toBe(700);-- // 810 MB disk: 110 free < 121 (15%); target 202. Removing "a" frees- // only its own 100 MB (b and c still link dawn.node): 210 free.- const j = janitor(root, fakeDisks(root, 810));- expect(names(await j.enforce(["c"]))).toEqual(["a"]);- expect(await used()).toBe(600);-- // Tighter disk: b goes too, but dawn.node stays for c.- const tight = janitor(root, fakeDisks(root, 650));- expect(names(await tight.enforce(["c"]))).toEqual(["b"]);- await expect(- fs.stat(path.join(root, "work/ws/c/0/node_modules/webgpu/dawn.node")),- ).resolves.toBeTruthy();- expect(await used()).toBe(500);- },- );+ it("frees shared module files only when their last workspace goes", async () => {+ const root = path.join(tmp, "shared-case");+ const store = moduleStoreDir(path.join(root, "work"));+ const slot = async (repo: string, ageMin: number) => {+ const dir = path.join(root, "work/ws", repo);+ await blob(path.join(dir, "0/node_modules/webgpu/dawn.node"), 400, 0, 7);+ await blob(path.join(dir, "0/node_modules", `${repo}.js`), 100);+ await shareModules(path.join(dir, "0"), store);+ await blob(path.join(dir, LAST_USED), 0, ageMin);+ };+ await slot("a", 60);+ await slot("b", 30);+ await slot("c", 1);+ // Stored once: 400 shared + 3 × 100 own = 700 MB.+ const used = async () =>+ Math.round((await usage([path.join(root, "work")])) / MB);+ expect(await used()).toBe(700);++ // 810 MB disk: 110 free < 121 (15%); target 202. Removing "a" frees+ // only its own 100 MB (b and c still link dawn.node): 210 free.+ const j = janitor(root, fakeDisks(root, 810));+ expect(names(await j.enforce(["c"]))).toEqual(["a"]);+ expect(await used()).toBe(600);++ // Tighter disk: b goes too, but dawn.node stays for c.+ const tight = janitor(root, fakeDisks(root, 650));+ expect(names(await tight.enforce(["c"]))).toEqual(["b"]);+ await expect(+ fs.stat(path.join(root, "work/ws/c/0/node_modules/webgpu/dawn.node")),+ ).resolves.toBeTruthy();+ expect(await used()).toBe(500);+ });it("on a separate cache disk: LFS objects, then mirrors, then package caches", async () => {const root = path.join(tmp, "cache-case");@@ -231,6 +220,18 @@ describe("CacheJanitor (free space)", () => {expect(names(await j.enforce([]))).toEqual(["a"]);});+ it("reports disks for /health from a background snapshot", async () => {+ const root = path.join(tmp, "health");+ await blob(path.join(root, "work/ws/r/0/big.bin"), 900);+ const j = janitor(root, fakeDisks(root, 1000));+ expect(j.disks()).toBeUndefined(); // first call starts the measurement+ await j.describeDisks();+ expect(j.disks()).toMatchObject({+ work: { lowOnSpace: true, path: path.join(root, "work") },+ cache: { sameDiskAs: "work" },+ });+ });+it("drops failed-job dirs older than a day", async () => {const root = path.join(tmp, "jobs-case");const old = path.join(root, "work/jobs/old-job");diff --git a/apps/renderer-service/src/worker/janitor.ts b/apps/renderer-service/src/worker/janitor.tsindex e19423fb..11bf6caa 100644--- a/apps/renderer-service/src/worker/janitor.ts+++ b/apps/renderer-service/src/worker/janitor.ts@@ -58,6 +58,18 @@ const statDisk: DiskProbe = async (dir) => {};};+/** One disk as /health and the eviction log report it. */+export interface DiskSummary {+ path: string;+ freeGb: number;+ totalGb: number;+ /** Eviction starts below this much free space. */+ evictBelowGb: number;+ lowOnSpace: boolean;+ /** Set when the cache dir is on the work dir's disk. */+ sameDiskAs?: string;+}+interface Candidate {path: string;bytes: number;@@ -113,23 +125,58 @@ export class CacheJanitor {}}+ private snapshot:+ | { at: number; disks: Record<string, DiskSummary> }+ | undefined;+ private refreshing: Promise<void> | undefined;++ /**+ * The last measured disks, for the synchronous /health handler. Starts a+ * background refresh when older than `maxAgeMs`; undefined until the+ * first measurement finishes.+ */+ disks(maxAgeMs = 10_000): Record<string, DiskSummary> | undefined {+ const stale = !this.snapshot || Date.now() - this.snapshot.at > maxAgeMs;+ if (stale && !this.refreshing) {+ this.refreshing = this.describeDisks()+ .then(+ () => undefined,+ () => undefined,+ )+ .finally(() => {+ this.refreshing = undefined;+ });+ }+ return this.snapshot?.disks;+ }+/** Free space per disk, for logs and /health. */- async describeDisks(): Promise<- Record<string, { freeGb: number; totalGb: number }>- > {+ async describeDisks(): Promise<Record<string, DiskSummary>> {const { workDir, cacheDir } = this.config.paths;- const out: Record<string, { freeGb: number; totalGb: number }> = {};- for (const [name, dir] of [- ["work", workDir],- ["cache", cacheDir],- ] as const) {- const d = await this.disk(dir);- out[name] = {- freeGb: Math.round((d.freeBytes / GB) * 10) / 10,- totalGb: Math.round((d.totalBytes / GB) * 10) / 10,+ const [work, cache] = await Promise.all([+ this.disk(workDir),+ this.disk(cacheDir),+ ]);+ const summary = (dir: string, d: DiskInfo): DiskSummary => {+ const { low } = this.thresholds(d);+ const gb = (bytes: number) => Math.round((bytes / GB) * 10) / 10;+ return {+ path: dir,+ freeGb: gb(d.freeBytes),+ totalGb: gb(d.totalBytes),+ evictBelowGb: gb(low),+ lowOnSpace: d.freeBytes < low,};- }- return out;+ };+ const disks = {+ work: summary(workDir, work),+ cache: {+ ...summary(cacheDir, cache),+ ...(cache.id === work.id && { sameDiskAs: "work" }),+ },+ };+ this.snapshot = { at: Date.now(), disks };+ return disks;}private thresholds(d: DiskInfo, extra = 0) {diff --git a/apps/renderer-service/vitest.config.ts b/apps/renderer-service/vitest.config.tsindex 64e5fe46..1c1245ab 100644--- a/apps/renderer-service/vitest.config.ts+++ b/apps/renderer-service/vitest.config.ts@@ -8,6 +8,9 @@ export default defineConfig({environment: "node",include: ["**/*.{test,spec}.{ts,tsx}"],pool: "forks",+ // Git, git-lfs and installs run for real; under a parallel run they+ // take longer than vitest's 5 s default.+ testTimeout: 30_000,},resolve: {alias: [