Code-first video & image, rendered natively on WebGPU.
feat(renderer-service): render repo code in-process (phase 5)
okanasl0cc02119fca6
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.mdindex 2852ccae..77c61b4a 100644--- a/apps/renderer-service/PLAN.md+++ b/apps/renderer-service/PLAN.md@@ -325,9 +325,12 @@ Rules that apply to every host:### 8.1 Loading the entry1. Resolve `entry` with `realpath` and check that it lies inside the workspace (`PATH_ESCAPE` otherwise, which also covers symlink escapes).-2. Import it with tsx's `tsImport(entryUrl, { parentURL, tsconfig })`.- - Every call gets its own module namespace, so job N+1 never sees job N's cached modules. A fix pushed to the same file is picked up even when the path is identical.+2. Import it with a per-job tsx loader (`register({ tsconfig })`) and a native `import()` of the entry URL plus `?ff-job=<jobId>`.+ - The engine alias hook passes that query on to every workspace module the entry imports, so job N+1 never sees job N's cached modules. A fix pushed to the same file is picked up even when the path is identical.+ - Engine modules get no query, so they stay the worker's single instance. tsx's `tsImport` was rejected because its namespace also applies to `framefields`, which loaded a second engine copy per job.- The project's `tsconfig.json` is honoured, including `paths`.+ - Workspace `.ts/.tsx/.js/.jsx` files always load as ESM, even without `"type": "module"`. In CommonJS mode, tsx transpiles framefields' `.mjs` into a second CJS copy of the engine. `.cjs`/`.cts` stay CommonJS, and their `require.cache` entries are cleared after each job.+ - The alias uses `module.registerHooks` (synchronous, in-thread), which covers both `import` and `require()`.- A missing module gives `ENTRY_NOT_FOUND`. A TS or syntax error gives `ENTRY_COMPILE_FAILED`, with the first frames of the error.3. Read the export (`EXPORT_NOT_FOUND`). If it is a function, call it with `args`. A throw here gives `COMPOSITION_BUILD_FAILED`.4. Normalise the result to a render plan (section 4). Anything that isn't a Composition, a record of compositions, or a plan gives `EXPORT_INVALID`.@@ -514,8 +517,12 @@ Each phase is its own PR-sized step with tests.- `resetRenderCaches()` (`@framefields/renderer`) clears media inputs and decoders, textures, LUTs and fonts. The worker calls it plus `FontManager.clear()` before each job. The GPU device and pipelines stay warm.- A GPU test (`src/engine.gpu.test.ts`, run with `FF_GPU_TESTS=1`) confirmed that without the reset, a changed file at the same workspace path is served stale.- Not yet verified: re-opening a *video* file at the same path after the reset. A comment in `renderImage` says a mediabunny `Input` can only be created once per URL. Phase 5 needs a GPU test that renders a video layer in two jobs back to back.-5. **InProcessRenderer.**- - The engine alias resolve hook, `tsImport` loading, export-shape normalisation, the job sandbox (8.2), the watchdog, the crash journal, and recycling.+5. **InProcessRenderer.** ✅ Done: the engine alias, per-job loading, export-shape normalisation, the job sandbox, stall detection, the abort grace period, the watchdog, the crash journal, and recycling. RunPod gets `refresh_worker` when the worker is draining.+ - Tests that need Node's real loader run in a subprocess (`test/load-harness.ts`, `test/render-harness.ts`), because vitest's VM bypasses loader hooks.+ - GPU-verified: back-to-back jobs, a changed video asset at the same path, frames with `select`, `process.exit` at import, cancel mid-render, and recovery afterwards.+ - Known limits:+ - Each job registers another tsx loader layer (deactivated afterwards). Recycling every `FF_MAX_JOBS_PER_WORKER` jobs bounds that.+ - `console.*` from repo code is not yet captured into a per-job log.6. **Outputs.** The s3, presigned, local and inline sinks, verification (ffprobe), multi-output keys, and idempotency.7. **Transports.**- HTTP v2 with SSE progress and cancel.diff --git a/apps/renderer-service/package.json b/apps/renderer-service/package.jsonindex f80818ca..9d793088 100644--- a/apps/renderer-service/package.json+++ b/apps/renderer-service/package.json@@ -25,13 +25,13 @@"framefields": "workspace:*","hono": "^4.13.7","semver": "^7.8.5",+ "tsx": "^4.21.0","zod": "^4.3.6"},"devDependencies": {"@framefields/tsconfig": "workspace:*","@types/node": "^24.1.0","@types/semver": "^7.8.0",- "tsx": "^4.20.3","typescript": "^5.8.3","vitest": "^4.1.5"}diff --git a/apps/renderer-service/runpod_worker.py b/apps/renderer-service/runpod_worker.pyindex e39b54e4..a5d316af 100644--- a/apps/renderer-service/runpod_worker.py+++ b/apps/renderer-service/runpod_worker.py@@ -51,6 +51,15 @@ def wait_ready():raise RuntimeError("Renderer did not become healthy in time")+def draining():+ """True when the worker decided to recycle (job count, memory, GPU loss)."""+ try:+ status, body = call("GET", "/health")+ return status != 200 and (body or {}).get("status") == "draining"+ except (urllib.error.URLError, ConnectionError):+ return True++def failure(error, refresh=False):"""RunPod marks a job failed when the output has an `error` key."""out = {"error": json.dumps(error), "refresh_worker": refresh}@@ -77,11 +86,14 @@ def handler(job):last_progress = progressif record["status"] == "COMPLETED":- return record["result"]+ result = dict(record["result"])+ if draining():+ result["refresh_worker"] = True+ return resultif record["status"] in ("FAILED", "CANCELLED"):error = record.get("error") or {"code": "INTERNAL", "message": "unknown failure"}# Device loss / crashes leave the worker unhealthy: ask for a fresh one.- refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED")+ refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED") or draining()return failure(error, refresh=refresh)diff --git a/apps/renderer-service/src/engine.gpu.test.ts b/apps/renderer-service/src/engine.gpu.test.tsindex cdd7c6ba..e3ffb4e9 100644--- a/apps/renderer-service/src/engine.gpu.test.ts+++ b/apps/renderer-service/src/engine.gpu.test.ts@@ -125,3 +125,56 @@ gpu("engine hooks", () => {expect(Buffer.compare(red, green)).not.toBe(0);}, 120_000);});++gpu("in-process render step (real Node loader)", () => {+ it("renders several jobs back to back in one process", async () => {+ const { execFileSync } = await import("node:child_process");+ const workDir = await fs.mkdtemp(path.join(os.tmpdir(), "ff-render-e2e-"));+ try {+ const out = execFileSync(+ path.resolve(import.meta.dirname, "../node_modules/.bin/tsx"),+ [+ path.resolve(import.meta.dirname, "../test/render-harness.ts"),+ workDir,+ ],+ {+ encoding: "utf8",+ env: { ...process.env, NODE_ENV: "test" },+ timeout: 300_000,+ },+ );+ const line = out+ .trim()+ .split("\n")+ .filter((l) => l.startsWith('{"video1"'))+ .at(-1);+ const r = JSON.parse(line ?? "{}");+ expect(r.video1).toMatchObject({+ ok: true,+ files: [{ name: "main", ext: "mp4" }],+ });+ // Same asset path with new bytes in the next job: decoded fresh.+ expect(r.video2.ok).toBe(true);+ expect(r.videosDiffer).toBe(true);+ expect(r.frames).toMatchObject({+ ok: true,+ files: [{ name: "teaser-f0" }, { name: "teaser-f12" }],+ });+ expect(r.exits).toMatchObject({ ok: false, code: "EXPORT_SIDE_EFFECT" });+ expect(r.cancel).toMatchObject({ ok: false, code: "CANCELLED" });+ expect(r.after.ok).toBe(true);+ for (const k of [+ "video1",+ "video2",+ "frames",+ "exits",+ "cancel",+ "after",+ ]) {+ expect(r[k].cwdRestored).toBe(true);+ }+ } finally {+ await fs.rm(workDir, { recursive: true, force: true });+ }+ }, 300_000);+});diff --git a/apps/renderer-service/src/index.ts b/apps/renderer-service/src/index.tsindex 3eb1d9a3..3c158820 100644--- a/apps/renderer-service/src/index.ts+++ b/apps/renderer-service/src/index.ts@@ -1,4 +1,6 @@+import { mkdirSync, realpathSync } from "node:fs";import { Server as HttpServer } from "node:http";+import path from "node:path";import { HeadlessMediaRenderer } from "@framefields/renderer";import "@framefields/renderer/offscreen-gl-polyfill";import { rendererLogger } from "@framefields/server-utils";@@ -8,9 +10,21 @@ import { RenderError } from "./errors.js";import { checkout } from "./git/checkout.js";import { createApp } from "./http/app.js";import { install } from "./install/installer.js";+import { CrashJournal } from "./jobs/journal.js";import type { Pipeline } from "./jobs/pipeline.js";import { JobRunner } from "./jobs/runner.js";import { ReadinessTracker } from "./readiness.js";+import { redact } from "./redact.js";+import { installEngineAlias } from "./render/engine-alias.js";+import { render } from "./render/render-step.js";+import { installProcessGuards, strayErrorCount } from "./render/sandbox.js";+import { workerHealth } from "./worker/health.js";+import { currentStats, recycleReason } from "./worker/recycle.js";+import { startWatchdog } from "./worker/watchdog.js";++// The job sandbox swaps process.exit while repo code runs; the worker itself+// always exits through the real one.+const exit = process.exit.bind(process);const config = loadConfig();scrubSecretEnv();@@ -19,6 +33,23 @@ rendererLogger.info("Renderer config loaded",);+// Workspaces live here; repo code under it gets the engine alias (§7).+const workspaces = path.join(config.paths.workDir, "ws");+mkdirSync(workspaces, { recursive: true });+mkdirSync(config.paths.cacheDir, { recursive: true });+installEngineAlias([realpathSync(workspaces)]);++installProcessGuards((msg, err) =>+ rendererLogger.error(+ {+ err: redact(+ err instanceof Error ? (err.stack ?? err.message) : String(err),+ ),+ },+ msg,+ ),+);+const readiness = new ReadinessTracker();let draining = false;@@ -32,11 +63,43 @@ const notYet = (step: string): never => {const pipeline: Pipeline = {checkout,install,- render: async () => notYet("render"),+ render,deliver: async () => notYet("deliver"),};-const runner = new JobRunner(config, pipeline);+const journal = new CrashJournal(+ path.join(config.paths.workDir, "inflight.json"),+);+let baselineHandles = 0;++const runner = new JobRunner(config, pipeline, {+ journal,+ afterJob: (completed) => {+ const reason = recycleReason(+ currentStats(+ completed,+ baselineHandles,+ strayErrorCount(),+ workerHealth.recycleReason,+ ),+ config,+ );+ if (reason) void recycle(reason);+ },+});++const crashed = journal.recover();+if (crashed) {+ runner.store.restore(crashed);+ rendererLogger.error(+ { jobId: crashed.id, error: crashed.error },+ "Previous process died during a job",+ );+}+const stopWatchdog = startWatchdog({+ blockSec: config.recycle.eventLoopBlockSec,+ journalPath: journal.path,+});const app = createApp(config, runner, {health: () => {@@ -59,6 +122,9 @@ const app = createApp(config, runner, {waiting: runner.queue.depth,},...(status.error && { error: status.error }),+ ...(workerHealth.recycleReason && {+ recycle: workerHealth.recycleReason,+ }),},};},@@ -87,6 +153,7 @@ async function bootstrap() {await HeadlessMediaRenderer.initialize();readiness.setRendererReady(true);readiness.setGpuReady(true);+ baselineHandles = process.getActiveResourcesInfo().length;rendererLogger.info("Renderer initialized.");} catch (error) {const errMsg = error instanceof Error ? error.message : String(error);@@ -96,12 +163,33 @@ async function bootstrap() {);readiness.setFailure(`GPU_UNAVAILABLE: ${errMsg}`);// RunPod should replace a worker without a usable GPU.- if (config.mode === "runpod-serverless") process.exit(1);+ if (config.mode === "runpod-serverless") exit(1);}}void bootstrap();+/**+ * Stops taking jobs and restarts the process once the queue is empty. On+ * RunPod serverless the shim asks for a fresh worker (refresh_worker) when+ * it sees /health report "draining", so the process just stays drained.+ */+async function recycle(reason: string) {+ if (draining) return;+ draining = true;+ rendererLogger.warn({ reason }, "Recycling worker");+ runner.queue.close(+ new RenderError("SHUTDOWN", `Worker is recycling (${reason}).`),+ false,+ );+ await runner.queue.idle();+ if (config.mode !== "runpod-serverless") {+ stopWatchdog();+ server.close();+ exit(0);+ }+}+// ─── Graceful Shutdown ──────────────────────────────────────────────────────────let shuttingDown = false;@@ -125,14 +213,15 @@ async function gracefulShutdown(signal: string) {const graceMs = (config.limits.killGraceSec + 5) * 1000;const forceExit = setTimeout(() => {rendererLogger.error("Shutdown timed out, forcing exit.");- process.exit(1);+ exit(1);}, graceMs);await runner.queue.idle();+ stopWatchdog();server.close();clearTimeout(forceExit);rendererLogger.info("Shutdown complete.");- process.exit(0);+ exit(0);}process.on("SIGTERM", () => void gracefulShutdown("SIGTERM"));diff --git a/apps/renderer-service/src/jobs/journal.ts b/apps/renderer-service/src/jobs/journal.tsnew file mode 100644index 00000000..78dcc417--- /dev/null+++ b/apps/renderer-service/src/jobs/journal.ts@@ -0,0 +1,65 @@+import { readFileSync, rmSync, writeFileSync } from "node:fs";+import { RenderError } from "../errors.js";+import type { JobRecord } from "./store.js";++/**+ * The running job, on local disk, so a process that dies mid-render (OOM,+ * native crash, watchdog kill) can still report it after restart (§8.3).+ * Sync I/O on purpose: it must land before repo code gets to run.+ */+export interface JournalEntry {+ jobId: string;+ startedAt: string;+ createdAt: string;+ /** Set by the watchdog right before it kills the process. */+ exitReason?: string;+}++export class CrashJournal {+ constructor(private readonly file: string) {}++ get path(): string {+ return this.file;+ }++ begin(entry: JournalEntry): void {+ writeFileSync(this.file, JSON.stringify(entry));+ }++ end(): void {+ rmSync(this.file, { force: true });+ }++ /**+ * Returns the job a previous process died on, as a failed record, and+ * clears the journal. Undefined when the last shutdown was clean.+ */+ recover(): JobRecord | undefined {+ let entry: JournalEntry;+ try {+ entry = JSON.parse(readFileSync(this.file, "utf8")) as JournalEntry;+ } catch {+ return undefined;+ } finally {+ rmSync(this.file, { force: true });+ }+ const oom = /heap|memory/i.test(entry.exitReason ?? "");+ const error = new RenderError(+ oom ? "RENDER_OOM" : "RENDER_CRASHED",+ `The worker process died during this job${entry.exitReason ? ` (${entry.exitReason})` : ""}.`,+ { phase: "render", hint: "A fresh worker can retry it." },+ );+ const now = new Date().toISOString();+ return {+ id: entry.jobId,+ status: "FAILED",+ progress: { phase: "done" },+ createdAt: entry.createdAt,+ startedAt: entry.startedAt,+ updatedAt: now,+ finishedAt: now,+ error: error.toJSON(),+ warnings: [],+ };+ }+}diff --git a/apps/renderer-service/src/jobs/runner.ts b/apps/renderer-service/src/jobs/runner.tsindex 07eb45c2..ed3dd636 100644--- a/apps/renderer-service/src/jobs/runner.ts+++ b/apps/renderer-service/src/jobs/runner.ts@@ -5,6 +5,7 @@ import { loggerContext, rendererLogger } from "@framefields/server-utils";import type { RendererConfig } from "../config/index.js";import { type JobPhase, RenderError, toRenderError } from "../errors.js";import { redact, redactValue } from "../redact.js";+import type { CrashJournal } from "./journal.js";import type { Pipeline, StepContext } from "./pipeline.js";import { JobQueue } from "./queue.js";import { type JobRecord, JobStore } from "./store.js";@@ -22,7 +23,10 @@ export class JobRunner {constructor(private readonly config: RendererConfig,private readonly pipeline: Pipeline,- private readonly hooks: { afterJob?: (completed: number) => void } = {},+ private readonly hooks: {+ afterJob?: (completed: number) => void;+ journal?: CrashJournal;+ } = {},) {this.store = new JobStore(config.limits.jobTtlSec * 1000,@@ -114,11 +118,16 @@ export class JobRunner {throw toRenderError(controller.signal.reason, phase);};- this.store.update(id, {+ const record = this.store.update(id, {status: "IN_PROGRESS",startedAt: new Date().toISOString(),progress: { phase: "resolving" },});+ this.hooks.journal?.begin({+ jobId: id,+ createdAt: record?.createdAt ?? new Date().toISOString(),+ startedAt: record?.startedAt ?? new Date().toISOString(),+ });rendererLogger.info({ jobId: id, repo: job.repo.id, ref: job.ref },"Job started",@@ -126,6 +135,7 @@ export class JobRunner {let release: (() => Promise<void>) | undefined;let succeeded = false;+ let settle: (() => unknown) | undefined;try {await fs.mkdir(path.join(jobDir, "out"), { recursive: true });@@ -163,34 +173,36 @@ export class JobRunner {checkAbort();timings.totalMs = Date.now() - started;- this.store.update(id, {- status: "COMPLETED",- progress: { phase: "done" },- result: redactValue({- jobId: id,- source: {- repo: job.repo.id,- ref: job.ref,- sha: checkout.sha,- workdir: job.input.source.workdir,- },- engine: {- ...rendered.engine,- projectDeclared: install.declaredEngine,- },- outputs,- timings,- cache,- warnings,- }),- });+ settle = () =>+ this.store.update(id, {+ status: "COMPLETED",+ progress: { phase: "done" },+ result: redactValue({+ jobId: id,+ source: {+ repo: job.repo.id,+ ref: job.ref,+ sha: checkout.sha,+ workdir: job.input.source.workdir,+ },+ engine: {+ ...rendered.engine,+ projectDeclared: install.declaredEngine,+ },+ outputs,+ timings,+ cache,+ warnings,+ }),+ });succeeded = true;- rendererLogger.info({ jobId: id, timings }, "Job completed");} catch (err) {const reason = controller.signal.aborted ? controller.signal.reason : err;- this.fail(id, toRenderError(reason, phase), { timings, cache });+ settle = () =>+ this.fail(id, toRenderError(reason, phase), { timings, cache });} finally {clearTimeout(timer);+ this.hooks.journal?.end();signal.removeEventListener("abort", onAbort);await release?.().catch((err) =>rendererLogger.warn(@@ -202,7 +214,12 @@ export class JobRunner {await fs.rm(jobDir, { recursive: true, force: true }).catch(() => {});}this.completedJobs++;+ // Decide on recycling before the job is visibly finished, so a client+ // that sees the result also sees /health report "draining".this.hooks.afterJob?.(this.completedJobs);+ settle?.();+ if (succeeded)+ rendererLogger.info({ jobId: id, timings }, "Job completed");}}diff --git a/apps/renderer-service/src/render/engine-alias.ts b/apps/renderer-service/src/render/engine-alias.tsnew file mode 100644index 00000000..a03d98a9--- /dev/null+++ b/apps/renderer-service/src/render/engine-alias.ts@@ -0,0 +1,95 @@+import { createRequire, registerHooks } from "node:module";+import path from "node:path";+import { pathToFileURL } from "node:url";++/** Query parameter that scopes workspace ES modules to one job. */+export const JOB_PARAM = "ff-job";++let roots: string[] = [];+let rootDirs: string[] = [];+let engineParent = "";+let installed = false;++const isEngine = (s: string) =>+ s === "framefields" ||+ s.startsWith("framefields/") ||+ s.startsWith("@framefields/");+const inWorkspace = (url: string) => roots.some((r) => url.startsWith(r));+/** Source files that become ESM even in a package without "type": "module". */+const FORCE_ESM = /\.(ts|tsx|js|jsx)$/;++function jobOf(url: string | undefined): string | undefined {+ if (!url?.includes("?")) return undefined;+ try {+ return new URL(url).searchParams.get(JOB_PARAM) ?? undefined;+ } catch {+ return undefined;+ }+}++/**+ * Installs the engine alias once per process (PLAN.md §7): imports and+ * requires of `framefields` / `@framefields/*` made from inside a workspace+ * resolve as if made by the worker's own framefields (ESM build), so repo+ * code shares the worker's single, already-initialised GPU stack. Workspace+ * ES modules also inherit a per-job query, so a job never reuses modules+ * cached by a previous one. Uses synchronous in-thread hooks, which apply to+ * both `import` and `require()`.+ */+export function installEngineAlias(workspaceRoots: string[]): void {+ if (installed) return;+ installed = true;+ rootDirs = workspaceRoots.map((r) => path.resolve(r) + path.sep);+ roots = rootDirs.map((r) => pathToFileURL(r).href);+ // framefields depends on every @framefields/* package, so resolving from+ // its entry finds them all.+ engineParent = import.meta.resolve("framefields");++ registerHooks({+ resolve(specifier, context, next) {+ const parent = context.parentURL;+ if (isEngine(specifier) && parent && inWorkspace(parent)) {+ try {+ // Always the ESM build, so require() and import share one instance.+ const conditions = (context.conditions ?? []).map((c) =>+ c === "require" ? "import" : c,+ );+ if (!conditions.includes("import")) conditions.push("import");+ return next(specifier, {+ ...context,+ conditions,+ parentURL: engineParent,+ });+ } catch {+ // Not part of the engine (e.g. a project-local @framefields/* package).+ }+ }+ const result = next(specifier, context);+ if (!result.url.startsWith("file:") || !inWorkspace(result.url))+ return result;+ let { url, format } = result;+ // Repo code always runs as ESM. In CommonJS mode tsx would transpile+ // framefields' .mjs into a second, CJS copy of the engine.+ if (format === "commonjs" && FORCE_ESM.test(new URL(url).pathname))+ format = "module";+ const job = jobOf(parent);+ if (job && !jobOf(url)) {+ const scoped = new URL(url);+ scoped.searchParams.set(JOB_PARAM, job);+ url = scoped.href;+ }+ return { ...result, url, format };+ },+ });+}++/**+ * Drops CommonJS modules loaded from the workspaces (ES modules are scoped by+ * the job query instead). Called when a job ends.+ */+export function clearWorkspaceRequireCache(): void {+ const cache = createRequire(import.meta.url).cache;+ for (const key of Object.keys(cache)) {+ if (rootDirs.some((r) => key.startsWith(r))) delete cache[key];+ }+}diff --git a/apps/renderer-service/src/render/load.ts b/apps/renderer-service/src/render/load.tsnew file mode 100644index 00000000..acd930bd--- /dev/null+++ b/apps/renderer-service/src/render/load.ts@@ -0,0 +1,242 @@+import { existsSync } from "node:fs";+import path from "node:path";+import { pathToFileURL } from "node:url";+import { register } from "tsx/esm/api";+import { RenderError } from "../errors.js";+import type { RenderSettings } from "../jobs/schema.js";+import { resolveInside } from "../paths.js";+import { JOB_PARAM } from "./engine-alias.js";++/** What the worker can render: a framefields Composition or Media. */+export interface Renderable {+ renderVideo(options?: Record<string, unknown>): Promise<{+ filePath: string;+ cleanup: () => Promise<void>;+ qa?: unknown;+ }>;+ renderFrame(options: Record<string, unknown>): Promise<Buffer>;+ renderFrameGrid?(options: Record<string, unknown>): Promise<Buffer>;+ toVirtualMedia?(): unknown;+ virtualMedia?: unknown;+}++export interface PlanOutput {+ name: string;+ target: Renderable;+ render: Partial<RenderSettings>;+}++const NAME = /^[A-Za-z0-9][\w.-]{0,63}$/;++export function isRenderable(value: unknown): value is Renderable {+ return (+ typeof value === "object" &&+ value !== null &&+ typeof (value as Renderable).renderVideo === "function" &&+ typeof (value as Renderable).renderFrame === "function"+ );+}++export interface LoadedEntry {+ mod: Record<string, unknown>;+ /** Removes this job's TypeScript loader; call when the job is done. */+ dispose(): Promise<void>;+}++/**+ * Imports the entry with tsx, using the workdir's tsconfig. The import URL+ * carries a job token that the engine alias propagates through the repo's+ * own modules, so a job never sees modules cached by a previous one, while+ * framefields itself stays the worker's single instance (PLAN.md §8.1).+ */+export async function importEntry(+ workdir: string,+ entry: string,+ jobToken: string,+): Promise<LoadedEntry> {+ const file = await resolveInside(workdir, entry).catch((err) => {+ if (err instanceof RenderError) throw err;+ throw new RenderError(+ "ENTRY_NOT_FOUND",+ `Entry ${entry} does not exist in the workdir.`,+ { phase: "load" },+ );+ });+ const tsconfig = path.join(workdir, "tsconfig.json");+ const unregister = register({+ tsconfig: existsSync(tsconfig) ? tsconfig : false,+ });+ const url = new URL(pathToFileURL(file).href);+ url.searchParams.set(JOB_PARAM, jobToken);+ try {+ const mod = (await import(/* @vite-ignore */ url.href)) as Record<+ string,+ unknown+ >;+ return { mod, dispose: unregister };+ } catch (err) {+ await unregister();+ throw classifyLoadError(err, file);+ }+}++export function classifyLoadError(+ err: unknown,+ entryFile: string,+): RenderError {+ if (err instanceof RenderError) return err;+ const e = err as NodeJS.ErrnoException & { url?: string };+ const message = e?.message ?? String(err);+ if (+ e?.code === "ERR_MODULE_NOT_FOUND" ||+ /Cannot find (module|package)/.test(message)+ ) {+ const missing =+ /Cannot find (?:module|package) '([^']+)'/.exec(message)?.[1] ?? "";+ const missingEntry =+ missing.split("?")[0] === entryFile ||+ missing.split("?")[0] === pathToFileURL(entryFile).href;+ return new RenderError(+ missingEntry ? "ENTRY_NOT_FOUND" : "ENTRY_COMPILE_FAILED",+ message,+ {+ phase: "load",+ hint: missingEntry+ ? undefined+ : "An import couldn't be resolved: is the dependency in package.json and the lockfile?",+ cause: err,+ },+ );+ }+ if (+ e instanceof SyntaxError ||+ /Transform failed|Expected .* but found|Unexpected token|ERR_UNKNOWN_FILE_EXTENSION/.test(+ message,+ )+ ) {+ return new RenderError("ENTRY_COMPILE_FAILED", message, {+ phase: "load",+ cause: err,+ });+ }+ return new RenderError("COMPOSITION_BUILD_FAILED", message, {+ phase: "load",+ cause: err,+ });+}++/**+ * Reads the export, calls it with `args` when it is a function, and turns the+ * result into a list of named outputs (PLAN.md §4 "Export shapes").+ */+export async function buildPlan(+ mod: Record<string, unknown>,+ exportName: string,+ args: unknown,+ select: string[] | undefined,+): Promise<PlanOutput[]> {+ if (!(exportName in mod)) {+ const available = Object.keys(mod).filter((k) => k !== "__esModule");+ throw new RenderError(+ "EXPORT_NOT_FOUND",+ `The entry has no export named "${exportName}".`,+ {+ phase: "load",+ hint: available.length+ ? `Exports: ${available.slice(0, 10).join(", ")}`+ : "The module exports nothing.",+ },+ );+ }+ let value = mod[exportName];+ if (typeof value === "function" && !isRenderable(value)) {+ try {+ value = await (value as (a: unknown) => unknown)(args);+ } catch (err) {+ if (err instanceof RenderError) throw err;+ throw new RenderError(+ "COMPOSITION_BUILD_FAILED",+ `${exportName}() threw: ${err instanceof Error ? err.message : String(err)}`,+ { phase: "load", cause: err },+ );+ }+ }++ let outputs: PlanOutput[];+ if (isRenderable(value)) {+ outputs = [{ name: "main", target: value, render: {} }];+ } else if (isPlan(value)) {+ outputs = value.outputs.map((o, i) => {+ if (!isRenderable(o?.composition)) {+ throw invalid(`outputs[${i}].composition is not a Composition.`);+ }+ return {+ name: String(o.name ?? `output-${i}`),+ target: o.composition,+ render: o.render ?? {},+ };+ });+ } else if (value && typeof value === "object" && !Array.isArray(value)) {+ const entries = Object.entries(value);+ if (entries.length === 0 || !entries.every(([, v]) => isRenderable(v))) {+ throw invalid(describe(value));+ }+ outputs = entries.map(([name, target]) => ({+ name,+ target: target as Renderable,+ render: {},+ }));+ } else {+ throw invalid(describe(value));+ }++ for (const o of outputs) {+ if (!NAME.test(o.name))+ throw invalid(`Output name "${o.name}" must match ${NAME}.`);+ }+ if (new Set(outputs.map((o) => o.name)).size !== outputs.length) {+ throw invalid("Output names must be unique.");+ }+ if (select) {+ const missing = select.filter((s) => !outputs.some((o) => o.name === s));+ if (missing.length) {+ throw new RenderError(+ "JOB_INVALID",+ `target.select names unknown outputs: ${missing.join(", ")}.`,+ {+ phase: "load",+ hint: `Available: ${outputs.map((o) => o.name).join(", ")}`,+ },+ );+ }+ outputs = outputs.filter((o) => select.includes(o.name));+ }+ return outputs;+}++interface PlanShape {+ outputs: {+ name?: unknown;+ composition?: unknown;+ render?: Partial<RenderSettings>;+ }[];+}++function isPlan(value: unknown): value is PlanShape {+ return (+ typeof value === "object" &&+ value !== null &&+ Array.isArray((value as PlanShape).outputs) &&+ (value as PlanShape).outputs.length > 0+ );+}++function describe(value: unknown): string {+ const kind =+ value === null ? "null" : Array.isArray(value) ? "an array" : typeof value;+ return `The export resolved to ${kind}, not a Composition, a record of compositions, or { outputs: [...] }.`;+}++function invalid(message: string): RenderError {+ return new RenderError("EXPORT_INVALID", message, { phase: "load" });+}diff --git a/apps/renderer-service/src/render/render-step.ts b/apps/renderer-service/src/render/render-step.tsnew file mode 100644index 00000000..642e4aff--- /dev/null+++ b/apps/renderer-service/src/render/render-step.ts@@ -0,0 +1,470 @@+import fs from "node:fs/promises";+import path from "node:path";+import {+ HeadlessMediaRenderer,+ resetRenderCaches,+} from "@framefields/renderer";+import { Composition, FontManager, Media } from "framefields";+import { RenderError } from "../errors.js";+import { bundledEngineVersion } from "../install/installer.js";+import type {+ CheckoutResult,+ Pipeline,+ RenderedFile,+ StepContext,+} from "../jobs/pipeline.js";+import type { JobInput, RenderSettings } from "../jobs/schema.js";+import { workerHealth } from "../worker/health.js";+import { clearWorkspaceRequireCache } from "./engine-alias.js";+import { buildPlan, importEntry, type PlanOutput } from "./load.js";+import { openSandbox } from "./sandbox.js";++const PROGRESS_INTERVAL_MS = 500;++/**+ * Pipeline step: imports the repo's entry into this process and renders every+ * output of its plan on the worker's warm GPU device (PLAN.md §8).+ */+export const render: Pipeline["render"] = async (checkout, input, ctx) => {+ // Path-keyed caches from the previous job may hold stale bytes (§8.2).+ resetRenderCaches();+ FontManager.clear();++ const outDir = path.join(ctx.jobDir, "out");+ await fs.mkdir(outDir, { recursive: true });++ // The render may be aborted by the job (cancel/timeout) or by the stall+ // detector; both go through this controller.+ const controller = new AbortController();+ const forward = () => controller.abort(ctx.signal.reason);+ if (ctx.signal.aborted) forward();+ ctx.signal.addEventListener("abort", forward, { once: true });++ const sandbox = openSandbox({+ cwd: checkout.workdir,+ argv1: path.join(checkout.workdir, input.target.entry),+ env: input.env,+ });+ const warnings: string[] = [];+ let dispose: (() => Promise<void>) | undefined;+ try {+ const work = (async () => {+ const files: RenderedFile[] = [];+ const plan = await withTimeout(+ loadPlan(checkout, input, ctx.jobId, (d) => {+ dispose = d;+ }),+ ctx.config.limits.timeouts.load,+ "load",+ );+ for (const output of plan) {+ if (+ !(+ output.target instanceof Composition ||+ output.target instanceof Media+ )+ ) {+ warnings.push(output.name);+ }+ }+ if (warnings.length) {+ ctx.warn(+ "ENGINE_FOREIGN_INSTANCE",+ `Outputs ${warnings.join(", ")} were built by another copy of framefields; rendering them with the worker's engine.`,+ );+ }+ for (const [index, output] of plan.entries()) {+ controller.signal.throwIfAborted();+ files.push(+ ...(await renderOutput(+ output,+ index,+ plan.length,+ input.render,+ outDir,+ ctx,+ controller,+ )),+ );+ }+ return files;+ })();++ const files = await raceWithGrace(+ work,+ sandbox.failed,+ controller.signal,+ ctx.config.limits.killGraceSec,+ );+ return {+ files,+ engine: { framefields: bundledEngineVersion(), source: "bundled" },+ };+ } catch (err) {+ throw classifyRenderError(+ controller.signal.aborted ? controller.signal.reason : err,+ checkout.root,+ );+ } finally {+ ctx.signal.removeEventListener("abort", forward);+ sandbox.restore();+ clearWorkspaceRequireCache();+ await dispose?.().catch(() => {});+ }+};++async function loadPlan(+ checkout: CheckoutResult,+ input: JobInput,+ jobId: string,+ onLoaded: (dispose: () => Promise<void>) => void,+): Promise<PlanOutput[]> {+ const { mod, dispose } = await importEntry(+ checkout.workdir,+ input.target.entry,+ jobId,+ );+ onLoaded(dispose);+ return buildPlan(+ mod,+ input.target.export,+ input.target.args,+ input.target.select,+ );+}++/** Renders one plan output into one or more files. */+async function renderOutput(+ output: PlanOutput,+ index: number,+ count: number,+ defaults: RenderSettings,+ outDir: string,+ ctx: StepContext,+ controller: AbortController,+): Promise<RenderedFile[]> {+ const settings = { ...defaults, ...output.render } as RenderSettings;+ const renderer = new HeadlessMediaRenderer();+ const stall = stallWatch(ctx.config.limits.stallTimeoutSec, controller);+ const base = path.join(outDir, `${index}-${output.name}`);++ try {+ switch (settings.kind) {+ case "video":+ case "audio": {+ const codec =+ settings.codec ?? (settings.kind === "audio" ? "aac" : "h264");+ const { ext, contentType } = containerFor(codec);+ const started = Date.now();+ let lastEmit = 0;+ const result = await output.target.renderVideo({+ outputPath: `${base}.${ext}`,+ codec,+ audioCodec: settings.audioCodec,+ quality: settings.quality,+ qa: settings.qa,+ renderer,+ signal: controller.signal,+ onProgress: (p: {+ phase: string;+ frame: number;+ totalFrames: number;+ }) => {+ stall.touch();+ const now = Date.now();+ if (+ now - lastEmit < PROGRESS_INTERVAL_MS &&+ p.frame < p.totalFrames+ )+ return;+ lastEmit = now;+ const fps = p.frame / Math.max(0.001, (now - started) / 1000);+ ctx.progress({+ phase: p.phase === "audio" ? "encoding" : "rendering",+ output: count > 1 ? index : undefined,+ frame: p.frame,+ total: p.totalFrames,+ fps: Math.round(fps * 10) / 10,+ etaSec: Math.round(+ (p.totalFrames - p.frame) / Math.max(fps, 0.001),+ ),+ });+ },+ });+ return [+ {+ name: output.name,+ path: result.filePath,+ ext,+ contentType,+ ...(result.qa !== undefined && { qa: result.qa }),+ },+ ];+ }+ case "frames": {+ const files: RenderedFile[] = [];+ const frames = settings.frames ?? [0];+ for (const [i, frame] of frames.entries()) {+ controller.signal.throwIfAborted();+ const png = await output.target.renderFrame({ frame, renderer });+ stall.touch();+ const file = `${base}-f${String(frame).padStart(5, "0")}.png`;+ await fs.writeFile(file, png);+ files.push({+ name: `${output.name}-f${frame}`,+ path: file,+ ext: "png",+ contentType: "image/png",+ });+ ctx.progress({+ phase: "rendering",+ output: count > 1 ? index : undefined,+ frame: i + 1,+ total: frames.length,+ });+ }+ return files;+ }+ case "frameGrid": {+ if (!output.target.renderFrameGrid) {+ throw new RenderError(+ "EXPORT_INVALID",+ `Output ${output.name} can't render a frame grid.`,+ { phase: "render" },+ );+ }+ const grid = settings.grid ?? {};+ const png = await output.target.renderFrameGrid({+ startFrame: grid.from,+ endFrame: grid.to,+ stepFrames: grid.step,+ ...(grid.columns && { columns: grid.columns }),+ renderer,+ });+ const file = `${base}-grid.png`;+ await fs.writeFile(file, png);+ return [+ {+ name: output.name,+ path: file,+ ext: "png",+ contentType: "image/png",+ },+ ];+ }+ case "lut": {+ const vm =+ output.target.toVirtualMedia?.() ?? output.target.virtualMedia;+ if (!vm) {+ throw new RenderError(+ "EXPORT_INVALID",+ `Output ${output.name} has no virtual media to bake a LUT from.`,+ {+ phase: "render",+ },+ );+ }+ const cube = await renderer.renderLut(+ vm as never,+ settings.frames?.[0] ?? 0,+ );+ const file = `${base}.cube`;+ await fs.writeFile(file, cube);+ return [+ {+ name: output.name,+ path: file,+ ext: "cube",+ contentType: "text/plain",+ },+ ];+ }+ }+ } finally {+ stall.stop();+ }+}++export function containerFor(codec: string): {+ ext: string;+ contentType: string;+} {+ switch (codec) {+ case "vp8":+ case "vp9":+ return { ext: "webm", contentType: "video/webm" };+ case "gif":+ return { ext: "gif", contentType: "image/gif" };+ case "mp3":+ return { ext: "mp3", contentType: "audio/mpeg" };+ case "aac":+ return { ext: "m4a", contentType: "audio/mp4" };+ case "opus":+ return { ext: "webm", contentType: "audio/webm" };+ default:+ return { ext: "mp4", contentType: "video/mp4" };+ }+}++/** Aborts with RENDER_STALLED when no progress arrives for `seconds`. */+function stallWatch(seconds: number, controller: AbortController) {+ let last = Date.now();+ const timer =+ seconds > 0+ ? setInterval(() => {+ if (+ Date.now() - last > seconds * 1000 &&+ !controller.signal.aborted+ ) {+ controller.abort(+ new RenderError(+ "RENDER_STALLED",+ `No render progress for ${seconds}s.`,+ { phase: "render" },+ ),+ );+ }+ }, 1000)+ : undefined;+ timer?.unref();+ return {+ touch: () => {+ last = Date.now();+ },+ stop: () => clearInterval(timer),+ };+}++function withTimeout<T>(+ promise: Promise<T>,+ seconds: number,+ what: string,+): Promise<T> {+ if (seconds <= 0) return promise;+ let timer: NodeJS.Timeout | undefined;+ return Promise.race([+ promise,+ new Promise<never>((_, reject) => {+ timer = setTimeout(+ () =>+ reject(+ new RenderError(+ "COMPOSITION_BUILD_FAILED",+ `Loading the entry took longer than ${seconds}s (${what}).`,+ {+ phase: "load",+ hint: "Is the export waiting on something that never resolves? Raise FF_TIMEOUT_LOAD_SEC if it's just slow.",+ },+ ),+ ),+ seconds * 1000,+ );+ }),+ ]).finally(() => clearTimeout(timer));+}++/**+ * Waits for the render, a stray error from repo code, or an abort. Code that+ * ignores the abort gets `graceSec`; after that the job is reported anyway and+ * the worker recycles, since it can't stop that code in-process (§8.3).+ */+async function raceWithGrace<T>(+ work: Promise<T>,+ failed: Promise<never>,+ signal: AbortSignal,+ graceSec: number,+): Promise<T> {+ let timer: NodeJS.Timeout | undefined;+ const abandoned = new Promise<never>((_, reject) => {+ const arm = () => {+ timer = setTimeout(() => {+ workerHealth.requestRecycle("render ignored abort");+ reject(signal.reason);+ }, graceSec * 1000);+ };+ if (signal.aborted) arm();+ else signal.addEventListener("abort", arm, { once: true });+ });+ work.catch(() => {}); // may settle after we've given up on it+ try {+ return await Promise.race([work, failed, abandoned]);+ } finally {+ clearTimeout(timer);+ }+}++export function classifyRenderError(+ err: unknown,+ repoRoot: string,+): RenderError {+ const scrub = (s: string) => s.replaceAll(repoRoot, "<repo>");+ if (err instanceof RenderError) {+ if (err.code === "GPU_DEVICE_LOST" || err.code === "RENDER_OOM")+ workerHealth.requestRecycle(err.code);+ return err;+ }+ const message = err instanceof Error ? err.message : String(err);+ const stack =+ err instanceof Error ? projectFrames(err.stack ?? "", repoRoot) : undefined;+ const details = stack ? { stack } : undefined;++ if (+ /device (was )?lost|GPUDevice.*lost|DeviceLost|VK_ERROR_DEVICE_LOST/i.test(+ message,+ )+ ) {+ workerHealth.requestRecycle("GPU device lost");+ return new RenderError("GPU_DEVICE_LOST", scrub(message), {+ phase: "render",+ details,+ cause: err,+ });+ }+ if (+ /out of memory|OutOfMemory|Failed to allocate|allocation failed|ERR_OUT_OF_MEMORY/i.test(+ message,+ )+ ) {+ workerHealth.requestRecycle("out of memory");+ return new RenderError("RENDER_OOM", scrub(message), {+ phase: "render",+ details,+ cause: err,+ });+ }+ if (/Cannot open local media file|ENOENT|no such file/i.test(message)) {+ return new RenderError("ASSET_NOT_FOUND", scrub(message), {+ phase: "render",+ details,+ hint: "If the file is tracked with LFS, make sure source.lfs.include covers it.",+ cause: err,+ });+ }+ if (+ /UnsupportedInputFormat|Unsupported.*format|Invalid data found/i.test(+ message,+ )+ ) {+ return new RenderError("ASSET_NOT_FOUND", scrub(message), {+ phase: "render",+ details,+ hint: "The file isn't decodable media. An undownloaded LFS pointer looks like this too.",+ cause: err,+ });+ }+ return new RenderError("RENDER_FAILED", scrub(message), {+ phase: "render",+ details,+ cause: err,+ });+}++/** Keeps stack frames from the project, with paths made repo-relative. */+function projectFrames(stack: string, repoRoot: string): string | undefined {+ const frames = stack+ .split("\n")+ .filter((line) => line.includes(repoRoot) && !line.includes("node_modules"))+ .slice(0, 15)+ .map((line) => line.replaceAll(repoRoot, "<repo>").trim());+ return frames.length ? frames.join("\n") : undefined;+}diff --git a/apps/renderer-service/src/render/render.test.ts b/apps/renderer-service/src/render/render.test.tsnew file mode 100644index 00000000..3b587a9e--- /dev/null+++ b/apps/renderer-service/src/render/render.test.ts@@ -0,0 +1,356 @@+import { execFileSync } from "node:child_process";+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { afterAll, beforeAll, describe, expect, it } from "vitest";+import { RenderError } from "../errors.js";+import { CrashJournal } from "../jobs/journal.js";+import { testConfig } from "../test-helpers.js";+import { recycleReason } from "../worker/recycle.js";+import { buildPlan, isRenderable } from "./load.js";+import { classifyRenderError } from "./render-step.js";+import { ExitTrap, openSandbox } from "./sandbox.js";++let ws: string;++async function write(rel: string, content: string) {+ const file = path.join(ws, rel);+ await fs.mkdir(path.dirname(file), { recursive: true });+ await fs.writeFile(file, content);+}++async function codeOf(promise: Promise<unknown>) {+ try {+ await promise;+ return "no error";+ } catch (err) {+ return err instanceof RenderError+ ? err.code+ : `not a RenderError: ${String(err)}`;+ }+}++beforeAll(async () => {+ ws = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "ff-ws-")));+ await write(+ "package.json",+ '{ "name": "film", "type": "module", "dependencies": { "framefields": "^0.0.1" } }',+ );+});+afterAll(async () => {+ await fs.rm(ws, { recursive: true, force: true });+});++const fake = () => ({+ renderVideo: async () => ({}),+ renderFrame: async () => Buffer.alloc(0),+});++/** Runs the loader in a real Node process (see test/load-harness.ts). */+function harness(entry: string, token: string, exportName?: string) {+ const tsx = path.resolve(import.meta.dirname, "../../node_modules/.bin/tsx");+ const script = path.resolve(+ import.meta.dirname,+ "../../test/load-harness.ts",+ );+ const out = execFileSync(+ tsx,+ [script, ws, entry, token, ...(exportName ? [exportName] : [])],+ {+ encoding: "utf8",+ env: { ...process.env, NODE_ENV: "test" },+ },+ );+ const line =+ out+ .trim()+ .split("\n")+ .filter((l) => l.startsWith("{"))+ .at(-1) ?? "{}";+ return JSON.parse(line) as Record<string, unknown>;+}++describe("entry loading (real Node loader)", () => {+ it("gives repo code the worker's framefields, even with none installed", async () => {+ await write(+ "src/film.ts",+ `import { Composition } from "framefields";+ const n: number = 1;+ export const Ctor = Composition;+ export const buildFilm = () => new Composition({ width: 64, height: 64, fps: 24, durationFrames: n });`,+ );+ expect(harness("src/film.ts", "job-a", "buildFilm")).toMatchObject({+ ok: true,+ sameComposition: true,+ plan: [{ name: "main", isWorkerComposition: true }],+ });+ }, 60_000);++ it('gives the same instance to a package without "type": "module"', async () => {+ await write("cjs/package.json", '{ "name": "legacy" }');+ await write(+ "cjs/film.ts",+ 'import { Composition } from "framefields"; export const Ctor = Composition;',+ );+ expect(harness("cjs/film.ts", "job-cjs")).toMatchObject({+ ok: true,+ sameComposition: true,+ });+ }, 60_000);++ it("loads changed code at the same path under a new job token", async () => {+ await write(+ "src/v.ts",+ 'import { dep } from "./dep-v.js"; export const v = dep;',+ );+ await write("src/dep-v.ts", "export const dep = 1;");+ expect(harness("src/v.ts", "job-b").v).toBe(1);+ await write("src/dep-v.ts", "export const dep = 2;");+ expect(harness("src/v.ts", "job-c").v).toBe(2);+ }, 60_000);++ it.each([+ ["src/missing.ts", "", "ENTRY_NOT_FOUND"],+ ["src/broken.ts", "export const = ;", "ENTRY_COMPILE_FAILED"],+ [+ "src/dep.ts",+ 'import x from "not-installed-anywhere"; export default x;',+ "ENTRY_COMPILE_FAILED",+ ],+ [+ "src/throws.ts",+ 'throw new Error("boom at import");',+ "COMPOSITION_BUILD_FAILED",+ ],+ [+ "src/quickstart.ts",+ "console.log('rendering…'); process.exit(0);",+ "EXPORT_SIDE_EFFECT",+ ],+ ["../outside.ts", "", "PATH_ESCAPE"],+ ])(+ "classifies %s",+ async (entry, source, code) => {+ if (source) await write(entry, source);+ expect(harness(entry, "job-d")).toMatchObject({ ok: false, code });+ },+ 60_000,+ );+});++describe("buildPlan", () => {+ it("accepts a composition, a record, and a render plan", async () => {+ expect(+ (+ await buildPlan({ default: fake() }, "default", undefined, undefined)+ ).map((o) => o.name),+ ).toEqual(["main"]);+ expect(+ (+ await buildPlan(+ { films: { main: fake(), teaser: fake() } },+ "films",+ undefined,+ ["teaser"],+ )+ ).map((o) => o.name),+ ).toEqual(["teaser"]);+ const plan = await buildPlan(+ {+ plan: async (args: { n: number }) => ({+ outputs: [+ {+ name: `cut-${args.n}`,+ composition: fake(),+ render: { kind: "frames", frames: [1] },+ },+ ],+ }),+ },+ "plan",+ { n: 3 },+ undefined,+ );+ expect(plan[0]).toMatchObject({+ name: "cut-3",+ render: { kind: "frames" },+ });+ });++ it.each([+ [{ a: 1 }, "nope", "EXPORT_NOT_FOUND"],+ [{ a: 42 }, "a", "EXPORT_INVALID"],+ [{ a: { x: fake(), y: 1 } }, "a", "EXPORT_INVALID"],+ [+ { a: { outputs: [{ name: "../evil", composition: fake() }] } },+ "a",+ "EXPORT_INVALID",+ ],+ [+ {+ a: () => {+ throw new Error("bad args");+ },+ },+ "a",+ "COMPOSITION_BUILD_FAILED",+ ],+ ])("rejects %o (export %s)", async (mod, name, code) => {+ expect(+ await codeOf(+ buildPlan(mod as Record<string, unknown>, name, undefined, undefined),+ ),+ ).toBe(code);+ });++ it("rejects unknown select names", async () => {+ expect(+ await codeOf(+ buildPlan({ f: { main: fake() } }, "f", undefined, ["nope"]),+ ),+ ).toBe("JOB_INVALID");+ });++ it("recognises renderables by shape", () => {+ expect(isRenderable(fake())).toBe(true);+ expect(isRenderable({ renderVideo: 1 })).toBe(false);+ });+});++describe("job sandbox", () => {+ it("sets and restores cwd, env, argv and process.exit", () => {+ const before = {+ cwd: process.cwd(),+ argv: process.argv,+ exit: process.exit,+ };+ const sandbox = openSandbox({+ cwd: ws,+ argv1: path.join(ws, "src/film.ts"),+ env: { LOCALE: "tr" },+ });+ expect(process.cwd()).toBe(ws);+ expect(process.env.LOCALE).toBe("tr");+ expect(process.argv[1]).toBe(path.join(ws, "src/film.ts"));+ expect(() => process.exit(1)).toThrow(ExitTrap);+ sandbox.restore();+ expect(process.cwd()).toBe(before.cwd);+ expect(process.env.LOCALE).toBeUndefined();+ expect(process.argv).toBe(before.argv);+ expect(process.exit).toBe(before.exit);+ });++ it("refuses to nest", () => {+ const sandbox = openSandbox({ cwd: ws, argv1: "x", env: {} });+ try {+ expect(() => openSandbox({ cwd: ws, argv1: "x", env: {} })).toThrow(+ /already open/,+ );+ } finally {+ sandbox.restore();+ }+ });+});++describe("render error classification", () => {+ it.each([+ ["GPUDevice was lost: reason unknown", "GPU_DEVICE_LOST"],+ ["Failed to allocate texture: out of memory", "RENDER_OOM"],+ ['Cannot open local media file "/ws/assets/a.mp4"', "ASSET_NOT_FOUND"],+ ["UnsupportedInputFormatError: cannot detect", "ASSET_NOT_FOUND"],+ ["something else", "RENDER_FAILED"],+ ])("%s → %s", (message, code) => {+ expect(classifyRenderError(new Error(message), "/ws").code).toBe(code);+ });++ it("keeps only project stack frames, repo-relative", () => {+ const err = new Error("x");+ err.stack =+ "Error: x\n at a (/ws/src/film.ts:3:1)\n at b (/ws/node_modules/z/i.js:1:1)\n at c (node:internal/x:1:1)";+ expect(classifyRenderError(err, "/ws").details?.stack).toBe(+ "at a (<repo>/src/film.ts:3:1)",+ );+ });+});++describe("recycling", () => {+ const config = testConfig({+ FF_MAX_JOBS_PER_WORKER: "3",+ FF_RECYCLE_HEAP_MB: "100",+ });+ const stats = {+ completedJobs: 1,+ heapUsedMb: 50,+ rssMb: 200,+ strayErrors: 0,+ leakedHandles: 0,+ };+ it.each([+ [{}, undefined],+ [{ completedJobs: 3 }, "completed 3 jobs"],+ [{ heapUsedMb: 150 }, "heap 150MB > 100MB"],+ [{ strayErrors: 3 }, "3 errors from repo code outside any job"],+ [{ leakedHandles: 51 }, "51 timers/handles left behind by repo code"],+ [{ requested: "GPU device lost" }, "GPU device lost"],+ ])("%o → %s", (patch, reason) => {+ expect(recycleReason({ ...stats, ...patch }, config)).toBe(reason);+ });+});++describe("event-loop watchdog", () => {+ it("kills a process stuck in a sync loop and records why", async () => {+ const journalPath = path.join(ws, "watchdog.json");+ const tsx = path.resolve(+ import.meta.dirname,+ "../../node_modules/.bin/tsx",+ );+ const script = path.resolve(+ import.meta.dirname,+ "../../test/watchdog-harness.ts",+ );+ let killed = false;+ try {+ execFileSync(tsx, [script, journalPath], {+ timeout: 30_000,+ stdio: "ignore",+ });+ } catch (err) {+ // tsx's CLI wrapper reports its child's SIGKILL as exit status 137.+ const e = err as { signal: string | null; status: number | null };+ killed = e.signal === "SIGKILL" || e.status === 137;+ }+ expect(killed).toBe(true);+ const record = new CrashJournal(journalPath).recover();+ expect(record?.error?.message).toMatch(/event loop blocked/);+ }, 40_000);+});++describe("crash journal", () => {+ it("reports the job a dead process left behind, once", async () => {+ const journal = new CrashJournal(path.join(ws, "inflight.json"));+ journal.begin({+ jobId: "j1",+ createdAt: "2026-01-01T00:00:00Z",+ startedAt: "2026-01-01T00:00:01Z",+ });+ const record = journal.recover();+ expect(record).toMatchObject({+ id: "j1",+ status: "FAILED",+ error: { code: "RENDER_CRASHED", retryable: true },+ });+ expect(journal.recover()).toBeUndefined();++ journal.begin({+ jobId: "j2",+ createdAt: "x",+ startedAt: "y",+ exitReason: "JavaScript heap out of memory",+ });+ expect(journal.recover()?.error?.code).toBe("RENDER_OOM");++ journal.begin({ jobId: "j3", createdAt: "x", startedAt: "y" });+ journal.end();+ expect(journal.recover()).toBeUndefined();+ });+});diff --git a/apps/renderer-service/src/render/sandbox.ts b/apps/renderer-service/src/render/sandbox.tsnew file mode 100644index 00000000..486b8eee--- /dev/null+++ b/apps/renderer-service/src/render/sandbox.ts@@ -0,0 +1,105 @@+import { RenderError } from "../errors.js";++/**+ * Thrown in place of exiting when repo code calls process.exit() while a job+ * runs in the worker (PLAN.md §8.2).+ */+export class ExitTrap extends RenderError {+ constructor(code: number | string | null | undefined) {+ super(+ "EXPORT_SIDE_EFFECT",+ `Repo code called process.exit(${code ?? ""}) while the worker was loading or rendering it.`,+ {+ phase: "load",+ hint: "Export a builder (a Composition or a function returning one); don't render or exit at import time.",+ },+ );+ }+}++export interface SandboxOptions {+ cwd: string;+ argv1: string;+ env: Record<string, string>;+}++export interface Sandbox {+ /** Rejects when repo code fails out of band (stray rejection / exception). */+ readonly failed: Promise<never>;+ restore(): void;+}++let active: { fail(err: unknown): void } | undefined;+let strayErrors = 0;++/** Errors raised by repo code after its job finished. */+export function strayErrorCount(): number {+ return strayErrors;+}++/**+ * Process-level handlers, installed once at boot: an unhandled rejection or+ * exception fails the active job instead of crashing the worker. With no job+ * running it is logged and counted (recycle trigger).+ */+export function installProcessGuards(+ log: (msg: string, err: unknown) => void,+): void {+ const onError = (err: unknown) => {+ if (active) active.fail(err);+ else {+ strayErrors++;+ log("Error outside any job", err);+ }+ };+ process.on("unhandledRejection", onError);+ process.on("uncaughtException", onError);+}++/**+ * Prepares the shared process for one job and returns how to undo it. Only+ * one sandbox can be open at a time (the queue runs one job at a time).+ */+export function openSandbox(options: SandboxOptions): Sandbox {+ if (active) throw new Error("A job sandbox is already open.");++ const previousCwd = process.cwd();+ const previousArgv = process.argv;+ const previousExit = process.exit;+ const previousEnv = new Map<string, string | undefined>();++ for (const [key, value] of Object.entries(options.env)) {+ previousEnv.set(key, process.env[key]);+ process.env[key] = value;+ }+ process.chdir(options.cwd);+ process.argv = [previousArgv[0], options.argv1];+ process.exit = ((code?: number | string | null) => {+ throw new ExitTrap(code);+ }) as typeof process.exit;++ let fail!: (err: unknown) => void;+ const failed = new Promise<never>((_, reject) => {+ fail = reject;+ });+ failed.catch(() => {}); // observed via race(); never unhandled+ active = { fail };++ return {+ failed,+ restore() {+ process.exit = previousExit;+ process.argv = previousArgv;+ try {+ process.chdir(previousCwd);+ } catch {+ process.chdir("/");+ }+ for (const [key, value] of previousEnv) {+ if (value === undefined) delete process.env[key];+ else process.env[key] = value;+ }+ active = undefined;+ },+ };+}diff --git a/apps/renderer-service/src/worker/health.ts b/apps/renderer-service/src/worker/health.tsnew file mode 100644index 00000000..ee0be2a8--- /dev/null+++ b/apps/renderer-service/src/worker/health.ts@@ -0,0 +1,18 @@+/**+ * Worker-level state that decides when the process should recycle itself+ * (PLAN.md §8.3). Recycling = stop taking jobs, finish the current one, exit.+ */+class WorkerHealth {+ private reason: string | undefined;++ /** Marks the worker as unable to safely run more jobs. */+ requestRecycle(reason: string): void {+ this.reason ??= reason;+ }++ get recycleReason(): string | undefined {+ return this.reason;+ }+}++export const workerHealth = new WorkerHealth();diff --git a/apps/renderer-service/src/worker/recycle.ts b/apps/renderer-service/src/worker/recycle.tsnew file mode 100644index 00000000..95169fb0--- /dev/null+++ b/apps/renderer-service/src/worker/recycle.ts@@ -0,0 +1,55 @@+import type { RendererConfig } from "../config/index.js";++export interface WorkerStats {+ completedJobs: number;+ heapUsedMb: number;+ rssMb: number;+ strayErrors: number;+ leakedHandles: number;+ requested?: string;+}++/**+ * Why this worker should stop taking jobs and restart, or undefined to keep+ * going (PLAN.md §8.3).+ */+export function recycleReason(+ stats: WorkerStats,+ config: RendererConfig,+): string | undefined {+ const r = config.recycle;+ if (stats.requested) return stats.requested;+ if (r.maxJobs > 0 && stats.completedJobs >= r.maxJobs)+ return `completed ${stats.completedJobs} jobs`;+ if (r.heapMb > 0 && stats.heapUsedMb > r.heapMb)+ return `heap ${stats.heapUsedMb}MB > ${r.heapMb}MB`;+ if (r.rssMb > 0 && stats.rssMb > r.rssMb)+ return `rss ${stats.rssMb}MB > ${r.rssMb}MB`;+ if (r.strayErrorLimit > 0 && stats.strayErrors >= r.strayErrorLimit) {+ return `${stats.strayErrors} errors from repo code outside any job`;+ }+ if (r.leakedHandleLimit > 0 && stats.leakedHandles > r.leakedHandleLimit) {+ return `${stats.leakedHandles} timers/handles left behind by repo code`;+ }+ return undefined;+}++export function currentStats(+ completedJobs: number,+ baselineHandles: number,+ strayErrors: number,+ requested: string | undefined,+): WorkerStats {+ const mem = process.memoryUsage();+ return {+ completedJobs,+ heapUsedMb: Math.round(mem.heapUsed / 1024 / 1024),+ rssMb: Math.round(mem.rss / 1024 / 1024),+ strayErrors,+ leakedHandles: Math.max(+ 0,+ process.getActiveResourcesInfo().length - baselineHandles,+ ),+ requested,+ };+}diff --git a/apps/renderer-service/src/worker/watchdog.ts b/apps/renderer-service/src/worker/watchdog.tsnew file mode 100644index 00000000..283ad719--- /dev/null+++ b/apps/renderer-service/src/worker/watchdog.ts@@ -0,0 +1,51 @@+import { Worker } from "node:worker_threads";++/**+ * Event-loop watchdog (PLAN.md §8.3). Repo code runs on the main thread, so+ * a synchronous infinite loop would freeze the worker forever. A helper+ * thread watches a heartbeat the main loop bumps every second; if it stops+ * for `blockSec`, the thread writes the crash journal and kills the process+ * so the supervisor (RunPod, Docker, k8s) starts a fresh one.+ */+const WATCHER = `+const { workerData } = require("node:worker_threads");+const fs = require("node:fs");+const beat = new Int32Array(workerData.buffer);+let last = Atomics.load(beat, 0);+let lastChange = Date.now();+setInterval(() => {+ const now = Atomics.load(beat, 0);+ if (now !== last) { last = now; lastChange = Date.now(); return; }+ if (Date.now() - lastChange < workerData.blockMs) return;+ try {+ const journal = JSON.parse(fs.readFileSync(workerData.journal, "utf8"));+ journal.exitReason = "event loop blocked for " + Math.round((Date.now() - lastChange) / 1000) + "s";+ fs.writeFileSync(workerData.journal, JSON.stringify(journal));+ } catch {}+ process.kill(process.pid, "SIGKILL");+}, 1000);+`;++export function startWatchdog(options: {+ blockSec: number;+ journalPath: string;+}): () => void {+ if (options.blockSec <= 0) return () => {};+ const buffer = new SharedArrayBuffer(4);+ const beat = new Int32Array(buffer);+ const heartbeat = setInterval(() => Atomics.add(beat, 0, 1), 1000);+ heartbeat.unref();+ const worker = new Worker(WATCHER, {+ eval: true,+ workerData: {+ buffer,+ blockMs: options.blockSec * 1000,+ journal: options.journalPath,+ },+ });+ worker.unref();+ return () => {+ clearInterval(heartbeat);+ void worker.terminate();+ };+}diff --git a/apps/renderer-service/test/load-harness.ts b/apps/renderer-service/test/load-harness.tsnew file mode 100644index 00000000..cd5e1b57--- /dev/null+++ b/apps/renderer-service/test/load-harness.ts@@ -0,0 +1,47 @@+/**+ * Runs the entry loader in a real Node process (vitest's VM can't do native+ * dynamic imports through loader hooks). Usage:+ * tsx test/load-harness.ts <workspace> <entry> <jobToken> [export]+ * Prints one JSON line describing the outcome.+ */+import * as worker from "framefields";+import { RenderError } from "../src/errors.js";+import { installEngineAlias } from "../src/render/engine-alias.js";+import { buildPlan, importEntry } from "../src/render/load.js";+import { openSandbox } from "../src/render/sandbox.js";++const [ws, entry, token, exportName] = process.argv.slice(2);+installEngineAlias([ws]);++const report = (value: unknown) => {+ process.stdout.write(`${JSON.stringify(value)}\n`);+};++const sandbox = openSandbox({ cwd: ws, argv1: entry, env: {} });+try {+ const { mod, dispose } = await importEntry(ws, entry, token);+ const out: Record<string, unknown> = {+ ok: true,+ exports: Object.keys(mod).sort(),+ v: mod.v,+ sameComposition:+ mod.Ctor === undefined ? undefined : mod.Ctor === worker.Composition,+ };+ if (exportName) {+ const plan = await buildPlan(mod, exportName, undefined, undefined);+ out.plan = plan.map((o) => ({+ name: o.name,+ isWorkerComposition: o.target instanceof worker.Composition,+ }));+ }+ await dispose();+ report(out);+} catch (err) {+ report({+ ok: false,+ code: err instanceof RenderError ? err.code : undefined,+ message: err instanceof Error ? err.message : String(err),+ });+} finally {+ sandbox.restore();+}diff --git a/apps/renderer-service/test/render-harness.ts b/apps/renderer-service/test/render-harness.tsnew file mode 100644index 00000000..b1da1af5--- /dev/null+++ b/apps/renderer-service/test/render-harness.ts@@ -0,0 +1,167 @@+/**+ * Runs the in-process render step for several jobs back to back in one real+ * Node process, the way a worker does. Usage:+ * tsx test/render-harness.ts <workDir>+ * Prints one JSON line with each scenario's outcome. Needs a GPU.+ */+import fs from "node:fs/promises";+import path from "node:path";+import { Composition } from "framefields";+import { buildConfig } from "../src/config/schema.js";+import { RenderError } from "../src/errors.js";+import type { CheckoutResult, StepContext } from "../src/jobs/pipeline.js";+import { jobSchema } from "../src/jobs/schema.js";+import { installEngineAlias } from "../src/render/engine-alias.js";+import { render } from "../src/render/render-step.js";+import { installProcessGuards } from "../src/render/sandbox.js";++const workDir = path.resolve(process.argv[2]);+const ws = path.join(workDir, "ws", "repo", "0");+await fs.mkdir(path.join(ws, "src"), { recursive: true });+await fs.mkdir(path.join(ws, "assets"), { recursive: true });+installEngineAlias([await fs.realpath(path.join(workDir, "ws"))]);+installProcessGuards(() => {});++const built = buildConfig(+ {+ FF_MODE: "cli",+ FF_REPO_ALLOWLIST: "github.com/acme/*",+ FF_WORK_DIR: workDir,+ FF_CACHE_DIR: path.join(workDir, "cache"),+ FF_KILL_GRACE_SEC: "2",+ FF_STALL_TIMEOUT_SEC: "30",+ },+ { exists: () => false, tmpdir: () => workDir, homedir: () => workDir },+);+if (!built.ok) throw new Error(JSON.stringify(built.problems));+const config = built.config;+const root = await fs.realpath(ws);+const checkout: CheckoutResult = {+ root,+ workdir: root,+ sha: "x",+ cache: {},+ release: async () => {},+};++async function clip(color: string) {+ const comp = new Composition({+ width: 64,+ height: 64,+ fps: 24,+ durationFrames: 24,+ backgroundColor: color,+ });+ const out = await comp.renderVideo({});+ await fs.copyFile(out.filePath, path.join(ws, "assets/clip.mp4"));+ await out.cleanup();+}++async function job(+ name: string,+ target: Record<string, unknown>,+ render_: Record<string, unknown> = {},+ abortAfterMs?: number,+) {+ const input = jobSchema.parse({+ version: 2,+ source: { repo: "https://github.com/acme/films" },+ target,+ render: render_,+ output: { type: "inline" },+ });+ const controller = new AbortController();+ const warnings: string[] = [];+ const ctx: StepContext = {+ jobId: name,+ config,+ signal: controller.signal,+ jobDir: path.join(workDir, "jobs", name),+ progress: () => {},+ warn: (code) => warnings.push(code),+ };+ if (abortAfterMs !== undefined) {+ setTimeout(+ () => controller.abort(new RenderError("CANCELLED", "test cancel")),+ abortAfterMs,+ );+ }+ try {+ const result = await render(checkout, input, ctx);+ const files = await Promise.all(+ result.files.map(async (f) => ({+ name: f.name,+ ext: f.ext,+ bytes: (await fs.stat(f.path)).size,+ path: f.path,+ })),+ );+ return { ok: true, files, warnings, cwdRestored: process.cwd() !== root };+ } catch (err) {+ return {+ ok: false,+ code: err instanceof RenderError ? err.code : String(err),+ warnings,+ cwdRestored: process.cwd() !== root,+ };+ }+}++await fs.writeFile(+ path.join(ws, "package.json"),+ '{ "name": "film", "type": "module" }',+);+await fs.writeFile(+ path.join(ws, "src/film.ts"),+ `import { Composition, Layer } from "framefields";+ export function buildFilm(args?: { color?: string }) {+ const comp = new Composition({ width: 64, height: 64, fps: 24, durationFrames: 24, backgroundColor: args?.color ?? "#000000" });+ comp.add(Layer.video("assets/clip.mp4", { width: 32, height: 32, muted: true }));+ return comp;+ }+ export const cuts = () => ({ main: buildFilm(), teaser: buildFilm({ color: "#ff0000" }) });+ export const slow = () => new Composition({ width: 640, height: 360, fps: 30, durationFrames: 3000, backgroundColor: "#123456" });`,+);+await fs.writeFile(path.join(ws, "src/exits.ts"), "process.exit(3);");++const results: Record<string, unknown> = {};+await clip("#00ff00");+results.video1 = await job("video1", {+ entry: "src/film.ts",+ export: "buildFilm",+});+// Same asset path, different bytes: the next job must decode the new file.+await clip("#0000ff");+results.video2 = await job("video2", {+ entry: "src/film.ts",+ export: "buildFilm",+});+results.frames = await job(+ "frames",+ { entry: "src/film.ts", export: "cuts", select: ["teaser"] },+ { kind: "frames", frames: [0, 12] },+);+results.exits = await job("exits", { entry: "src/exits.ts" });+results.cancel = await job(+ "cancel",+ { entry: "src/film.ts", export: "slow" },+ {},+ 1500,+);+results.after = await job("after", {+ entry: "src/film.ts",+ export: "buildFilm",+});++const bytes = async (r: unknown) => {+ const f = (r as { files?: { path: string }[] }).files?.[0];+ return f ? fs.readFile(f.path) : undefined;+};+const [a, b] = await Promise.all([+ bytes(results.video1),+ bytes(results.video2),+]);+results.videosDiffer = Boolean(a && b && Buffer.compare(a, b) !== 0);++process.stdout.write(`${JSON.stringify(results)}\n`);+process.exit(0);diff --git a/apps/renderer-service/test/watchdog-harness.ts b/apps/renderer-service/test/watchdog-harness.tsnew file mode 100644index 00000000..7b1490f1--- /dev/null+++ b/apps/renderer-service/test/watchdog-harness.ts@@ -0,0 +1,11 @@+/** Blocks the event loop under the watchdog. Usage: tsx test/watchdog-harness.ts <journal> */+import { CrashJournal } from "../src/jobs/journal.js";+import { startWatchdog } from "../src/worker/watchdog.js";++const journal = new CrashJournal(process.argv[2]);+journal.begin({ jobId: "stuck", createdAt: "x", startedAt: "y" });+startWatchdog({ blockSec: 2, journalPath: journal.path });+setTimeout(() => {+ // Repo code stuck in a synchronous loop.+ for (;;) {}+}, 1500);diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 8979859f..1c463a15 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -202,6 +202,9 @@ importers:semver:specifier: ^7.8.5version: 7.8.5+ tsx:+ specifier: ^4.21.0+ version: 4.21.0zod:specifier: ^4.3.6version: 4.3.6@@ -215,9 +218,6 @@ importers:'@types/semver':specifier: ^7.8.0version: 7.8.0- tsx:- specifier: ^4.20.3- version: 4.21.0typescript:specifier: ^5.8.3version: 5.9.3