Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(renderer-service): render repo code in-process (phase 5)

okanasl0cc02119fca6
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.md
index 2852ccae..77c61b4a 100644
--- a/apps/renderer-service/PLAN.md
+++ b/apps/renderer-service/PLAN.md
@@ -325,9 +325,12 @@ Rules that apply to every host:
### 8.1 Loading the entry
1. Resolve `entry` with `realpath` and check that it lies inside the workspace (`PATH_ESCAPE` otherwise, which also covers symlink escapes).
-2. Import it with tsx's `tsImport(entryUrl, { parentURL, tsconfig })`.
- - Every call gets its own module namespace, so job N+1 never sees job N's cached modules. A fix pushed to the same file is picked up even when the path is identical.
+2. Import it with a per-job tsx loader (`register({ tsconfig })`) and a native `import()` of the entry URL plus `?ff-job=<jobId>`.
+ - The engine alias hook passes that query on to every workspace module the entry imports, so job N+1 never sees job N's cached modules. A fix pushed to the same file is picked up even when the path is identical.
+ - Engine modules get no query, so they stay the worker's single instance. tsx's `tsImport` was rejected because its namespace also applies to `framefields`, which loaded a second engine copy per job.
- The project's `tsconfig.json` is honoured, including `paths`.
+ - Workspace `.ts/.tsx/.js/.jsx` files always load as ESM, even without `"type": "module"`. In CommonJS mode, tsx transpiles framefields' `.mjs` into a second CJS copy of the engine. `.cjs`/`.cts` stay CommonJS, and their `require.cache` entries are cleared after each job.
+ - The alias uses `module.registerHooks` (synchronous, in-thread), which covers both `import` and `require()`.
- A missing module gives `ENTRY_NOT_FOUND`. A TS or syntax error gives `ENTRY_COMPILE_FAILED`, with the first frames of the error.
3. Read the export (`EXPORT_NOT_FOUND`). If it is a function, call it with `args`. A throw here gives `COMPOSITION_BUILD_FAILED`.
4. Normalise the result to a render plan (section 4). Anything that isn't a Composition, a record of compositions, or a plan gives `EXPORT_INVALID`.
@@ -514,8 +517,12 @@ Each phase is its own PR-sized step with tests.
- `resetRenderCaches()` (`@framefields/renderer`) clears media inputs and decoders, textures, LUTs and fonts. The worker calls it plus `FontManager.clear()` before each job. The GPU device and pipelines stay warm.
- A GPU test (`src/engine.gpu.test.ts`, run with `FF_GPU_TESTS=1`) confirmed that without the reset, a changed file at the same workspace path is served stale.
- Not yet verified: re-opening a *video* file at the same path after the reset. A comment in `renderImage` says a mediabunny `Input` can only be created once per URL. Phase 5 needs a GPU test that renders a video layer in two jobs back to back.
-5. **InProcessRenderer.**
- - The engine alias resolve hook, `tsImport` loading, export-shape normalisation, the job sandbox (8.2), the watchdog, the crash journal, and recycling.
+5. **InProcessRenderer.** ✅ Done: the engine alias, per-job loading, export-shape normalisation, the job sandbox, stall detection, the abort grace period, the watchdog, the crash journal, and recycling. RunPod gets `refresh_worker` when the worker is draining.
+ - Tests that need Node's real loader run in a subprocess (`test/load-harness.ts`, `test/render-harness.ts`), because vitest's VM bypasses loader hooks.
+ - GPU-verified: back-to-back jobs, a changed video asset at the same path, frames with `select`, `process.exit` at import, cancel mid-render, and recovery afterwards.
+ - Known limits:
+ - Each job registers another tsx loader layer (deactivated afterwards). Recycling every `FF_MAX_JOBS_PER_WORKER` jobs bounds that.
+ - `console.*` from repo code is not yet captured into a per-job log.
6. **Outputs.** The s3, presigned, local and inline sinks, verification (ffprobe), multi-output keys, and idempotency.
7. **Transports.**
- HTTP v2 with SSE progress and cancel.
diff --git a/apps/renderer-service/package.json b/apps/renderer-service/package.json
index f80818ca..9d793088 100644
--- a/apps/renderer-service/package.json
+++ b/apps/renderer-service/package.json
@@ -25,13 +25,13 @@
"framefields": "workspace:*",
"hono": "^4.13.7",
"semver": "^7.8.5",
+ "tsx": "^4.21.0",
"zod": "^4.3.6"
},
"devDependencies": {
"@framefields/tsconfig": "workspace:*",
"@types/node": "^24.1.0",
"@types/semver": "^7.8.0",
- "tsx": "^4.20.3",
"typescript": "^5.8.3",
"vitest": "^4.1.5"
}
diff --git a/apps/renderer-service/runpod_worker.py b/apps/renderer-service/runpod_worker.py
index e39b54e4..a5d316af 100644
--- a/apps/renderer-service/runpod_worker.py
+++ b/apps/renderer-service/runpod_worker.py
@@ -51,6 +51,15 @@ def wait_ready():
raise RuntimeError("Renderer did not become healthy in time")
+def draining():
+ """True when the worker decided to recycle (job count, memory, GPU loss)."""
+ try:
+ status, body = call("GET", "/health")
+ return status != 200 and (body or {}).get("status") == "draining"
+ except (urllib.error.URLError, ConnectionError):
+ return True
+
+
def failure(error, refresh=False):
"""RunPod marks a job failed when the output has an `error` key."""
out = {"error": json.dumps(error), "refresh_worker": refresh}
@@ -77,11 +86,14 @@ def handler(job):
last_progress = progress
if record["status"] == "COMPLETED":
- return record["result"]
+ result = dict(record["result"])
+ if draining():
+ result["refresh_worker"] = True
+ return result
if record["status"] in ("FAILED", "CANCELLED"):
error = record.get("error") or {"code": "INTERNAL", "message": "unknown failure"}
# Device loss / crashes leave the worker unhealthy: ask for a fresh one.
- refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED")
+ refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED") or draining()
return failure(error, refresh=refresh)
diff --git a/apps/renderer-service/src/engine.gpu.test.ts b/apps/renderer-service/src/engine.gpu.test.ts
index cdd7c6ba..e3ffb4e9 100644
--- a/apps/renderer-service/src/engine.gpu.test.ts
+++ b/apps/renderer-service/src/engine.gpu.test.ts
@@ -125,3 +125,56 @@ gpu("engine hooks", () => {
expect(Buffer.compare(red, green)).not.toBe(0);
}, 120_000);
});
+
+gpu("in-process render step (real Node loader)", () => {
+ it("renders several jobs back to back in one process", async () => {
+ const { execFileSync } = await import("node:child_process");
+ const workDir = await fs.mkdtemp(path.join(os.tmpdir(), "ff-render-e2e-"));
+ try {
+ const out = execFileSync(
+ path.resolve(import.meta.dirname, "../node_modules/.bin/tsx"),
+ [
+ path.resolve(import.meta.dirname, "../test/render-harness.ts"),
+ workDir,
+ ],
+ {
+ encoding: "utf8",
+ env: { ...process.env, NODE_ENV: "test" },
+ timeout: 300_000,
+ },
+ );
+ const line = out
+ .trim()
+ .split("\n")
+ .filter((l) => l.startsWith('{"video1"'))
+ .at(-1);
+ const r = JSON.parse(line ?? "{}");
+ expect(r.video1).toMatchObject({
+ ok: true,
+ files: [{ name: "main", ext: "mp4" }],
+ });
+ // Same asset path with new bytes in the next job: decoded fresh.
+ expect(r.video2.ok).toBe(true);
+ expect(r.videosDiffer).toBe(true);
+ expect(r.frames).toMatchObject({
+ ok: true,
+ files: [{ name: "teaser-f0" }, { name: "teaser-f12" }],
+ });
+ expect(r.exits).toMatchObject({ ok: false, code: "EXPORT_SIDE_EFFECT" });
+ expect(r.cancel).toMatchObject({ ok: false, code: "CANCELLED" });
+ expect(r.after.ok).toBe(true);
+ for (const k of [
+ "video1",
+ "video2",
+ "frames",
+ "exits",
+ "cancel",
+ "after",
+ ]) {
+ expect(r[k].cwdRestored).toBe(true);
+ }
+ } finally {
+ await fs.rm(workDir, { recursive: true, force: true });
+ }
+ }, 300_000);
+});
diff --git a/apps/renderer-service/src/index.ts b/apps/renderer-service/src/index.ts
index 3eb1d9a3..3c158820 100644
--- a/apps/renderer-service/src/index.ts
+++ b/apps/renderer-service/src/index.ts
@@ -1,4 +1,6 @@
+import { mkdirSync, realpathSync } from "node:fs";
import { Server as HttpServer } from "node:http";
+import path from "node:path";
import { HeadlessMediaRenderer } from "@framefields/renderer";
import "@framefields/renderer/offscreen-gl-polyfill";
import { rendererLogger } from "@framefields/server-utils";
@@ -8,9 +10,21 @@ import { RenderError } from "./errors.js";
import { checkout } from "./git/checkout.js";
import { createApp } from "./http/app.js";
import { install } from "./install/installer.js";
+import { CrashJournal } from "./jobs/journal.js";
import type { Pipeline } from "./jobs/pipeline.js";
import { JobRunner } from "./jobs/runner.js";
import { ReadinessTracker } from "./readiness.js";
+import { redact } from "./redact.js";
+import { installEngineAlias } from "./render/engine-alias.js";
+import { render } from "./render/render-step.js";
+import { installProcessGuards, strayErrorCount } from "./render/sandbox.js";
+import { workerHealth } from "./worker/health.js";
+import { currentStats, recycleReason } from "./worker/recycle.js";
+import { startWatchdog } from "./worker/watchdog.js";
+
+// The job sandbox swaps process.exit while repo code runs; the worker itself
+// always exits through the real one.
+const exit = process.exit.bind(process);
const config = loadConfig();
scrubSecretEnv();
@@ -19,6 +33,23 @@ rendererLogger.info(
"Renderer config loaded",
);
+// Workspaces live here; repo code under it gets the engine alias (§7).
+const workspaces = path.join(config.paths.workDir, "ws");
+mkdirSync(workspaces, { recursive: true });
+mkdirSync(config.paths.cacheDir, { recursive: true });
+installEngineAlias([realpathSync(workspaces)]);
+
+installProcessGuards((msg, err) =>
+ rendererLogger.error(
+ {
+ err: redact(
+ err instanceof Error ? (err.stack ?? err.message) : String(err),
+ ),
+ },
+ msg,
+ ),
+);
+
const readiness = new ReadinessTracker();
let draining = false;
@@ -32,11 +63,43 @@ const notYet = (step: string): never => {
const pipeline: Pipeline = {
checkout,
install,
- render: async () => notYet("render"),
+ render,
deliver: async () => notYet("deliver"),
};
-const runner = new JobRunner(config, pipeline);
+const journal = new CrashJournal(
+ path.join(config.paths.workDir, "inflight.json"),
+);
+let baselineHandles = 0;
+
+const runner = new JobRunner(config, pipeline, {
+ journal,
+ afterJob: (completed) => {
+ const reason = recycleReason(
+ currentStats(
+ completed,
+ baselineHandles,
+ strayErrorCount(),
+ workerHealth.recycleReason,
+ ),
+ config,
+ );
+ if (reason) void recycle(reason);
+ },
+});
+
+const crashed = journal.recover();
+if (crashed) {
+ runner.store.restore(crashed);
+ rendererLogger.error(
+ { jobId: crashed.id, error: crashed.error },
+ "Previous process died during a job",
+ );
+}
+const stopWatchdog = startWatchdog({
+ blockSec: config.recycle.eventLoopBlockSec,
+ journalPath: journal.path,
+});
const app = createApp(config, runner, {
health: () => {
@@ -59,6 +122,9 @@ const app = createApp(config, runner, {
waiting: runner.queue.depth,
},
...(status.error && { error: status.error }),
+ ...(workerHealth.recycleReason && {
+ recycle: workerHealth.recycleReason,
+ }),
},
};
},
@@ -87,6 +153,7 @@ async function bootstrap() {
await HeadlessMediaRenderer.initialize();
readiness.setRendererReady(true);
readiness.setGpuReady(true);
+ baselineHandles = process.getActiveResourcesInfo().length;
rendererLogger.info("Renderer initialized.");
} catch (error) {
const errMsg = error instanceof Error ? error.message : String(error);
@@ -96,12 +163,33 @@ async function bootstrap() {
);
readiness.setFailure(`GPU_UNAVAILABLE: ${errMsg}`);
// RunPod should replace a worker without a usable GPU.
- if (config.mode === "runpod-serverless") process.exit(1);
+ if (config.mode === "runpod-serverless") exit(1);
}
}
void bootstrap();
+/**
+ * Stops taking jobs and restarts the process once the queue is empty. On
+ * RunPod serverless the shim asks for a fresh worker (refresh_worker) when
+ * it sees /health report "draining", so the process just stays drained.
+ */
+async function recycle(reason: string) {
+ if (draining) return;
+ draining = true;
+ rendererLogger.warn({ reason }, "Recycling worker");
+ runner.queue.close(
+ new RenderError("SHUTDOWN", `Worker is recycling (${reason}).`),
+ false,
+ );
+ await runner.queue.idle();
+ if (config.mode !== "runpod-serverless") {
+ stopWatchdog();
+ server.close();
+ exit(0);
+ }
+}
+
// ─── Graceful Shutdown ──────────────────────────────────────────────────────────
let shuttingDown = false;
@@ -125,14 +213,15 @@ async function gracefulShutdown(signal: string) {
const graceMs = (config.limits.killGraceSec + 5) * 1000;
const forceExit = setTimeout(() => {
rendererLogger.error("Shutdown timed out, forcing exit.");
- process.exit(1);
+ exit(1);
}, graceMs);
await runner.queue.idle();
+ stopWatchdog();
server.close();
clearTimeout(forceExit);
rendererLogger.info("Shutdown complete.");
- process.exit(0);
+ exit(0);
}
process.on("SIGTERM", () => void gracefulShutdown("SIGTERM"));
diff --git a/apps/renderer-service/src/jobs/journal.ts b/apps/renderer-service/src/jobs/journal.ts
new file mode 100644
index 00000000..78dcc417
--- /dev/null
+++ b/apps/renderer-service/src/jobs/journal.ts
@@ -0,0 +1,65 @@
+import { readFileSync, rmSync, writeFileSync } from "node:fs";
+import { RenderError } from "../errors.js";
+import type { JobRecord } from "./store.js";
+
+/**
+ * The running job, on local disk, so a process that dies mid-render (OOM,
+ * native crash, watchdog kill) can still report it after restart (§8.3).
+ * Sync I/O on purpose: it must land before repo code gets to run.
+ */
+export interface JournalEntry {
+ jobId: string;
+ startedAt: string;
+ createdAt: string;
+ /** Set by the watchdog right before it kills the process. */
+ exitReason?: string;
+}
+
+export class CrashJournal {
+ constructor(private readonly file: string) {}
+
+ get path(): string {
+ return this.file;
+ }
+
+ begin(entry: JournalEntry): void {
+ writeFileSync(this.file, JSON.stringify(entry));
+ }
+
+ end(): void {
+ rmSync(this.file, { force: true });
+ }
+
+ /**
+ * Returns the job a previous process died on, as a failed record, and
+ * clears the journal. Undefined when the last shutdown was clean.
+ */
+ recover(): JobRecord | undefined {
+ let entry: JournalEntry;
+ try {
+ entry = JSON.parse(readFileSync(this.file, "utf8")) as JournalEntry;
+ } catch {
+ return undefined;
+ } finally {
+ rmSync(this.file, { force: true });
+ }
+ const oom = /heap|memory/i.test(entry.exitReason ?? "");
+ const error = new RenderError(
+ oom ? "RENDER_OOM" : "RENDER_CRASHED",
+ `The worker process died during this job${entry.exitReason ? ` (${entry.exitReason})` : ""}.`,
+ { phase: "render", hint: "A fresh worker can retry it." },
+ );
+ const now = new Date().toISOString();
+ return {
+ id: entry.jobId,
+ status: "FAILED",
+ progress: { phase: "done" },
+ createdAt: entry.createdAt,
+ startedAt: entry.startedAt,
+ updatedAt: now,
+ finishedAt: now,
+ error: error.toJSON(),
+ warnings: [],
+ };
+ }
+}
diff --git a/apps/renderer-service/src/jobs/runner.ts b/apps/renderer-service/src/jobs/runner.ts
index 07eb45c2..ed3dd636 100644
--- a/apps/renderer-service/src/jobs/runner.ts
+++ b/apps/renderer-service/src/jobs/runner.ts
@@ -5,6 +5,7 @@ import { loggerContext, rendererLogger } from "@framefields/server-utils";
import type { RendererConfig } from "../config/index.js";
import { type JobPhase, RenderError, toRenderError } from "../errors.js";
import { redact, redactValue } from "../redact.js";
+import type { CrashJournal } from "./journal.js";
import type { Pipeline, StepContext } from "./pipeline.js";
import { JobQueue } from "./queue.js";
import { type JobRecord, JobStore } from "./store.js";
@@ -22,7 +23,10 @@ export class JobRunner {
constructor(
private readonly config: RendererConfig,
private readonly pipeline: Pipeline,
- private readonly hooks: { afterJob?: (completed: number) => void } = {},
+ private readonly hooks: {
+ afterJob?: (completed: number) => void;
+ journal?: CrashJournal;
+ } = {},
) {
this.store = new JobStore(
config.limits.jobTtlSec * 1000,
@@ -114,11 +118,16 @@ export class JobRunner {
throw toRenderError(controller.signal.reason, phase);
};
- this.store.update(id, {
+ const record = this.store.update(id, {
status: "IN_PROGRESS",
startedAt: new Date().toISOString(),
progress: { phase: "resolving" },
});
+ this.hooks.journal?.begin({
+ jobId: id,
+ createdAt: record?.createdAt ?? new Date().toISOString(),
+ startedAt: record?.startedAt ?? new Date().toISOString(),
+ });
rendererLogger.info(
{ jobId: id, repo: job.repo.id, ref: job.ref },
"Job started",
@@ -126,6 +135,7 @@ export class JobRunner {
let release: (() => Promise<void>) | undefined;
let succeeded = false;
+ let settle: (() => unknown) | undefined;
try {
await fs.mkdir(path.join(jobDir, "out"), { recursive: true });
@@ -163,34 +173,36 @@ export class JobRunner {
checkAbort();
timings.totalMs = Date.now() - started;
- this.store.update(id, {
- status: "COMPLETED",
- progress: { phase: "done" },
- result: redactValue({
- jobId: id,
- source: {
- repo: job.repo.id,
- ref: job.ref,
- sha: checkout.sha,
- workdir: job.input.source.workdir,
- },
- engine: {
- ...rendered.engine,
- projectDeclared: install.declaredEngine,
- },
- outputs,
- timings,
- cache,
- warnings,
- }),
- });
+ settle = () =>
+ this.store.update(id, {
+ status: "COMPLETED",
+ progress: { phase: "done" },
+ result: redactValue({
+ jobId: id,
+ source: {
+ repo: job.repo.id,
+ ref: job.ref,
+ sha: checkout.sha,
+ workdir: job.input.source.workdir,
+ },
+ engine: {
+ ...rendered.engine,
+ projectDeclared: install.declaredEngine,
+ },
+ outputs,
+ timings,
+ cache,
+ warnings,
+ }),
+ });
succeeded = true;
- rendererLogger.info({ jobId: id, timings }, "Job completed");
} catch (err) {
const reason = controller.signal.aborted ? controller.signal.reason : err;
- this.fail(id, toRenderError(reason, phase), { timings, cache });
+ settle = () =>
+ this.fail(id, toRenderError(reason, phase), { timings, cache });
} finally {
clearTimeout(timer);
+ this.hooks.journal?.end();
signal.removeEventListener("abort", onAbort);
await release?.().catch((err) =>
rendererLogger.warn(
@@ -202,7 +214,12 @@ export class JobRunner {
await fs.rm(jobDir, { recursive: true, force: true }).catch(() => {});
}
this.completedJobs++;
+ // Decide on recycling before the job is visibly finished, so a client
+ // that sees the result also sees /health report "draining".
this.hooks.afterJob?.(this.completedJobs);
+ settle?.();
+ if (succeeded)
+ rendererLogger.info({ jobId: id, timings }, "Job completed");
}
}
diff --git a/apps/renderer-service/src/render/engine-alias.ts b/apps/renderer-service/src/render/engine-alias.ts
new file mode 100644
index 00000000..a03d98a9
--- /dev/null
+++ b/apps/renderer-service/src/render/engine-alias.ts
@@ -0,0 +1,95 @@
+import { createRequire, registerHooks } from "node:module";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+
+/** Query parameter that scopes workspace ES modules to one job. */
+export const JOB_PARAM = "ff-job";
+
+let roots: string[] = [];
+let rootDirs: string[] = [];
+let engineParent = "";
+let installed = false;
+
+const isEngine = (s: string) =>
+ s === "framefields" ||
+ s.startsWith("framefields/") ||
+ s.startsWith("@framefields/");
+const inWorkspace = (url: string) => roots.some((r) => url.startsWith(r));
+/** Source files that become ESM even in a package without "type": "module". */
+const FORCE_ESM = /\.(ts|tsx|js|jsx)$/;
+
+function jobOf(url: string | undefined): string | undefined {
+ if (!url?.includes("?")) return undefined;
+ try {
+ return new URL(url).searchParams.get(JOB_PARAM) ?? undefined;
+ } catch {
+ return undefined;
+ }
+}
+
+/**
+ * Installs the engine alias once per process (PLAN.md §7): imports and
+ * requires of `framefields` / `@framefields/*` made from inside a workspace
+ * resolve as if made by the worker's own framefields (ESM build), so repo
+ * code shares the worker's single, already-initialised GPU stack. Workspace
+ * ES modules also inherit a per-job query, so a job never reuses modules
+ * cached by a previous one. Uses synchronous in-thread hooks, which apply to
+ * both `import` and `require()`.
+ */
+export function installEngineAlias(workspaceRoots: string[]): void {
+ if (installed) return;
+ installed = true;
+ rootDirs = workspaceRoots.map((r) => path.resolve(r) + path.sep);
+ roots = rootDirs.map((r) => pathToFileURL(r).href);
+ // framefields depends on every @framefields/* package, so resolving from
+ // its entry finds them all.
+ engineParent = import.meta.resolve("framefields");
+
+ registerHooks({
+ resolve(specifier, context, next) {
+ const parent = context.parentURL;
+ if (isEngine(specifier) && parent && inWorkspace(parent)) {
+ try {
+ // Always the ESM build, so require() and import share one instance.
+ const conditions = (context.conditions ?? []).map((c) =>
+ c === "require" ? "import" : c,
+ );
+ if (!conditions.includes("import")) conditions.push("import");
+ return next(specifier, {
+ ...context,
+ conditions,
+ parentURL: engineParent,
+ });
+ } catch {
+ // Not part of the engine (e.g. a project-local @framefields/* package).
+ }
+ }
+ const result = next(specifier, context);
+ if (!result.url.startsWith("file:") || !inWorkspace(result.url))
+ return result;
+ let { url, format } = result;
+ // Repo code always runs as ESM. In CommonJS mode tsx would transpile
+ // framefields' .mjs into a second, CJS copy of the engine.
+ if (format === "commonjs" && FORCE_ESM.test(new URL(url).pathname))
+ format = "module";
+ const job = jobOf(parent);
+ if (job && !jobOf(url)) {
+ const scoped = new URL(url);
+ scoped.searchParams.set(JOB_PARAM, job);
+ url = scoped.href;
+ }
+ return { ...result, url, format };
+ },
+ });
+}
+
+/**
+ * Drops CommonJS modules loaded from the workspaces (ES modules are scoped by
+ * the job query instead). Called when a job ends.
+ */
+export function clearWorkspaceRequireCache(): void {
+ const cache = createRequire(import.meta.url).cache;
+ for (const key of Object.keys(cache)) {
+ if (rootDirs.some((r) => key.startsWith(r))) delete cache[key];
+ }
+}
diff --git a/apps/renderer-service/src/render/load.ts b/apps/renderer-service/src/render/load.ts
new file mode 100644
index 00000000..acd930bd
--- /dev/null
+++ b/apps/renderer-service/src/render/load.ts
@@ -0,0 +1,242 @@
+import { existsSync } from "node:fs";
+import path from "node:path";
+import { pathToFileURL } from "node:url";
+import { register } from "tsx/esm/api";
+import { RenderError } from "../errors.js";
+import type { RenderSettings } from "../jobs/schema.js";
+import { resolveInside } from "../paths.js";
+import { JOB_PARAM } from "./engine-alias.js";
+
+/** What the worker can render: a framefields Composition or Media. */
+export interface Renderable {
+ renderVideo(options?: Record<string, unknown>): Promise<{
+ filePath: string;
+ cleanup: () => Promise<void>;
+ qa?: unknown;
+ }>;
+ renderFrame(options: Record<string, unknown>): Promise<Buffer>;
+ renderFrameGrid?(options: Record<string, unknown>): Promise<Buffer>;
+ toVirtualMedia?(): unknown;
+ virtualMedia?: unknown;
+}
+
+export interface PlanOutput {
+ name: string;
+ target: Renderable;
+ render: Partial<RenderSettings>;
+}
+
+const NAME = /^[A-Za-z0-9][\w.-]{0,63}$/;
+
+export function isRenderable(value: unknown): value is Renderable {
+ return (
+ typeof value === "object" &&
+ value !== null &&
+ typeof (value as Renderable).renderVideo === "function" &&
+ typeof (value as Renderable).renderFrame === "function"
+ );
+}
+
+export interface LoadedEntry {
+ mod: Record<string, unknown>;
+ /** Removes this job's TypeScript loader; call when the job is done. */
+ dispose(): Promise<void>;
+}
+
+/**
+ * Imports the entry with tsx, using the workdir's tsconfig. The import URL
+ * carries a job token that the engine alias propagates through the repo's
+ * own modules, so a job never sees modules cached by a previous one, while
+ * framefields itself stays the worker's single instance (PLAN.md §8.1).
+ */
+export async function importEntry(
+ workdir: string,
+ entry: string,
+ jobToken: string,
+): Promise<LoadedEntry> {
+ const file = await resolveInside(workdir, entry).catch((err) => {
+ if (err instanceof RenderError) throw err;
+ throw new RenderError(
+ "ENTRY_NOT_FOUND",
+ `Entry ${entry} does not exist in the workdir.`,
+ { phase: "load" },
+ );
+ });
+ const tsconfig = path.join(workdir, "tsconfig.json");
+ const unregister = register({
+ tsconfig: existsSync(tsconfig) ? tsconfig : false,
+ });
+ const url = new URL(pathToFileURL(file).href);
+ url.searchParams.set(JOB_PARAM, jobToken);
+ try {
+ const mod = (await import(/* @vite-ignore */ url.href)) as Record<
+ string,
+ unknown
+ >;
+ return { mod, dispose: unregister };
+ } catch (err) {
+ await unregister();
+ throw classifyLoadError(err, file);
+ }
+}
+
+export function classifyLoadError(
+ err: unknown,
+ entryFile: string,
+): RenderError {
+ if (err instanceof RenderError) return err;
+ const e = err as NodeJS.ErrnoException & { url?: string };
+ const message = e?.message ?? String(err);
+ if (
+ e?.code === "ERR_MODULE_NOT_FOUND" ||
+ /Cannot find (module|package)/.test(message)
+ ) {
+ const missing =
+ /Cannot find (?:module|package) '([^']+)'/.exec(message)?.[1] ?? "";
+ const missingEntry =
+ missing.split("?")[0] === entryFile ||
+ missing.split("?")[0] === pathToFileURL(entryFile).href;
+ return new RenderError(
+ missingEntry ? "ENTRY_NOT_FOUND" : "ENTRY_COMPILE_FAILED",
+ message,
+ {
+ phase: "load",
+ hint: missingEntry
+ ? undefined
+ : "An import couldn't be resolved: is the dependency in package.json and the lockfile?",
+ cause: err,
+ },
+ );
+ }
+ if (
+ e instanceof SyntaxError ||
+ /Transform failed|Expected .* but found|Unexpected token|ERR_UNKNOWN_FILE_EXTENSION/.test(
+ message,
+ )
+ ) {
+ return new RenderError("ENTRY_COMPILE_FAILED", message, {
+ phase: "load",
+ cause: err,
+ });
+ }
+ return new RenderError("COMPOSITION_BUILD_FAILED", message, {
+ phase: "load",
+ cause: err,
+ });
+}
+
+/**
+ * Reads the export, calls it with `args` when it is a function, and turns the
+ * result into a list of named outputs (PLAN.md §4 "Export shapes").
+ */
+export async function buildPlan(
+ mod: Record<string, unknown>,
+ exportName: string,
+ args: unknown,
+ select: string[] | undefined,
+): Promise<PlanOutput[]> {
+ if (!(exportName in mod)) {
+ const available = Object.keys(mod).filter((k) => k !== "__esModule");
+ throw new RenderError(
+ "EXPORT_NOT_FOUND",
+ `The entry has no export named "${exportName}".`,
+ {
+ phase: "load",
+ hint: available.length
+ ? `Exports: ${available.slice(0, 10).join(", ")}`
+ : "The module exports nothing.",
+ },
+ );
+ }
+ let value = mod[exportName];
+ if (typeof value === "function" && !isRenderable(value)) {
+ try {
+ value = await (value as (a: unknown) => unknown)(args);
+ } catch (err) {
+ if (err instanceof RenderError) throw err;
+ throw new RenderError(
+ "COMPOSITION_BUILD_FAILED",
+ `${exportName}() threw: ${err instanceof Error ? err.message : String(err)}`,
+ { phase: "load", cause: err },
+ );
+ }
+ }
+
+ let outputs: PlanOutput[];
+ if (isRenderable(value)) {
+ outputs = [{ name: "main", target: value, render: {} }];
+ } else if (isPlan(value)) {
+ outputs = value.outputs.map((o, i) => {
+ if (!isRenderable(o?.composition)) {
+ throw invalid(`outputs[${i}].composition is not a Composition.`);
+ }
+ return {
+ name: String(o.name ?? `output-${i}`),
+ target: o.composition,
+ render: o.render ?? {},
+ };
+ });
+ } else if (value && typeof value === "object" && !Array.isArray(value)) {
+ const entries = Object.entries(value);
+ if (entries.length === 0 || !entries.every(([, v]) => isRenderable(v))) {
+ throw invalid(describe(value));
+ }
+ outputs = entries.map(([name, target]) => ({
+ name,
+ target: target as Renderable,
+ render: {},
+ }));
+ } else {
+ throw invalid(describe(value));
+ }
+
+ for (const o of outputs) {
+ if (!NAME.test(o.name))
+ throw invalid(`Output name "${o.name}" must match ${NAME}.`);
+ }
+ if (new Set(outputs.map((o) => o.name)).size !== outputs.length) {
+ throw invalid("Output names must be unique.");
+ }
+ if (select) {
+ const missing = select.filter((s) => !outputs.some((o) => o.name === s));
+ if (missing.length) {
+ throw new RenderError(
+ "JOB_INVALID",
+ `target.select names unknown outputs: ${missing.join(", ")}.`,
+ {
+ phase: "load",
+ hint: `Available: ${outputs.map((o) => o.name).join(", ")}`,
+ },
+ );
+ }
+ outputs = outputs.filter((o) => select.includes(o.name));
+ }
+ return outputs;
+}
+
+interface PlanShape {
+ outputs: {
+ name?: unknown;
+ composition?: unknown;
+ render?: Partial<RenderSettings>;
+ }[];
+}
+
+function isPlan(value: unknown): value is PlanShape {
+ return (
+ typeof value === "object" &&
+ value !== null &&
+ Array.isArray((value as PlanShape).outputs) &&
+ (value as PlanShape).outputs.length > 0
+ );
+}
+
+function describe(value: unknown): string {
+ const kind =
+ value === null ? "null" : Array.isArray(value) ? "an array" : typeof value;
+ return `The export resolved to ${kind}, not a Composition, a record of compositions, or { outputs: [...] }.`;
+}
+
+function invalid(message: string): RenderError {
+ return new RenderError("EXPORT_INVALID", message, { phase: "load" });
+}
diff --git a/apps/renderer-service/src/render/render-step.ts b/apps/renderer-service/src/render/render-step.ts
new file mode 100644
index 00000000..642e4aff
--- /dev/null
+++ b/apps/renderer-service/src/render/render-step.ts
@@ -0,0 +1,470 @@
+import fs from "node:fs/promises";
+import path from "node:path";
+import {
+ HeadlessMediaRenderer,
+ resetRenderCaches,
+} from "@framefields/renderer";
+import { Composition, FontManager, Media } from "framefields";
+import { RenderError } from "../errors.js";
+import { bundledEngineVersion } from "../install/installer.js";
+import type {
+ CheckoutResult,
+ Pipeline,
+ RenderedFile,
+ StepContext,
+} from "../jobs/pipeline.js";
+import type { JobInput, RenderSettings } from "../jobs/schema.js";
+import { workerHealth } from "../worker/health.js";
+import { clearWorkspaceRequireCache } from "./engine-alias.js";
+import { buildPlan, importEntry, type PlanOutput } from "./load.js";
+import { openSandbox } from "./sandbox.js";
+
+const PROGRESS_INTERVAL_MS = 500;
+
+/**
+ * Pipeline step: imports the repo's entry into this process and renders every
+ * output of its plan on the worker's warm GPU device (PLAN.md §8).
+ */
+export const render: Pipeline["render"] = async (checkout, input, ctx) => {
+ // Path-keyed caches from the previous job may hold stale bytes (§8.2).
+ resetRenderCaches();
+ FontManager.clear();
+
+ const outDir = path.join(ctx.jobDir, "out");
+ await fs.mkdir(outDir, { recursive: true });
+
+ // The render may be aborted by the job (cancel/timeout) or by the stall
+ // detector; both go through this controller.
+ const controller = new AbortController();
+ const forward = () => controller.abort(ctx.signal.reason);
+ if (ctx.signal.aborted) forward();
+ ctx.signal.addEventListener("abort", forward, { once: true });
+
+ const sandbox = openSandbox({
+ cwd: checkout.workdir,
+ argv1: path.join(checkout.workdir, input.target.entry),
+ env: input.env,
+ });
+ const warnings: string[] = [];
+ let dispose: (() => Promise<void>) | undefined;
+ try {
+ const work = (async () => {
+ const files: RenderedFile[] = [];
+ const plan = await withTimeout(
+ loadPlan(checkout, input, ctx.jobId, (d) => {
+ dispose = d;
+ }),
+ ctx.config.limits.timeouts.load,
+ "load",
+ );
+ for (const output of plan) {
+ if (
+ !(
+ output.target instanceof Composition ||
+ output.target instanceof Media
+ )
+ ) {
+ warnings.push(output.name);
+ }
+ }
+ if (warnings.length) {
+ ctx.warn(
+ "ENGINE_FOREIGN_INSTANCE",
+ `Outputs ${warnings.join(", ")} were built by another copy of framefields; rendering them with the worker's engine.`,
+ );
+ }
+ for (const [index, output] of plan.entries()) {
+ controller.signal.throwIfAborted();
+ files.push(
+ ...(await renderOutput(
+ output,
+ index,
+ plan.length,
+ input.render,
+ outDir,
+ ctx,
+ controller,
+ )),
+ );
+ }
+ return files;
+ })();
+
+ const files = await raceWithGrace(
+ work,
+ sandbox.failed,
+ controller.signal,
+ ctx.config.limits.killGraceSec,
+ );
+ return {
+ files,
+ engine: { framefields: bundledEngineVersion(), source: "bundled" },
+ };
+ } catch (err) {
+ throw classifyRenderError(
+ controller.signal.aborted ? controller.signal.reason : err,
+ checkout.root,
+ );
+ } finally {
+ ctx.signal.removeEventListener("abort", forward);
+ sandbox.restore();
+ clearWorkspaceRequireCache();
+ await dispose?.().catch(() => {});
+ }
+};
+
+async function loadPlan(
+ checkout: CheckoutResult,
+ input: JobInput,
+ jobId: string,
+ onLoaded: (dispose: () => Promise<void>) => void,
+): Promise<PlanOutput[]> {
+ const { mod, dispose } = await importEntry(
+ checkout.workdir,
+ input.target.entry,
+ jobId,
+ );
+ onLoaded(dispose);
+ return buildPlan(
+ mod,
+ input.target.export,
+ input.target.args,
+ input.target.select,
+ );
+}
+
+/** Renders one plan output into one or more files. */
+async function renderOutput(
+ output: PlanOutput,
+ index: number,
+ count: number,
+ defaults: RenderSettings,
+ outDir: string,
+ ctx: StepContext,
+ controller: AbortController,
+): Promise<RenderedFile[]> {
+ const settings = { ...defaults, ...output.render } as RenderSettings;
+ const renderer = new HeadlessMediaRenderer();
+ const stall = stallWatch(ctx.config.limits.stallTimeoutSec, controller);
+ const base = path.join(outDir, `${index}-${output.name}`);
+
+ try {
+ switch (settings.kind) {
+ case "video":
+ case "audio": {
+ const codec =
+ settings.codec ?? (settings.kind === "audio" ? "aac" : "h264");
+ const { ext, contentType } = containerFor(codec);
+ const started = Date.now();
+ let lastEmit = 0;
+ const result = await output.target.renderVideo({
+ outputPath: `${base}.${ext}`,
+ codec,
+ audioCodec: settings.audioCodec,
+ quality: settings.quality,
+ qa: settings.qa,
+ renderer,
+ signal: controller.signal,
+ onProgress: (p: {
+ phase: string;
+ frame: number;
+ totalFrames: number;
+ }) => {
+ stall.touch();
+ const now = Date.now();
+ if (
+ now - lastEmit < PROGRESS_INTERVAL_MS &&
+ p.frame < p.totalFrames
+ )
+ return;
+ lastEmit = now;
+ const fps = p.frame / Math.max(0.001, (now - started) / 1000);
+ ctx.progress({
+ phase: p.phase === "audio" ? "encoding" : "rendering",
+ output: count > 1 ? index : undefined,
+ frame: p.frame,
+ total: p.totalFrames,
+ fps: Math.round(fps * 10) / 10,
+ etaSec: Math.round(
+ (p.totalFrames - p.frame) / Math.max(fps, 0.001),
+ ),
+ });
+ },
+ });
+ return [
+ {
+ name: output.name,
+ path: result.filePath,
+ ext,
+ contentType,
+ ...(result.qa !== undefined && { qa: result.qa }),
+ },
+ ];
+ }
+ case "frames": {
+ const files: RenderedFile[] = [];
+ const frames = settings.frames ?? [0];
+ for (const [i, frame] of frames.entries()) {
+ controller.signal.throwIfAborted();
+ const png = await output.target.renderFrame({ frame, renderer });
+ stall.touch();
+ const file = `${base}-f${String(frame).padStart(5, "0")}.png`;
+ await fs.writeFile(file, png);
+ files.push({
+ name: `${output.name}-f${frame}`,
+ path: file,
+ ext: "png",
+ contentType: "image/png",
+ });
+ ctx.progress({
+ phase: "rendering",
+ output: count > 1 ? index : undefined,
+ frame: i + 1,
+ total: frames.length,
+ });
+ }
+ return files;
+ }
+ case "frameGrid": {
+ if (!output.target.renderFrameGrid) {
+ throw new RenderError(
+ "EXPORT_INVALID",
+ `Output ${output.name} can't render a frame grid.`,
+ { phase: "render" },
+ );
+ }
+ const grid = settings.grid ?? {};
+ const png = await output.target.renderFrameGrid({
+ startFrame: grid.from,
+ endFrame: grid.to,
+ stepFrames: grid.step,
+ ...(grid.columns && { columns: grid.columns }),
+ renderer,
+ });
+ const file = `${base}-grid.png`;
+ await fs.writeFile(file, png);
+ return [
+ {
+ name: output.name,
+ path: file,
+ ext: "png",
+ contentType: "image/png",
+ },
+ ];
+ }
+ case "lut": {
+ const vm =
+ output.target.toVirtualMedia?.() ?? output.target.virtualMedia;
+ if (!vm) {
+ throw new RenderError(
+ "EXPORT_INVALID",
+ `Output ${output.name} has no virtual media to bake a LUT from.`,
+ {
+ phase: "render",
+ },
+ );
+ }
+ const cube = await renderer.renderLut(
+ vm as never,
+ settings.frames?.[0] ?? 0,
+ );
+ const file = `${base}.cube`;
+ await fs.writeFile(file, cube);
+ return [
+ {
+ name: output.name,
+ path: file,
+ ext: "cube",
+ contentType: "text/plain",
+ },
+ ];
+ }
+ }
+ } finally {
+ stall.stop();
+ }
+}
+
+export function containerFor(codec: string): {
+ ext: string;
+ contentType: string;
+} {
+ switch (codec) {
+ case "vp8":
+ case "vp9":
+ return { ext: "webm", contentType: "video/webm" };
+ case "gif":
+ return { ext: "gif", contentType: "image/gif" };
+ case "mp3":
+ return { ext: "mp3", contentType: "audio/mpeg" };
+ case "aac":
+ return { ext: "m4a", contentType: "audio/mp4" };
+ case "opus":
+ return { ext: "webm", contentType: "audio/webm" };
+ default:
+ return { ext: "mp4", contentType: "video/mp4" };
+ }
+}
+
+/** Aborts with RENDER_STALLED when no progress arrives for `seconds`. */
+function stallWatch(seconds: number, controller: AbortController) {
+ let last = Date.now();
+ const timer =
+ seconds > 0
+ ? setInterval(() => {
+ if (
+ Date.now() - last > seconds * 1000 &&
+ !controller.signal.aborted
+ ) {
+ controller.abort(
+ new RenderError(
+ "RENDER_STALLED",
+ `No render progress for ${seconds}s.`,
+ { phase: "render" },
+ ),
+ );
+ }
+ }, 1000)
+ : undefined;
+ timer?.unref();
+ return {
+ touch: () => {
+ last = Date.now();
+ },
+ stop: () => clearInterval(timer),
+ };
+}
+
+function withTimeout<T>(
+ promise: Promise<T>,
+ seconds: number,
+ what: string,
+): Promise<T> {
+ if (seconds <= 0) return promise;
+ let timer: NodeJS.Timeout | undefined;
+ return Promise.race([
+ promise,
+ new Promise<never>((_, reject) => {
+ timer = setTimeout(
+ () =>
+ reject(
+ new RenderError(
+ "COMPOSITION_BUILD_FAILED",
+ `Loading the entry took longer than ${seconds}s (${what}).`,
+ {
+ phase: "load",
+ hint: "Is the export waiting on something that never resolves? Raise FF_TIMEOUT_LOAD_SEC if it's just slow.",
+ },
+ ),
+ ),
+ seconds * 1000,
+ );
+ }),
+ ]).finally(() => clearTimeout(timer));
+}
+
+/**
+ * Waits for the render, a stray error from repo code, or an abort. Code that
+ * ignores the abort gets `graceSec`; after that the job is reported anyway and
+ * the worker recycles, since it can't stop that code in-process (§8.3).
+ */
+async function raceWithGrace<T>(
+ work: Promise<T>,
+ failed: Promise<never>,
+ signal: AbortSignal,
+ graceSec: number,
+): Promise<T> {
+ let timer: NodeJS.Timeout | undefined;
+ const abandoned = new Promise<never>((_, reject) => {
+ const arm = () => {
+ timer = setTimeout(() => {
+ workerHealth.requestRecycle("render ignored abort");
+ reject(signal.reason);
+ }, graceSec * 1000);
+ };
+ if (signal.aborted) arm();
+ else signal.addEventListener("abort", arm, { once: true });
+ });
+ work.catch(() => {}); // may settle after we've given up on it
+ try {
+ return await Promise.race([work, failed, abandoned]);
+ } finally {
+ clearTimeout(timer);
+ }
+}
+
+export function classifyRenderError(
+ err: unknown,
+ repoRoot: string,
+): RenderError {
+ const scrub = (s: string) => s.replaceAll(repoRoot, "<repo>");
+ if (err instanceof RenderError) {
+ if (err.code === "GPU_DEVICE_LOST" || err.code === "RENDER_OOM")
+ workerHealth.requestRecycle(err.code);
+ return err;
+ }
+ const message = err instanceof Error ? err.message : String(err);
+ const stack =
+ err instanceof Error ? projectFrames(err.stack ?? "", repoRoot) : undefined;
+ const details = stack ? { stack } : undefined;
+
+ if (
+ /device (was )?lost|GPUDevice.*lost|DeviceLost|VK_ERROR_DEVICE_LOST/i.test(
+ message,
+ )
+ ) {
+ workerHealth.requestRecycle("GPU device lost");
+ return new RenderError("GPU_DEVICE_LOST", scrub(message), {
+ phase: "render",
+ details,
+ cause: err,
+ });
+ }
+ if (
+ /out of memory|OutOfMemory|Failed to allocate|allocation failed|ERR_OUT_OF_MEMORY/i.test(
+ message,
+ )
+ ) {
+ workerHealth.requestRecycle("out of memory");
+ return new RenderError("RENDER_OOM", scrub(message), {
+ phase: "render",
+ details,
+ cause: err,
+ });
+ }
+ if (/Cannot open local media file|ENOENT|no such file/i.test(message)) {
+ return new RenderError("ASSET_NOT_FOUND", scrub(message), {
+ phase: "render",
+ details,
+ hint: "If the file is tracked with LFS, make sure source.lfs.include covers it.",
+ cause: err,
+ });
+ }
+ if (
+ /UnsupportedInputFormat|Unsupported.*format|Invalid data found/i.test(
+ message,
+ )
+ ) {
+ return new RenderError("ASSET_NOT_FOUND", scrub(message), {
+ phase: "render",
+ details,
+ hint: "The file isn't decodable media. An undownloaded LFS pointer looks like this too.",
+ cause: err,
+ });
+ }
+ return new RenderError("RENDER_FAILED", scrub(message), {
+ phase: "render",
+ details,
+ cause: err,
+ });
+}
+
+/** Keeps stack frames from the project, with paths made repo-relative. */
+function projectFrames(stack: string, repoRoot: string): string | undefined {
+ const frames = stack
+ .split("\n")
+ .filter((line) => line.includes(repoRoot) && !line.includes("node_modules"))
+ .slice(0, 15)
+ .map((line) => line.replaceAll(repoRoot, "<repo>").trim());
+ return frames.length ? frames.join("\n") : undefined;
+}
diff --git a/apps/renderer-service/src/render/render.test.ts b/apps/renderer-service/src/render/render.test.ts
new file mode 100644
index 00000000..3b587a9e
--- /dev/null
+++ b/apps/renderer-service/src/render/render.test.ts
@@ -0,0 +1,356 @@
+import { execFileSync } from "node:child_process";
+import fs from "node:fs/promises";
+import os from "node:os";
+import path from "node:path";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { RenderError } from "../errors.js";
+import { CrashJournal } from "../jobs/journal.js";
+import { testConfig } from "../test-helpers.js";
+import { recycleReason } from "../worker/recycle.js";
+import { buildPlan, isRenderable } from "./load.js";
+import { classifyRenderError } from "./render-step.js";
+import { ExitTrap, openSandbox } from "./sandbox.js";
+
+let ws: string;
+
+async function write(rel: string, content: string) {
+ const file = path.join(ws, rel);
+ await fs.mkdir(path.dirname(file), { recursive: true });
+ await fs.writeFile(file, content);
+}
+
+async function codeOf(promise: Promise<unknown>) {
+ try {
+ await promise;
+ return "no error";
+ } catch (err) {
+ return err instanceof RenderError
+ ? err.code
+ : `not a RenderError: ${String(err)}`;
+ }
+}
+
+beforeAll(async () => {
+ ws = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "ff-ws-")));
+ await write(
+ "package.json",
+ '{ "name": "film", "type": "module", "dependencies": { "framefields": "^0.0.1" } }',
+ );
+});
+afterAll(async () => {
+ await fs.rm(ws, { recursive: true, force: true });
+});
+
+const fake = () => ({
+ renderVideo: async () => ({}),
+ renderFrame: async () => Buffer.alloc(0),
+});
+
+/** Runs the loader in a real Node process (see test/load-harness.ts). */
+function harness(entry: string, token: string, exportName?: string) {
+ const tsx = path.resolve(import.meta.dirname, "../../node_modules/.bin/tsx");
+ const script = path.resolve(
+ import.meta.dirname,
+ "../../test/load-harness.ts",
+ );
+ const out = execFileSync(
+ tsx,
+ [script, ws, entry, token, ...(exportName ? [exportName] : [])],
+ {
+ encoding: "utf8",
+ env: { ...process.env, NODE_ENV: "test" },
+ },
+ );
+ const line =
+ out
+ .trim()
+ .split("\n")
+ .filter((l) => l.startsWith("{"))
+ .at(-1) ?? "{}";
+ return JSON.parse(line) as Record<string, unknown>;
+}
+
+describe("entry loading (real Node loader)", () => {
+ it("gives repo code the worker's framefields, even with none installed", async () => {
+ await write(
+ "src/film.ts",
+ `import { Composition } from "framefields";
+ const n: number = 1;
+ export const Ctor = Composition;
+ export const buildFilm = () => new Composition({ width: 64, height: 64, fps: 24, durationFrames: n });`,
+ );
+ expect(harness("src/film.ts", "job-a", "buildFilm")).toMatchObject({
+ ok: true,
+ sameComposition: true,
+ plan: [{ name: "main", isWorkerComposition: true }],
+ });
+ }, 60_000);
+
+ it('gives the same instance to a package without "type": "module"', async () => {
+ await write("cjs/package.json", '{ "name": "legacy" }');
+ await write(
+ "cjs/film.ts",
+ 'import { Composition } from "framefields"; export const Ctor = Composition;',
+ );
+ expect(harness("cjs/film.ts", "job-cjs")).toMatchObject({
+ ok: true,
+ sameComposition: true,
+ });
+ }, 60_000);
+
+ it("loads changed code at the same path under a new job token", async () => {
+ await write(
+ "src/v.ts",
+ 'import { dep } from "./dep-v.js"; export const v = dep;',
+ );
+ await write("src/dep-v.ts", "export const dep = 1;");
+ expect(harness("src/v.ts", "job-b").v).toBe(1);
+ await write("src/dep-v.ts", "export const dep = 2;");
+ expect(harness("src/v.ts", "job-c").v).toBe(2);
+ }, 60_000);
+
+ it.each([
+ ["src/missing.ts", "", "ENTRY_NOT_FOUND"],
+ ["src/broken.ts", "export const = ;", "ENTRY_COMPILE_FAILED"],
+ [
+ "src/dep.ts",
+ 'import x from "not-installed-anywhere"; export default x;',
+ "ENTRY_COMPILE_FAILED",
+ ],
+ [
+ "src/throws.ts",
+ 'throw new Error("boom at import");',
+ "COMPOSITION_BUILD_FAILED",
+ ],
+ [
+ "src/quickstart.ts",
+ "console.log('rendering…'); process.exit(0);",
+ "EXPORT_SIDE_EFFECT",
+ ],
+ ["../outside.ts", "", "PATH_ESCAPE"],
+ ])(
+ "classifies %s",
+ async (entry, source, code) => {
+ if (source) await write(entry, source);
+ expect(harness(entry, "job-d")).toMatchObject({ ok: false, code });
+ },
+ 60_000,
+ );
+});
+
+describe("buildPlan", () => {
+ it("accepts a composition, a record, and a render plan", async () => {
+ expect(
+ (
+ await buildPlan({ default: fake() }, "default", undefined, undefined)
+ ).map((o) => o.name),
+ ).toEqual(["main"]);
+ expect(
+ (
+ await buildPlan(
+ { films: { main: fake(), teaser: fake() } },
+ "films",
+ undefined,
+ ["teaser"],
+ )
+ ).map((o) => o.name),
+ ).toEqual(["teaser"]);
+ const plan = await buildPlan(
+ {
+ plan: async (args: { n: number }) => ({
+ outputs: [
+ {
+ name: `cut-${args.n}`,
+ composition: fake(),
+ render: { kind: "frames", frames: [1] },
+ },
+ ],
+ }),
+ },
+ "plan",
+ { n: 3 },
+ undefined,
+ );
+ expect(plan[0]).toMatchObject({
+ name: "cut-3",
+ render: { kind: "frames" },
+ });
+ });
+
+ it.each([
+ [{ a: 1 }, "nope", "EXPORT_NOT_FOUND"],
+ [{ a: 42 }, "a", "EXPORT_INVALID"],
+ [{ a: { x: fake(), y: 1 } }, "a", "EXPORT_INVALID"],
+ [
+ { a: { outputs: [{ name: "../evil", composition: fake() }] } },
+ "a",
+ "EXPORT_INVALID",
+ ],
+ [
+ {
+ a: () => {
+ throw new Error("bad args");
+ },
+ },
+ "a",
+ "COMPOSITION_BUILD_FAILED",
+ ],
+ ])("rejects %o (export %s)", async (mod, name, code) => {
+ expect(
+ await codeOf(
+ buildPlan(mod as Record<string, unknown>, name, undefined, undefined),
+ ),
+ ).toBe(code);
+ });
+
+ it("rejects unknown select names", async () => {
+ expect(
+ await codeOf(
+ buildPlan({ f: { main: fake() } }, "f", undefined, ["nope"]),
+ ),
+ ).toBe("JOB_INVALID");
+ });
+
+ it("recognises renderables by shape", () => {
+ expect(isRenderable(fake())).toBe(true);
+ expect(isRenderable({ renderVideo: 1 })).toBe(false);
+ });
+});
+
+describe("job sandbox", () => {
+ it("sets and restores cwd, env, argv and process.exit", () => {
+ const before = {
+ cwd: process.cwd(),
+ argv: process.argv,
+ exit: process.exit,
+ };
+ const sandbox = openSandbox({
+ cwd: ws,
+ argv1: path.join(ws, "src/film.ts"),
+ env: { LOCALE: "tr" },
+ });
+ expect(process.cwd()).toBe(ws);
+ expect(process.env.LOCALE).toBe("tr");
+ expect(process.argv[1]).toBe(path.join(ws, "src/film.ts"));
+ expect(() => process.exit(1)).toThrow(ExitTrap);
+ sandbox.restore();
+ expect(process.cwd()).toBe(before.cwd);
+ expect(process.env.LOCALE).toBeUndefined();
+ expect(process.argv).toBe(before.argv);
+ expect(process.exit).toBe(before.exit);
+ });
+
+ it("refuses to nest", () => {
+ const sandbox = openSandbox({ cwd: ws, argv1: "x", env: {} });
+ try {
+ expect(() => openSandbox({ cwd: ws, argv1: "x", env: {} })).toThrow(
+ /already open/,
+ );
+ } finally {
+ sandbox.restore();
+ }
+ });
+});
+
+describe("render error classification", () => {
+ it.each([
+ ["GPUDevice was lost: reason unknown", "GPU_DEVICE_LOST"],
+ ["Failed to allocate texture: out of memory", "RENDER_OOM"],
+ ['Cannot open local media file "/ws/assets/a.mp4"', "ASSET_NOT_FOUND"],
+ ["UnsupportedInputFormatError: cannot detect", "ASSET_NOT_FOUND"],
+ ["something else", "RENDER_FAILED"],
+ ])("%s → %s", (message, code) => {
+ expect(classifyRenderError(new Error(message), "/ws").code).toBe(code);
+ });
+
+ it("keeps only project stack frames, repo-relative", () => {
+ const err = new Error("x");
+ err.stack =
+ "Error: x\n at a (/ws/src/film.ts:3:1)\n at b (/ws/node_modules/z/i.js:1:1)\n at c (node:internal/x:1:1)";
+ expect(classifyRenderError(err, "/ws").details?.stack).toBe(
+ "at a (<repo>/src/film.ts:3:1)",
+ );
+ });
+});
+
+describe("recycling", () => {
+ const config = testConfig({
+ FF_MAX_JOBS_PER_WORKER: "3",
+ FF_RECYCLE_HEAP_MB: "100",
+ });
+ const stats = {
+ completedJobs: 1,
+ heapUsedMb: 50,
+ rssMb: 200,
+ strayErrors: 0,
+ leakedHandles: 0,
+ };
+ it.each([
+ [{}, undefined],
+ [{ completedJobs: 3 }, "completed 3 jobs"],
+ [{ heapUsedMb: 150 }, "heap 150MB > 100MB"],
+ [{ strayErrors: 3 }, "3 errors from repo code outside any job"],
+ [{ leakedHandles: 51 }, "51 timers/handles left behind by repo code"],
+ [{ requested: "GPU device lost" }, "GPU device lost"],
+ ])("%o → %s", (patch, reason) => {
+ expect(recycleReason({ ...stats, ...patch }, config)).toBe(reason);
+ });
+});
+
+describe("event-loop watchdog", () => {
+ it("kills a process stuck in a sync loop and records why", async () => {
+ const journalPath = path.join(ws, "watchdog.json");
+ const tsx = path.resolve(
+ import.meta.dirname,
+ "../../node_modules/.bin/tsx",
+ );
+ const script = path.resolve(
+ import.meta.dirname,
+ "../../test/watchdog-harness.ts",
+ );
+ let killed = false;
+ try {
+ execFileSync(tsx, [script, journalPath], {
+ timeout: 30_000,
+ stdio: "ignore",
+ });
+ } catch (err) {
+ // tsx's CLI wrapper reports its child's SIGKILL as exit status 137.
+ const e = err as { signal: string | null; status: number | null };
+ killed = e.signal === "SIGKILL" || e.status === 137;
+ }
+ expect(killed).toBe(true);
+ const record = new CrashJournal(journalPath).recover();
+ expect(record?.error?.message).toMatch(/event loop blocked/);
+ }, 40_000);
+});
+
+describe("crash journal", () => {
+ it("reports the job a dead process left behind, once", async () => {
+ const journal = new CrashJournal(path.join(ws, "inflight.json"));
+ journal.begin({
+ jobId: "j1",
+ createdAt: "2026-01-01T00:00:00Z",
+ startedAt: "2026-01-01T00:00:01Z",
+ });
+ const record = journal.recover();
+ expect(record).toMatchObject({
+ id: "j1",
+ status: "FAILED",
+ error: { code: "RENDER_CRASHED", retryable: true },
+ });
+ expect(journal.recover()).toBeUndefined();
+
+ journal.begin({
+ jobId: "j2",
+ createdAt: "x",
+ startedAt: "y",
+ exitReason: "JavaScript heap out of memory",
+ });
+ expect(journal.recover()?.error?.code).toBe("RENDER_OOM");
+
+ journal.begin({ jobId: "j3", createdAt: "x", startedAt: "y" });
+ journal.end();
+ expect(journal.recover()).toBeUndefined();
+ });
+});
diff --git a/apps/renderer-service/src/render/sandbox.ts b/apps/renderer-service/src/render/sandbox.ts
new file mode 100644
index 00000000..486b8eee
--- /dev/null
+++ b/apps/renderer-service/src/render/sandbox.ts
@@ -0,0 +1,105 @@
+import { RenderError } from "../errors.js";
+
+/**
+ * Thrown in place of exiting when repo code calls process.exit() while a job
+ * runs in the worker (PLAN.md §8.2).
+ */
+export class ExitTrap extends RenderError {
+ constructor(code: number | string | null | undefined) {
+ super(
+ "EXPORT_SIDE_EFFECT",
+ `Repo code called process.exit(${code ?? ""}) while the worker was loading or rendering it.`,
+ {
+ phase: "load",
+ hint: "Export a builder (a Composition or a function returning one); don't render or exit at import time.",
+ },
+ );
+ }
+}
+
+export interface SandboxOptions {
+ cwd: string;
+ argv1: string;
+ env: Record<string, string>;
+}
+
+export interface Sandbox {
+ /** Rejects when repo code fails out of band (stray rejection / exception). */
+ readonly failed: Promise<never>;
+ restore(): void;
+}
+
+let active: { fail(err: unknown): void } | undefined;
+let strayErrors = 0;
+
+/** Errors raised by repo code after its job finished. */
+export function strayErrorCount(): number {
+ return strayErrors;
+}
+
+/**
+ * Process-level handlers, installed once at boot: an unhandled rejection or
+ * exception fails the active job instead of crashing the worker. With no job
+ * running it is logged and counted (recycle trigger).
+ */
+export function installProcessGuards(
+ log: (msg: string, err: unknown) => void,
+): void {
+ const onError = (err: unknown) => {
+ if (active) active.fail(err);
+ else {
+ strayErrors++;
+ log("Error outside any job", err);
+ }
+ };
+ process.on("unhandledRejection", onError);
+ process.on("uncaughtException", onError);
+}
+
+/**
+ * Prepares the shared process for one job and returns how to undo it. Only
+ * one sandbox can be open at a time (the queue runs one job at a time).
+ */
+export function openSandbox(options: SandboxOptions): Sandbox {
+ if (active) throw new Error("A job sandbox is already open.");
+
+ const previousCwd = process.cwd();
+ const previousArgv = process.argv;
+ const previousExit = process.exit;
+ const previousEnv = new Map<string, string | undefined>();
+
+ for (const [key, value] of Object.entries(options.env)) {
+ previousEnv.set(key, process.env[key]);
+ process.env[key] = value;
+ }
+ process.chdir(options.cwd);
+ process.argv = [previousArgv[0], options.argv1];
+ process.exit = ((code?: number | string | null) => {
+ throw new ExitTrap(code);
+ }) as typeof process.exit;
+
+ let fail!: (err: unknown) => void;
+ const failed = new Promise<never>((_, reject) => {
+ fail = reject;
+ });
+ failed.catch(() => {}); // observed via race(); never unhandled
+ active = { fail };
+
+ return {
+ failed,
+ restore() {
+ process.exit = previousExit;
+ process.argv = previousArgv;
+ try {
+ process.chdir(previousCwd);
+ } catch {
+ process.chdir("/");
+ }
+ for (const [key, value] of previousEnv) {
+ if (value === undefined) delete process.env[key];
+ else process.env[key] = value;
+ }
+ active = undefined;
+ },
+ };
+}
diff --git a/apps/renderer-service/src/worker/health.ts b/apps/renderer-service/src/worker/health.ts
new file mode 100644
index 00000000..ee0be2a8
--- /dev/null
+++ b/apps/renderer-service/src/worker/health.ts
@@ -0,0 +1,18 @@
+/**
+ * Worker-level state that decides when the process should recycle itself
+ * (PLAN.md §8.3). Recycling = stop taking jobs, finish the current one, exit.
+ */
+class WorkerHealth {
+ private reason: string | undefined;
+
+ /** Marks the worker as unable to safely run more jobs. */
+ requestRecycle(reason: string): void {
+ this.reason ??= reason;
+ }
+
+ get recycleReason(): string | undefined {
+ return this.reason;
+ }
+}
+
+export const workerHealth = new WorkerHealth();
diff --git a/apps/renderer-service/src/worker/recycle.ts b/apps/renderer-service/src/worker/recycle.ts
new file mode 100644
index 00000000..95169fb0
--- /dev/null
+++ b/apps/renderer-service/src/worker/recycle.ts
@@ -0,0 +1,55 @@
+import type { RendererConfig } from "../config/index.js";
+
+export interface WorkerStats {
+ completedJobs: number;
+ heapUsedMb: number;
+ rssMb: number;
+ strayErrors: number;
+ leakedHandles: number;
+ requested?: string;
+}
+
+/**
+ * Why this worker should stop taking jobs and restart, or undefined to keep
+ * going (PLAN.md §8.3).
+ */
+export function recycleReason(
+ stats: WorkerStats,
+ config: RendererConfig,
+): string | undefined {
+ const r = config.recycle;
+ if (stats.requested) return stats.requested;
+ if (r.maxJobs > 0 && stats.completedJobs >= r.maxJobs)
+ return `completed ${stats.completedJobs} jobs`;
+ if (r.heapMb > 0 && stats.heapUsedMb > r.heapMb)
+ return `heap ${stats.heapUsedMb}MB > ${r.heapMb}MB`;
+ if (r.rssMb > 0 && stats.rssMb > r.rssMb)
+ return `rss ${stats.rssMb}MB > ${r.rssMb}MB`;
+ if (r.strayErrorLimit > 0 && stats.strayErrors >= r.strayErrorLimit) {
+ return `${stats.strayErrors} errors from repo code outside any job`;
+ }
+ if (r.leakedHandleLimit > 0 && stats.leakedHandles > r.leakedHandleLimit) {
+ return `${stats.leakedHandles} timers/handles left behind by repo code`;
+ }
+ return undefined;
+}
+
+export function currentStats(
+ completedJobs: number,
+ baselineHandles: number,
+ strayErrors: number,
+ requested: string | undefined,
+): WorkerStats {
+ const mem = process.memoryUsage();
+ return {
+ completedJobs,
+ heapUsedMb: Math.round(mem.heapUsed / 1024 / 1024),
+ rssMb: Math.round(mem.rss / 1024 / 1024),
+ strayErrors,
+ leakedHandles: Math.max(
+ 0,
+ process.getActiveResourcesInfo().length - baselineHandles,
+ ),
+ requested,
+ };
+}
diff --git a/apps/renderer-service/src/worker/watchdog.ts b/apps/renderer-service/src/worker/watchdog.ts
new file mode 100644
index 00000000..283ad719
--- /dev/null
+++ b/apps/renderer-service/src/worker/watchdog.ts
@@ -0,0 +1,51 @@
+import { Worker } from "node:worker_threads";
+
+/**
+ * Event-loop watchdog (PLAN.md §8.3). Repo code runs on the main thread, so
+ * a synchronous infinite loop would freeze the worker forever. A helper
+ * thread watches a heartbeat the main loop bumps every second; if it stops
+ * for `blockSec`, the thread writes the crash journal and kills the process
+ * so the supervisor (RunPod, Docker, k8s) starts a fresh one.
+ */
+const WATCHER = `
+const { workerData } = require("node:worker_threads");
+const fs = require("node:fs");
+const beat = new Int32Array(workerData.buffer);
+let last = Atomics.load(beat, 0);
+let lastChange = Date.now();
+setInterval(() => {
+ const now = Atomics.load(beat, 0);
+ if (now !== last) { last = now; lastChange = Date.now(); return; }
+ if (Date.now() - lastChange < workerData.blockMs) return;
+ try {
+ const journal = JSON.parse(fs.readFileSync(workerData.journal, "utf8"));
+ journal.exitReason = "event loop blocked for " + Math.round((Date.now() - lastChange) / 1000) + "s";
+ fs.writeFileSync(workerData.journal, JSON.stringify(journal));
+ } catch {}
+ process.kill(process.pid, "SIGKILL");
+}, 1000);
+`;
+
+export function startWatchdog(options: {
+ blockSec: number;
+ journalPath: string;
+}): () => void {
+ if (options.blockSec <= 0) return () => {};
+ const buffer = new SharedArrayBuffer(4);
+ const beat = new Int32Array(buffer);
+ const heartbeat = setInterval(() => Atomics.add(beat, 0, 1), 1000);
+ heartbeat.unref();
+ const worker = new Worker(WATCHER, {
+ eval: true,
+ workerData: {
+ buffer,
+ blockMs: options.blockSec * 1000,
+ journal: options.journalPath,
+ },
+ });
+ worker.unref();
+ return () => {
+ clearInterval(heartbeat);
+ void worker.terminate();
+ };
+}
diff --git a/apps/renderer-service/test/load-harness.ts b/apps/renderer-service/test/load-harness.ts
new file mode 100644
index 00000000..cd5e1b57
--- /dev/null
+++ b/apps/renderer-service/test/load-harness.ts
@@ -0,0 +1,47 @@
+/**
+ * Runs the entry loader in a real Node process (vitest's VM can't do native
+ * dynamic imports through loader hooks). Usage:
+ * tsx test/load-harness.ts <workspace> <entry> <jobToken> [export]
+ * Prints one JSON line describing the outcome.
+ */
+import * as worker from "framefields";
+import { RenderError } from "../src/errors.js";
+import { installEngineAlias } from "../src/render/engine-alias.js";
+import { buildPlan, importEntry } from "../src/render/load.js";
+import { openSandbox } from "../src/render/sandbox.js";
+
+const [ws, entry, token, exportName] = process.argv.slice(2);
+installEngineAlias([ws]);
+
+const report = (value: unknown) => {
+ process.stdout.write(`${JSON.stringify(value)}\n`);
+};
+
+const sandbox = openSandbox({ cwd: ws, argv1: entry, env: {} });
+try {
+ const { mod, dispose } = await importEntry(ws, entry, token);
+ const out: Record<string, unknown> = {
+ ok: true,
+ exports: Object.keys(mod).sort(),
+ v: mod.v,
+ sameComposition:
+ mod.Ctor === undefined ? undefined : mod.Ctor === worker.Composition,
+ };
+ if (exportName) {
+ const plan = await buildPlan(mod, exportName, undefined, undefined);
+ out.plan = plan.map((o) => ({
+ name: o.name,
+ isWorkerComposition: o.target instanceof worker.Composition,
+ }));
+ }
+ await dispose();
+ report(out);
+} catch (err) {
+ report({
+ ok: false,
+ code: err instanceof RenderError ? err.code : undefined,
+ message: err instanceof Error ? err.message : String(err),
+ });
+} finally {
+ sandbox.restore();
+}
diff --git a/apps/renderer-service/test/render-harness.ts b/apps/renderer-service/test/render-harness.ts
new file mode 100644
index 00000000..b1da1af5
--- /dev/null
+++ b/apps/renderer-service/test/render-harness.ts
@@ -0,0 +1,167 @@
+/**
+ * Runs the in-process render step for several jobs back to back in one real
+ * Node process, the way a worker does. Usage:
+ * tsx test/render-harness.ts <workDir>
+ * Prints one JSON line with each scenario's outcome. Needs a GPU.
+ */
+import fs from "node:fs/promises";
+import path from "node:path";
+import { Composition } from "framefields";
+import { buildConfig } from "../src/config/schema.js";
+import { RenderError } from "../src/errors.js";
+import type { CheckoutResult, StepContext } from "../src/jobs/pipeline.js";
+import { jobSchema } from "../src/jobs/schema.js";
+import { installEngineAlias } from "../src/render/engine-alias.js";
+import { render } from "../src/render/render-step.js";
+import { installProcessGuards } from "../src/render/sandbox.js";
+
+const workDir = path.resolve(process.argv[2]);
+const ws = path.join(workDir, "ws", "repo", "0");
+await fs.mkdir(path.join(ws, "src"), { recursive: true });
+await fs.mkdir(path.join(ws, "assets"), { recursive: true });
+installEngineAlias([await fs.realpath(path.join(workDir, "ws"))]);
+installProcessGuards(() => {});
+
+const built = buildConfig(
+ {
+ FF_MODE: "cli",
+ FF_REPO_ALLOWLIST: "github.com/acme/*",
+ FF_WORK_DIR: workDir,
+ FF_CACHE_DIR: path.join(workDir, "cache"),
+ FF_KILL_GRACE_SEC: "2",
+ FF_STALL_TIMEOUT_SEC: "30",
+ },
+ { exists: () => false, tmpdir: () => workDir, homedir: () => workDir },
+);
+if (!built.ok) throw new Error(JSON.stringify(built.problems));
+const config = built.config;
+const root = await fs.realpath(ws);
+const checkout: CheckoutResult = {
+ root,
+ workdir: root,
+ sha: "x",
+ cache: {},
+ release: async () => {},
+};
+
+async function clip(color: string) {
+ const comp = new Composition({
+ width: 64,
+ height: 64,
+ fps: 24,
+ durationFrames: 24,
+ backgroundColor: color,
+ });
+ const out = await comp.renderVideo({});
+ await fs.copyFile(out.filePath, path.join(ws, "assets/clip.mp4"));
+ await out.cleanup();
+}
+
+async function job(
+ name: string,
+ target: Record<string, unknown>,
+ render_: Record<string, unknown> = {},
+ abortAfterMs?: number,
+) {
+ const input = jobSchema.parse({
+ version: 2,
+ source: { repo: "https://github.com/acme/films" },
+ target,
+ render: render_,
+ output: { type: "inline" },
+ });
+ const controller = new AbortController();
+ const warnings: string[] = [];
+ const ctx: StepContext = {
+ jobId: name,
+ config,
+ signal: controller.signal,
+ jobDir: path.join(workDir, "jobs", name),
+ progress: () => {},
+ warn: (code) => warnings.push(code),
+ };
+ if (abortAfterMs !== undefined) {
+ setTimeout(
+ () => controller.abort(new RenderError("CANCELLED", "test cancel")),
+ abortAfterMs,
+ );
+ }
+ try {
+ const result = await render(checkout, input, ctx);
+ const files = await Promise.all(
+ result.files.map(async (f) => ({
+ name: f.name,
+ ext: f.ext,
+ bytes: (await fs.stat(f.path)).size,
+ path: f.path,
+ })),
+ );
+ return { ok: true, files, warnings, cwdRestored: process.cwd() !== root };
+ } catch (err) {
+ return {
+ ok: false,
+ code: err instanceof RenderError ? err.code : String(err),
+ warnings,
+ cwdRestored: process.cwd() !== root,
+ };
+ }
+}
+
+await fs.writeFile(
+ path.join(ws, "package.json"),
+ '{ "name": "film", "type": "module" }',
+);
+await fs.writeFile(
+ path.join(ws, "src/film.ts"),
+ `import { Composition, Layer } from "framefields";
+ export function buildFilm(args?: { color?: string }) {
+ const comp = new Composition({ width: 64, height: 64, fps: 24, durationFrames: 24, backgroundColor: args?.color ?? "#000000" });
+ comp.add(Layer.video("assets/clip.mp4", { width: 32, height: 32, muted: true }));
+ return comp;
+ }
+ export const cuts = () => ({ main: buildFilm(), teaser: buildFilm({ color: "#ff0000" }) });
+ export const slow = () => new Composition({ width: 640, height: 360, fps: 30, durationFrames: 3000, backgroundColor: "#123456" });`,
+);
+await fs.writeFile(path.join(ws, "src/exits.ts"), "process.exit(3);");
+
+const results: Record<string, unknown> = {};
+await clip("#00ff00");
+results.video1 = await job("video1", {
+ entry: "src/film.ts",
+ export: "buildFilm",
+});
+// Same asset path, different bytes: the next job must decode the new file.
+await clip("#0000ff");
+results.video2 = await job("video2", {
+ entry: "src/film.ts",
+ export: "buildFilm",
+});
+results.frames = await job(
+ "frames",
+ { entry: "src/film.ts", export: "cuts", select: ["teaser"] },
+ { kind: "frames", frames: [0, 12] },
+);
+results.exits = await job("exits", { entry: "src/exits.ts" });
+results.cancel = await job(
+ "cancel",
+ { entry: "src/film.ts", export: "slow" },
+ {},
+ 1500,
+);
+results.after = await job("after", {
+ entry: "src/film.ts",
+ export: "buildFilm",
+});
+
+const bytes = async (r: unknown) => {
+ const f = (r as { files?: { path: string }[] }).files?.[0];
+ return f ? fs.readFile(f.path) : undefined;
+};
+const [a, b] = await Promise.all([
+ bytes(results.video1),
+ bytes(results.video2),
+]);
+results.videosDiffer = Boolean(a && b && Buffer.compare(a, b) !== 0);
+
+process.stdout.write(`${JSON.stringify(results)}\n`);
+process.exit(0);
diff --git a/apps/renderer-service/test/watchdog-harness.ts b/apps/renderer-service/test/watchdog-harness.ts
new file mode 100644
index 00000000..7b1490f1
--- /dev/null
+++ b/apps/renderer-service/test/watchdog-harness.ts
@@ -0,0 +1,11 @@
+/** Blocks the event loop under the watchdog. Usage: tsx test/watchdog-harness.ts <journal> */
+import { CrashJournal } from "../src/jobs/journal.js";
+import { startWatchdog } from "../src/worker/watchdog.js";
+
+const journal = new CrashJournal(process.argv[2]);
+journal.begin({ jobId: "stuck", createdAt: "x", startedAt: "y" });
+startWatchdog({ blockSec: 2, journalPath: journal.path });
+setTimeout(() => {
+ // Repo code stuck in a synchronous loop.
+ for (;;) {}
+}, 1500);
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 8979859f..1c463a15 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -202,6 +202,9 @@ importers:
semver:
specifier: ^7.8.5
version: 7.8.5
+ tsx:
+ specifier: ^4.21.0
+ version: 4.21.0
zod:
specifier: ^4.3.6
version: 4.3.6
@@ -215,9 +218,6 @@ importers:
'@types/semver':
specifier: ^7.8.0
version: 7.8.0
- tsx:
- specifier: ^4.20.3
- version: 4.21.0
typescript:
specifier: ^5.8.3
version: 5.9.3