Code-first video & image, rendered natively on WebGPU.
feat(renderer-service): self-deploy groundwork — job API v2 and git checkout
okanasl9a4e717ed34b
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.mdnew file mode 100644index 00000000..549a334a--- /dev/null+++ b/apps/renderer-service/PLAN.md@@ -0,0 +1,558 @@+# Renderer service: self-deployment plan & spec++Status: **draft, decisions recorded**, 2026-10-14. Branch: `feat/renderer-self-deploy`.++### Decisions (2026-10-14)++1. **Rendering runs in the main service process** (revised 2026-10-14; this replaces the earlier child-process design). The repo's entry module is imported into the long-lived worker and rendered with the worker's warm GPU device. Section 8 covers what this costs and how each cost is handled.+2. **Engine source: `bundled`.** This follows from decision 1, because one process can only hold one GPU stack. Imports of `framefields` and `@framefields/*` from repo code resolve to the worker's own copy. The project's lockfile version is still checked (section 7). The earlier `project` default, where each job used the framefields version in its lockfile, is dropped.+3. **Script mode is dropped.** A package script always runs as its own process, so it can't render in the main process. Its use cases (several outputs, contact sheets, parametrised variants) are covered by letting an export return several compositions or a render plan (section 4).+4. **The inline `virtualMedia` path is removed.** That covers the legacy input mapping, the v1 `/run` and `/status` routes, `cli.ts --job <json>`, `configureAssetUrls` and `BASE_URL`, and the `R2_*` env names. There are no callers left.+5. **Git hosts:** GitHub, GitLab, Bitbucket and self-hosted servers (GitLab self-managed, Gitea/Forgejo, plain git over HTTPS/SSH) are all in scope from the start (section 6.5).+6. **Image:** `ghcr.io/gatewai-dev/framefields-renderer`.+7. **RunPod worker:** a thin Python shim in front of a long-lived Node server.++## 1. Goal++Turn `@framefields/renderer-service` into a GPU render worker that anyone can deploy themselves:++- as a **RunPod template** (serverless endpoint or a long-running pod),+- or as a plain **Docker image** on any NVIDIA/Vulkan host.++A job tells the worker which **git repo + ref** holds a framefields project. The worker fetches the code, pulls the assets with **git LFS**, installs dependencies, builds the composition the job names, renders it, and uploads the result. Every knob that differs between deployments is configured by env vars or a config file, never by a code change.++### Non-goals (for this iteration)++- Splitting one render across several workers by frame range. Section 13 covers this as future work.+- A queue or scheduler of our own. RunPod or the caller does the queueing.+- Repo encryption filters such as `git-crypt`. These are detected and rejected with a clear error.+- Running untrusted third-party repos on a shared multi-tenant worker. The trust model is in section 9.++## 2. Current state (what exists today)++| Piece | State |+|---|---|+| `src/index.ts` | Hono HTTP server with `/run`, `/status/:id`, `/health` and `/ping`. Jobs are kept in an in-memory `Map` that is never pruned. There is **no auth**: `RENDERER_API_KEY` is set in `.env` but never read. |+| `src/cli.ts` | One-shot render that takes the job as **JSON in argv**. `ARG_MAX` limits that payload, so large `virtualMedia` inputs fail. |+| `runpod_worker.py` | Spawns a fresh `node cli.js` for each job, so every job pays the GPU init again. It finds the result by scanning stdout for a line that starts with `{"success":`. It points at the old path `apps/gatewai-renderer/...`. |+| `renderer-entrypoint.sh` | Also points at `apps/gatewai-renderer`. |+| `src/di-setup.ts` | Sets up an inversify container (`ServerMediaService`, `R2StorageService`, which is an alias of `S3StorageService`) that rendering doesn't need. The media service is optional, because `@framefields/media` falls back to `sharp` without it. |+| `Dockerfile.renderer` | The runtime image has no `git`, `git-lfs` or `openssh-client`. The user is named `gatewai`. |+| Job input | Only takes a pre-compiled `virtualMedia` AST plus `bucket`/`key`. It has no notion of a project or repo. |+| Storage | Assumes R2 (env names `R2_*`) and only supports the bucket/key output target. |++Phase 0 (section 13) fixes these before any new feature work, and removes the inline `virtualMedia` path (`job-processor.ts`, the `virtualMedia` branch of `render-upload.ts`, and `cli.ts`).++## 3. Architecture++```+ ┌──────────────── transport ────────────────┐+ RunPod ──▶ │ runpod_worker.py (thin shim) │+ HTTP ──▶ │ Hono server (/v2/jobs, /health, ...) │──┐+ CLI ──▶ │ cli.ts (--job-file / stdin) │ │+ └───────────────────────────────────────────┘ │+ ▼+ ┌──────── JobRunner (service process) ───────────┐+ │ validate → resolve → fetch → lfs → install → │+ │ load → render → verify → upload → report │+ └───┬───────────┬────────────┬──────────────┬────┘+ │ │ │ │+ GitWorkspace Installer InProcessRenderer OutputSink+ (mirror cache,(pnpm/npm/ (tsImport entry, (s3 / presigned+ worktrees, yarn, store engine alias hook, / local / inline)+ LFS, auth) cache) job sandbox,+ warm GPU device)+```++Key decisions:++1. **One long-lived Node process per worker.** The RunPod Python shim talks to it over `localhost` HTTP instead of spawning `node` for every job. GPU and renderer init then happens once per worker, not once per job.+2. **Repo code is rendered in the service process.** The GPU device, compiled pipelines, the font and model caches, and the node registry all stay warm across jobs. Only git, LFS and the package manager run as subprocesses.+3. **One job at a time per worker.** `cwd`, `process.env` and the GPU are process-wide, so jobs can't safely overlap. Further jobs wait in a queue.+4. **A job sandbox wraps each job** (8.2). It sets `cwd` and the job env, traps `process.exit`, attributes stray rejections to the job, and restores everything afterwards.+5. **The worker recycles itself when it can't recover in-process:** a hard timeout, GPU device loss, heap growth past a limit, or N jobs done. It reports the job first, then exits, and the supervisor (RunPod, Docker `restart`, k8s) starts a fresh one. This replaces the child process's hard-kill guarantees.+6. **Everything is pinned to a commit SHA once it has been resolved.** A ref is resolved exactly once per job. Every later step uses the SHA, and the SHA is returned in the result.++## 4. Job API (v2)++There is one schema, validated with zod and shared by every transport. RunPod wraps it as `{ "input": <job> }`. HTTP posts it as the body.++```jsonc+{+ "version": 2,++ // Where the project comes from. Git is the only source type.+ "source": {+ "type": "git",+ "repo": "https://github.com/acme/promo-videos.git", // or [email protected]:acme/promo-videos.git; default: FF_DEFAULT_REPO+ "ref": "main", // branch | tag | full ref (refs/heads/x) | sha (>= 7 chars); default: FF_DEFAULT_REF+ "commit": "9f2c1e…", // optional: fail with COMMIT_MISMATCH if ref no longer points here+ "workdir": "videos/launch", // optional: package dir inside the repo (monorepos); default "."+ "lfs": {+ "enabled": true, // default true when the repo has .gitattributes with filter=lfs+ "include": ["videos/launch/assets/**"], // only pull what this render needs+ "exclude": []+ },+ "submodules": false,+ "sparse": ["videos/launch", "shared/fonts"] // optional sparse checkout (cone mode)+ },++ // What to render. Same {entry, export} convention as startPreview().+ "target": {+ "entry": "src/film.ts", // relative to workdir; .ts/.mts/.js/.mjs+ "export": "buildFilm", // see "Export shapes" below; default "default"+ "args": { "locale": "tr" }, // optional: passed to the export when it is a function (parametrised renders)+ "select": ["main", "teaser"] // optional: which named compositions to render when the export returns several+ },++ // Default render settings. A render plan from the export can override them per output.+ "render": {+ "kind": "video", // video | frames | frameGrid | lut | audio+ "codec": "h264", "audioCodec": "aac", "quality": "high",+ "frames": [0, 80, 320], // kind=frames+ "range": { "from": 0, "to": 240 }, // optional sub-range for video+ "qa": { "failOn": ["black", "frozen", "clipping"] } // optional; report always returned when set+ },++ "output": {+ "type": "s3", // s3 | presigned | local | inline+ "bucket": "renders", // must be in FF_OUTPUT_BUCKETS allowlist; default STORAGE_BUCKET+ "key": "acme/{shortSha}/{jobId}.{ext}" // template vars below+ // presigned: { "type": "presigned", "urls": ["..."], "headers": {...} } (one URL per output, in plan order)+ // local: { "type": "local", "path": "launch/{jobId}.{ext}" } (under FF_OUTPUT_DIR)+ // inline: { "type": "inline" } base64 in the result, capped by FF_INLINE_MAX_BYTES+ },++ "env": { "LOCALE": "tr" }, // non-secret vars set in process.env for the job only; names must match FF_JOB_ENV_ALLOW+ "timeoutSec": 3600, // capped by FF_JOB_TIMEOUT_MAX_SEC+ "idempotencyKey": "acme-launch-v12" // optional, see 8.6+}+```++**Export shapes.** The export, or the value its function returns (sync or async), can be one of three things:++- **A `Composition`.** It is rendered with `render` and becomes one output, named `main`.+- **A record of named compositions,** `{ main: Composition, teaser: Composition }`. Each entry becomes one output, named by its key, and `target.select` narrows which are rendered.+- **A render plan:** `{ outputs: [{ name, composition, render?: {...overrides} }] }`. This covers contact sheets (`render.kind = "frameGrid"`), frame stills, and different codecs per output.++Outputs are rendered one after another on the same warm device.++**Key template variables:** `{jobId}`, `{sha}`, `{shortSha}`, `{ref}` (slugified), `{date}` (UTC `YYYY-MM-DD`), `{ext}`, `{index}` (frames or outputs, from 0), `{name}` (the output name). Rendered keys are normalised. Any key that contains `..`, starts with `/`, or contains control characters is rejected.++A payload without `version: 2` is rejected with `JOB_INVALID`. There is no legacy mapping.++When a job has several outputs, the `output` key template is applied once per output. A template that would map two outputs to the same key is rejected before rendering starts (`JOB_INVALID`).++**Repo defaults file (optional).** The project root may contain `framefields.render.json` with the non-security fields of `source.workdir`, `source.lfs`, `target`, `render` and `install` (section 7). Precedence:++> job > repo file > env defaults > built-in defaults++Security-relevant settings (allowlists, credentials, limits, output buckets) can **never** come from the job or the repo file.++### 4.1 Result++```jsonc+{+ "status": "COMPLETED",+ "jobId": "…",+ "source": { "repo": "https://github.com/acme/promo-videos.git", "ref": "main", "sha": "9f2c1e…", "workdir": "videos/launch" },+ "engine": { "framefields": "2.1.3", "source": "bundled", "projectDeclared": "^2.1.0" },+ "outputs": [{ "key": "acme/9f2c1e4/….mp4", "url": "https://…", "bytes": 48213311, "sha256": "…", "contentType": "video/mp4", "durationMs": 12000, "frames": 720 }],+ "qa": { /* VideoQaReport when requested */ },+ "timings": { "resolveMs": 210, "fetchMs": 1830, "lfsMs": 9400, "installMs": 0, "loadMs": 900, "renderMs": 61000, "uploadMs": 3100 },+ "cache": { "mirror": "hit", "lfs": "partial", "install": "hit", "workspace": "warm" },+ "warnings": ["engine 2.0.9 is older than the image's tested range ^2.1.0"]+}+```++A failure returns `{ "status": "FAILED", "error": { "code", "message", "phase", "retryable", "hint", "details" } }`. The error codes are listed in section 10.++### 4.2 Progress++The worker emits phase events:++> `queued → resolving → fetching → lfs → installing → loading → rendering{frame,total,fps,etaSec} → encoding → uploading → done`++- **HTTP:** `GET /v2/jobs/:id` returns the latest event. `GET /v2/jobs/:id/events` streams them as SSE.+- **RunPod:** the shim forwards them with `runpod.serverless.progress_update`.++## 5. Transports & deployment modes++`FF_MODE` selects the mode:++| Mode | Process layout | Use |+|---|---|---|+| `runpod-serverless` | Entrypoint starts the Node server on `127.0.0.1:${RENDERER_PORT}` and waits for `/health` = ok, then `exec`s `runpod_worker.py`. The shim POSTs the job, polls or streams progress, and returns the result. | RunPod serverless endpoint |+| `http` | Node server on `0.0.0.0`. **Bearer auth with `RENDERER_API_KEY` is required.** The server refuses to start if `RENDERER_API_KEY` is unset, unless `FF_HTTP_INSECURE=true`. | RunPod GPU pod, docker compose, k8s |+| `cli` | `node dist/src/cli.js --job-file job.json` (or `--job -` for stdin). Writes the result JSON to `--result-file` or to fd 3. stdout is left free for logs. | CI, one-off renders, debugging |++**HTTP API v2:**++- `POST /v2/jobs` returns `202 {id}`.+- `GET /v2/jobs/:id` returns the job.+- `GET /v2/jobs/:id/events` streams events as SSE.+- `POST /v2/jobs/:id/cancel` cancels a job.+- `GET /health` reports readiness, including the git, git-lfs, disk, cache and GPU checks.+- `GET /ping` is the RunPod LB health check.++The v1 routes `/run` and `/status/:id` are removed.++Finished jobs are evicted from memory after `FF_JOB_TTL_SEC` (default 1h). At most `FF_JOB_HISTORY_MAX` jobs are kept.++**Concurrency.** There is exactly one active job per worker (section 3). Extra jobs queue in-process up to `FF_QUEUE_MAX`, and anything beyond that gets `429 BUSY`. To scale, run more workers.++**Cancellation is cooperative:**++- HTTP cancel, a RunPod job cancel, or SIGTERM aborts the job's `AbortSignal`.+ - Git, LFS and install subprocesses are killed straight away.+ - The renderer checks the signal between frames (phase 4 adds `signal` to `renderVideo`/`renderFrame`).+- If the render has not stopped within `FF_KILL_GRACE_SEC`, the job is reported as cancelled anyway. The worker then marks itself unhealthy and exits to recycle (8.3).+- Partial outputs are deleted.+- The job ends as `CANCELLED`, or as `SHUTDOWN` when the worker itself was stopped.++## 6. Git workspace manager++### 6.1 Layout++`FF_CACHE_DIR` defaults to `/runpod-volume/framefields` when that path exists, and to `/var/cache/framefields` otherwise.++```+$FF_CACHE_DIR/+ git/<worker-id>/<repo-hash>.git # partial (blob:none) mirror per repo, per worker+ lfs/objects/… # shared LFS object store (content-addressed; lfs.storage)+ pnpm-store/ npm-cache/ yarn-cache/ # shared package caches (content-addressed)+ models/ # FRAMEFIELDS_MODELS_DIR (pinned vision models, sha256-checked)+$FF_WORK_DIR/ (local container disk, default /tmp/framefields)+ ws/<repo-hash>/<slot>/ # warm checkout + node_modules, leased one job at a time+ jobs/<jobId>/{out,logs}+ inflight.json # the running job, so a crash can be reported after restart (8.3)+```++Why the files are split this way:++- **Mirrors are per worker.** RunPod network volumes can be mounted by several workers at once, and `flock` over the volume is not reliable. Git mirrors are not safe for concurrent writers.+- **LFS objects and package stores are content-addressed with tmp+rename writes, so sharing them is safe.**+- Checkouts and `node_modules` live on local disk. Network volumes are slow at many-small-file IO, and installs there would be the bottleneck.++### 6.2 Fetch sequence++1. **Validate the repo.** Normalise the URL to `host/owner/name` and match it against `FF_REPO_ALLOWLIST`. Reject `file://`, `ext::` and local paths unless `FF_ALLOW_LOCAL_REPOS=true` (used by tests).+2. **Resolve the ref** with `git ls-remote` (or the mirror after a fetch):+ - A full ref is used as is.+ - A bare name is checked against `refs/heads/` and `refs/tags/`. If it matches both, the job fails with `REF_AMBIGUOUS`.+ - A SHA prefix must be unique in the mirror (`REF_AMBIGUOUS` otherwise).+ - Annotated tags are peeled to the commit.+ - If `commit` is given and differs from the resolved SHA, the job fails with `COMMIT_MISMATCH`.+3. **Update the mirror** under an in-process mutex for that repo:+ - Run `git fetch --prune --filter=blob:none origin '+refs/heads/*:refs/heads/*' '+refs/tags/*:refs/tags/*'`, or fetch only the one SHA when `FF_GIT_FETCH=single`.+ - If the mirror is unreadable (`rev-parse` or `fsck --connectivity-only` fails), delete it and re-clone **once**.+ - If the resolved SHA is still missing after the fetch, the job fails with `REF_NOT_FOUND`. This happens when a force-push between resolve and fetch makes the old SHA unreachable.+4. **Lease a workspace slot** for this repo.+ - **Warm slot:** run `git checkout --force --detach <sha>`, then `git clean -ffdx -e node_modules -e .ff-install-stamp`.+ - **Cold slot:** run `git worktree add --detach` from the mirror.+ - Apply sparse checkout and submodules (`--recursive`, same auth, same allowlist check per submodule URL) if they were requested.+5. **Pull LFS objects:**+ - The checkout runs with `GIT_LFS_SKIP_SMUDGE=1`.+ - Then run `git lfs pull --include=… --exclude=…` with `lfs.storage=$FF_CACHE_DIR/lfs` and `lfs.concurrenttransfers=$FF_LFS_CONCURRENCY`.+ - Transient failures are retried with backoff.+ - **Preflight:** `git lfs ls-files --size` gives the bytes to fetch. If that plus a margin exceeds free disk, the job fails with `DISK_FULL` before downloading anything. LRU eviction of the cache runs first (6.4).+ - **Postflight:** scan the included paths for files that are still LFS pointers (they start with `version https://git-lfs.github.com/spec/v1`). If any remain, fail with `LFS_POINTER_UNRESOLVED` and list the files. Otherwise the renderer would try to decode a 130-byte "video".+6. **Reject unsupported filters.** If `.gitattributes` declares a filter other than `lfs` (for example `git-crypt`) on a path the render needs, fail with `UNSUPPORTED_GIT_FILTER`.++### 6.3 Auth (never in URLs, logs, `.git/config` or `process.env` during a render)++- **HTTPS token:** `GIT_TOKEN` (+ optional `GIT_USERNAME`; the default depends on the host, see 6.5). A custom credential helper (`git -c credential.helper=…`) answers **only for hosts in the allowlist**. A `.lfsconfig` that points LFS at another host therefore never receives the token.++ Per-host tokens use `FF_GIT_CREDENTIALS` (JSON `{host: token}`) or the config file.+- **SSH:** `GIT_SSH_KEY` (base64 or raw PEM) is written to a `0600` tmpfs file. `GIT_SSH_KNOWN_HOSTS` holds the known hosts. Host key checking is `yes` by default; `FF_SSH_ACCEPT_NEW=true` relaxes it. `GIT_SSH_COMMAND` is built by us. LFS over SSH remotes uses `git-lfs-authenticate` automatically.+- **Per-job credentials** (`source.auth`) are **off** by default. When `FF_ALLOW_JOB_CREDENTIALS=true` enables them, they get the same host check and are redacted everywhere.+- **Hardening flags on every git call:**+ - `-c core.hooksPath=/dev/null`+ - `-c protocol.file.allow=never`+ - `-c protocol.ext.allow=never`+ - `-c submodule.recurse=false` (we recurse explicitly)+ - `-c core.symlinks=true`+ - `GIT_TERMINAL_PROMPT=0` (never hang waiting for a password prompt)+ - `GIT_ASKPASS=/bin/false`++### 6.4 Cache eviction++`FF_CACHE_MAX_GB` and `FF_WORK_MAX_GB` set the size limits. Eviction runs before each job, and again whenever a preflight needs space:++- Least-recently-leased workspaces are evicted first, then LFS objects by atime, then whole mirrors.+- A workspace that is currently leased is never evicted.++### 6.5 Git hosts++Git itself behaves the same everywhere; the hosts differ in their auth conventions, LFS endpoints and error messages. Each host gets a small `HostProfile` (`src/git/hosts.ts`). The profile is picked by hostname, or by `FF_GIT_HOSTS` for self-hosted servers (`{"git.acme.dev": "gitlab", "code.acme.dev": "gitea"}`).++| Host | HTTPS username default (token as password) | Token type | LFS notes | Host-specific errors we map |+|---|---|---|---|---|+| GitHub (`github.com`, GHES via `FF_GIT_HOSTS`) | `x-access-token` | fine-grained PAT, GitHub App installation token, classic PAT | LFS on `github.com`; separate bandwidth and storage quota | "This repository is over its data quota" → `LFS_QUOTA_EXCEEDED`; SAML SSO not authorized → `GIT_AUTH_FAILED` with an SSO hint |+| GitLab (`gitlab.com`, self-managed) | `oauth2` | project, group or personal access token, scope `read_repository` | LFS on the same host | 403 "LFS objects are not available" / project size limit → `LFS_QUOTA_EXCEEDED` |+| Bitbucket Cloud (`bitbucket.org`) | `x-token-auth` (access tokens); the Atlassian user for API tokens or app passwords | repository/workspace access token, API token | LFS served from a different host (`bbuseruploads`/media). The credential helper must allow it, and it does not need our token. | "Repository storage limit" → `LFS_QUOTA_EXCEEDED` |+| Gitea / Forgejo | the token owner's username (`GIT_USERNAME` required) | access token, scope `read:repository` | LFS on the same host, if the server has it enabled | LFS disabled on the server → `LFS_UNAVAILABLE` |+| Generic HTTPS / SSH | `GIT_USERNAME` required | anything git accepts | `.lfsconfig` `url`, which must also be an allowed host | generic mapping only |++Rules that apply to every host:++- **The allowlist covers every host we talk to**, including LFS endpoints and submodule remotes. Credentials are handed out per host. A CDN that the LFS API redirects to receives **no** credentials, because the credential helper only answers for allowed hosts and git-lfs only sends them where it asked.+- **SSH known hosts:** the image ships a pinned `known_hosts` for github.com, gitlab.com and bitbucket.org at `apps/renderer-service/deploy/known_hosts`. It is generated at image build with `ssh-keyscan`, and the build fails unless the fingerprints match the ones each host publishes. Self-hosted servers must supply `GIT_SSH_KNOWN_HOSTS`, or opt into `FF_SSH_ACCEPT_NEW=true`.+- **Self-signed TLS on a self-hosted server:** `FF_GIT_CA_BUNDLE` (PEM) is passed as `http.sslCAInfo`. Turning off TLS verification is not offered.+- **Ref resolution** uses only `git ls-remote` and `fetch`, never a host API. This keeps one code path for every host.+- **Test matrix:** an integration test against local bare repos for the generic path, plus a nightly opt-in smoke test against a real private repo on each of GitHub, GitLab and Bitbucket (tokens from CI secrets), and a Gitea container.++## 7. Dependency install++- **Package manager:**+ - The `packageManager` field wins, then the lockfile (`pnpm-lock.yaml` → pnpm, `yarn.lock` → yarn, `package-lock.json` → npm), via corepack.+ - The install root is the nearest workspace root above `workdir`. pnpm installs only that package and its dependencies with `--filter ./<workdir>...`.+- **Lockfile** is always frozen (`--frozen-lockfile` / `npm ci` / `--immutable`). A missing lockfile fails with `LOCKFILE_MISSING`, unless `FF_ALLOW_UNLOCKED_INSTALL=true`.+- **Skip the install** when the slot's `.ff-install-stamp` matches the hash of (lockfile, package manager version, Node major, platform, install flags).+- **Caches:** shared stores on the volume (`--store-dir`, `--prefer-offline`).+- **Install scripts** are on by default, because the native GPU, skia and webcodecs modules need them. `FF_INSTALL_SCRIPTS=false` turns them off. The build toolchain is only present in the `-full` image (section 11).+- **Private registries:** `FF_NPMRC` (secret, file contents) is written to a temp `.npmrc` that is used only for the install.+- **Prepare steps (optional):** `install.prepare: ["fonts"]` in the repo file runs those package scripts after the install, under the same timeout. It is used for fetching fonts and similar setup.+- **Engine alias (always on).** A `module.register` resolve hook, installed once at boot, maps every `framefields` and `@framefields/*` import from inside a workspace to the worker's own copy. This happens however the project declares the dependency (npm range or `workspace:*`). Repo code therefore shares the worker's single, already-initialised GPU stack, and a second copy of Dawn or skia never gets loaded.+- **The project's own copy of framefields is installed but never loaded.** Leaving it out of the install would mean changing the overrides, which breaks the frozen-lockfile check, so the install runs as is and the alias wins at import time. The cost is mostly prebuild downloads, which the shared store absorbs after the first install.+- **Skip the install** entirely when the project's only runtime dependencies are framefields packages, which is the common case. This is the fast path.+- **Engine check:** compare the version range the project declares with the bundled version.+ - If the bundled version doesn't satisfy it, apply `FF_ENGINE_POLICY=warn|fail` (default `warn`, which adds `ENGINE_MISMATCH` to the warnings).+ - `engines.node` mismatches only produce a warning.++## 8. Render execution (in-process)++### 8.1 Loading the entry++1. Resolve `entry` with `realpath` and check that it lies inside the workspace (`PATH_ESCAPE` otherwise, which also covers symlink escapes).+2. Import it with tsx's `tsImport(entryUrl, { parentURL, tsconfig })`.+ - Every call gets its own module namespace, so job N+1 never sees job N's cached modules. A fix pushed to the same file is picked up even when the path is identical.+ - The project's `tsconfig.json` is honoured, including `paths`.+ - A missing module gives `ENTRY_NOT_FOUND`. A TS or syntax error gives `ENTRY_COMPILE_FAILED`, with the first frames of the error.+3. Read the export (`EXPORT_NOT_FOUND`). If it is a function, call it with `args`. A throw here gives `COMPOSITION_BUILD_FAILED`.+4. Normalise the result to a render plan (section 4). Anything that isn't a Composition, a record of compositions, or a plan gives `EXPORT_INVALID`.+5. **Composition from a different engine instance.** If the result was built by some other copy of framefields (the alias was bypassed, for example by a bundled `dist/` inside the repo), `instanceof` fails. The worker then falls back to `toVirtualMedia()` and renders that with its own renderer, adding the warning `ENGINE_FOREIGN_INSTANCE`.++### 8.2 Job sandbox++Repo code shares the worker's process, so every job runs inside a sandbox that is set up before loading and torn down in a `finally`:++| Shared state | What the sandbox does |+|---|---|+| `process.cwd()` | `chdir` to `<workspace>/<workdir>`, because relative asset paths resolve against it; restored afterwards. |+| `process.env` | Adds the filtered `job.env` and `FRAMEFIELDS_*` overrides; removed or restored afterwards. **Secrets are never in `process.env` while repo code runs:** at boot the config is parsed into a frozen object, and `GIT_*`, `STORAGE_*`, `RENDERER_API_KEY`, `FF_GIT_CREDENTIALS` and `FF_NPMRC` are deleted from `process.env`. Git subprocesses get credentials through their own env, never the shared one. |+| `process.exit` | Replaced with a trap that throws `EXPORT_SIDE_EFFECT` ("export a builder; don't render or exit at import time"), e.g. for the README quickstart pattern. It is restored afterwards. |+| `process.argv` | Set to `[node, entry]`, so scripts that parse argv don't see the worker's flags. |+| Unhandled rejections and uncaught exceptions | Process handlers attribute them to the active job. The job fails with `RENDER_FAILED` and the original stack, and the worker stays up. If one arrives when no job is active, it is logged and counted, and it triggers a recycle after `FF_STRAY_ERROR_LIMIT`. |+| Timers and handles left by repo code | They can't be cleaned up reliably in-process. After the job, active handles that the worker didn't own are counted (`process.getActiveResourcesInfo()`). Leaks above `FF_LEAKED_HANDLE_LIMIT` trigger a recycle. |+| Engine caches (fonts, textures, compiled timelines) | Cleared per job through the renderer's reset API (phase 4). The GPU device and pipeline cache are kept, because keeping them warm is the point of this design. |+| Console output | `console.*` from repo code is captured into the job log, tagged with the jobId. |++### 8.3 Limits, failure detection & recycling++- **Phase timeouts:** `FF_TIMEOUT_GIT_SEC`, `FF_TIMEOUT_LFS_SEC`, `FF_TIMEOUT_INSTALL_SEC`, `FF_TIMEOUT_LOAD_SEC` and `FF_TIMEOUT_RENDER_SEC`, plus the job's total `timeoutSec`. Git, LFS and install are subprocesses and are killed hard. Load and render are in-process, and are aborted through the job's `AbortSignal`.+- **Stalls:** if no progress event arrives for `FF_STALL_TIMEOUT_SEC` while rendering, the job fails with `RENDER_STALLED`.+- **Code that ignores the signal:** for example user code stuck in `await new Promise(() => {})` or a sync loop.+ - The job is reported `RENDER_TIMEOUT` or `CANCELLED` as soon as the grace period ends. The reporting runs on a separate timer; a sync loop is blocking the event loop, so it is detected by an event-loop watchdog instead (see below).+ - The worker then becomes unhealthy, stops accepting work, and exits to recycle.+- **Event-loop watchdog:** a `worker_threads` heartbeat checks the main loop. If the loop is blocked longer than `FF_EVENT_LOOP_BLOCK_SEC` (default 120), the watchdog writes the failure to `inflight.json` and kills the process.+- **Crash journal.** Before loading, the job's id, phase and input are written to `$FF_WORK_DIR/inflight.json`, and the file is removed when the job settles.+ - On boot, a leftover journal means the previous process died mid-job (OOM, native crash or watchdog). The job is recorded as `RENDER_CRASHED`, or `RENDER_OOM` when the exit reason is known. In HTTP mode `GET /v2/jobs/:id` still answers after the restart.+ - In RunPod serverless the platform itself sees the worker die and fails or retries the job, so the journal is only used for logs.+- **GPU device lost:** a `device lost` or VRAM allocation failure gives `GPU_DEVICE_LOST`, then a recycle.+- **Memory:** after each job, if `heapUsed` or RSS is above `FF_RECYCLE_HEAP_MB` or `FF_RECYCLE_RSS_MB`, the worker recycles. A V8 OOM kills the process, and the crash journal reports it.+- **Planned recycling:** `FF_MAX_JOBS_PER_WORKER` (default 25) bounds slow leaks from module graphs that can't be unloaded, and from native code.+- **What recycling does:** stop accepting jobs, let the queue drain to other workers (HTTP `503`, RunPod `refresh_worker`), report the current job, then exit 0. RunPod, Docker `restart: unless-stopped`, or k8s starts a fresh worker. In `cli` mode the process exits after one job anyway.++### 8.4 Render++Each output in the plan is rendered into `jobs/<jobId>/out/`, in plan order, according to its `render.kind`:++- `renderVideo({ outputPath, codec, quality, qa, signal, onProgress })`,+- `renderFrame` per frame,+- `renderFrameGrid`,+- or LUT/audio.++Progress callbacks become the phase events in 4.2.++### 8.5 Output verification & upload++- **Check the output:**+ - It must exist, be non-empty, and be at most `FF_OUTPUT_MAX_BYTES`.+ - The container is probed (ffprobe) for duration, frames and streams.+ - If the probe fails or the duration differs from the expected value by more than 1 frame, the job fails with `OUTPUT_INVALID`.+ - If `render.qa.failOn` matches a QA finding, the job fails with `QA_FAILED` and the report is attached. The output is still uploaded unless `FF_QA_BLOCK_UPLOAD=true`.+- **Upload targets:**+ - **s3:** multipart upload with retries. The bucket must be in `FF_OUTPUT_BUCKETS`. Object metadata `x-ff-sha`, `x-ff-job` and `x-ff-idempotency` is set.+ - **presigned:** PUT with the given headers. A 403/400 on an expired signature gives `UPLOAD_URL_EXPIRED`, which is not retryable. The hint is to sign with an expiry ≥ the job timeout.+ - **local:** an atomic rename into `FF_OUTPUT_DIR`.+ - **inline:** base64, only up to `FF_INLINE_MAX_BYTES` (default 5 MB, below RunPod's payload limits); above that the job fails with `OUTPUT_TOO_LARGE`.+- The job dir is deleted afterwards. Set `FF_KEEP_FAILED_JOBS=true` to keep failed ones for debugging.++### 8.6 Idempotency & retries++RunPod delivers jobs at least once, and callers retry. With an `idempotencyKey` and an s3 output, the worker `HEAD`s the key before resolving. If the object exists with the same `x-ff-idempotency` **and** the same resolved SHA, the worker returns the existing output with `cache.output = "hit"` and does not render. Without a key, the output is overwritten (last writer wins).++## 9. Trust & security model++The worker **executes code from the repo**. Anyone who can submit a job and point it at a repo they control gets code execution on the GPU host, and that host can reach whatever its network can. The defaults are therefore closed:++- `FF_REPO_ALLOWLIST` is **required** in git mode (e.g. `github.com/acme/*`). With `FF_DEFAULT_REPO` set and no allowlist, only that repo is allowed.+- HTTP mode requires `RENDERER_API_KEY`. RunPod endpoints already require the RunPod API key.+- **Repo code runs in the same process as the worker's credentials.** Secrets are removed from `process.env` before any repo code runs (8.2), and they are never written to disk in the workspace. Even so, code running in-process can in principle reach anything in memory. **The repos on the allowlist must be trusted to the same degree as the worker image.** Use a storage key scoped to the output bucket, and a read-only git token scoped to the allowed repos. With a `presigned` output the worker needs no storage credentials at all.+- The process runs as non-root user `framefields`. The workspace and cache are owned by it, and nothing outside them is writable.+- Logs and results go through a **redactor**. It strips tokens, URL userinfo, presigned query strings, and the `job.env` values the config marks secret.+- Documented risk: install scripts and repo code have network access. Self-hosters who need more can run with egress firewalling; that is out of scope here.++## 10. Error model++Every failure maps to a stable `code`. `retryable` is what RunPod or the caller should act on. Boot-time failures make `/health` return 503 and, in serverless mode, exit non-zero so RunPod recycles the worker.++| Phase | Code | Retryable | Typical cause / hint |+|---|---|---|---|+| boot | `CONFIG_INVALID` | – | Bad or missing env. The message lists every invalid key. |+| boot | `GPU_UNAVAILABLE` | – (worker recycled) | No Vulkan device, or the driver doesn't match the image's CUDA/Vulkan. |+| boot | `TOOL_MISSING` | – | `git` or `git-lfs` is not on PATH (custom image). |+| validate | `JOB_INVALID` | no | Schema error, with zod paths. |+| validate | `REPO_NOT_ALLOWED` / `BUCKET_NOT_ALLOWED` / `PATH_ESCAPE` | no | |+| validate | `BUSY` | yes | The queue is full (HTTP 429). |+| resolve | `GIT_AUTH_FAILED` | no | Bad or expired token, or a missing SSH key or known_hosts entry. |+| resolve | `REPO_NOT_FOUND` | no | Also returned for private repos without credentials. |+| resolve | `REF_NOT_FOUND` / `REF_AMBIGUOUS` / `COMMIT_MISMATCH` | no | A branch was deleted, force-pushed, or a short SHA matches several commits. |+| resolve | `GIT_TLS_FAILED` | no | Certificate verification failed, usually a self-signed server. The hint points to `FF_GIT_CA_BUNDLE`. |+| fetch | `GIT_NETWORK` | yes | Timeouts, 5xx, connection reset. Retried internally first. |+| fetch | `GIT_CACHE_CORRUPT` | yes | Raised only if the automatic re-clone also fails. |+| lfs | `LFS_AUTH_FAILED` | no | |+| lfs | `LFS_QUOTA_EXCEEDED` | no | The host's LFS bandwidth or storage quota (GitHub, GitLab or Bitbucket, see 6.5). The hint points to an alternative LFS server. |+| lfs | `LFS_UNAVAILABLE` | no | The server has LFS turned off (common on self-hosted Gitea). |+| lfs | `LFS_OBJECT_MISSING` | no | A pointer was committed without its object being pushed. The message lists the paths. |+| lfs | `LFS_POINTER_UNRESOLVED` | no | The include patterns didn't cover a needed asset, or LFS is disabled. |+| lfs | `UNSUPPORTED_GIT_FILTER` | no | `git-crypt` and similar. |+| any | `DISK_FULL` | yes (other worker) | Raised by the preflight or by ENOSPC. |+| install | `LOCKFILE_MISSING` / `INSTALL_FAILED` / `INSTALL_TIMEOUT` | no / no / yes | The tail of the install log is attached. |+| install | `ENGINE_INCOMPATIBLE` | no | Only with `FF_ENGINE_POLICY=fail`. |+| load | `ENTRY_NOT_FOUND` / `ENTRY_COMPILE_FAILED` / `EXPORT_NOT_FOUND` / `EXPORT_INVALID` / `EXPORT_SIDE_EFFECT` / `COMPOSITION_BUILD_FAILED` | no | The user's stack trace is attached and trimmed to the project's files. |+| render | `ASSET_NOT_FOUND` | no | A referenced file is missing. Often an LFS include that is too narrow, so the hint says so. |+| render | `RENDER_FAILED` / `RENDER_STALLED` / `RENDER_TIMEOUT` | no / yes / no | |+| render | `RENDER_OOM` / `GPU_DEVICE_LOST` / `RENDER_CRASHED` | yes (fresh worker) | The worker recycles. A crash is reported from the crash journal after the restart (8.3). |+| verify | `OUTPUT_MISSING` / `OUTPUT_INVALID` / `OUTPUT_TOO_LARGE` / `QA_FAILED` | no | |+| upload | `UPLOAD_FAILED` | yes | Retried internally first. |+| upload | `UPLOAD_URL_EXPIRED` | no | |+| any | `CANCELLED` / `SHUTDOWN` | no / yes | `SHUTDOWN` means the worker was preempted or scaled down mid-job. |++Warnings don't fail a job; they are listed in the result's `warnings`. They include `ENGINE_MISMATCH` (the declared framefields range isn't satisfied by the bundled engine, under `warn` policy), `ENGINE_FOREIGN_INSTANCE` (8.1), and `NODE_ENGINE_MISMATCH`.++## 11. Configuration reference (env)++The config is one zod schema (`src/config/schema.ts`). `FF_CONFIG_FILE` can point at a JSON or YAML file for structured values, such as per-host credentials or allowlists. Env vars override the file.++On boot the service validates **all** keys and fails with the complete list of problems. It then logs the effective config with secrets redacted.++| Group | Vars (defaults) |+|---|---|+| Mode | `FF_MODE` (`runpod-serverless` if `RUNPOD_POD_ID` is set, else `http`), `RENDERER_PORT` (8082), `RENDERER_API_KEY`, `FF_HTTP_INSECURE` (false), `LOG_LEVEL` (info), `LOG_FORMAT` (json) |+| Source | `FF_DEFAULT_REPO`, `FF_DEFAULT_REF` (`main`), `FF_REPO_ALLOWLIST` (comma globs), `FF_ALLOW_LOCAL_REPOS` (false), `FF_GIT_FETCH` (`mirror`\|`single`), `FF_LFS_CONCURRENCY` (8), `FF_LFS_DEFAULT_INCLUDE` |+| Git auth & hosts | `GIT_TOKEN`, `GIT_USERNAME`, `FF_GIT_CREDENTIALS` (`{host: token \| {username, token}}`), `FF_GIT_HOSTS` (`{host: github\|gitlab\|bitbucket\|gitea\|generic}`), `FF_GIT_CA_BUNDLE`, `GIT_SSH_KEY`, `GIT_SSH_KNOWN_HOSTS`, `FF_SSH_ACCEPT_NEW` (false), `FF_ALLOW_JOB_CREDENTIALS` (false) |+| Install | `FF_INSTALL_SCRIPTS` (true), `FF_ALLOW_UNLOCKED_INSTALL` (false), `FF_NPMRC`, `FF_ENGINE_POLICY` (`warn`) |+| Storage | `STORAGE_DRIVER` (`s3`\|`local`\|`none`), `STORAGE_ENDPOINT`, `STORAGE_REGION` (`auto`), `STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, `STORAGE_BUCKET`, `STORAGE_PUBLIC_BASE_URL`, `STORAGE_FORCE_PATH_STYLE` (MinIO), `FF_OUTPUT_BUCKETS`, `FF_OUTPUT_DIR`, `FF_INLINE_MAX_BYTES`, `FF_OUTPUT_MAX_BYTES`. The `R2_*` names are gone, with no aliases. |+| Paths & cache | `FF_CACHE_DIR`, `FF_WORK_DIR`, `FF_CACHE_MAX_GB` (50), `FF_WORK_MAX_GB` (40), `FRAMEFIELDS_MODELS_DIR` (`$FF_CACHE_DIR/models`), `FRAMEFIELDS_FONTS_DIR` |+| Limits | `FF_QUEUE_MAX` (8), `FF_JOB_TIMEOUT_MAX_SEC` (7200), `FF_TIMEOUT_{GIT,LFS,INSTALL,LOAD,RENDER}_SEC`, `FF_STALL_TIMEOUT_SEC` (300), `FF_KILL_GRACE_SEC` (20), `FF_JOB_TTL_SEC` (3600), `FF_JOB_HISTORY_MAX` (500) |+| Recycling | `FF_MAX_JOBS_PER_WORKER` (25; 0 = unlimited), `FF_RECYCLE_HEAP_MB` (4096), `FF_RECYCLE_RSS_MB` (0 = off), `FF_EVENT_LOOP_BLOCK_SEC` (120), `FF_STRAY_ERROR_LIMIT` (3), `FF_LEAKED_HANDLE_LIMIT` (50). `NODE_OPTIONS=--max-old-space-size` is set by the image. |+| Job env | `FF_JOB_ENV_ALLOW` (`*` minus a reserved denylist), `FF_KEEP_FAILED_JOBS` (false), `FF_QA_BLOCK_UPLOAD` (false) |++`BASE_URL`, `R2_*` and `R2_CUSTOM_DOMAIN` are removed together with the inline path. Repo assets are local files in the checkout, so `configureAssetUrls` is no longer called. A project that loads assets over HTTP uses absolute URLs.++The service no longer uses `S3StorageService` from `@framefields/server-utils`, which reads `R2_*`. Its output sink (phase 6) has its own S3 client configured from `STORAGE_*`.++## 12. Image, RunPod template, docs++**Image** (`Dockerfile.renderer`, renamed to `apps/renderer-service/Dockerfile`, with the build context still at the repo root):++- **Runtime packages added:** `git`, `git-lfs`, `openssh-client`, `ca-certificates`, `tini` (PID 1, reaps git and install subprocesses), `jq`. Corepack is pre-warmed for pnpm, yarn and npm, and `tsx` ships in the service's prod dependencies (its `tsImport` API is used in-process).+- **Two variants:**+ - `:<ver>` (slim) has no compiler. It relies on prebuilds and the bundled native modules.+ - `:<ver>-full` adds `build-essential` and `python3-dev`, for projects whose native dependencies need compiling.+- The user is renamed from `gatewai` to `framefields`. The `HEALTHCHECK` hits `/health`.+- **Entrypoint fixes:**+ - correct paths (`apps/renderer-service`);+ - `xvfb-run` is used only when `FF_XVFB=auto` and no headless Vulkan/EGL is available;+ - pre-boot checks: GPU visible (`nvidia-smi`, warn only), cache dirs writable, `git lfs install --skip-repo`.+- **Published to** `ghcr.io/gatewai-dev/framefields-renderer` by a release workflow, tagged from the package version. Multi-arch is not needed because RunPod is amd64.++**RunPod template:** `apps/renderer-service/deploy/runpod/`.++- `template.json` holds the image, `containerDiskInGb` (40), `volumeMountPath` (`/runpod-volume`), and the env with `{{ RUNPOD_SECRET_* }}` references for `GIT_TOKEN`, `GIT_SSH_KEY`, `STORAGE_*` and `RENDERER_API_KEY`. The template is created through the UI or the GraphQL API.+- `README.md` covers:+ - the serverless vs pod choice;+ - **setting the endpoint execution timeout** (the RunPod default is far below a long render);+ - network volume sizing, and keeping it in the same data center as the GPUs;+ - FlashBoot;+ - sample `curl` calls for `/run`, `/status` and `/stream`;+ - recommended GPU types.+- `deploy/docker-compose.yml` runs the service with a local MinIO for self-hosting off RunPod.++**Docs:** a "Self-hosted rendering" page on the docs site, plus a `framefields.render.json` schema reference.++## 13. Implementation phases++Each phase is its own PR-sized step with tests.++0. **Unbreak and remove the inline path.**+ - Drop the DI container (`di-setup.ts`, inversify, `ServerMediaService`, `R2StorageService`). Storage returns in phase 6 as the service's own output sink.+ - Delete the `virtualMedia` path: `job-processor.ts`, `render-upload.ts`, `cli.ts --job <json>`, the `/run` and `/status` routes, `configureAssetUrls`/`BASE_URL`, and the old `render.test.ts` cases. Keep `/health`, `/ping`, readiness and graceful shutdown.+ - Fix the `gatewai-renderer` paths in the entrypoint and the Python shim.+ - Get `pnpm --filter @framefields/renderer-service typecheck test` green.+1. **Config + job schema v2 + error model.**+ - `src/config/` (frozen config, secrets scrubbed from `process.env`), `src/jobs/schema.ts`, `src/errors.ts` (`RenderError` with `code`/`phase`/`retryable`), and the redactor.+ - API-key auth on HTTP, the job store with TTL eviction, and the single-slot queue.+2. **GitWorkspace.**+ - Mirror cache, ref resolution, workspace leasing, auth helpers and host profiles (6.5), LFS pull with pre- and postflight, eviction.+ - Integration tests use local bare repos and a file-based LFS remote (`lfs.standalonetransferagent`), so no network is needed.+3. **Installer.** Package manager detection, frozen install with framefields filtered out, skipping the install when nothing else is needed, the stamp check, shared stores, and the engine range check.+4. **Engine hooks (in `@framefields/renderer` and `framefields`).**+ - `signal: AbortSignal` and `onProgress` on `renderVideo`/`renderFrame`/`renderFrameGrid`, checked between frames.+ - A per-job cache reset (fonts, textures, compiled timelines) that keeps the device and pipelines.+5. **InProcessRenderer.**+ - The engine alias resolve hook, `tsImport` loading, export-shape normalisation, the job sandbox (8.2), the watchdog, the crash journal, and recycling.+6. **Outputs.** The s3, presigned, local and inline sinks, verification (ffprobe), multi-output keys, and idempotency.+7. **Transports.**+ - HTTP v2 with SSE progress and cancel.+ - New `runpod_worker.py` (localhost client, `progress_update`, error passthrough with `retryable`, `refresh_worker` on recycle).+ - CLI `--job-file` and `--result-file`.+8. **Image + template + compose + docs.**+ - Release workflow.+ - GPU smoke test: render the fixture repo on a RunPod pod. Run it by hand first, then as a nightly job.+ - Nightly host smoke test against GitHub, GitLab, Bitbucket and a Gitea container (6.5).+9. **(Later)**+ - Chunked distributed rendering: split a frame range across N jobs and concat with a stream copy.+ - Webhooks on completion (`output.webhook` with an HMAC signature).++## 14. Test plan++- **Unit tests:**+ - config parsing;+ - ref classification;+ - host profile selection and per-host credential answers, including no credentials for unlisted LFS/CDN hosts;+ - key templates and path guards;+ - the redactor (tokens in URLs, presigned queries);+ - error classification from git, LFS (per host), pnpm and subprocess exit signatures, using captured stderr fixtures;+ - export-shape normalisation;+ - the job sandbox restoring `cwd`, env, `process.exit` and argv after success, a throw and a timeout.+- **Integration (no GPU):** `test/fixtures/project/` is a tiny framefields project with one LFS-tracked PNG. Tests build a bare repo and a file-based LFS store at test time, then cover:+ - branch, tag, SHA, ambiguous ref, force-push between resolve and fetch, deleted branch;+ - LFS include too narrow, an object missing on the server;+ - a corrupt mirror that gets re-cloned;+ - a missing lockfile, and a warm-slot install skip;+ - every `EXPORT_*` failure;+ - the engine alias: the fixture declares `framefields@^2` and must get the worker's instance;+ - two jobs back to back on the same path with different code, which must not hit the module cache;+ - secrets absent from `process.env` inside repo code;+ - `process.exit` in user code, a never-resolving promise (timeout, then recycle flag), a sync infinite loop (watchdog kill and crash journal report on restart), and cancel mid-render.++ The GPU render is stubbed behind an injected renderer interface.+- **GPU (CI optional, manual on RunPod):** an end-to-end render of the fixture to MinIO, with a QA report; a cold vs warm timing comparison; worker recycling after `GPU_DEVICE_LOST`, simulated with a debug hook.+- **Image:** a `docker run … --version` and a `/health` check in CI. `hadolint`.++## 15. Open questions++None at the moment. The decisions are recorded at the top of this document.diff --git a/apps/renderer-service/package.json b/apps/renderer-service/package.jsonindex a81db9da..ff7d7700 100644--- a/apps/renderer-service/package.json+++ b/apps/renderer-service/package.json@@ -14,20 +14,15 @@"dev": "tsx watch --tsconfig tsconfig.dev.json --conditions=development src/index.ts","typecheck": "tsc -b tsconfig.json","build": "tsc -b tsconfig.json",- "start-cli": "node dist/src/cli.js","test": "vitest run"},"dependencies": {- "@framefields/client-utils": "workspace:*","@framefields/core": "workspace:*","@framefields/renderer": "workspace:*",- "@framefields/media": "workspace:*","@framefields/server-utils": "workspace:*","@hono/node-server": "^1.19.11","dotenv": "^17.0.1","hono": "^4.13.7",- "inversify": "^7.11.0",- "reflect-metadata": "^0.2.2","zod": "^4.3.6"},"devDependencies": {diff --git a/apps/renderer-service/runpod_worker.py b/apps/renderer-service/runpod_worker.pyindex 1b2e162e..e39b54e4 100644--- a/apps/renderer-service/runpod_worker.py+++ b/apps/renderer-service/runpod_worker.py@@ -1,30 +1,90 @@-import runpod-import subprocess+"""RunPod serverless shim (PLAN.md §5).++The Node renderer runs as a long-lived server on localhost, so the GPU is+initialised once per worker. This handler only forwards each job to it,+relays progress, and returns the result or a structured error.+"""+import json-import sys+import os+import time+import urllib.error+import urllib.request-def handler(job):- job_input = job['input']-- # Run Node.js CLI command to process the render job- result = subprocess.run(- ["node", "apps/gatewai-renderer/dist/src/cli.js", "--job", json.dumps(job_input)],- capture_output=True,- text=True- )-- if result.returncode != 0:- raise Exception(f"Node.js rendering process failed: {result.stderr or result.stdout}")-+import runpod++BASE = f"http://127.0.0.1:{os.environ.get('RENDERER_PORT', '8082')}"+API_KEY = os.environ.get("RENDERER_API_KEY", "")+POLL_SEC = float(os.environ.get("FF_RUNPOD_POLL_SEC", "1"))+BOOT_TIMEOUT_SEC = float(os.environ.get("FF_BOOT_TIMEOUT_SEC", "300"))+++def call(method, path, body=None):+ data = json.dumps(body).encode() if body is not None else None+ req = urllib.request.Request(BASE + path, data=data, method=method)+ req.add_header("content-type", "application/json")+ if API_KEY:+ req.add_header("authorization", f"Bearer {API_KEY}")try:- # Find the JSON result line from stdout- output_lines = result.stdout.strip().split('\n')- for line in reversed(output_lines):- line_str = line.strip()- if line_str.startswith('{"success":'):- return json.loads(line_str)- raise Exception(f"No JSON success payload returned in Node.js output. Raw stdout: {result.stdout}")- except Exception as e:- raise Exception(f"Failed to parse Node.js output: {str(e)}. Raw stdout: {result.stdout}")--runpod.serverless.start({"handler": handler})+ with urllib.request.urlopen(req, timeout=30) as res:+ return res.status, json.loads(res.read() or b"null")+ except urllib.error.HTTPError as err:+ payload = err.read()+ try:+ return err.code, json.loads(payload)+ except ValueError:+ return err.code, {"error": {"code": "INTERNAL", "message": payload.decode(errors="replace")}}+++def wait_ready():+ deadline = time.monotonic() + BOOT_TIMEOUT_SEC+ while time.monotonic() < deadline:+ try:+ status, body = call("GET", "/health")+ if status == 200:+ return+ if body and body.get("status") == "failed":+ raise RuntimeError(f"Renderer failed to start: {body.get('error')}")+ except (urllib.error.URLError, ConnectionError):+ pass+ time.sleep(1)+ raise RuntimeError("Renderer did not become healthy in time")+++def failure(error, refresh=False):+ """RunPod marks a job failed when the output has an `error` key."""+ out = {"error": json.dumps(error), "refresh_worker": refresh}+ return out+++def handler(job):+ status, body = call("POST", "/v2/jobs", job["input"])+ if status != 202:+ error = (body or {}).get("error", {"code": "INTERNAL", "message": str(body)})+ return failure(error, refresh=status == 503)++ job_id = body["id"]+ last_progress = None+ while True:+ time.sleep(POLL_SEC)+ status, record = call("GET", f"/v2/jobs/{job_id}")+ if status != 200:+ return failure({"code": "INTERNAL", "message": f"lost job {job_id} ({status})"}, refresh=True)++ progress = record.get("progress")+ if progress and progress != last_progress:+ runpod.serverless.progress_update(job, progress)+ last_progress = progress++ if record["status"] == "COMPLETED":+ return record["result"]+ if record["status"] in ("FAILED", "CANCELLED"):+ error = record.get("error") or {"code": "INTERNAL", "message": "unknown failure"}+ # Device loss / crashes leave the worker unhealthy: ask for a fresh one.+ refresh = error.get("code") in ("GPU_DEVICE_LOST", "RENDER_OOM", "RENDER_CRASHED")+ return failure(error, refresh=refresh)+++if __name__ == "__main__":+ wait_ready()+ runpod.serverless.start({"handler": handler})diff --git a/apps/renderer-service/src/cli.ts b/apps/renderer-service/src/cli.tsdeleted file mode 100644index 25c41ee9..00000000--- a/apps/renderer-service/src/cli.ts+++ /dev/null@@ -1,101 +0,0 @@-import "reflect-metadata";-import { configureAssetUrls } from "@framefields/client-utils";-import { HeadlessMediaRenderer } from "@framefields/renderer";-import "@framefields/renderer/offscreen-gl-polyfill";-import {- container,- rendererLogger,- type StorageService,- TOKENS,-} from "@framefields/server-utils";-import { registerBackendServices } from "./di-setup.js";-import { RENDERER_ENV_CONFIG } from "./env-config.js";-import { renderAndUpload } from "./render-upload.js";--async function main() {- const jobArgIndex = process.argv.indexOf("--job");- if (jobArgIndex === -1 || !process.argv[jobArgIndex + 1]) {- console.error("Missing --job parameter containing JSON input payload");- process.exit(1);- }-- let jobData: any;- try {- jobData = JSON.parse(process.argv[jobArgIndex + 1]);- } catch (e: any) {- console.error("Failed to parse JSON job input:", e.message);- process.exit(1);- }-- const { virtualMedia, type, frame, bucket, key, codec, audioCodec, quality } =- jobData;-- if (!virtualMedia || !type || !bucket || !key) {- console.error(- "Missing required job data fields (virtualMedia, type, bucket, key)",- );- process.exit(1);- }-- try {- // Initialize configurations- configureAssetUrls({- baseUrl: RENDERER_ENV_CONFIG.BASE_URL,- r2CustomDomain: RENDERER_ENV_CONFIG.R2_CUSTOM_DOMAIN,- });-- // Boot DI services- await registerBackendServices();-- // Eagerly initialize renderer- await HeadlessMediaRenderer.initialize();-- const normalizedType = type.toLowerCase();- rendererLogger.info(- { type: normalizedType, key },- "[CLI] Starting render process...",- );-- let probedMeta: any;-- const contentType = await renderAndUpload({- virtualMedia,- type,- frame,- codec,- audioCodec,- quality,- key,- bucket,- logPrefix: "[CLI]",- });-- const storage = container.get<StorageService>(TOKENS.STORAGE);- const publicUrl = storage.getPublicUrl?.(key, bucket);- rendererLogger.info(- { key, bucket, contentType },- "[CLI] Render complete and uploaded to R2 successfully.",- );-- // Print the final result in JSON format as stdout for the wrapper script to read- console.log(- JSON.stringify({- success: true,- key,- bucket,- publicUrl,- metadata: probedMeta,- }),- );- process.exit(0);- } catch (error: any) {- rendererLogger.error(- { err: error.message || String(error) },- "[CLI] Rendering job failed.",- );- console.error(error.message || String(error));- process.exit(1);- }-}--main();diff --git a/apps/renderer-service/src/config/index.ts b/apps/renderer-service/src/config/index.tsnew file mode 100644index 00000000..8f26aad6--- /dev/null+++ b/apps/renderer-service/src/config/index.ts@@ -0,0 +1,59 @@+import { config as loadDotenv } from "dotenv";+import { registerSecret } from "../redact.js";+import { buildConfig, type RendererConfig, SECRET_ENV_KEYS } from "./schema.js";++export type { RendererConfig } from "./schema.js";++let current: RendererConfig | undefined;++/**+ * Parses the environment once. On failure, prints every problem and exits, so+ * a misconfigured worker never reports healthy.+ */+export function loadConfig(): RendererConfig {+ if (current) return current;+ loadDotenv({ quiet: true });++ const result = buildConfig(process.env);+ if (!result.ok) {+ const lines = result.problems.map((p) => ` - ${p.key}: ${p.message}`);+ console.error(`❌ CONFIG_INVALID\n${lines.join("\n")}`);+ process.exit(1);+ }+ current = result.config;++ for (const key of SECRET_ENV_KEYS) registerSecret(process.env[key]);+ for (const cred of Object.values(current.git.credentials)) {+ registerSecret(typeof cred === "string" ? cred : cred.token);+ }+ return current;+}++/**+ * Removes secrets from process.env so repo code rendered in this process+ * can't read them (PLAN.md §8.2). Call after loadConfig().+ */+export function scrubSecretEnv(): void {+ for (const key of SECRET_ENV_KEYS) delete process.env[key];+}++/** The config with secrets replaced, for the boot log. */+export function describeConfig(config: RendererConfig): unknown {+ const mask = (v: unknown) => (v ? "***" : undefined);+ return {+ ...config,+ http: { ...config.http, apiKey: mask(config.http.apiKey) },+ git: {+ ...config.git,+ token: mask(config.git.token),+ sshKey: mask(config.git.sshKey),+ credentials: Object.keys(config.git.credentials),+ },+ install: { ...config.install, npmrc: mask(config.install.npmrc) },+ storage: {+ ...config.storage,+ accessKeyId: mask(config.storage.accessKeyId),+ secretAccessKey: mask(config.storage.secretAccessKey),+ },+ };+}diff --git a/apps/renderer-service/src/config/schema.test.ts b/apps/renderer-service/src/config/schema.test.tsnew file mode 100644index 00000000..c1051b9c--- /dev/null+++ b/apps/renderer-service/src/config/schema.test.ts@@ -0,0 +1,106 @@+import { describe, expect, it } from "vitest";+import { buildConfig } from "./schema.js";++const probe = (exists = false) => ({+ exists: () => exists,+ tmpdir: () => "/tmp",+ homedir: () => "/home/u",+});++describe("buildConfig", () => {+ it("reports every problem at once", () => {+ const result = buildConfig(+ { FF_MODE: "http", STORAGE_DRIVER: "s3" },+ probe(),+ );+ expect(result.ok).toBe(false);+ if (result.ok) return;+ const keys = result.problems.map((p) => p.key);+ expect(keys).toEqual(+ expect.arrayContaining([+ "RENDERER_API_KEY",+ "FF_REPO_ALLOWLIST",+ "STORAGE_ENDPOINT",+ "STORAGE_ACCESS_KEY_ID",+ "STORAGE_SECRET_ACCESS_KEY",+ "STORAGE_BUCKET",+ ]),+ );+ });++ it("rejects ambiguous booleans instead of treating them as true", () => {+ const result = buildConfig(+ { FF_HTTP_INSECURE: "nope", FF_REPO_ALLOWLIST: "github.com/a/*" },+ probe(),+ );+ expect(result.ok).toBe(false);+ });++ it("derives the allowlist from FF_DEFAULT_REPO", () => {+ const result = buildConfig(+ {+ FF_MODE: "runpod-serverless",+ FF_DEFAULT_REPO: "[email protected]:Acme/Films.git",+ },+ probe(),+ );+ expect(result.ok && result.config.source.allowlist).toEqual([+ "github.com/Acme/Films",+ ]);+ });++ it("defaults to serverless mode on RunPod, bound to localhost", () => {+ const result = buildConfig(+ { RUNPOD_POD_ID: "abc", FF_REPO_ALLOWLIST: "github.com/a/*" },+ probe(true),+ );+ expect(result.ok).toBe(true);+ if (!result.ok) return;+ expect(result.config.mode).toBe("runpod-serverless");+ expect(result.config.http.host).toBe("127.0.0.1");+ expect(result.config.paths.cacheDir).toBe("/runpod-volume/framefields");+ });++ it("parses JSON credentials and host kinds", () => {+ const result = buildConfig(+ {+ FF_MODE: "cli",+ FF_REPO_ALLOWLIST: "git.acme.dev/**",+ FF_GIT_CREDENTIALS:+ '{"git.acme.dev":{"username":"bot","token":"t0ken123"}}',+ FF_GIT_HOSTS: '{"git.acme.dev":"gitea"}',+ },+ probe(),+ );+ expect(result.ok).toBe(true);+ if (!result.ok) return;+ expect(result.config.git.hosts["git.acme.dev"]).toBe("gitea");+ expect(Object.isFrozen(result.config.git)).toBe(true);+ });++ it("flags malformed JSON", () => {+ const result = buildConfig(+ { FF_MODE: "cli", FF_REPO_ALLOWLIST: "x/*", FF_GIT_HOSTS: "{oops" },+ probe(),+ );+ expect(result.ok).toBe(false);+ });++ it("defaults output buckets to STORAGE_BUCKET", () => {+ const result = buildConfig(+ {+ FF_MODE: "cli",+ FF_REPO_ALLOWLIST: "github.com/a/*",+ STORAGE_ENDPOINT: "https://s3.example.com",+ STORAGE_ACCESS_KEY_ID: "key",+ STORAGE_SECRET_ACCESS_KEY: "secret",+ STORAGE_BUCKET: "renders",+ },+ probe(),+ );+ expect(result.ok && result.config.storage).toMatchObject({+ driver: "s3",+ outputBuckets: ["renders"],+ });+ });+});diff --git a/apps/renderer-service/src/config/schema.ts b/apps/renderer-service/src/config/schema.tsnew file mode 100644index 00000000..c76e5e65--- /dev/null+++ b/apps/renderer-service/src/config/schema.ts@@ -0,0 +1,378 @@+import { existsSync } from "node:fs";+import os from "node:os";+import path from "node:path";+import z from "zod";+import { parseRepoUrl } from "../git/repo-url.js";++/** Env booleans: only explicit true/false spellings are accepted. */+const bool = (fallback: boolean) =>+ z+ .enum(["true", "false", "1", "0", "yes", "no"])+ .optional()+ .transform((v) =>+ v === undefined ? fallback : ["true", "1", "yes"].includes(v),+ );++const int = (fallback: number) =>+ z.coerce.number().int().min(0).default(fallback);++const list = z+ .string()+ .optional()+ .transform((v) =>+ (v ?? "")+ .split(",")+ .map((s) => s.trim())+ .filter(Boolean),+ );++const json = <T extends z.ZodType>(schema: T) =>+ z+ .string()+ .optional()+ .transform((v, ctx) => {+ if (!v) return undefined;+ try {+ return JSON.parse(v) as unknown;+ } catch {+ ctx.addIssue({ code: "custom", message: "must be valid JSON" });+ return z.NEVER;+ }+ })+ .pipe(schema.optional());++const credential = z.union([+ z.string().min(1),+ z.object({ username: z.string().optional(), token: z.string().min(1) }),+]);++export const HOST_KINDS = [+ "github",+ "gitlab",+ "bitbucket",+ "gitea",+ "generic",+] as const;++export const envSchema = z.object({+ // Mode+ NODE_ENV: z+ .enum(["development", "production", "staging", "test"])+ .default("production"),+ FF_MODE: z.enum(["runpod-serverless", "http", "cli"]).optional(),+ RUNPOD_POD_ID: z.string().optional(),+ RENDERER_HOST: z.string().optional(),+ RENDERER_PORT: z.coerce.number().int().positive().optional(),+ PORT: z.coerce.number().int().positive().optional(),+ RENDERER_API_KEY: z.string().min(16, "use at least 16 characters").optional(),+ FF_HTTP_INSECURE: bool(false),++ // Source+ FF_DEFAULT_REPO: z.string().optional(),+ FF_DEFAULT_REF: z.string().default("main"),+ FF_REPO_ALLOWLIST: list,+ FF_ALLOW_LOCAL_REPOS: bool(false),+ FF_GIT_FETCH: z.enum(["mirror", "single"]).default("mirror"),+ FF_LFS_CONCURRENCY: int(8),+ FF_LFS_DEFAULT_INCLUDE: list,++ // Git auth & hosts+ GIT_TOKEN: z.string().optional(),+ GIT_USERNAME: z.string().optional(),+ FF_GIT_CREDENTIALS: json(z.record(z.string(), credential)),+ FF_GIT_HOSTS: json(z.record(z.string(), z.enum(HOST_KINDS))),+ FF_GIT_CA_BUNDLE: z.string().optional(),+ GIT_SSH_KEY: z.string().optional(),+ GIT_SSH_KNOWN_HOSTS: z.string().optional(),+ FF_SSH_ACCEPT_NEW: bool(false),+ FF_ALLOW_JOB_CREDENTIALS: bool(false),++ // Install+ FF_INSTALL_SCRIPTS: bool(true),+ FF_ALLOW_UNLOCKED_INSTALL: bool(false),+ FF_NPMRC: z.string().optional(),+ FF_ENGINE_POLICY: z.enum(["warn", "fail"]).default("warn"),++ // Storage+ STORAGE_DRIVER: z.enum(["s3", "local", "none"]).optional(),+ STORAGE_ENDPOINT: z.string().optional(),+ STORAGE_REGION: z.string().default("auto"),+ STORAGE_ACCESS_KEY_ID: z.string().optional(),+ STORAGE_SECRET_ACCESS_KEY: z.string().optional(),+ STORAGE_BUCKET: z.string().optional(),+ STORAGE_PUBLIC_BASE_URL: z.string().optional(),+ STORAGE_FORCE_PATH_STYLE: bool(false),+ FF_OUTPUT_BUCKETS: list,+ FF_OUTPUT_DIR: z.string().optional(),+ FF_INLINE_MAX_BYTES: int(5 * 1024 * 1024),+ FF_OUTPUT_MAX_BYTES: int(20 * 1024 * 1024 * 1024),++ // Paths & cache+ FF_CACHE_DIR: z.string().optional(),+ FF_WORK_DIR: z.string().optional(),+ FF_CACHE_MAX_GB: int(50),+ FF_WORK_MAX_GB: int(40),++ // Limits+ FF_QUEUE_MAX: int(8),+ FF_JOB_TIMEOUT_MAX_SEC: int(7200),+ FF_TIMEOUT_GIT_SEC: int(600),+ FF_TIMEOUT_LFS_SEC: int(1800),+ FF_TIMEOUT_INSTALL_SEC: int(900),+ FF_TIMEOUT_LOAD_SEC: int(120),+ FF_TIMEOUT_RENDER_SEC: int(7200),+ FF_STALL_TIMEOUT_SEC: int(300),+ FF_KILL_GRACE_SEC: int(20),+ FF_JOB_TTL_SEC: int(3600),+ FF_JOB_HISTORY_MAX: int(500),++ // Recycling+ FF_MAX_JOBS_PER_WORKER: int(25),+ FF_RECYCLE_HEAP_MB: int(4096),+ FF_RECYCLE_RSS_MB: int(0),+ FF_EVENT_LOOP_BLOCK_SEC: int(120),+ FF_STRAY_ERROR_LIMIT: int(3),+ FF_LEAKED_HANDLE_LIMIT: int(50),++ // Job env+ FF_JOB_ENV_ALLOW: list,+ FF_KEEP_FAILED_JOBS: bool(false),+ FF_QA_BLOCK_UPLOAD: bool(false),+});++export type RawEnv = z.infer<typeof envSchema>;++/** Env vars removed from process.env before repo code runs (PLAN.md §8.2). */+export const SECRET_ENV_KEYS = [+ "GIT_TOKEN",+ "GIT_SSH_KEY",+ "FF_GIT_CREDENTIALS",+ "FF_NPMRC",+ "STORAGE_ACCESS_KEY_ID",+ "STORAGE_SECRET_ACCESS_KEY",+ "RENDERER_API_KEY",+] as const;++/** Job env names repo code may never set, whatever FF_JOB_ENV_ALLOW says. */+export const RESERVED_JOB_ENV = [+ /^PATH$/,+ /^HOME$/,+ /^NODE_/,+ /^LD_/,+ /^DYLD_/,+ /^GIT_/,+ /^STORAGE_/,+ /^FF_/,+ /^RUNPOD_/,+ /^RENDERER_/,+ /^NVIDIA_/,+ /^VK_/,+ /^DISPLAY$/,+ /^npm_/i,+];++export interface Probe {+ exists(p: string): boolean;+ tmpdir(): string;+ homedir(): string;+}++const defaultProbe: Probe = {+ exists: existsSync,+ tmpdir: os.tmpdir,+ homedir: os.homedir,+};++export interface ConfigProblem {+ key: string;+ message: string;+}++/**+ * Validates the whole environment and derives the effective config. Returns+ * every problem at once instead of failing on the first.+ */+export function buildConfig(+ env: NodeJS.ProcessEnv,+ probe: Probe = defaultProbe,+) {+ const parsed = envSchema.safeParse(env);+ if (!parsed.success) {+ return {+ ok: false as const,+ problems: parsed.error.issues.map((i) => ({+ key: i.path.join(".") || "(env)",+ message: i.message,+ })),+ };+ }+ const e = parsed.data;+ const problems: ConfigProblem[] = [];++ const mode = e.FF_MODE ?? (e.RUNPOD_POD_ID ? "runpod-serverless" : "http");++ if (mode === "http" && !e.RENDERER_API_KEY && !e.FF_HTTP_INSECURE) {+ problems.push({+ key: "RENDERER_API_KEY",+ message: "required in http mode (or set FF_HTTP_INSECURE=true)",+ });+ }++ const allowlist = [...e.FF_REPO_ALLOWLIST];+ if (e.FF_DEFAULT_REPO) {+ try {+ const repo = parseRepoUrl(e.FF_DEFAULT_REPO, {+ allowLocal: e.FF_ALLOW_LOCAL_REPOS,+ });+ if (allowlist.length === 0) allowlist.push(repo.id);+ } catch (err) {+ problems.push({+ key: "FF_DEFAULT_REPO",+ message: err instanceof Error ? err.message : String(err),+ });+ }+ }+ if (allowlist.length === 0) {+ problems.push({+ key: "FF_REPO_ALLOWLIST",+ message: "required, e.g. github.com/acme/* (or set FF_DEFAULT_REPO)",+ });+ }++ const storageDriver =+ e.STORAGE_DRIVER ??+ (e.STORAGE_ENDPOINT ? "s3" : e.FF_OUTPUT_DIR ? "local" : "none");+ if (storageDriver === "s3") {+ for (const key of [+ "STORAGE_ENDPOINT",+ "STORAGE_ACCESS_KEY_ID",+ "STORAGE_SECRET_ACCESS_KEY",+ "STORAGE_BUCKET",+ ] as const) {+ if (!e[key])+ problems.push({ key, message: "required when STORAGE_DRIVER=s3" });+ }+ }+ if (storageDriver === "local" && !e.FF_OUTPUT_DIR) {+ problems.push({+ key: "FF_OUTPUT_DIR",+ message: "required when STORAGE_DRIVER=local",+ });+ }++ if (problems.length > 0) return { ok: false as const, problems };++ const cacheDir =+ e.FF_CACHE_DIR ??+ (probe.exists("/runpod-volume")+ ? "/runpod-volume/framefields"+ : path.join(probe.homedir(), ".cache", "framefields-renderer"));+ const workDir = e.FF_WORK_DIR ?? path.join(probe.tmpdir(), "framefields");+ const outputBuckets = e.FF_OUTPUT_BUCKETS.length+ ? e.FF_OUTPUT_BUCKETS+ : e.STORAGE_BUCKET+ ? [e.STORAGE_BUCKET]+ : [];++ const config = {+ nodeEnv: e.NODE_ENV,+ mode,+ http: {+ host:+ e.RENDERER_HOST ??+ (mode === "runpod-serverless" ? "127.0.0.1" : "0.0.0.0"),+ port: e.RENDERER_PORT ?? e.PORT ?? 8082,+ apiKey: e.RENDERER_API_KEY,+ insecure: e.FF_HTTP_INSECURE,+ },+ source: {+ defaultRepo: e.FF_DEFAULT_REPO,+ defaultRef: e.FF_DEFAULT_REF,+ allowlist,+ allowLocal: e.FF_ALLOW_LOCAL_REPOS,+ fetch: e.FF_GIT_FETCH,+ lfsConcurrency: e.FF_LFS_CONCURRENCY,+ lfsDefaultInclude: e.FF_LFS_DEFAULT_INCLUDE,+ },+ git: {+ token: e.GIT_TOKEN,+ username: e.GIT_USERNAME,+ credentials: e.FF_GIT_CREDENTIALS ?? {},+ hosts: e.FF_GIT_HOSTS ?? {},+ caBundle: e.FF_GIT_CA_BUNDLE,+ sshKey: e.GIT_SSH_KEY,+ sshKnownHosts: e.GIT_SSH_KNOWN_HOSTS,+ sshAcceptNew: e.FF_SSH_ACCEPT_NEW,+ allowJobCredentials: e.FF_ALLOW_JOB_CREDENTIALS,+ },+ install: {+ scripts: e.FF_INSTALL_SCRIPTS,+ allowUnlocked: e.FF_ALLOW_UNLOCKED_INSTALL,+ npmrc: e.FF_NPMRC,+ enginePolicy: e.FF_ENGINE_POLICY,+ },+ storage: {+ driver: storageDriver,+ endpoint: e.STORAGE_ENDPOINT,+ region: e.STORAGE_REGION,+ accessKeyId: e.STORAGE_ACCESS_KEY_ID,+ secretAccessKey: e.STORAGE_SECRET_ACCESS_KEY,+ bucket: e.STORAGE_BUCKET,+ publicBaseUrl: e.STORAGE_PUBLIC_BASE_URL,+ forcePathStyle: e.STORAGE_FORCE_PATH_STYLE,+ outputBuckets,+ outputDir: e.FF_OUTPUT_DIR,+ inlineMaxBytes: e.FF_INLINE_MAX_BYTES,+ outputMaxBytes: e.FF_OUTPUT_MAX_BYTES,+ },+ paths: {+ cacheDir,+ workDir,+ cacheMaxGb: e.FF_CACHE_MAX_GB,+ workMaxGb: e.FF_WORK_MAX_GB,+ },+ limits: {+ queueMax: e.FF_QUEUE_MAX,+ jobTimeoutMaxSec: e.FF_JOB_TIMEOUT_MAX_SEC,+ timeouts: {+ git: e.FF_TIMEOUT_GIT_SEC,+ lfs: e.FF_TIMEOUT_LFS_SEC,+ install: e.FF_TIMEOUT_INSTALL_SEC,+ load: e.FF_TIMEOUT_LOAD_SEC,+ render: e.FF_TIMEOUT_RENDER_SEC,+ },+ stallTimeoutSec: e.FF_STALL_TIMEOUT_SEC,+ killGraceSec: e.FF_KILL_GRACE_SEC,+ jobTtlSec: e.FF_JOB_TTL_SEC,+ jobHistoryMax: e.FF_JOB_HISTORY_MAX,+ },+ recycle: {+ maxJobs: e.FF_MAX_JOBS_PER_WORKER,+ heapMb: e.FF_RECYCLE_HEAP_MB,+ rssMb: e.FF_RECYCLE_RSS_MB,+ eventLoopBlockSec: e.FF_EVENT_LOOP_BLOCK_SEC,+ strayErrorLimit: e.FF_STRAY_ERROR_LIMIT,+ leakedHandleLimit: e.FF_LEAKED_HANDLE_LIMIT,+ },+ jobEnv: {+ allow: e.FF_JOB_ENV_ALLOW.length ? e.FF_JOB_ENV_ALLOW : ["*"],+ },+ keepFailedJobs: e.FF_KEEP_FAILED_JOBS,+ qaBlockUpload: e.FF_QA_BLOCK_UPLOAD,+ };++ return { ok: true as const, config: deepFreeze(config) };+}++export type RendererConfig = Extract<+ ReturnType<typeof buildConfig>,+ { ok: true }+>["config"];++function deepFreeze<T>(value: T): T {+ if (value && typeof value === "object") {+ for (const v of Object.values(value)) deepFreeze(v);+ Object.freeze(value);+ }+ return value;+}diff --git a/apps/renderer-service/src/di-setup.ts b/apps/renderer-service/src/di-setup.tsdeleted file mode 100644index 166ac8f3..00000000--- a/apps/renderer-service/src/di-setup.ts+++ /dev/null@@ -1,25 +0,0 @@-import { ServerMediaService } from "@framefields/media/server";-import {- container,- logger,- R2StorageService,- TOKENS,-} from "@framefields/server-utils";-import { RENDERER_ENV_CONFIG } from "./env-config.js";--/**- * Register backend-specific services into the DI container.- */-export async function registerBackendServices() {- // Register generic values- container.bind(TOKENS.ENV).toConstantValue(RENDERER_ENV_CONFIG);- container.bind(TOKENS.LOGGER).toConstantValue(logger);-- container.bind(TOKENS.STORAGE).to(R2StorageService).inSingletonScope();-- // Register Media- container.bind(TOKENS.MEDIA).to(ServerMediaService).inSingletonScope();-- logger.info("Renderer services registered successfully.");- return container;-}diff --git a/apps/renderer-service/src/env-config.ts b/apps/renderer-service/src/env-config.tsdeleted file mode 100644index 995bfc82..00000000--- a/apps/renderer-service/src/env-config.ts+++ /dev/null@@ -1,40 +0,0 @@-import { rendererLogger } from "@framefields/server-utils";-import { config } from "dotenv";-import z from "zod";--const envSchema = z.object({- NODE_ENV: z- .enum(["development", "production", "staging", "test"])- .default("development"),- BASE_URL: z.url(),- RENDERER_PORT: z.coerce- .number()- .default(() => Number(process.env.PORT || 8082)),- R2_ACCESS_KEY_ID: z.string(),- R2_SECRET_ACCESS_KEY: z.string(),- R2_S3_API_ENDPOINT: z.string(),- R2_ASSETS_BUCKET: z.string(),- R2_CUSTOM_DOMAIN: z.string().optional(),-});--config();--const parsed = envSchema.safeParse(process.env);--if (!parsed.success) {- const formatted = parsed.error.format();- console.error(- "❌ Invalid environment variables:",- JSON.stringify(formatted, null, 2),- );- rendererLogger.error(- { err: z.treeifyError(parsed.error) },- "❌ Invalid environment variables",- );- throw new Error(- `Invalid environment variables: ${JSON.stringify(formatted)}`,- );-}-export const RENDERER_ENV_CONFIG = parsed.data;--export type RendererEnvConfig = z.infer<typeof envSchema>;diff --git a/apps/renderer-service/src/errors.ts b/apps/renderer-service/src/errors.tsnew file mode 100644index 00000000..3b165a42--- /dev/null+++ b/apps/renderer-service/src/errors.ts@@ -0,0 +1,144 @@+/**+ * Stable error codes for render jobs (PLAN.md §10). `retryable` tells RunPod+ * or the caller whether running the same job again can succeed.+ */++export type JobPhase =+ | "boot"+ | "validate"+ | "resolve"+ | "fetch"+ | "lfs"+ | "install"+ | "load"+ | "render"+ | "verify"+ | "upload"+ | "any";++interface CodeInfo {+ phase: JobPhase;+ retryable: boolean;+}++const CODES = {+ CONFIG_INVALID: { phase: "boot", retryable: false },+ GPU_UNAVAILABLE: { phase: "boot", retryable: false },+ TOOL_MISSING: { phase: "boot", retryable: false },++ JOB_INVALID: { phase: "validate", retryable: false },+ REPO_NOT_ALLOWED: { phase: "validate", retryable: false },+ BUCKET_NOT_ALLOWED: { phase: "validate", retryable: false },+ PATH_ESCAPE: { phase: "validate", retryable: false },+ BUSY: { phase: "validate", retryable: true },++ GIT_AUTH_FAILED: { phase: "resolve", retryable: false },+ REPO_NOT_FOUND: { phase: "resolve", retryable: false },+ REF_NOT_FOUND: { phase: "resolve", retryable: false },+ REF_AMBIGUOUS: { phase: "resolve", retryable: false },+ COMMIT_MISMATCH: { phase: "resolve", retryable: false },+ GIT_TLS_FAILED: { phase: "resolve", retryable: false },++ GIT_NETWORK: { phase: "fetch", retryable: true },+ GIT_CACHE_CORRUPT: { phase: "fetch", retryable: true },++ LFS_AUTH_FAILED: { phase: "lfs", retryable: false },+ LFS_QUOTA_EXCEEDED: { phase: "lfs", retryable: false },+ LFS_UNAVAILABLE: { phase: "lfs", retryable: false },+ LFS_OBJECT_MISSING: { phase: "lfs", retryable: false },+ LFS_POINTER_UNRESOLVED: { phase: "lfs", retryable: false },+ UNSUPPORTED_GIT_FILTER: { phase: "lfs", retryable: false },++ DISK_FULL: { phase: "any", retryable: true },++ LOCKFILE_MISSING: { phase: "install", retryable: false },+ INSTALL_FAILED: { phase: "install", retryable: false },+ INSTALL_TIMEOUT: { phase: "install", retryable: true },+ ENGINE_INCOMPATIBLE: { phase: "install", retryable: false },++ ENTRY_NOT_FOUND: { phase: "load", retryable: false },+ ENTRY_COMPILE_FAILED: { phase: "load", retryable: false },+ EXPORT_NOT_FOUND: { phase: "load", retryable: false },+ EXPORT_INVALID: { phase: "load", retryable: false },+ EXPORT_SIDE_EFFECT: { phase: "load", retryable: false },+ COMPOSITION_BUILD_FAILED: { phase: "load", retryable: false },++ ASSET_NOT_FOUND: { phase: "render", retryable: false },+ RENDER_FAILED: { phase: "render", retryable: false },+ RENDER_STALLED: { phase: "render", retryable: true },+ RENDER_TIMEOUT: { phase: "render", retryable: false },+ RENDER_OOM: { phase: "render", retryable: true },+ GPU_DEVICE_LOST: { phase: "render", retryable: true },+ RENDER_CRASHED: { phase: "render", retryable: true },++ OUTPUT_MISSING: { phase: "verify", retryable: false },+ OUTPUT_INVALID: { phase: "verify", retryable: false },+ OUTPUT_TOO_LARGE: { phase: "verify", retryable: false },+ QA_FAILED: { phase: "verify", retryable: false },++ UPLOAD_FAILED: { phase: "upload", retryable: true },+ UPLOAD_URL_EXPIRED: { phase: "upload", retryable: false },++ CANCELLED: { phase: "any", retryable: false },+ SHUTDOWN: { phase: "any", retryable: true },+ INTERNAL: { phase: "any", retryable: false },+} as const satisfies Record<string, CodeInfo>;++export type ErrorCode = keyof typeof CODES;++export interface RenderErrorJSON {+ code: ErrorCode;+ message: string;+ phase: JobPhase;+ retryable: boolean;+ hint?: string;+ details?: Record<string, unknown>;+}++export class RenderError extends Error {+ readonly code: ErrorCode;+ readonly phase: JobPhase;+ readonly retryable: boolean;+ readonly hint?: string;+ readonly details?: Record<string, unknown>;++ constructor(+ code: ErrorCode,+ message: string,+ options: {+ phase?: JobPhase;+ hint?: string;+ details?: Record<string, unknown>;+ cause?: unknown;+ } = {},+ ) {+ super(message, { cause: options.cause });+ this.name = "RenderError";+ this.code = code;+ this.phase = options.phase ?? CODES[code].phase;+ this.retryable = CODES[code].retryable;+ this.hint = options.hint;+ this.details = options.details;+ }++ toJSON(): RenderErrorJSON {+ return {+ code: this.code,+ message: this.message,+ phase: this.phase,+ retryable: this.retryable,+ ...(this.hint && { hint: this.hint }),+ ...(this.details && { details: this.details }),+ };+ }+}++/** Wraps anything thrown into a RenderError, keeping RenderErrors as they are. */+export function toRenderError(err: unknown, phase: JobPhase): RenderError {+ if (err instanceof RenderError) return err;+ const message = err instanceof Error ? err.message : String(err);+ if (/ENOSPC/.test(message)) {+ return new RenderError("DISK_FULL", message, { phase, cause: err });+ }+ return new RenderError("INTERNAL", message, { phase, cause: err });+}diff --git a/apps/renderer-service/src/git/auth.ts b/apps/renderer-service/src/git/auth.tsnew file mode 100644index 00000000..176b6ac4--- /dev/null+++ b/apps/renderer-service/src/git/auth.ts@@ -0,0 +1,228 @@+import { existsSync } from "node:fs";+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { fileURLToPath } from "node:url";+import type { RendererConfig } from "../config/index.js";+import { RenderError } from "../errors.js";+import { registerSecret } from "../redact.js";+import { hostProfile } from "./hosts.js";+import { isHostAllowed, type RepoRef } from "./repo-url.js";++export interface JobCredentials {+ username?: string;+ token?: string;+ sshKey?: string;+}++/**+ * Environment + `-c` flags for every git / git-lfs call of one job. Secrets+ * live only in this env (passed to the git subprocess), never in URLs,+ * `.git/config`, files in the workspace, or the worker's process.env.+ */+export interface GitSession {+ env: NodeJS.ProcessEnv;+ configArgs: string[];+ dispose(): Promise<void>;+}++/** Shell credential helper: answers `get` from FF_CRED_<HOST>_USER/_PASS. */+const HELPER = `#!/bin/sh+[ "$1" = get ] || exit 0+host=""+while IFS= read -r line; do+ [ -z "$line" ] && break+ case "$line" in host=*) host="\${line#host=}";; esac+done+[ -n "$host" ] || exit 0+key=$(printf '%s' "$host" | tr -c 'A-Za-z0-9' '_' | tr 'a-z' 'A-Z')+eval "user=\\\${FF_CRED_\${key}_USER:-}"+eval "pass=\\\${FF_CRED_\${key}_PASS:-}"+[ -n "$pass" ] || exit 0+printf 'username=%s\\npassword=%s\\n' "$user" "$pass"+`;++export function credEnvKey(host: string): string {+ return host.replace(/[^A-Za-z0-9]/g, "_").toUpperCase();+}++/** Bundled known_hosts, generated at image build (PLAN.md §6.5). */+const BUNDLED_KNOWN_HOSTS = path.resolve(+ path.dirname(fileURLToPath(import.meta.url)),+ "../../../deploy/known_hosts",+);++/**+ * Builds the per-host credential map: the configured token for the job's+ * repo host, FF_GIT_CREDENTIALS entries for allowlisted hosts, and per-job+ * credentials when enabled. Nothing is answered for other hosts (LFS CDNs).+ */+export function credentialMap(+ repo: RepoRef,+ config: RendererConfig,+ job?: JobCredentials,+): Map<string, { username: string; password: string }> {+ const map = new Map<string, { username: string; password: string }>();+ const put = (host: string, token: string, username?: string) => {+ if (!isHostAllowed(host.split(":")[0], config.source.allowlist)) return;+ const user =+ username ?? hostProfile(host.split(":")[0], config).defaultUsername;+ if (!user) {+ throw new RenderError(+ "GIT_AUTH_FAILED",+ `No username for ${host}: set GIT_USERNAME or {"username", "token"} in FF_GIT_CREDENTIALS.`,+ { phase: "resolve" },+ );+ }+ map.set(host.toLowerCase(), { username: user, password: token });+ };++ for (const [host, cred] of Object.entries(config.git.credentials)) {+ if (typeof cred === "string") put(host, cred);+ else put(host, cred.token, cred.username);+ }+ const repoHost = repo.port ? `${repo.host}:${repo.port}` : repo.host;+ if (config.git.token && !map.has(repoHost)) {+ put(repoHost, config.git.token, config.git.username);+ }+ if (job?.token) {+ registerSecret(job.token);+ put(repoHost, job.token, job.username);+ }+ return map;+}++export async function openGitSession(+ repo: RepoRef,+ config: RendererConfig,+ job?: JobCredentials,+): Promise<GitSession> {+ const dir = await fs.mkdtemp(path.join(secretTmpRoot(), "ff-git-"));+ await fs.chmod(dir, 0o700);++ const helper = path.join(dir, "credential-helper");+ await fs.writeFile(helper, HELPER, { mode: 0o700 });++ const env: NodeJS.ProcessEnv = {+ PATH: process.env.PATH,+ HOME: dir,+ LANG: "C",+ LC_ALL: "C",+ TZ: "UTC",+ // Isolate from the host's git config (credential stores, insteadOf, hooks).+ GIT_CONFIG_NOSYSTEM: "1",+ GIT_CONFIG_GLOBAL: path.join(dir, "gitconfig"),+ GIT_TERMINAL_PROMPT: "0",+ GIT_ASKPASS: "/bin/false",+ SSH_ASKPASS: "/bin/false",+ GIT_LFS_SKIP_SMUDGE: "1",+ GCM_INTERACTIVE: "never",+ };+ await fs.writeFile(env.GIT_CONFIG_GLOBAL as string, "", { mode: 0o600 });++ for (const [host, cred] of credentialMap(repo, config, job)) {+ env[`FF_CRED_${credEnvKey(host)}_USER`] = cred.username;+ env[`FF_CRED_${credEnvKey(host)}_PASS`] = cred.password;+ }++ const configArgs = [+ "-c",+ "credential.helper=",+ "-c",+ `credential.helper=${helper}`,+ "-c",+ "credential.useHttpPath=false",+ "-c",+ "core.hooksPath=/dev/null",+ "-c",+ "core.fsmonitor=false",+ "-c",+ "core.symlinks=true",+ "-c",+ "core.longpaths=true",+ "-c",+ `protocol.file.allow=${config.source.allowLocal ? "always" : "never"}`,+ "-c",+ "protocol.ext.allow=never",+ "-c",+ "submodule.recurse=false",+ "-c",+ "fetch.recurseSubmodules=false",+ "-c",+ "advice.detachedHead=false",+ "-c",+ "gc.auto=0",+ "-c",+ "maintenance.auto=false",+ "-c",+ "filter.lfs.process=git-lfs filter-process --skip",+ "-c",+ "filter.lfs.smudge=git-lfs smudge --skip -- %f",+ "-c",+ "filter.lfs.clean=git-lfs clean -- %f",+ "-c",+ "filter.lfs.required=false",+ ];++ if (config.git.caBundle) {+ const ca = path.join(dir, "ca.pem");+ await fs.writeFile(ca, config.git.caBundle, { mode: 0o600 });+ configArgs.push("-c", `http.sslCAInfo=${ca}`);+ }++ if (repo.protocol === "ssh") {+ const sshKey = job?.sshKey ?? config.git.sshKey;+ const knownHosts = path.join(dir, "known_hosts");+ const bundled = existsSync(BUNDLED_KNOWN_HOSTS)+ ? await fs.readFile(BUNDLED_KNOWN_HOSTS, "utf8")+ : "";+ await fs.writeFile(+ knownHosts,+ `${bundled}\n${config.git.sshKnownHosts ?? ""}\n`,+ {+ mode: 0o600,+ },+ );+ const ssh = [+ "ssh",+ "-o",+ "BatchMode=yes",+ "-o",+ `UserKnownHostsFile=${knownHosts}`,+ "-o",+ `StrictHostKeyChecking=${config.git.sshAcceptNew ? "accept-new" : "yes"}`,+ "-o",+ "ConnectTimeout=30",+ ];+ if (sshKey) {+ const keyFile = path.join(dir, "id_key");+ await fs.writeFile(keyFile, decodeKey(sshKey), { mode: 0o600 });+ ssh.push("-o", "IdentitiesOnly=yes", "-i", keyFile);+ }+ env.GIT_SSH_COMMAND = ssh.map(shellQuote).join(" ");+ }++ return {+ env,+ configArgs,+ dispose: () => fs.rm(dir, { recursive: true, force: true }),+ };+}++/** tmpfs where available, so keys never hit a persistent disk. */+function secretTmpRoot(): string {+ return existsSync("/dev/shm") ? "/dev/shm" : os.tmpdir();+}++function decodeKey(value: string): string {+ const text = value.includes("PRIVATE KEY")+ ? value+ : Buffer.from(value, "base64").toString("utf8");+ return text.endsWith("\n") ? text : `${text}\n`;+}++function shellQuote(arg: string): string {+ return /^[\w@%+=:,./-]+$/.test(arg) ? arg : `'${arg.replace(/'/g, "'\\''")}'`;+}++export const _internal = { HELPER };diff --git a/apps/renderer-service/src/git/checkout.test.ts b/apps/renderer-service/src/git/checkout.test.tsnew file mode 100644index 00000000..11f722aa--- /dev/null+++ b/apps/renderer-service/src/git/checkout.test.ts@@ -0,0 +1,273 @@+import { execFileSync } from "node:child_process";+import { existsSync } from "node:fs";+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { afterAll, beforeAll, describe, expect, it } from "vitest";+import { RenderError } from "../errors.js";+import type { StepContext } from "../jobs/pipeline.js";+import { validateJob } from "../jobs/validate.js";+import { testConfig, testJob } from "../test-helpers.js";+import { checkout } from "./checkout.js";+import { mirrorPath } from "./mirror.js";+import { parseRepoUrl, type RepoRef } from "./repo-url.js";+import { resolveRelativeUrl } from "./workspace.js";++let tmp: string;+let origin: string;+let originUrl: string;+const shas: Record<string, string> = {};++const gitEnv = {+ ...process.env,+ GIT_CONFIG_NOSYSTEM: "1",+ GIT_CONFIG_GLOBAL: "/dev/null",+ GIT_AUTHOR_NAME: "t",+ GIT_AUTHOR_EMAIL: "[email protected]",+ GIT_COMMITTER_NAME: "t",+ GIT_COMMITTER_EMAIL: "[email protected]",+};+const sh = (args: string[], cwd = origin) =>+ execFileSync("git", args, { cwd, env: gitEnv, encoding: "utf8" }).trim();++async function write(rel: string, content: string) {+ await fs.mkdir(path.dirname(path.join(origin, rel)), { recursive: true });+ await fs.writeFile(path.join(origin, rel), content);+}++beforeAll(async () => {+ tmp = await fs.realpath(+ await fs.mkdtemp(path.join(os.tmpdir(), "ff-git-test-")),+ );+ origin = path.join(tmp, "origin");+ await fs.mkdir(origin);+ originUrl = `file://${origin}`;+ sh(["init", "--quiet", "--initial-branch=main"]);++ await write("videos/launch/package.json", '{"name":"launch"}');+ await write("videos/launch/src/film.ts", "export const v = 1;\n");+ await write("other/readme.md", "other\n");+ sh(["add", "-A"]);+ sh(["commit", "--quiet", "-m", "one"]);+ shas.one = sh(["rev-parse", "HEAD"]);+ sh(["tag", "-a", "v1", "-m", "v1"]);++ await write("videos/launch/src/film.ts", "export const v = 2;\n");+ sh(["commit", "--quiet", "-am", "two"]);+ shas.two = sh(["rev-parse", "HEAD"]);++ // "release" is a branch at `two` and a tag at `one` → ambiguous.+ sh(["branch", "release"]);+ sh(["tag", "release", shas.one]);++ sh(["checkout", "--quiet", "-b", "feature/intro"]);+ await write("videos/launch/src/film.ts", "export const v = 3;\n");+ sh(["commit", "--quiet", "-am", "three"]);+ shas.three = sh(["rev-parse", "HEAD"]);++ sh(["checkout", "--quiet", "-b", "crypt", shas.one]);+ await write(".gitattributes", "secrets/** filter=git-crypt diff=git-crypt\n");+ await write("secrets/key.bin", "encrypted\n");+ await write(+ "videos/launch/assets/hero.mp4",+ "version https://git-lfs.github.com/spec/v1\noid sha256:abc\nsize 12\n",+ );+ sh(["add", "-A"]);+ sh(["commit", "--quiet", "-m", "crypt"]);++ sh(["checkout", "--quiet", "main"]);+});++afterAll(async () => {+ await fs.rm(tmp, { recursive: true, force: true });+});++function setup(+ overrides: Record<string, unknown> = {},+ env: Record<string, string> = {},+) {+ const config = testConfig({+ FF_ALLOW_LOCAL_REPOS: "true",+ FF_CACHE_DIR: path.join(tmp, "cache"),+ FF_WORK_DIR: path.join(tmp, "work"),+ ...env,+ });+ const job = validateJob(+ testJob({+ source: {+ repo: originUrl,+ ref: "main",+ workdir: "videos/launch",+ ...overrides,+ },+ }),+ config,+ );+ const warnings: string[] = [];+ const controller = new AbortController();+ const ctx: StepContext = {+ jobId: "test",+ config,+ signal: controller.signal,+ jobDir: path.join(tmp, "job"),+ progress: () => {},+ warn: (code) => warnings.push(code),+ };+ return { job, ctx, warnings, controller, config };+}++async function codeOf(promise: Promise<unknown>): Promise<string> {+ try {+ await promise;+ return "no error";+ } catch (err) {+ return err instanceof RenderError+ ? err.code+ : `not a RenderError: ${String(err)}`;+ }+}++const film = (root: string) =>+ fs.readFile(path.join(root, "videos/launch/src/film.ts"), "utf8");++describe("checkout (local repos)", () => {+ it("checks out a branch and pins the SHA", async () => {+ const { job, ctx } = setup();+ const result = await checkout(job, ctx);+ expect(result.sha).toBe(shas.two);+ expect(result.workdir).toBe(path.join(result.root, "videos/launch"));+ expect(await film(result.root)).toContain("v = 2");+ expect(result.cache).toMatchObject({ workspace: "cold", lfs: "off" });+ });++ it("resolves branches with slashes, annotated tags and SHA prefixes", async () => {+ expect(+ (await checkout(setup({ ref: "feature/intro" }).job, setup().ctx)).sha,+ ).toBe(shas.three);+ expect((await checkout(setup({ ref: "v1" }).job, setup().ctx)).sha).toBe(+ shas.one,+ );+ expect(+ (await checkout(setup({ ref: "refs/tags/v1" }).job, setup().ctx)).sha,+ ).toBe(shas.one);+ expect(+ (await checkout(setup({ ref: shas.one.slice(0, 10) }).job, setup().ctx))+ .sha,+ ).toBe(shas.one);+ });++ it("reuses the warm slot, keeping node_modules and dropping other untracked files", async () => {+ const first = await checkout(setup().job, setup().ctx);+ await fs.mkdir(path.join(first.workdir, "node_modules/x"), {+ recursive: true,+ });+ await fs.writeFile(path.join(first.workdir, "output.mp4"), "stale");++ const { job, ctx } = setup({ ref: "feature/intro" });+ const second = await checkout(job, ctx);+ expect(second.cache.workspace).toBe("warm");+ expect(second.cache.mirror).toBe("hit");+ expect(await film(second.root)).toContain("v = 3");+ expect(existsSync(path.join(second.workdir, "node_modules/x"))).toBe(true);+ expect(existsSync(path.join(second.workdir, "output.mp4"))).toBe(false);+ });++ it.each([+ [{ ref: "release" }, "REF_AMBIGUOUS"],+ [{ ref: "nope" }, "REF_NOT_FOUND"],+ [{ ref: "refs/heads/nope" }, "REF_NOT_FOUND"],+ [{ ref: "deadbeefdeadbeef" }, "REF_NOT_FOUND"],+ [{ ref: "main", commit: "0000000" }, "COMMIT_MISMATCH"],+ [{ ref: "main", workdir: "missing/dir" }, "JOB_INVALID"],+ [{ ref: "crypt", workdir: "." }, "UNSUPPORTED_GIT_FILTER"],+ ])("fails %o with %s", async (overrides, code) => {+ const { job, ctx } = setup(overrides);+ expect(await codeOf(checkout(job, ctx))).toBe(code);+ });++ it("accepts the commit check when it matches", async () => {+ const { job, ctx } = setup({ commit: shas.two.slice(0, 8) });+ expect((await checkout(job, ctx)).sha).toBe(shas.two);+ });++ it("warns about LFS pointers left in the workdir when LFS is off", async () => {+ const { job, ctx, warnings } = setup({+ ref: "crypt",+ lfs: { enabled: false },+ });+ // The crypt branch's filtered files are outside this workdir.+ await checkout(job, ctx);+ expect(warnings).toContain("LFS_POINTERS_LEFT");+ });++ it("rejects a workdir that escapes through a symlink", async () => {+ sh(["checkout", "--quiet", "-b", "escape", "main"]);+ await fs.symlink("/tmp", path.join(origin, "videos/out"));+ sh(["add", "-A"]);+ sh(["commit", "--quiet", "-m", "symlink"]);+ sh(["checkout", "--quiet", "main"]);+ const { job, ctx } = setup({ ref: "escape", workdir: "videos/out" });+ expect(await codeOf(checkout(job, ctx))).toBe("PATH_ESCAPE");+ });++ it("re-clones a corrupt mirror", async () => {+ const { job, ctx, config } = setup();+ await checkout(job, ctx);+ const repo = parseRepoUrl(originUrl, { allowLocal: true });+ await fs.writeFile(path.join(mirrorPath(config, repo), "HEAD"), "garbage");+ const again = setup();+ const result = await checkout(again.job, again.ctx);+ expect(result.sha).toBe(shas.two);+ expect(result.cache.mirror).toBe("recloned");+ expect(result.cache.workspace).toBe("cold");+ });++ it("fetches only the sparse paths plus the workdir", async () => {+ const { job, ctx } = setup({ sparse: ["videos"] });+ const result = await checkout(job, ctx);+ expect(+ existsSync(path.join(result.root, "videos/launch/src/film.ts")),+ ).toBe(true);+ expect(existsSync(path.join(result.root, "other/readme.md"))).toBe(false);++ const full = setup();+ const after = await checkout(full.job, full.ctx);+ expect(existsSync(path.join(after.root, "other/readme.md"))).toBe(true);+ });++ it("stops when the job is aborted", async () => {+ const { job, ctx, controller } = setup();+ controller.abort(new RenderError("CANCELLED", "x"));+ expect(await codeOf(checkout(job, ctx))).toBe("CANCELLED");+ });++ it("refuses file:// repos unless allowed", () => {+ expect(() =>+ validateJob(testJob({ source: { repo: originUrl } }), testConfig()),+ ).toThrow(/FF_ALLOW_LOCAL_REPOS/);+ });+});++describe("submodule URLs", () => {+ const parent = (url: string): RepoRef => parseRepoUrl(url);+ it("resolves relative URLs against https and scp-like parents", () => {+ expect(+ resolveRelativeUrl(+ "../shared.git",+ parent("https://github.com/acme/films.git"),+ ),+ ).toBe("https://github.com/acme/shared.git");+ expect(+ resolveRelativeUrl(+ "../shared.git",+ parent("[email protected]:acme/films.git"),+ ),+ ).toBe("[email protected]:acme/shared.git");+ expect(+ resolveRelativeUrl(+ "https://x.dev/a/b",+ parent("https://github.com/acme/films"),+ ),+ ).toBe("https://x.dev/a/b");+ });+});diff --git a/apps/renderer-service/src/git/checkout.ts b/apps/renderer-service/src/git/checkout.tsnew file mode 100644index 00000000..4b2bafd4--- /dev/null+++ b/apps/renderer-service/src/git/checkout.ts@@ -0,0 +1,172 @@+import fs from "node:fs/promises";+import path from "node:path";+import { RenderError } from "../errors.js";+import type { CheckoutResult, Pipeline } from "../jobs/pipeline.js";+import { resolveInside } from "../paths.js";+import { openGitSession } from "./auth.js";+import { hostProfile } from "./hosts.js";+import { findPointers, pullLfs } from "./lfs.js";+import {+ type GitContext,+ git,+ listRemote,+ resolveRef,+ syncMirror,+} from "./mirror.js";+import { checkoutWorkspace, initSubmodules } from "./workspace.js";++/**+ * Pipeline step: resolve → fetch → checkout → submodules → LFS (PLAN.md §6.2).+ * The ref is resolved once; everything after uses the SHA.+ */+export const checkout: Pipeline["checkout"] = async (job, ctx) => {+ const { input, repo, ref } = job;+ const source = input.source;+ const session = await openGitSession(repo, ctx.config, source.auth);+ const gctx: GitContext = {+ config: ctx.config,+ session,+ profile: hostProfile(repo.host, ctx.config),+ signal: ctx.signal,+ };++ try {+ ctx.progress({ phase: "resolving" });+ const spec = resolveRef(ref, await listRemote(gctx, repo));++ ctx.progress({ phase: "fetching" });+ const mirror = await syncMirror(gctx, repo, spec);+ if (source.commit && !mirror.sha.startsWith(source.commit.toLowerCase())) {+ throw new RenderError(+ "COMMIT_MISMATCH",+ `${ref} is at ${mirror.sha.slice(0, 12)}, not ${source.commit}.`,+ {+ phase: "resolve",+ hint: "The branch moved since the job was created.",+ },+ );+ }++ const ws = await checkoutWorkspace(gctx, repo, mirror.path, mirror.sha, {+ sparse: sparsePaths(source.sparse, source.workdir),+ });+ if (source.submodules) await initSubmodules(gctx, ws.root, repo);++ const workdir = await resolveInside(ws.root, source.workdir).catch(+ (err) => {+ if (err instanceof RenderError) throw err;+ throw new RenderError(+ "JOB_INVALID",+ `workdir "${source.workdir}" does not exist at ${mirror.sha.slice(0, 12)}.`,+ {+ phase: "fetch",+ },+ );+ },+ );+ const scope = source.workdir === "." ? [] : [source.workdir];+ await assertSupportedFilters(gctx, ws.root, scope);++ ctx.progress({ phase: "lfs" });+ const include = source.lfs.include.length+ ? source.lfs.include+ : ctx.config.source.lfsDefaultInclude;+ const lfs = await pullLfs(ws.root, gctx, {+ enabled: source.lfs.enabled,+ include,+ exclude: source.lfs.exclude,+ scope: include.length ? [] : scope,+ });+ const leftover = await findPointers(ws.root, source.workdir);+ if (leftover.length > 0) {+ ctx.warn(+ "LFS_POINTERS_LEFT",+ `LFS pointer files in the workdir were not downloaded (${leftover.slice(0, 5).join(", ")}${leftover.length > 5 ? ", …" : ""}). Widen source.lfs.include if the render needs them.`,+ );+ }++ return {+ root: ws.root,+ workdir,+ sha: mirror.sha,+ cache: {+ mirror: mirror.cache,+ workspace: ws.warm ? "warm" : "cold",+ lfs: lfs.cache,+ },+ release: async () => {},+ } satisfies CheckoutResult;+ } finally {+ await session.dispose();+ }+};++/** Sparse checkouts always include the workdir itself. */+function sparsePaths(+ sparse: string[] | undefined,+ workdir: string,+): string[] | undefined {+ if (!sparse?.length) return undefined;+ return workdir === "." || sparse.includes(workdir)+ ? sparse+ : [...sparse, workdir];+}++/**+ * Files in scope routed through a filter other than LFS (git-crypt, …) would+ * be checked out encrypted or as placeholders: refuse them up front.+ */+async function assertSupportedFilters(+ ctx: GitContext,+ root: string,+ scope: string[],+): Promise<void> {+ const { stdout: attrFiles } = await git(ctx, [+ "-C",+ root,+ "ls-files",+ "-z",+ "--",+ ".gitattributes",+ ":(glob)**/.gitattributes",+ ]).catch(() => ({ stdout: "" }));+ let suspicious = false;+ for (const file of attrFiles.split("\0").filter(Boolean)) {+ const text = await fs+ .readFile(path.join(root, file), "utf8")+ .catch(() => "");+ if (/filter=(?!lfs\b)[\w-]+/.test(text)) suspicious = true;+ }+ if (!suspicious) return;++ const { stdout: tracked } = await git(ctx, [+ "-C",+ root,+ "ls-files",+ "-z",+ "--",+ ...(scope.length ? scope : ["."]),+ ]);+ const { stdout } = await git(+ ctx,+ ["-C", root, "check-attr", "--stdin", "-z", "filter"],+ { input: tracked },+ );+ const parts = stdout.split("\0");+ const offending: string[] = [];+ for (let i = 0; i + 2 < parts.length; i += 3) {+ const [file, , value] = [parts[i], parts[i + 1], parts[i + 2]];+ if (!["unspecified", "unset", "lfs", "set", ""].includes(value))+ offending.push(`${file} (${value})`);+ }+ if (offending.length > 0) {+ throw new RenderError(+ "UNSUPPORTED_GIT_FILTER",+ `Files use git filters the renderer can't apply: ${offending.slice(0, 5).join(", ")}${offending.length > 5 ? ", …" : ""}`,+ {+ phase: "lfs",+ hint: "Only LFS is supported; decrypted assets must be committed or served over HTTPS.",+ },+ );+ }+}diff --git a/apps/renderer-service/src/git/exec.ts b/apps/renderer-service/src/git/exec.tsnew file mode 100644index 00000000..20bbac8c--- /dev/null+++ b/apps/renderer-service/src/git/exec.ts@@ -0,0 +1,239 @@+import { spawn } from "node:child_process";+import { type ErrorCode, type JobPhase, RenderError } from "../errors.js";+import { redact } from "../redact.js";++export interface ExecOptions {+ cwd?: string;+ env?: NodeJS.ProcessEnv;+ timeoutMs?: number;+ signal?: AbortSignal;+ input?: string;+ /** Cap on captured stdout, to avoid buffering huge listings. */+ maxStdoutBytes?: number;+}++export interface ExecResult {+ stdout: string;+ stderr: string;+ code: number;+}++export class CommandError extends Error {+ constructor(+ readonly command: string,+ readonly result: ExecResult & { timedOut: boolean; aborted: boolean },+ ) {+ super(+ `${command} ${result.timedOut ? "timed out" : `exited with ${result.code}`}: ${tail(result.stderr || result.stdout)}`,+ );+ this.name = "CommandError";+ }+}++/**+ * Spawns a command without a shell. Kills it (whole process group) on+ * timeout or abort. Rejects with CommandError on a non-zero exit.+ */+export function exec(+ file: string,+ args: readonly string[],+ options: ExecOptions = {},+): Promise<ExecResult> {+ return new Promise((resolve, reject) => {+ if (options.signal?.aborted) {+ reject(options.signal.reason);+ return;+ }+ const child = spawn(file, args, {+ cwd: options.cwd,+ env: options.env ?? process.env,+ stdio: ["pipe", "pipe", "pipe"],+ detached: process.platform !== "win32",+ });+ const max = options.maxStdoutBytes ?? 64 * 1024 * 1024;+ let stdout = "";+ let stderr = "";+ let timedOut = false;+ let aborted = false;++ child.stdout.setEncoding("utf8").on("data", (d: string) => {+ if (stdout.length < max) stdout += d;+ });+ child.stderr.setEncoding("utf8").on("data", (d: string) => {+ stderr = (stderr + d).slice(-64 * 1024);+ });++ const kill = () => {+ try {+ if (child.pid && process.platform !== "win32")+ process.kill(-child.pid, "SIGKILL");+ else child.kill("SIGKILL");+ } catch {+ // already gone+ }+ };+ const timer = options.timeoutMs+ ? setTimeout(() => {+ timedOut = true;+ kill();+ }, options.timeoutMs)+ : undefined;+ const onAbort = () => {+ aborted = true;+ kill();+ };+ options.signal?.addEventListener("abort", onAbort, { once: true });++ child.on("error", (err) => {+ clearTimeout(timer);+ options.signal?.removeEventListener("abort", onAbort);+ reject(err);+ });+ child.on("close", (code) => {+ clearTimeout(timer);+ options.signal?.removeEventListener("abort", onAbort);+ if (aborted) {+ reject(options.signal?.reason);+ return;+ }+ const result = { stdout, stderr, code: code ?? -1 };+ if (code === 0 && !timedOut) resolve(result);+ else+ reject(+ new CommandError(`${file} ${args[0] ?? ""}`.trim(), {+ ...result,+ timedOut,+ aborted,+ }),+ );+ });++ child.stdin.end(options.input ?? "");+ });+}++function tail(text: string, lines = 12): string {+ return redact(text.trim().split("\n").slice(-lines).join("\n"));+}++interface Rule {+ test: RegExp;+ code: ErrorCode;+ hint?: string;+}++/** Git and git-lfs messages → stable codes. Host-specific rules come first. */+const COMMON_RULES: Rule[] = [+ { test: /No space left on device|ENOSPC/i, code: "DISK_FULL" },+ {+ test: /SSL certificate problem|unable to get local issuer certificate|certificate verify failed/i,+ code: "GIT_TLS_FAILED",+ hint: "For a self-signed server, set FF_GIT_CA_BUNDLE to its CA certificate (PEM).",+ },+ {+ test: /Host key verification failed|REMOTE HOST IDENTIFICATION HAS CHANGED/i,+ code: "GIT_AUTH_FAILED",+ hint: "Add the server's key to GIT_SSH_KNOWN_HOSTS (or set FF_SSH_ACCEPT_NEW=true).",+ },+ {+ test: /SAML|single sign-on|SSO/i,+ code: "GIT_AUTH_FAILED",+ hint: "Authorize the token for the organization's SSO.",+ },+ {+ test: /Authentication failed|could not read (Username|Password)|terminal prompts disabled|Permission denied \(publickey|HTTP Basic: Access denied|invalid credentials|\b401\b|Invalid username or (password|token)/i,+ code: "GIT_AUTH_FAILED",+ hint: "Check GIT_TOKEN / FF_GIT_CREDENTIALS (or GIT_SSH_KEY) and that it can read this repo.",+ },+ {+ test: /Repository not found|repository '.*' not found|does not appear to be a git repository|Project not found|The project you were looking for could not be found|\b404\b/i,+ code: "REPO_NOT_FOUND",+ hint: "Private repos also report 'not found' when the token lacks access.",+ },+ {+ test: /Could not resolve host|Connection (timed out|refused|reset)|Operation timed out|early EOF|RPC failed|unexpected disconnect|\b50[234]\b|SSL_ERROR_SYSCALL|TLS connection was non-properly terminated|Failed to connect/i,+ code: "GIT_NETWORK",+ },+];++const LFS_RULES: Rule[] = [+ { test: /No space left on device|ENOSPC/i, code: "DISK_FULL" },+ {+ test: /over its data quota|exceeded its LFS (budget|bandwidth)|bandwidth limit|storage limit|LFS objects are not available|repository storage limit|quota/i,+ code: "LFS_QUOTA_EXCEEDED",+ hint: "The host's LFS quota is used up; buy more bandwidth/storage or move LFS to another server (.lfsconfig).",+ },+ {+ test: /LFS (is )?(not enabled|disabled)|LFS support is disabled|Git LFS is disabled/i,+ code: "LFS_UNAVAILABLE",+ },+ {+ test: /Object does not exist on the server|\[404\] Object|object .* not found|missing object/i,+ code: "LFS_OBJECT_MISSING",+ hint: "A pointer was committed without pushing its object; run `git lfs push --all` from a machine that has it.",+ },+ {+ test: /Authentication failed|\b401\b|\b403\b|credentials|Permission denied/i,+ code: "LFS_AUTH_FAILED",+ },+ {+ test: /Could not resolve host|timed out|Connection (refused|reset)|\b50[234]\b|EOF/i,+ code: "GIT_NETWORK",+ },+];++/**+ * Maps a failed git / git-lfs command to a RenderError. `extraRules` are host+ * specific and checked first.+ */+export function classifyGitError(+ err: unknown,+ phase: JobPhase,+ options: {+ lfs?: boolean;+ extraRules?: readonly Rule[];+ fallback?: ErrorCode;+ } = {},+): RenderError {+ if (err instanceof RenderError) return err;+ if (!(err instanceof CommandError)) {+ const message = err instanceof Error ? err.message : String(err);+ if (/ENOENT/.test(message)) {+ return new RenderError(+ "TOOL_MISSING",+ `A required tool is missing: ${message}`,+ { phase },+ );+ }+ return new RenderError(options.fallback ?? "INTERNAL", redact(message), {+ phase,+ cause: err,+ });+ }+ if (err.result.timedOut) {+ return new RenderError("GIT_NETWORK", `${err.command} timed out.`, {+ phase,+ hint: "Raise FF_TIMEOUT_GIT_SEC / FF_TIMEOUT_LFS_SEC for very large repos.",+ });+ }+ const output = `${err.result.stderr}\n${err.result.stdout}`;+ const rules = [+ ...(options.extraRules ?? []),+ ...(options.lfs ? LFS_RULES : COMMON_RULES),+ ];+ for (const rule of rules) {+ if (rule.test.test(output)) {+ return new RenderError(rule.code, err.message, {+ phase,+ hint: rule.hint,+ cause: err,+ });+ }+ }+ return new RenderError(options.fallback ?? "INTERNAL", err.message, {+ phase,+ cause: err,+ });+}++export type { Rule as ErrorRule };diff --git a/apps/renderer-service/src/git/hosts.ts b/apps/renderer-service/src/git/hosts.tsnew file mode 100644index 00000000..d4230beb--- /dev/null+++ b/apps/renderer-service/src/git/hosts.ts@@ -0,0 +1,79 @@+import type { RendererConfig } from "../config/index.js";+import type { HOST_KINDS } from "../config/schema.js";+import type { ErrorRule } from "./exec.js";++export type HostKind = (typeof HOST_KINDS)[number];++export interface HostProfile {+ kind: HostKind;+ /** HTTPS username to pair with a token; undefined means GIT_USERNAME is required. */+ defaultUsername?: string;+ /** Host-specific error rules, checked before the generic ones. */+ rules: ErrorRule[];+ lfsRules: ErrorRule[];+}++const KNOWN_HOSTS: Record<string, HostKind> = {+ "github.com": "github",+ "gitlab.com": "gitlab",+ "bitbucket.org": "bitbucket",+};++const PROFILES: Record<HostKind, Omit<HostProfile, "kind">> = {+ github: {+ defaultUsername: "x-access-token",+ rules: [+ {+ test: /SAML|organization has enabled or enforced SAML SSO/i,+ code: "GIT_AUTH_FAILED",+ hint: "Authorize the token for the organization's SAML SSO in GitHub settings.",+ },+ ],+ lfsRules: [+ {+ test: /over its data quota|exceeded its LFS budget/i,+ code: "LFS_QUOTA_EXCEEDED",+ hint: "GitHub LFS bandwidth/storage is used up; buy a data pack or move LFS elsewhere.",+ },+ ],+ },+ gitlab: {+ defaultUsername: "oauth2",+ rules: [],+ lfsRules: [+ {+ test: /LFS objects are not available|size limit|storage limit/i,+ code: "LFS_QUOTA_EXCEEDED",+ hint: "The GitLab project or namespace hit its storage limit.",+ },+ ],+ },+ bitbucket: {+ defaultUsername: "x-token-auth",+ rules: [],+ lfsRules: [+ {+ test: /repository storage limit|over (the|its) .*limit/i,+ code: "LFS_QUOTA_EXCEEDED",+ hint: "The Bitbucket workspace hit its LFS storage limit.",+ },+ ],+ },+ gitea: {+ rules: [],+ lfsRules: [+ {+ test: /LFS (is )?(not enabled|disabled)|\b404\b.*batch/i,+ code: "LFS_UNAVAILABLE",+ hint: "Enable LFS on the Gitea/Forgejo server (LFS_START_SERVER = true).",+ },+ ],+ },+ generic: { rules: [], lfsRules: [] },+};++export function hostProfile(host: string, config: RendererConfig): HostProfile {+ const h = host.toLowerCase();+ const kind: HostKind = config.git.hosts[h] ?? KNOWN_HOSTS[h] ?? "generic";+ return { kind, ...PROFILES[kind] };+}diff --git a/apps/renderer-service/src/git/lfs.ts b/apps/renderer-service/src/git/lfs.tsnew file mode 100644index 00000000..4bf0f56f--- /dev/null+++ b/apps/renderer-service/src/git/lfs.ts@@ -0,0 +1,242 @@+import fs from "node:fs/promises";+import path from "node:path";+import { RenderError } from "../errors.js";+import { CommandError, classifyGitError } from "./exec.js";+import { type GitContext, git } from "./mirror.js";++const POINTER_HEADER = "version https://git-lfs.github.com/spec/v1";+const DISK_MARGIN_BYTES = 2 * 1024 * 1024 * 1024;++export interface LfsOptions {+ enabled?: boolean;+ include: string[];+ exclude: string[];+ /** Directories that must be fully usable (workdir, sparse paths). */+ scope: string[];+}++export interface LfsResult {+ used: boolean;+ files: number;+ bytes: number;+ cache: "hit" | "partial" | "miss" | "off";+}++interface LfsFile {+ name: string;+ size: number;+ downloaded?: boolean;+}++/** True when any tracked .gitattributes routes files through LFS. */+export async function usesLfs(root: string, ctx: GitContext): Promise<boolean> {+ const { stdout } = await git(ctx, [+ "-C",+ root,+ "ls-files",+ "-z",+ "--",+ ":(glob)**/.gitattributes",+ ".gitattributes",+ ]).catch(() => ({ stdout: "" }));+ for (const file of stdout.split("\0").filter(Boolean)) {+ const text = await fs+ .readFile(path.join(root, file), "utf8")+ .catch(() => "");+ if (/filter=lfs/.test(text)) return true;+ }+ return false;+}++/**+ * Pulls LFS objects for the include/exclude patterns into the shared object+ * store, with a disk-space preflight and a pointer-file postflight (§6.2.5).+ */+export async function pullLfs(+ root: string,+ ctx: GitContext,+ options: LfsOptions,+): Promise<LfsResult> {+ const enabled = options.enabled ?? (await usesLfs(root, ctx));+ if (!enabled) return { used: false, files: 0, bytes: 0, cache: "off" };++ const storage = path.join(ctx.config.paths.cacheDir, "lfs");+ await fs.mkdir(storage, { recursive: true });+ const lfsArgs = [+ "-C",+ root,+ "-c",+ `lfs.storage=${storage}`,+ "-c",+ `lfs.concurrenttransfers=${Math.max(1, ctx.config.source.lfsConcurrency)}`,+ "-c",+ "lfs.fetchrecentalways=false",+ "-c",+ "lfs.skipdownloaderrors=false",+ ];+ const filters = [+ ...(options.include.length ? ["-I", options.include.join(",")] : []),+ ...(options.exclude.length ? ["-X", options.exclude.join(",")] : []),+ ];+ const lfsError = (err: unknown) =>+ err instanceof RenderError+ ? err+ : classifyGitError(err, "lfs", {+ lfs: true,+ extraRules: ctx.profile.lfsRules,+ fallback: "GIT_NETWORK",+ });++ // Preflight: what would be downloaded, and does it fit?+ let files: LfsFile[];+ try {+ const { stdout } = await git(ctx, [+ ...lfsArgs,+ "lfs",+ "ls-files",+ "--json",+ ...filters,+ ]);+ files = (JSON.parse(stdout || "{}").files ?? []) as LfsFile[];+ } catch (err) {+ if (+ err instanceof CommandError &&+ /unknown flag: --json/.test(err.result.stderr)+ ) {+ throw new RenderError(+ "TOOL_MISSING",+ "git-lfs ≥ 3.3 is required (ls-files --json).",+ { phase: "lfs" },+ );+ }+ throw lfsError(err);+ }+ const missing = files.filter((f) => !f.downloaded);+ const needed = missing.reduce((sum, f) => sum + (f.size ?? 0), 0);+ const total = files.reduce((sum, f) => sum + (f.size ?? 0), 0);+ // Objects land in the store, then get copied into the checkout.+ await assertFreeSpace(storage, needed);+ await assertFreeSpace(root, total);++ const attempts = 3;+ for (let attempt = 1; ; attempt++) {+ try {+ await git(ctx, [...lfsArgs, "lfs", "pull", ...filters], {+ timeoutSec: ctx.config.limits.timeouts.lfs,+ });+ break;+ } catch (err) {+ const classified = lfsError(err);+ if (!classified.retryable || attempt >= attempts || ctx.signal.aborted)+ throw classified;+ await new Promise((r) => setTimeout(r, 1000 * 2 ** attempt));+ }+ }++ await assertNoPointers(root, files, options.scope);++ return {+ used: true,+ files: files.length,+ bytes: total,+ cache:+ missing.length === 0+ ? "hit"+ : missing.length === files.length+ ? "miss"+ : "partial",+ };+}++/**+ * After the pull, any file in scope that still holds a pointer is an asset+ * the render would choke on. Fail with the exact paths instead.+ */+async function assertNoPointers(+ root: string,+ files: LfsFile[],+ scope: string[],+): Promise<void> {+ const inScope = (name: string) =>+ scope.length === 0 ||+ scope.some(+ (s) =>+ s === "." || name === s || name.startsWith(`${s.replace(/\/$/, "")}/`),+ );+ const unresolved: string[] = [];+ for (const file of files) {+ if (!inScope(file.name)) continue;+ const handle = await fs+ .open(path.join(root, file.name), "r")+ .catch(() => undefined);+ if (!handle) continue;+ try {+ const buf = Buffer.alloc(POINTER_HEADER.length);+ await handle.read(buf, 0, buf.length, 0);+ if (buf.toString("utf8") === POINTER_HEADER) unresolved.push(file.name);+ } finally {+ await handle.close();+ }+ }+ if (unresolved.length > 0) {+ throw new RenderError(+ "LFS_POINTER_UNRESOLVED",+ `${unresolved.length} LFS file(s) were not downloaded: ${unresolved.slice(0, 10).join(", ")}${unresolved.length > 10 ? ", …" : ""}`,+ { phase: "lfs", details: { files: unresolved.slice(0, 100) } },+ );+ }+}++/**+ * Pointer files that are outside the job's include patterns but inside its+ * workdir: likely assets the render needs. Reported as a warning up front.+ */+export async function findPointers(+ root: string,+ scope: string,+ limit = 20,+): Promise<string[]> {+ const found: string[] = [];+ const walk = async (dir: string): Promise<void> => {+ if (found.length >= limit) return;+ const entries = await fs+ .readdir(dir, { withFileTypes: true })+ .catch(() => []);+ for (const entry of entries) {+ if (found.length >= limit) return;+ if (entry.name === ".git" || entry.name === "node_modules") continue;+ const full = path.join(dir, entry.name);+ if (entry.isDirectory()) await walk(full);+ else if (entry.isFile()) {+ const stat = await fs.stat(full);+ if (stat.size > 200) continue;+ const text = await fs.readFile(full, "utf8").catch(() => "");+ if (text.startsWith(POINTER_HEADER))+ found.push(path.relative(root, full));+ }+ }+ };+ await walk(path.join(root, scope));+ return found;+}++export async function assertFreeSpace(+ dir: string,+ bytes: number,+): Promise<void> {+ if (bytes <= 0) return;+ const stat = await fs.statfs(dir);+ const free = Number(stat.bavail) * Number(stat.bsize);+ if (free < bytes + DISK_MARGIN_BYTES) {+ throw new RenderError(+ "DISK_FULL",+ `Need ${gb(bytes)} (+${gb(DISK_MARGIN_BYTES)} margin) in ${dir} but only ${gb(free)} is free.`,+ {+ phase: "lfs",+ hint: "Raise containerDiskInGb / the volume size, or narrow source.lfs.include.",+ },+ );+ }+}++const gb = (bytes: number) => `${(bytes / 1024 ** 3).toFixed(2)} GB`;diff --git a/apps/renderer-service/src/git/mirror.ts b/apps/renderer-service/src/git/mirror.tsnew file mode 100644index 00000000..92f53731--- /dev/null+++ b/apps/renderer-service/src/git/mirror.ts@@ -0,0 +1,314 @@+import { createHash } from "node:crypto";+import { existsSync } from "node:fs";+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import type { RendererConfig } from "../config/index.js";+import { RenderError } from "../errors.js";+import type { GitSession } from "./auth.js";+import {+ CommandError,+ classifyGitError,+ type ExecOptions,+ exec,+} from "./exec.js";+import type { HostProfile } from "./hosts.js";+import type { RepoRef } from "./repo-url.js";++export interface GitContext {+ config: RendererConfig;+ session: GitSession;+ profile: HostProfile;+ signal: AbortSignal;+}++export function git(+ ctx: GitContext,+ args: string[],+ options: Omit<ExecOptions, "env" | "signal"> & { timeoutSec?: number } = {},+) {+ return exec("git", [...ctx.session.configArgs, ...args], {+ ...options,+ env: ctx.session.env,+ signal: ctx.signal,+ timeoutMs: (options.timeoutSec ?? ctx.config.limits.timeouts.git) * 1000,+ });+}++export function repoHash(repo: RepoRef): string {+ return createHash("sha256")+ .update(repo.id.toLowerCase())+ .digest("hex")+ .slice(0, 16);+}++/** Mirrors are per worker: a shared volume isn't safe for concurrent git writers (§6.1). */+export function mirrorPath(config: RendererConfig, repo: RepoRef): string {+ const worker = (process.env.RUNPOD_POD_ID || os.hostname()).replace(+ /[^\w.-]/g,+ "_",+ );+ return path.join(+ config.paths.cacheDir,+ "git",+ worker,+ `${repoHash(repo)}.git`,+ );+}++export interface RemoteRef {+ name: string;+ sha: string;+}++/** `git ls-remote`, with annotated tags peeled to their commit. */+export async function listRemote(+ ctx: GitContext,+ repo: RepoRef,+): Promise<RemoteRef[]> {+ try {+ const { stdout } = await git(ctx, ["ls-remote", "--", repo.url]);+ const refs = new Map<string, string>();+ for (const line of stdout.split("\n")) {+ const [sha, name] = line.trim().split(/\s+/);+ if (!sha || !name) continue;+ if (name.endsWith("^{}"))+ refs.set(name.slice(0, -3), sha); // peeled wins+ else if (!refs.has(name)) refs.set(name, sha);+ }+ return [...refs].map(([name, sha]) => ({ name, sha }));+ } catch (err) {+ throw classifyGitError(err, "resolve", { extraRules: ctx.profile.rules });+ }+}++export type RefSpec =+ | { kind: "ref"; name: string; sha: string }+ | { kind: "sha"; sha: string };++const HEX = /^[0-9a-f]{7,64}$/i;++/**+ * Resolves a job ref against the remote's refs. Full refs match exactly; bare+ * names match a branch or a tag (both → REF_AMBIGUOUS); hex strings that match+ * no ref are treated as a commit (prefix resolved after fetching).+ */+export function resolveRef(ref: string, remote: RemoteRef[]): RefSpec {+ const byName = new Map(remote.map((r) => [r.name, r.sha]));+ if (ref.startsWith("refs/")) {+ const sha = byName.get(ref);+ if (!sha)+ throw new RenderError(+ "REF_NOT_FOUND",+ `Ref ${ref} does not exist on the remote.`,+ );+ return { kind: "ref", name: ref, sha };+ }+ const head = byName.get(`refs/heads/${ref}`);+ const tag = byName.get(`refs/tags/${ref}`);+ if (head && tag && head !== tag) {+ throw new RenderError(+ "REF_AMBIGUOUS",+ `"${ref}" is both a branch and a tag pointing at different commits.`,+ { hint: `Use refs/heads/${ref} or refs/tags/${ref}.` },+ );+ }+ if (head) return { kind: "ref", name: `refs/heads/${ref}`, sha: head };+ if (tag) return { kind: "ref", name: `refs/tags/${ref}`, sha: tag };+ if (ref === "HEAD" && byName.get("HEAD")) {+ return { kind: "ref", name: "HEAD", sha: byName.get("HEAD") as string };+ }+ if (HEX.test(ref)) return { kind: "sha", sha: ref.toLowerCase() };+ throw new RenderError(+ "REF_NOT_FOUND",+ `No branch or tag named "${ref}" on the remote.`,+ {+ hint: "Check the ref; deleted branches and typos end up here.",+ },+ );+}++/** Serialises mirror access within this process. */+const locks = new Map<string, Promise<unknown>>();+async function withLock<T>(key: string, fn: () => Promise<T>): Promise<T> {+ const prev = locks.get(key) ?? Promise.resolve();+ const next = prev.catch(() => {}).then(fn);+ locks.set(key, next);+ try {+ return await next;+ } finally {+ if (locks.get(key) === next) locks.delete(key);+ }+}++export interface MirrorResult {+ path: string;+ sha: string;+ cache: "hit" | "miss" | "recloned";+}++/**+ * Creates/updates the bare mirror and makes sure `spec` is present in it.+ * A mirror that git can't read is deleted and re-cloned once.+ */+export async function syncMirror(+ ctx: GitContext,+ repo: RepoRef,+ spec: RefSpec,+): Promise<MirrorResult> {+ const dir = mirrorPath(ctx.config, repo);+ return withLock(dir, async () => {+ let cache: MirrorResult["cache"] = existsSync(dir) ? "hit" : "miss";+ try {+ return { path: dir, sha: await fetchInto(ctx, repo, dir, spec), cache };+ } catch (err) {+ if (!isCorruption(err)) throw err;+ await fs.rm(dir, { recursive: true, force: true });+ cache = "recloned";+ try {+ return { path: dir, sha: await fetchInto(ctx, repo, dir, spec), cache };+ } catch (retryErr) {+ if (isCorruption(retryErr)) {+ throw new RenderError(+ "GIT_CACHE_CORRUPT",+ "The git mirror is unreadable even after re-cloning.",+ {+ cause: retryErr,+ phase: "fetch",+ },+ );+ }+ throw retryErr;+ }+ }+ });+}++class CorruptMirror extends Error {}++function isCorruption(err: unknown): boolean {+ if (err instanceof CorruptMirror) return true;+ if (err instanceof RenderError && err.cause instanceof CommandError) {+ return /corrupt|bad object|did not send all necessary objects|unable to read|loose object .* is corrupt|fatal: not a git repository|missing blob|index file corrupt|packfile .* cannot be accessed/i.test(+ err.cause.result.stderr,+ );+ }+ return false;+}++async function fetchInto(+ ctx: GitContext,+ repo: RepoRef,+ dir: string,+ spec: RefSpec,+): Promise<string> {+ const g = (args: string[], timeoutSec?: number) =>+ git(ctx, ["--git-dir", dir, ...args], { timeoutSec }).catch((err) => {+ throw classifyGitError(err, "fetch", { extraRules: ctx.profile.rules });+ });++ if (!existsSync(path.join(dir, "HEAD"))) {+ await fs.mkdir(dir, { recursive: true });+ await g(["init", "--bare", "--quiet"]);+ await g(["config", "remote.origin.url", repo.url]);+ await g(["config", "remote.origin.promisor", "true"]);+ await g(["config", "remote.origin.partialclonefilter", "blob:none"]);+ } else {+ // Detect a broken mirror before trusting it.+ await g(["rev-parse", "--git-dir"]).catch((err) => {+ throw new CorruptMirror(String(err));+ });+ await g(["config", "remote.origin.url", repo.url]);+ }++ const fetchArgs = [+ "fetch",+ "--quiet",+ "--prune",+ "--no-tags",+ "--filter=blob:none",+ "origin",+ ];+ const pinnable = spec.kind === "ref" || spec.sha.length >= 40;+ if (ctx.config.source.fetch === "single" && pinnable) {+ await g(+ [...fetchArgs, `+${spec.sha}:refs/ff/pinned`],+ ctx.config.limits.timeouts.git,+ );+ } else {+ await g(+ [...fetchArgs, "+refs/heads/*:refs/heads/*", "+refs/tags/*:refs/tags/*"],+ ctx.config.limits.timeouts.git,+ );+ // A commit outside branches/tags (e.g. a PR head) by full SHA.+ if (+ spec.kind === "sha" &&+ spec.sha.length >= 40 &&+ !(await hasCommit(ctx, dir, spec.sha))+ ) {+ await g([...fetchArgs, spec.sha]).catch(() => undefined);+ }+ }++ if (spec.kind === "ref") {+ if (!(await hasCommit(ctx, dir, spec.sha))) {+ throw new RenderError(+ "REF_NOT_FOUND",+ `${spec.name} moved while fetching (force-push?); ${spec.sha.slice(0, 12)} is no longer reachable.`,+ { phase: "fetch" },+ );+ }+ return spec.sha;+ }+ return resolveShaPrefix(ctx, dir, spec.sha);+}++async function hasCommit(+ ctx: GitContext,+ dir: string,+ sha: string,+): Promise<boolean> {+ try {+ await git(ctx, ["--git-dir", dir, "cat-file", "-e", `${sha}^{commit}`]);+ return true;+ } catch {+ return false;+ }+}++async function resolveShaPrefix(+ ctx: GitContext,+ dir: string,+ prefix: string,+): Promise<string> {+ try {+ const { stdout } = await git(ctx, [+ "--git-dir",+ dir,+ "rev-parse",+ "--verify",+ "--end-of-options",+ `${prefix}^{commit}`,+ ]);+ return stdout.trim();+ } catch (err) {+ const stderr = err instanceof CommandError ? err.result.stderr : "";+ if (/ambiguous/i.test(stderr)) {+ throw new RenderError(+ "REF_AMBIGUOUS",+ `Commit prefix ${prefix} matches several commits; use more characters.`,+ {+ phase: "resolve",+ },+ );+ }+ throw new RenderError(+ "REF_NOT_FOUND",+ `Commit ${prefix} is not on any branch or tag of the remote.`,+ {+ phase: "resolve",+ hint: "Push the commit, or pass its full 40-character SHA.",+ },+ );+ }+}diff --git a/apps/renderer-service/src/git/repo-url.test.ts b/apps/renderer-service/src/git/repo-url.test.tsnew file mode 100644index 00000000..1fb2d87e--- /dev/null+++ b/apps/renderer-service/src/git/repo-url.test.ts@@ -0,0 +1,85 @@+import { describe, expect, it } from "vitest";+import { isHostAllowed, isRepoAllowed, parseRepoUrl } from "./repo-url.js";++describe("parseRepoUrl", () => {+ it.each([+ ["https://github.com/acme/films.git", "github.com/acme/films", "https"],+ [+ "https://gitlab.com/group/sub/films",+ "gitlab.com/group/sub/films",+ "https",+ ],+ ["[email protected]:acme/films.git", "github.com/acme/films", "ssh"],+ [+ "ssh://[email protected]:2222/acme/films.git",+ "git.acme.dev/acme/films",+ "ssh",+ ],+ ["https://bitbucket.org/acme/films/", "bitbucket.org/acme/films", "https"],+ ])("%s → %s", (url, id, protocol) => {+ const repo = parseRepoUrl(url);+ expect(repo.id).toBe(id);+ expect(repo.protocol).toBe(protocol);+ });++ it("strips credentials from https URLs", () => {+ const repo = parseRepoUrl(+ "https://user:[email protected]/acme/films.git",+ );+ expect(repo.url).toBe("https://github.com/acme/films.git");+ });++ it.each([+ "ext::sh -c touch% /tmp/pwned",+ "/srv/repos/films",+ "../films",+ "films",+ ])("rejects %s", (url) => {+ expect(() => parseRepoUrl(url)).toThrow();+ });++ it("rejects file:// unless local repos are allowed", () => {+ expect(() => parseRepoUrl("file:///srv/films.git")).toThrow(+ /FF_ALLOW_LOCAL_REPOS/,+ );+ expect(+ parseRepoUrl("file:///srv/films.git", { allowLocal: true }).protocol,+ ).toBe("file");+ });+});++describe("allowlist", () => {+ it("matches * within a segment and ** across segments", () => {+ expect(isRepoAllowed("github.com/acme/films", ["github.com/acme/*"])).toBe(+ true,+ );+ expect(isRepoAllowed("github.com/other/films", ["github.com/acme/*"])).toBe(+ false,+ );+ expect(+ isRepoAllowed("gitlab.com/acme/sub/films", ["gitlab.com/acme/*"]),+ ).toBe(false);+ expect(+ isRepoAllowed("gitlab.com/acme/sub/films", ["gitlab.com/acme/**"]),+ ).toBe(true);+ });++ it("is case-insensitive and anchored", () => {+ expect(+ isRepoAllowed("github.com/Acme/Films", ["github.com/acme/films"]),+ ).toBe(true);+ expect(+ isRepoAllowed("evilgithub.com/acme/films", ["github.com/acme/*"]),+ ).toBe(false);+ expect(+ isRepoAllowed("github.com/acme/films-fork", ["github.com/acme/films"]),+ ).toBe(false);+ });++ it("checks hosts for credential answers", () => {+ expect(isHostAllowed("github.com", ["github.com/acme/*"])).toBe(true);+ expect(isHostAllowed("lfs-cdn.example.com", ["github.com/acme/*"])).toBe(+ false,+ );+ });+});diff --git a/apps/renderer-service/src/git/repo-url.ts b/apps/renderer-service/src/git/repo-url.tsnew file mode 100644index 00000000..ceb2df9b--- /dev/null+++ b/apps/renderer-service/src/git/repo-url.ts@@ -0,0 +1,126 @@+import { RenderError } from "../errors.js";++export interface RepoRef {+ /** The URL handed to git, credentials stripped. */+ url: string;+ protocol: "https" | "http" | "ssh" | "file";+ /** Lowercased hostname, "" for file:// */+ host: string;+ port?: number;+ /** owner/name without a trailing .git */+ path: string;+ /** host/owner/name, what the allowlist matches against */+ id: string;+}++const SCP_LIKE = /^(?:([^@/\s]+)@)?([^:/\s]+):(?!\/)(.+)$/;++/**+ * Parses https://, ssh:// and scp-like (git@host:owner/repo) URLs. Anything+ * else (ext::, local paths, …) is rejected; file:// only when allowed.+ */+export function parseRepoUrl(+ raw: string,+ options: { allowLocal?: boolean } = {},+): RepoRef {+ const input = raw.trim();+ if (/^ext::/i.test(input) || /[\0\n\r]/.test(input)) {+ throw invalid(raw, "unsupported transport");+ }++ if (/^file:\/\//i.test(input)) {+ if (!options.allowLocal) {+ throw new RenderError(+ "REPO_NOT_ALLOWED",+ "Local repositories are disabled (FF_ALLOW_LOCAL_REPOS).",+ );+ }+ const url = new URL(input);+ const path = stripGit(decodeURIComponent(url.pathname));+ return { url: input, protocol: "file", host: "", path, id: `file:${path}` };+ }++ if (/^(https?|ssh):\/\//i.test(input)) {+ let url: URL;+ try {+ url = new URL(input);+ } catch {+ throw invalid(raw, "malformed URL");+ }+ const protocol = url.protocol.slice(0, -1) as "https" | "http" | "ssh";+ const path = stripGit(url.pathname.replace(/^\/+/, ""));+ if (!path) throw invalid(raw, "missing repository path");+ const host = url.hostname.toLowerCase();+ const port = url.port ? Number(url.port) : undefined;+ // Credentials never travel in the URL; they come from the helper.+ const user = protocol === "ssh" ? url.username || "git" : "";+ const clean = `${protocol}://${user ? `${user}@` : ""}${url.host}/${url.pathname.replace(/^\/+/, "")}`;+ return { url: clean, protocol, host, port, path, id: `${host}/${path}` };+ }++ const scp = SCP_LIKE.exec(input);+ if (scp && !input.startsWith("/") && !input.startsWith(".")) {+ const [, user = "git", hostRaw, rest] = scp;+ const host = hostRaw.toLowerCase();+ const path = stripGit(rest.replace(/^\/+/, ""));+ if (!path) throw invalid(raw, "missing repository path");+ return {+ url: `${user}@${hostRaw}:${rest}`,+ protocol: "ssh",+ host,+ path,+ id: `${host}/${path}`,+ };+ }++ throw invalid(raw, "expected an https://, ssh:// or git@host:owner/repo URL");+}++function stripGit(path: string): string {+ return path.replace(/\/+$/, "").replace(/\.git$/, "");+}++function invalid(raw: string, why: string): RenderError {+ return new RenderError("JOB_INVALID", `Invalid repo URL "${raw}": ${why}.`);+}++/**+ * Matches `host/owner/name` against allowlist globs. `*` matches within one+ * path segment, `**` across segments. Matching is case-insensitive.+ */+export function isRepoAllowed(+ id: string,+ patterns: readonly string[],+): boolean {+ const subject = id.toLowerCase();+ return patterns.some((p) => globToRegExp(p.toLowerCase()).test(subject));+}++export function isHostAllowed(+ host: string,+ patterns: readonly string[],+): boolean {+ const h = host.toLowerCase();+ return patterns.some((p) => {+ const patternHost = p.toLowerCase().split("/")[0];+ return globToRegExp(patternHost).test(h);+ });+}++function globToRegExp(glob: string): RegExp {+ let re = "";+ for (let i = 0; i < glob.length; i++) {+ const ch = glob[i];+ if (ch === "*") {+ if (glob[i + 1] === "*") {+ re += ".*";+ i++;+ } else {+ re += "[^/]*";+ }+ } else {+ re += ch.replace(/[.+?^${}()|[\]\\]/g, "\\$&");+ }+ }+ return new RegExp(`^${re.replace(/\/$/, "")}$`);+}diff --git a/apps/renderer-service/src/git/workspace.ts b/apps/renderer-service/src/git/workspace.tsnew file mode 100644index 00000000..87273007--- /dev/null+++ b/apps/renderer-service/src/git/workspace.ts@@ -0,0 +1,207 @@+import { existsSync } from "node:fs";+import fs from "node:fs/promises";+import path from "node:path";+import { RenderError } from "../errors.js";+import { classifyGitError } from "./exec.js";+import { type GitContext, git, repoHash } from "./mirror.js";+import { isRepoAllowed, parseRepoUrl, type RepoRef } from "./repo-url.js";++export interface Workspace {+ root: string;+ warm: boolean;+}++/** Files that survive `git clean` between jobs in a warm slot. */+export const KEEP_ON_CLEAN = ["node_modules", ".ff-install-stamp"];++/**+ * Checks `sha` out into this repo's workspace slot on local disk. A slot that+ * still belongs to the current mirror is reused (warm: keeps node_modules);+ * anything else is rebuilt from scratch.+ */+export async function checkoutWorkspace(+ ctx: GitContext,+ repo: RepoRef,+ mirror: string,+ sha: string,+ options: { sparse?: string[] },+): Promise<Workspace> {+ const root = path.join(ctx.config.paths.workDir, "ws", repoHash(repo), "0");+ const inSlot = (args: string[], timeoutSec?: number) =>+ git(ctx, ["-C", root, ...args], { timeoutSec }).catch((err) => {+ throw classifyGitError(err, "fetch", { extraRules: ctx.profile.rules });+ });++ let warm = await isLiveWorktree(ctx, root, mirror);+ if (!warm) {+ await fs.rm(root, { recursive: true, force: true });+ await fs.mkdir(path.dirname(root), { recursive: true });+ await git(ctx, ["--git-dir", mirror, "worktree", "prune"]).catch(+ () => undefined,+ );+ await git(ctx, [+ "--git-dir",+ mirror,+ "worktree",+ "add",+ "--force",+ "--detach",+ "--no-checkout",+ root,+ sha,+ ]).catch((err) => {+ throw classifyGitError(err, "fetch", { extraRules: ctx.profile.rules });+ });+ }++ if (options.sparse?.length) {+ await inSlot(["sparse-checkout", "set", "--cone", "--", ...options.sparse]);+ } else if (warm && existsSync(path.join(root, ".git"))) {+ await inSlot(["sparse-checkout", "disable"]).catch(() => undefined);+ }++ // Blobs of a partial mirror are fetched on demand here.+ await inSlot(+ ["checkout", "--force", "--detach", sha],+ ctx.config.limits.timeouts.git,+ );+ if (warm) {+ await inSlot([+ "clean",+ "-ffdx",+ "--quiet",+ ...KEEP_ON_CLEAN.flatMap((p) => ["-e", p]),+ ]);+ }+ const { stdout } = await inSlot(["rev-parse", "HEAD"]);+ if (stdout.trim() !== sha) {+ warm = false;+ throw new RenderError(+ "GIT_CACHE_CORRUPT",+ `Checkout ended on ${stdout.trim()}, expected ${sha}.`,+ {+ phase: "fetch",+ },+ );+ }+ return { root, warm };+}++async function isLiveWorktree(+ ctx: GitContext,+ root: string,+ mirror: string,+): Promise<boolean> {+ if (!existsSync(path.join(root, ".git"))) return false;+ try {+ const { stdout } = await git(ctx, [+ "-C",+ root,+ "rev-parse",+ "--git-common-dir",+ ]);+ const common = path.resolve(root, stdout.trim());+ return (await fs.realpath(common)) === (await fs.realpath(mirror));+ } catch {+ return false;+ }+}++/**+ * Initialises submodules level by level, checking every submodule URL against+ * the allowlist before git touches it (relative URLs resolve against the parent).+ */+export async function initSubmodules(+ ctx: GitContext,+ dir: string,+ parent: RepoRef,+ depth = 0,+): Promise<void> {+ if (depth > 8) {+ throw new RenderError(+ "JOB_INVALID",+ "Submodules nest deeper than 8 levels.",+ { phase: "fetch" },+ );+ }+ if (!existsSync(path.join(dir, ".gitmodules"))) return;++ const { stdout } = await git(ctx, [+ "-C",+ dir,+ "config",+ "-f",+ ".gitmodules",+ "--get-regexp",+ "^submodule\\..*\\.url$",+ ]).catch(() => ({ stdout: "" }));+ for (const line of stdout.split("\n").filter(Boolean)) {+ const url = line.slice(line.indexOf(" ") + 1).trim();+ const sub = parseRepoUrl(resolveRelativeUrl(url, parent), {+ allowLocal: ctx.config.source.allowLocal,+ });+ if (+ sub.protocol !== "file" &&+ !isRepoAllowed(sub.id, ctx.config.source.allowlist)+ ) {+ throw new RenderError(+ "REPO_NOT_ALLOWED",+ `Submodule ${sub.id} is not in FF_REPO_ALLOWLIST.`,+ {+ phase: "fetch",+ },+ );+ }+ }++ await git(ctx, ["-C", dir, "submodule", "sync", "--quiet"]);+ await git(+ ctx,+ ["-C", dir, "submodule", "update", "--init", "--force", "--checkout"],+ {+ timeoutSec: ctx.config.limits.timeouts.git,+ },+ ).catch((err) => {+ throw classifyGitError(err, "fetch", { extraRules: ctx.profile.rules });+ });++ const { stdout: paths } = await git(ctx, [+ "-C",+ dir,+ "submodule",+ "--quiet",+ "foreach",+ "--quiet",+ "pwd",+ ]).catch(() => ({ stdout: "" }));+ for (const subDir of paths.split("\n").filter(Boolean)) {+ const { stdout: subUrl } = await git(ctx, [+ "-C",+ subDir,+ "config",+ "remote.origin.url",+ ]).catch(() => ({+ stdout: "",+ }));+ if (!subUrl.trim()) continue;+ await initSubmodules(+ ctx,+ subDir,+ parseRepoUrl(subUrl.trim(), { allowLocal: ctx.config.source.allowLocal }),+ depth + 1,+ );+ }+}++/** `../other.git` relative to `https://host/acme/films` → `https://host/acme/other.git` */+export function resolveRelativeUrl(url: string, parent: RepoRef): string {+ if (!url.startsWith("./") && !url.startsWith("../")) return url;+ const base = parent.url.replace(/\/+$/, "");+ if (parent.protocol === "ssh" && !base.includes("://")) {+ const [hostPart, repoPath] = base.split(/:(.*)/s);+ return `${hostPart}:${path.posix.normalize(path.posix.join(repoPath, url))}`;+ }+ const u = new URL(base);+ u.pathname = path.posix.normalize(path.posix.join(u.pathname, url));+ return u.toString();+}diff --git a/apps/renderer-service/src/http/app.test.ts b/apps/renderer-service/src/http/app.test.tsnew file mode 100644index 00000000..0836b37b--- /dev/null+++ b/apps/renderer-service/src/http/app.test.ts@@ -0,0 +1,193 @@+import { describe, expect, it } from "vitest";+import { RenderError } from "../errors.js";+import type { Pipeline } from "../jobs/pipeline.js";+import { JobRunner } from "../jobs/runner.js";+import { testConfig, testJob } from "../test-helpers.js";+import { createApp } from "./app.js";++const API_KEY = "k".repeat(32);++function fakePipeline(overrides: Partial<Pipeline> = {}): Pipeline {+ return {+ checkout: async () => ({+ root: "/tmp/ws",+ workdir: "/tmp/ws",+ sha: "9f2c1e4abcdef0123456789abcdef0123456789",+ cache: { mirror: "hit" },+ release: async () => {},+ }),+ install: async () => ({ skipped: true, cache: { install: "skipped" } }),+ render: async () => ({+ files: [+ {+ name: "main",+ path: "/tmp/x.mp4",+ ext: "mp4",+ contentType: "video/mp4",+ },+ ],+ engine: { framefields: "2.1.3", source: "bundled" },+ }),+ deliver: async () => [+ {+ name: "main",+ bytes: 10,+ sha256: "abc",+ contentType: "video/mp4",+ base64: "AAAA",+ },+ ],+ ...overrides,+ };+}++function setup(pipeline = fakePipeline(), ready = true) {+ const config = testConfig({+ FF_HTTP_INSECURE: "false",+ RENDERER_API_KEY: API_KEY,+ });+ const runner = new JobRunner(config, pipeline);+ const app = createApp(config, runner, {+ health: () => ({+ ok: ready,+ body: { status: ready ? "ok" : "initializing" },+ }),+ draining: () => false,+ });+ const call = (+ path: string,+ init: RequestInit = {},+ key: string | null = API_KEY,+ ) =>+ app.request(path, {+ ...init,+ headers: {+ "content-type": "application/json",+ ...(key && { authorization: `Bearer ${key}` }),+ },+ });+ return { runner, call };+}++const waitDone = async (runner: JobRunner, id: string) => {+ for (let i = 0; i < 100; i++) {+ const job = runner.store.get(id);+ if (job && ["COMPLETED", "FAILED", "CANCELLED"].includes(job.status))+ return job;+ await new Promise((r) => setTimeout(r, 5));+ }+ throw new Error("job did not finish");+};++describe("HTTP v2", () => {+ it("requires the API key on job routes but not on health", async () => {+ const { call } = setup();+ expect(+ (await call("/v2/jobs", { method: "POST", body: "{}" }, null)).status,+ ).toBe(401);+ expect(+ (await call("/v2/jobs", { method: "POST", body: "{}" }, "wrong")).status,+ ).toBe(401);+ expect((await call("/health", {}, null)).status).toBe(200);+ });++ it("runs a job to completion and reports the result", async () => {+ const { call, runner } = setup();+ const res = await call("/v2/jobs", {+ method: "POST",+ body: JSON.stringify(testJob()),+ });+ expect(res.status).toBe(202);+ const { id } = (await res.json()) as { id: string };++ const job = await waitDone(runner, id);+ expect(job.status).toBe("COMPLETED");+ expect(job.result).toMatchObject({+ source: { repo: "github.com/acme/films", ref: "main" },+ outputs: [{ name: "main" }],+ cache: { mirror: "hit", install: "skipped" },+ });++ const fetched = await call(`/v2/jobs/${id}`);+ expect(((await fetched.json()) as { status: string }).status).toBe(+ "COMPLETED",+ );+ });++ it("returns structured validation errors", async () => {+ const { call } = setup();+ const res = await call("/v2/jobs", {+ method: "POST",+ body: JSON.stringify(+ testJob({ source: { repo: "https://github.com/evil/x" } }),+ ),+ });+ expect(res.status).toBe(403);+ expect(await res.json()).toMatchObject({+ error: { code: "REPO_NOT_ALLOWED", retryable: false },+ });+ });++ it("refuses jobs until the GPU is ready", async () => {+ const { call } = setup(fakePipeline(), false);+ const res = await call("/v2/jobs", {+ method: "POST",+ body: JSON.stringify(testJob()),+ });+ expect(res.status).toBe(503);+ });++ it("records step failures with their code and redacts secrets", async () => {+ const { call, runner } = setup(+ fakePipeline({+ checkout: async () => {+ throw new RenderError(+ "GIT_AUTH_FAILED",+ "fatal: could not read from https://x-access-token:[email protected]/acme/films",+ );+ },+ }),+ );+ const res = await call("/v2/jobs", {+ method: "POST",+ body: JSON.stringify(testJob()),+ });+ const { id } = (await res.json()) as { id: string };+ const job = await waitDone(runner, id);+ expect(job.status).toBe("FAILED");+ expect(job.error?.code).toBe("GIT_AUTH_FAILED");+ expect(job.error?.message).not.toContain("ghp_x1");+ });++ it("cancels a running job", async () => {+ const { call, runner } = setup(+ fakePipeline({+ render: (_c, _i, ctx) =>+ new Promise((_resolve, reject) => {+ ctx.signal.addEventListener("abort", () =>+ reject(ctx.signal.reason),+ );+ }),+ }),+ );+ const res = await call("/v2/jobs", {+ method: "POST",+ body: JSON.stringify(testJob()),+ });+ const { id } = (await res.json()) as { id: string };+ await new Promise((r) => setTimeout(r, 10));+ expect(+ (await call(`/v2/jobs/${id}/cancel`, { method: "POST" })).status,+ ).toBe(202);+ const job = await waitDone(runner, id);+ expect(job.status).toBe("CANCELLED");+ expect(+ (await call(`/v2/jobs/${id}/cancel`, { method: "POST" })).status,+ ).toBe(409);+ });++ it("404s unknown jobs", async () => {+ const { call } = setup();+ expect((await call("/v2/jobs/nope")).status).toBe(404);+ });+});diff --git a/apps/renderer-service/src/http/app.ts b/apps/renderer-service/src/http/app.tsnew file mode 100644index 00000000..d34aef84--- /dev/null+++ b/apps/renderer-service/src/http/app.ts@@ -0,0 +1,148 @@+import { createHash, timingSafeEqual } from "node:crypto";+import { rendererLogger } from "@framefields/server-utils";+import { type Context, Hono, type MiddlewareHandler } from "hono";+import { HTTPException } from "hono/http-exception";+import { streamSSE } from "hono/streaming";+import type { RendererConfig } from "../config/index.js";+import { RenderError } from "../errors.js";+import type { JobRunner } from "../jobs/runner.js";+import { isTerminal, type JobRecord } from "../jobs/store.js";+import { loggerMiddleware } from "../middlewares.js";+import { redact } from "../redact.js";++export interface HealthSource {+ /** Overall readiness plus per-component detail. */+ health(): { ok: boolean; body: Record<string, unknown> };+ /** True when the worker is recycling and must not take new jobs. */+ draining(): boolean;+}++const STATUS_BY_CODE: Partial<Record<RenderError["code"], number>> = {+ JOB_INVALID: 400,+ PATH_ESCAPE: 400,+ REPO_NOT_ALLOWED: 403,+ BUCKET_NOT_ALLOWED: 403,+ BUSY: 429,+ SHUTDOWN: 503,+};++export function createApp(+ config: RendererConfig,+ runner: JobRunner,+ health: HealthSource,+) {+ const requireKey = bearerAuth(config.http.apiKey);++ return new Hono()+ .onError((err: Error, c: Context) => {+ if (err instanceof RenderError) {+ return c.json(+ { error: err.toJSON() },+ (STATUS_BY_CODE[err.code] ?? 500) as 500,+ );+ }+ const status = err instanceof HTTPException ? err.status : 500;+ if (status >= 500) {+ rendererLogger.error(+ { err: redact(err.message), url: c.req.path, status },+ "HTTP error",+ );+ }+ return c.json(+ {+ error:+ err instanceof HTTPException+ ? err.message+ : "Internal Server Error",+ },+ status,+ );+ })+ .use(loggerMiddleware)+ .use(async (c, next) => {+ c.header("X-Content-Type-Options", "nosniff");+ c.header("Cache-Control", "no-store");+ await next();+ })+ .get("/ping", (c) =>+ health.health().ok ? c.text("pong", 200) : c.body(null, 204),+ )+ .get("/health", (c) => {+ const { ok, body } = health.health();+ return c.json(body, ok ? 200 : 503);+ })+ .post("/v2/jobs", requireKey, async (c) => {+ if (health.draining() || !health.health().ok) {+ throw new RenderError("SHUTDOWN", "This worker is not accepting jobs.");+ }+ let body: unknown;+ try {+ body = await c.req.json();+ } catch {+ throw new RenderError("JOB_INVALID", "Request body must be JSON.");+ }+ const record = runner.submit(body);+ return c.json({ id: record.id, status: record.status }, 202);+ })+ .get("/v2/jobs/:id", requireKey, (c) =>+ c.json(findJob(runner, c.req.param("id"))),+ )+ .post("/v2/jobs/:id/cancel", requireKey, (c) => {+ const job = findJob(runner, c.req.param("id"));+ if (isTerminal(job.status)) return c.json(job, 409);+ runner.cancel(job.id);+ return c.json(runner.store.get(job.id) ?? job, 202);+ })+ .get("/v2/jobs/:id/events", requireKey, (c) => {+ const job = findJob(runner, c.req.param("id"));+ return streamSSE(c, async (stream) => {+ const send = (record: JobRecord) =>+ stream.writeSSE({ event: "job", data: JSON.stringify(record) });+ await send(job);+ if (isTerminal(job.status)) return;++ await new Promise<void>((resolve) => {+ const onUpdate = (record: JobRecord) => {+ if (record.id !== job.id) return;+ void send(record);+ if (isTerminal(record.status)) done();+ };+ const keepAlive = setInterval(+ () => void stream.writeSSE({ event: "ping", data: "" }),+ 15_000,+ );+ const done = () => {+ clearInterval(keepAlive);+ runner.store.off("update", onUpdate);+ resolve();+ };+ runner.store.on("update", onUpdate);+ stream.onAbort(done);+ });+ });+ });+}++function findJob(runner: JobRunner, id: string): JobRecord {+ const job = runner.store.get(id);+ if (!job) throw new HTTPException(404, { message: "Job not found" });+ return job;+}++function bearerAuth(apiKey: string | undefined): MiddlewareHandler {+ const expected = apiKey ? digest(apiKey) : undefined;+ return async (c, next) => {+ if (expected) {+ const header = c.req.header("authorization") ?? "";+ const token = /^Bearer\s+(.+)$/i.exec(header)?.[1] ?? "";+ if (!timingSafeEqual(digest(token), expected)) {+ throw new HTTPException(401, { message: "Unauthorized" });+ }+ }+ await next();+ };+}++function digest(value: string): Buffer {+ return createHash("sha256").update(value).digest();+}diff --git a/apps/renderer-service/src/index.ts b/apps/renderer-service/src/index.tsindex f959fb1d..40430ca7 100644--- a/apps/renderer-service/src/index.ts+++ b/apps/renderer-service/src/index.ts@@ -1,244 +1,141 @@-// Keep this at top. (Force reload 1)-import "reflect-metadata";import { Server as HttpServer } from "node:http";-import { configureAssetUrls } from "@framefields/client-utils";-import {- HeadlessMediaRenderer,- renderSemaphore,-} from "@framefields/renderer";+import { HeadlessMediaRenderer } from "@framefields/renderer";import "@framefields/renderer/offscreen-gl-polyfill";import { rendererLogger } from "@framefields/server-utils";import { serve } from "@hono/node-server";-import { type Context, Hono } from "hono";-import { HTTPException } from "hono/http-exception";-import { registerBackendServices } from "./di-setup.js";-import { RENDERER_ENV_CONFIG } from "./env-config.js";-import {- executeRenderJob,- hasActiveJobs,- jobs,- type RenderJob,- type RenderJobInput,-} from "./job-processor.js";-import { loggerMiddleware } from "./middlewares.js";+import { describeConfig, loadConfig, scrubSecretEnv } from "./config/index.js";+import { RenderError } from "./errors.js";+import { checkout } from "./git/checkout.js";+import { createApp } from "./http/app.js";+import type { Pipeline } from "./jobs/pipeline.js";+import { JobRunner } from "./jobs/runner.js";import { ReadinessTracker } from "./readiness.js";-const readiness = new ReadinessTracker();-let podState: "initializing" | "idle" | "processing" = "initializing";--// RenderJob types and Map are imported from job-processor.js--const app = new Hono()- .onError((err: Error, c: Context) => {- const isHTTPException = err instanceof HTTPException;- const status = isHTTPException ? (err as HTTPException).status : 500;-- rendererLogger.error(- {- err: err.message,- stack:- RENDERER_ENV_CONFIG.NODE_ENV === "production" ? undefined : err.stack,- url: c.req.url,- status,- },- isHTTPException ? "HTTP Exception" : "Unhandled Exception",- );+const config = loadConfig();+scrubSecretEnv();+rendererLogger.info(+ { config: describeConfig(config) },+ "Renderer config loaded",+);- return c.json(- {- error: isHTTPException ? err.message : "Internal Server Error",- },- status,- );- })- .use(async (c, next) => {- return loggerMiddleware(c, next);- })- .use(async (c, next) => {- c.header("X-Frame-Options", "SAMEORIGIN");- c.header("X-Content-Type-Options", "nosniff");- c.header("Referrer-Policy", "strict-origin-when-cross-origin");- c.header(- "Strict-Transport-Security",- "max-age=31536000; includeSubDomains",- );- await next();- })- .get("/ping", (c) => {- if (!readiness.isReady) {- return c.body(null, 204);- }- return c.text("pong", 200);- })- .get("/health", async (c) => {+const readiness = new ReadinessTracker();+let draining = false;++const notYet = (step: string): never => {+ throw new RenderError(+ "INTERNAL",+ `The ${step} step is not implemented in this build yet.`,+ );+};+// Steps land phase by phase (PLAN.md §13).+const pipeline: Pipeline = {+ checkout,+ install: async () => notYet("install"),+ render: async () => notYet("render"),+ deliver: async () => notYet("deliver"),+};++const runner = new JobRunner(config, pipeline);++const app = createApp(config, runner, {+ health: () => {const status = readiness.status;- const health: Record<string, unknown> = {- status: readiness.isReady- ? "ok"- : status.error- ? "failed"- : "initializing",- timestamp: new Date().toISOString(),- components: status.components,- renderQueue: renderSemaphore.stats,- };- if (status.error) {- health.error = status.error;- }-- return c.json(health, readiness.isReady ? 200 : 503);- })- .post("/run", async (c) => {- const body = await c.req.json<{ input: RenderJobInput }>();- const jobData = body.input;- const jobId = `local-job-${Date.now()}-${Math.random().toString(36).slice(2, 11)}`;-- const job: RenderJob = {- id: jobId,- status: "IN_QUEUE",+ return {+ ok: readiness.isReady && !draining,+ body: {+ status: draining+ ? "draining"+ : readiness.isReady+ ? "ok"+ : status.error+ ? "failed"+ : "initializing",+ timestamp: new Date().toISOString(),+ mode: config.mode,+ components: status.components,+ queue: {+ active: runner.queue.activeId ?? null,+ waiting: runner.queue.depth,+ },+ ...(status.error && { error: status.error }),+ },};- jobs.set(jobId, job);-- // Execute rendering in the background asynchronously- executeRenderJob(jobId, jobData, (state) => {- podState = state;- });-- return c.json({ id: jobId, status: "IN_QUEUE" }, 200);- })- .get("/status/:id", (c) => {- const jobId = c.req.param("id");- const job = jobs.get(jobId);- if (!job) {- return c.json({ error: "Job not found" }, 404);- }- return c.json(job, 200);- });+ },+ draining: () => draining,+});const server = serve(- {- fetch: app.fetch,- port: RENDERER_ENV_CONFIG.RENDERER_PORT,- hostname: "0.0.0.0",- },+ { fetch: app.fetch, port: config.http.port, hostname: config.http.host },(info) => {- rendererLogger.info(`Server is running on port ${info.port} (0.0.0.0)`);+ rendererLogger.info(+ `Server is running on ${config.http.host}:${info.port}`,+ );},);-const TWO_HOURS_MS = 2 * 60 * 60 * 1000;+// Long SSE streams and slow clients; renders themselves never block a request.if (server instanceof HttpServer) {- server.timeout = TWO_HOURS_MS;- server.requestTimeout = TWO_HOURS_MS;- server.headersTimeout = TWO_HOURS_MS;- server.keepAliveTimeout = TWO_HOURS_MS;+ server.requestTimeout = 0;+ server.headersTimeout = 60_000;+ server.keepAliveTimeout = 65_000;}-let shutdownRequested = false;--const sleep = (time: number) =>- new Promise((resolve) => setTimeout(resolve, time));-async function bootstrap() {try {- // Configure asset and font URLs for headless rendering- configureAssetUrls({- baseUrl: RENDERER_ENV_CONFIG.BASE_URL,- r2CustomDomain: RENDERER_ENV_CONFIG.R2_CUSTOM_DOMAIN,- });-- // Initialize Dependency Injection Container- await registerBackendServices();-- // Sleep for 2 seconds to allow HMR for node builds on dev.- if (RENDERER_ENV_CONFIG.NODE_ENV !== "production") {- await sleep(2000);- }- } catch (err) {- rendererLogger.fatal(- { err },- "❌ Failed to start Gatewai Backend services",- );- readiness.setFailure(err instanceof Error ? err.message : String(err));- process.exit(1);- }-- if (shutdownRequested) {- rendererLogger.info(- "Shutdown was requested during container boot. Aborting startup.",- );- return;- }-- // Initialize renderer (GPU, node discovery)- try {- rendererLogger.info(- "Eagerly initializing renderer (GPU, node discovery)...",- );+ rendererLogger.info("Initializing renderer (GPU, node discovery)...");await HeadlessMediaRenderer.initialize();readiness.setRendererReady(true);readiness.setGpuReady(true);- podState = "idle";- rendererLogger.info(- "Eager renderer initialization completed successfully.",- );+ rendererLogger.info("Renderer initialized.");} catch (error) {const errMsg = error instanceof Error ? error.message : String(error);rendererLogger.fatal({ err: error },"❌ GPU / Renderer initialization failed",);- readiness.setFailure(errMsg);+ readiness.setFailure(`GPU_UNAVAILABLE: ${errMsg}`);+ // RunPod should replace a worker without a usable GPU.+ if (config.mode === "runpod-serverless") process.exit(1);}}-bootstrap();+void bootstrap();// ─── Graceful Shutdown ──────────────────────────────────────────────────────────-async function gracefulShutdown(signal: string) {- rendererLogger.info({ signal, podState }, "Starting graceful shutdown...");- shutdownRequested = true;-- const IDLE_TIMEOUT_MS = 30_000;- const INIT_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes- const MAX_RENDER_TIMEOUT_MS = 2.5 * 60 * 60 * 1000; // 2.5 hours-- const shutdownTimeoutMs =- podState === "processing"- ? MAX_RENDER_TIMEOUT_MS- : podState === "initializing"- ? INIT_TIMEOUT_MS- : IDLE_TIMEOUT_MS;+let shuttingDown = false;- const timeout = setTimeout(() => {+async function gracefulShutdown(signal: string) {+ if (shuttingDown) return;+ shuttingDown = true;+ draining = true;+ rendererLogger.info(+ { signal, active: runner.queue.activeId },+ "Shutting down...",+ );++ // The platform is stopping us: abort the running job so it's reported as+ // SHUTDOWN (retryable) instead of vanishing mid-render.+ runner.queue.close(+ new RenderError("SHUTDOWN", `Worker received ${signal}.`),+ true,+ );++ const graceMs = (config.limits.killGraceSec + 5) * 1000;+ const forceExit = setTimeout(() => {rendererLogger.error("Shutdown timed out, forcing exit.");process.exit(1);- }, shutdownTimeoutMs);-- try {- // Stop HTTP server- server.close();- rendererLogger.info(- "HTTP server closed, waiting for active jobs to complete...",- );-- while (hasActiveJobs()) {- rendererLogger.info("Active jobs still running, waiting...");- await sleep(1000);- }+ }, graceMs);- rendererLogger.info("Graceful shutdown complete. Bye!");- clearTimeout(timeout);- process.exit(0);- } catch (err) {- rendererLogger.error({ err }, "Error during graceful shutdown");- process.exit(1);- }+ await runner.queue.idle();+ server.close();+ clearTimeout(forceExit);+ rendererLogger.info("Shutdown complete.");+ process.exit(0);}-process.on("SIGTERM", () => gracefulShutdown("SIGTERM"));-process.on("SIGINT", () => gracefulShutdown("SIGINT"));+process.on("SIGTERM", () => void gracefulShutdown("SIGTERM"));+process.on("SIGINT", () => void gracefulShutdown("SIGINT"));export { app };export type AppType = typeof app;diff --git a/apps/renderer-service/src/job-processor.ts b/apps/renderer-service/src/job-processor.tsdeleted file mode 100644index 8a53d08d..00000000--- a/apps/renderer-service/src/job-processor.ts+++ /dev/null@@ -1,112 +0,0 @@-import type { VirtualMediaData } from "@framefields/core";-import { rendererLogger } from "@framefields/server-utils";-import { renderAndUpload } from "./render-upload.js";--export interface RenderJobInput {- virtualMedia?: VirtualMediaData;- type: "Video" | "Image" | "LUT";- frame?: number;- fps?: number;- width?: number;- height?: number;- bucket: string;- key: string;- codec?: string;- audioCodec?: string;- quality?: string;-}--export interface RenderJob {- id: string;- status: "IN_QUEUE" | "IN_PROGRESS" | "COMPLETED" | "FAILED" | "CANCELLED";- output?: { success: boolean; key: string; error?: string };- error?: string;-}--export const jobs = new Map<string, RenderJob>();--export const hasActiveJobs = (): boolean => {- for (const job of jobs.values()) {- if (job.status === "IN_PROGRESS" || job.status === "IN_QUEUE") {- return true;- }- }- return false;-};--export async function executeRenderJob(- jobId: string,- jobData: RenderJobInput,- onStateChange?: (state: "processing" | "idle") => void,-): Promise<void> {- const job = jobs.get(jobId);- if (!job) return;-- try {- onStateChange?.("processing");- job.status = "IN_PROGRESS";- const {- virtualMedia,- type,- frame,- bucket,- key,- codec,- audioCodec,- quality,- } = jobData;-- let probedMeta: any;-- const normalizedType = type.toLowerCase();- rendererLogger.info(- { type: normalizedType, key },- "[HTTP Local] Starting render process...",- );-- // Standard Virtual Media Rendering path - enforce metadata presence- if (!virtualMedia) {- throw new Error("virtualMedia is required for standard media rendering.");- }- if (!virtualMedia.metadata) {- throw new Error(- "virtualMedia.metadata is required for standard media rendering.",- );- }-- await renderAndUpload({- virtualMedia,- type,- frame,- codec,- audioCodec,- quality,- key,- bucket,- logPrefix: "[HTTP Local]",- });-- job.status = "COMPLETED";- job.output = {- success: true,- key,- metadata: probedMeta,- } as any;- rendererLogger.info(- { jobId, key },- "[HTTP Local] Render complete and uploaded successfully.",- );- } catch (err) {- const errMsg = err instanceof Error ? err.message : String(err);- rendererLogger.error(- { jobId, err: errMsg },- "[HTTP Local] Rendering job failed.",- );- job.status = "FAILED";- job.error = errMsg;- } finally {- if (!hasActiveJobs()) {- onStateChange?.("idle");- }- }-}diff --git a/apps/renderer-service/src/jobs/jobs.test.ts b/apps/renderer-service/src/jobs/jobs.test.tsnew file mode 100644index 00000000..943b0736--- /dev/null+++ b/apps/renderer-service/src/jobs/jobs.test.ts@@ -0,0 +1,299 @@+import { describe, expect, it } from "vitest";+import { RenderError } from "../errors.js";+import { redact, registerSecret } from "../redact.js";+import { testConfig, testJob } from "../test-helpers.js";+import { renderKey, renderKeys, slug } from "./key-template.js";+import { JobQueue } from "./queue.js";+import { JobStore } from "./store.js";+import { validateJob } from "./validate.js";++const codeOf = (fn: () => unknown) => {+ try {+ fn();+ } catch (err) {+ return err instanceof RenderError ? err.code : String(err);+ }+ return "no error";+};++describe("validateJob", () => {+ const config = testConfig();++ it("accepts a minimal job and fills defaults", () => {+ const job = validateJob(testJob(), config);+ expect(job.repo.id).toBe("github.com/acme/films");+ expect(job.input.render.kind).toBe("video");+ expect(job.input.source.workdir).toBe(".");+ expect(job.timeoutSec).toBe(config.limits.jobTimeoutMaxSec);+ });++ it("points legacy virtualMedia payloads at v2", () => {+ try {+ validateJob(+ { virtualMedia: {}, type: "Video", bucket: "b", key: "k" },+ config,+ );+ } catch (err) {+ expect((err as RenderError).hint).toMatch(/virtualMedia/);+ return;+ }+ throw new Error("expected JOB_INVALID");+ });++ it.each([+ [+ "a repo outside the allowlist",+ { source: { repo: "https://github.com/evil/x" } },+ "REPO_NOT_ALLOWED",+ ],+ [+ "plain http",+ { source: { repo: "http://github.com/acme/films" } },+ "REPO_NOT_ALLOWED",+ ],+ [+ "a workdir escaping the repo",+ { source: { repo: "https://github.com/acme/f", workdir: "../x" } },+ "JOB_INVALID",+ ],+ [+ "an absolute entry",+ { target: { entry: "/etc/passwd.js" } },+ "JOB_INVALID",+ ],+ [+ "a non-module entry",+ { target: { entry: "src/film.json" } },+ "JOB_INVALID",+ ],+ [+ "a bad export name",+ { target: { entry: "a.ts", export: "x; process.exit()" } },+ "JOB_INVALID",+ ],+ [+ "a ref with ..",+ { source: { repo: "https://github.com/acme/f", ref: "main..x" } },+ "JOB_INVALID",+ ],+ [+ "a ref that looks like an option",+ { source: { repo: "https://github.com/acme/f", ref: "--upload-pack=x" } },+ "JOB_INVALID",+ ],+ ["reserved env", { env: { NODE_OPTIONS: "--require x" } }, "JOB_INVALID"],+ [+ "job credentials when disabled",+ { source: { repo: "https://github.com/acme/f", auth: { token: "x" } } },+ "JOB_INVALID",+ ],+ [+ "s3 output without s3 storage",+ { output: { type: "s3", key: "a.mp4" } },+ "JOB_INVALID",+ ],+ [+ "frames without a frame list",+ { render: { kind: "frames" } },+ "JOB_INVALID",+ ],+ ])("rejects %s", (_, overrides, code) => {+ expect(codeOf(() => validateJob(testJob(overrides), config))).toBe(code);+ });++ it("enforces the bucket allowlist", () => {+ const s3 = testConfig({+ STORAGE_ENDPOINT: "https://s3.example.com",+ STORAGE_ACCESS_KEY_ID: "key",+ STORAGE_SECRET_ACCESS_KEY: "secret",+ STORAGE_BUCKET: "renders",+ });+ expect(+ codeOf(() =>+ validateJob(+ testJob({ output: { type: "s3", bucket: "other", key: "x" } }),+ s3,+ ),+ ),+ ).toBe("BUCKET_NOT_ALLOWED");+ expect(+ codeOf(() =>+ validateJob(+ testJob({ output: { type: "s3", key: "{jobId}.{ext}" } }),+ s3,+ ),+ ),+ ).toBe("no error");+ expect(+ codeOf(() =>+ validateJob(testJob({ output: { type: "s3", key: "{nope}" } }), s3),+ ),+ ).toBe("JOB_INVALID");+ });++ it("caps the timeout", () => {+ const job = validateJob(testJob({ timeoutSec: 999_999 }), config);+ expect(job.timeoutSec).toBe(config.limits.jobTimeoutMaxSec);+ });+});++describe("key templates", () => {+ const vars = {+ jobId: "j1",+ sha: "9f2c1e4abcdef",+ ref: "refs/heads/Feature/Intro v2",+ date: new Date("2026-10-14T10:00:00Z"),+ };++ it("renders variables", () => {+ expect(+ renderKey("acme/{ref}/{shortSha}/{date}/{name}-{index}.{ext}", {+ ...vars,+ name: "main",+ ext: "mp4",+ index: 0,+ }),+ ).toBe("acme/feature-intro-v2/9f2c1e4/2026-10-14/main-0.mp4");+ });++ it("rejects keys that escape", () => {+ expect(() =>+ renderKey("/{name}", { ...vars, name: "a", ext: "mp4", index: 0 }),+ ).toThrow();+ expect(() =>+ renderKey("a/../{name}", { ...vars, name: "a", ext: "mp4", index: 0 }),+ ).toThrow();+ });++ it("rejects templates that collide across outputs", () => {+ const outputs = [+ { name: "main", ext: "mp4" },+ { name: "teaser", ext: "mp4" },+ ];+ expect(() => renderKeys("{jobId}.{ext}", outputs, vars)).toThrow(+ /several outputs/,+ );+ expect(renderKeys("{jobId}/{name}.{ext}", outputs, vars)).toEqual([+ "j1/main.mp4",+ "j1/teaser.mp4",+ ]);+ });++ it("slugs refs", () => {+ expect(slug("refs/tags/v1.2.0")).toBe("v1.2.0");+ });+});++describe("redact", () => {+ it("strips URL credentials, signed queries and registered secrets", () => {+ registerSecret("ghp_supersecret123");+ expect(+ redact("fetch https://x-access-token:[email protected]/a/b.git failed"),+ ).toBe("fetch https://***@github.com/a/b.git failed");+ expect(+ redact("PUT https://b.r2.dev/k.mp4?X-Amz-Signature=abc&X-Amz-Date=1 403"),+ ).toBe("PUT https://b.r2.dev/k.mp4?*** 403");+ expect(redact("token ghp_supersecret123 leaked")).toBe("token *** leaked");+ });+});++describe("JobStore", () => {+ it("evicts finished jobs after the TTL and never touches active ones", () => {+ let now = 0;+ const store = new JobStore(1000, 10, () => now);+ store.create("a");+ store.create("b");+ store.update("a", { status: "COMPLETED" });+ now = 5000;+ store.prune();+ expect(store.get("a")).toBeUndefined();+ expect(store.get("b")).toBeDefined();+ });++ it("caps history by evicting the oldest finished jobs", () => {+ const store = new JobStore(60_000, 2);+ for (const id of ["a", "b", "c"]) {+ store.create(id);+ store.update(id, { status: "FAILED" });+ }+ store.create("d");+ expect(store.size).toBe(2);+ expect(store.get("d")).toBeDefined();+ });++ it("ignores updates after a job finished", () => {+ const store = new JobStore(60_000, 10);+ store.create("a");+ store.update("a", { status: "CANCELLED" });+ store.update("a", { status: "COMPLETED" });+ expect(store.get("a")?.status).toBe("CANCELLED");+ });+});++describe("JobQueue", () => {+ const deferred = () => {+ let resolve!: () => void;+ const promise = new Promise<void>((r) => {+ resolve = r;+ });+ return { promise, resolve };+ };++ it("runs one job at a time and rejects past the cap", async () => {+ const queue = new JobQueue(1);+ const order: string[] = [];+ const gate = deferred();+ queue.enqueue({+ id: "a",+ run: async () => {+ order.push("a:start");+ await gate.promise;+ order.push("a:end");+ },+ onCancelledWhileQueued: () => {},+ });+ queue.enqueue({+ id: "b",+ run: async () => void order.push("b"),+ onCancelledWhileQueued: () => {},+ });+ expect(() =>+ queue.enqueue({+ id: "c",+ run: async () => {},+ onCancelledWhileQueued: () => {},+ }),+ ).toThrow(/Queue full/);++ gate.resolve();+ await queue.idle();+ expect(order).toEqual(["a:start", "a:end", "b"]);+ });++ it("cancels waiting jobs without running them and aborts the active one", async () => {+ const queue = new JobQueue(5);+ const cancelled: string[] = [];+ let activeSignal: AbortSignal | undefined;+ queue.enqueue({+ id: "a",+ run: (signal) =>+ new Promise<void>((resolve) => {+ activeSignal = signal;+ signal.addEventListener("abort", () => resolve());+ }),+ onCancelledWhileQueued: () => cancelled.push("a"),+ });+ queue.enqueue({+ id: "b",+ run: async () => {},+ onCancelledWhileQueued: () => cancelled.push("b"),+ });++ expect(queue.cancel("b", new RenderError("CANCELLED", "x"))).toBe(true);+ expect(queue.cancel("a", new RenderError("CANCELLED", "x"))).toBe(true);+ await queue.idle();+ expect(cancelled).toEqual(["b"]);+ expect(activeSignal?.aborted).toBe(true);+ expect(queue.cancel("zzz", new RenderError("CANCELLED", "x"))).toBe(false);+ });+});diff --git a/apps/renderer-service/src/jobs/key-template.ts b/apps/renderer-service/src/jobs/key-template.tsnew file mode 100644index 00000000..583434f3--- /dev/null+++ b/apps/renderer-service/src/jobs/key-template.ts@@ -0,0 +1,102 @@+import { RenderError } from "../errors.js";+import { hasControlChars } from "../paths.js";++export interface KeyVars {+ jobId: string;+ sha: string;+ ref: string;+ ext: string;+ name: string;+ index: number;+ date?: Date;+}++const VAR = /\{(\w+)\}/g;+const KNOWN = new Set([+ "jobId",+ "sha",+ "shortSha",+ "ref",+ "date",+ "ext",+ "name",+ "index",+]);++/** Throws JOB_INVALID for templates that use unknown variables. */+export function checkKeyTemplate(template: string): void {+ for (const [, name] of template.matchAll(VAR)) {+ if (!KNOWN.has(name)) {+ throw new RenderError(+ "JOB_INVALID",+ `Unknown variable {${name}} in output key "${template}".`,+ { hint: `Use one of ${[...KNOWN].map((k) => `{${k}}`).join(", ")}.` },+ );+ }+ }+}++/** Lowercase, `/`-free slug for refs like `feature/Intro v2`. */+export function slug(value: string): string {+ return (+ value+ .toLowerCase()+ .replace(/^refs\/(heads|tags)\//, "")+ .replace(/[^a-z0-9._-]+/g, "-")+ .replace(/^-+|-+$/g, "") || "x"+ );+}++/**+ * Renders an output key (or local path) and rejects results that could+ * escape their bucket/dir: `..` segments, a leading `/`, control characters.+ */+export function renderKey(template: string, vars: KeyVars): string {+ checkKeyTemplate(template);+ const date = (vars.date ?? new Date()).toISOString().slice(0, 10);+ const values: Record<string, string> = {+ jobId: vars.jobId,+ sha: vars.sha,+ shortSha: vars.sha.slice(0, 7),+ ref: slug(vars.ref),+ date,+ ext: vars.ext,+ name: slug(vars.name),+ index: String(vars.index),+ };+ const key = template+ .replace(VAR, (_, name: string) => values[name])+ .replace(/\/{2,}/g, "/");++ if (+ key.startsWith("/") ||+ hasControlChars(key) ||+ key.split("/").some((seg) => seg === ".." || seg === ".")+ ) {+ throw new RenderError("PATH_ESCAPE", `Output key "${key}" is not allowed.`);+ }+ return key;+}++/** Renders one key per output and fails if two outputs would collide. */+export function renderKeys(+ template: string,+ outputs: readonly { name: string; ext: string }[],+ vars: Omit<KeyVars, "name" | "ext" | "index">,+): string[] {+ const keys = outputs.map((o, index) =>+ renderKey(template, { ...vars, name: o.name, ext: o.ext, index }),+ );+ const seen = new Set<string>();+ for (const key of keys) {+ if (seen.has(key)) {+ throw new RenderError(+ "JOB_INVALID",+ `Output key template "${template}" maps several outputs to "${key}".`,+ { hint: "Add {name} or {index} to the key." },+ );+ }+ seen.add(key);+ }+ return keys;+}diff --git a/apps/renderer-service/src/jobs/pipeline.ts b/apps/renderer-service/src/jobs/pipeline.tsnew file mode 100644index 00000000..76f629e8--- /dev/null+++ b/apps/renderer-service/src/jobs/pipeline.ts@@ -0,0 +1,75 @@+import type { RendererConfig } from "../config/index.js";+import type { RepoRef } from "../git/repo-url.js";+import type { JobInput } from "./schema.js";+import type { JobProgress } from "./store.js";++/** What every pipeline step gets from the runner. */+export interface StepContext {+ jobId: string;+ config: RendererConfig;+ signal: AbortSignal;+ progress(p: JobProgress): void;+ warn(code: string, message: string): void;+ /** Per-job scratch dir on local disk (outputs, logs). */+ jobDir: string;+}++export interface CheckoutResult {+ /** Absolute path of the repo checkout. */+ root: string;+ /** Absolute path of `source.workdir` inside it. */+ workdir: string;+ sha: string;+ cache: Record<string, string>;+ /** Returns the workspace slot to the pool. */+ release(): Promise<void>;+}++export interface InstallResult {+ skipped: boolean;+ packageManager?: string;+ cache: Record<string, string>;+ /** Range the project declares for framefields, if any. */+ declaredEngine?: string;+}++export interface RenderedFile {+ name: string;+ path: string;+ ext: string;+ contentType: string;+ durationMs?: number;+ frames?: number;+ qa?: unknown;+}++export interface DeliveredFile {+ name: string;+ bytes: number;+ sha256: string;+ contentType: string;+ key?: string;+ url?: string;+ path?: string;+ base64?: string;+ durationMs?: number;+ frames?: number;+}++export interface Pipeline {+ checkout(+ job: { input: JobInput; repo: RepoRef; ref: string },+ ctx: StepContext,+ ): Promise<CheckoutResult>;+ install(checkout: CheckoutResult, ctx: StepContext): Promise<InstallResult>;+ render(+ checkout: CheckoutResult,+ input: JobInput,+ ctx: StepContext,+ ): Promise<{ files: RenderedFile[]; engine: Record<string, unknown> }>;+ deliver(+ files: RenderedFile[],+ job: { input: JobInput; sha: string; ref: string },+ ctx: StepContext,+ ): Promise<DeliveredFile[]>;+}diff --git a/apps/renderer-service/src/jobs/queue.ts b/apps/renderer-service/src/jobs/queue.tsnew file mode 100644index 00000000..afc59ae1--- /dev/null+++ b/apps/renderer-service/src/jobs/queue.ts@@ -0,0 +1,100 @@+import { RenderError } from "../errors.js";++export interface QueuedJob {+ id: string;+ run(signal: AbortSignal): Promise<void>;+ /** Called instead of run() when the job is cancelled while waiting. */+ onCancelledWhileQueued(reason: RenderError): void;+}++/**+ * Runs one job at a time (PLAN.md §3: cwd, env and the GPU are process-wide).+ * Waiting jobs are capped at `max`; past that, enqueue throws BUSY.+ */+export class JobQueue {+ private readonly waiting: QueuedJob[] = [];+ private active?: { job: QueuedJob; controller: AbortController };+ private closed = false;+ private idleResolvers: (() => void)[] = [];++ constructor(private readonly max: number) {}++ enqueue(job: QueuedJob): void {+ if (this.closed) {+ throw new RenderError("SHUTDOWN", "This worker is shutting down.");+ }+ if (this.waiting.length >= this.max) {+ throw new RenderError("BUSY", `Queue full (${this.max} waiting).`);+ }+ this.waiting.push(job);+ this.pump();+ }++ /** Cancels a waiting or running job. Returns false if unknown. */+ cancel(id: string, reason: RenderError): boolean {+ const index = this.waiting.findIndex((j) => j.id === id);+ if (index >= 0) {+ const [job] = this.waiting.splice(index, 1);+ job.onCancelledWhileQueued(reason);+ return true;+ }+ if (this.active?.job.id === id) {+ this.active.controller.abort(reason);+ return true;+ }+ return false;+ }++ /**+ * Stops accepting jobs. Waiting jobs are cancelled with `reason`; the+ * running one is aborted too when `abortActive` is set.+ */+ close(reason: RenderError, abortActive: boolean): void {+ this.closed = true;+ for (const job of this.waiting.splice(0))+ job.onCancelledWhileQueued(reason);+ if (abortActive) this.active?.controller.abort(reason);+ }++ get busy(): boolean {+ return this.active !== undefined;+ }++ get activeId(): string | undefined {+ return this.active?.job.id;+ }++ get depth(): number {+ return this.waiting.length;+ }++ get isClosed(): boolean {+ return this.closed;+ }++ /** Resolves once nothing is running or waiting. */+ idle(): Promise<void> {+ if (!this.active && this.waiting.length === 0) return Promise.resolve();+ return new Promise((resolve) => this.idleResolvers.push(resolve));+ }++ private pump(): void {+ if (this.active) return;+ const job = this.waiting.shift();+ if (!job) {+ for (const resolve of this.idleResolvers.splice(0)) resolve();+ return;+ }+ const controller = new AbortController();+ this.active = { job, controller };+ job+ .run(controller.signal)+ .catch(() => {+ // run() reports its own failures; never let one stop the queue.+ })+ .finally(() => {+ this.active = undefined;+ this.pump();+ });+ }+}diff --git a/apps/renderer-service/src/jobs/runner.ts b/apps/renderer-service/src/jobs/runner.tsnew file mode 100644index 00000000..07eb45c2--- /dev/null+++ b/apps/renderer-service/src/jobs/runner.ts@@ -0,0 +1,222 @@+import { randomUUID } from "node:crypto";+import fs from "node:fs/promises";+import path from "node:path";+import { loggerContext, rendererLogger } from "@framefields/server-utils";+import type { RendererConfig } from "../config/index.js";+import { type JobPhase, RenderError, toRenderError } from "../errors.js";+import { redact, redactValue } from "../redact.js";+import type { Pipeline, StepContext } from "./pipeline.js";+import { JobQueue } from "./queue.js";+import { type JobRecord, JobStore } from "./store.js";+import { type ValidatedJob, validateJob } from "./validate.js";++/**+ * Accepts jobs, queues them one at a time, and drives each through the+ * pipeline: checkout → install → render → deliver (PLAN.md §3).+ */+export class JobRunner {+ readonly store: JobStore;+ readonly queue: JobQueue;+ private completedJobs = 0;++ constructor(+ private readonly config: RendererConfig,+ private readonly pipeline: Pipeline,+ private readonly hooks: { afterJob?: (completed: number) => void } = {},+ ) {+ this.store = new JobStore(+ config.limits.jobTtlSec * 1000,+ config.limits.jobHistoryMax,+ );+ this.queue = new JobQueue(config.limits.queueMax);+ }++ /** Validates and queues a job. Throws RenderError (JOB_INVALID, BUSY, …). */+ submit(raw: unknown): JobRecord {+ const job = validateJob(raw, this.config);+ const id = randomUUID();+ const record = this.store.create(id);+ try {+ this.queue.enqueue({+ id,+ // Job logs carry the jobId, not the submitting request's context.+ run: (signal) =>+ loggerContext.run({ component: "job", jobId: id }, () =>+ this.execute(id, job, signal),+ ),+ onCancelledWhileQueued: (reason) => this.fail(id, reason),+ });+ } catch (err) {+ this.fail(id, toRenderError(err, "validate"));+ throw err;+ }+ return record;+ }++ cancel(id: string): boolean {+ return this.queue.cancel(+ id,+ new RenderError("CANCELLED", "Cancelled by request."),+ );+ }++ private async execute(+ id: string,+ job: ValidatedJob,+ signal: AbortSignal,+ ): Promise<void> {+ const started = Date.now();+ const timings: Record<string, number> = {};+ const cache: Record<string, string> = {};+ const warnings: string[] = [];+ const jobDir = path.join(this.config.paths.workDir, "jobs", id);+ let phase: JobPhase = "resolve";++ // Total timeout: abort with RENDER_TIMEOUT unless something else aborts first.+ const controller = new AbortController();+ const onAbort = () => controller.abort(signal.reason);+ signal.addEventListener("abort", onAbort, { once: true });+ const timer = setTimeout(+ () =>+ controller.abort(+ new RenderError(+ "RENDER_TIMEOUT",+ `Job exceeded its ${job.timeoutSec}s timeout.`,+ {+ phase,+ },+ ),+ ),+ job.timeoutSec * 1000,+ );++ const ctx: StepContext = {+ jobId: id,+ config: this.config,+ signal: controller.signal,+ jobDir,+ progress: (p) => this.store.update(id, { progress: p }),+ warn: (code, message) => {+ warnings.push(`${code}: ${redact(message)}`);+ this.store.update(id, { warnings: [...warnings] });+ },+ };+ const timed = async <T>(name: string, fn: () => Promise<T>): Promise<T> => {+ const t0 = Date.now();+ try {+ return await fn();+ } finally {+ timings[`${name}Ms`] = Date.now() - t0;+ }+ };+ const checkAbort = () => {+ if (controller.signal.aborted)+ throw toRenderError(controller.signal.reason, phase);+ };++ this.store.update(id, {+ status: "IN_PROGRESS",+ startedAt: new Date().toISOString(),+ progress: { phase: "resolving" },+ });+ rendererLogger.info(+ { jobId: id, repo: job.repo.id, ref: job.ref },+ "Job started",+ );++ let release: (() => Promise<void>) | undefined;+ let succeeded = false;+ try {+ await fs.mkdir(path.join(jobDir, "out"), { recursive: true });++ const checkout = await timed("checkout", () =>+ this.pipeline.checkout(job, ctx),+ );+ release = checkout.release;+ Object.assign(cache, checkout.cache);+ checkAbort();++ phase = "install";+ ctx.progress({ phase: "installing" });+ const install = await timed("install", () =>+ this.pipeline.install(checkout, ctx),+ );+ Object.assign(cache, install.cache);+ checkAbort();++ phase = "render";+ ctx.progress({ phase: "loading" });+ const rendered = await timed("render", () =>+ this.pipeline.render(checkout, job.input, ctx),+ );+ checkAbort();++ phase = "upload";+ ctx.progress({ phase: "uploading" });+ const outputs = await timed("upload", () =>+ this.pipeline.deliver(+ rendered.files,+ { input: job.input, sha: checkout.sha, ref: job.ref },+ ctx,+ ),+ );+ checkAbort();++ timings.totalMs = Date.now() - started;+ this.store.update(id, {+ status: "COMPLETED",+ progress: { phase: "done" },+ result: redactValue({+ jobId: id,+ source: {+ repo: job.repo.id,+ ref: job.ref,+ sha: checkout.sha,+ workdir: job.input.source.workdir,+ },+ engine: {+ ...rendered.engine,+ projectDeclared: install.declaredEngine,+ },+ outputs,+ timings,+ cache,+ warnings,+ }),+ });+ succeeded = true;+ rendererLogger.info({ jobId: id, timings }, "Job completed");+ } catch (err) {+ const reason = controller.signal.aborted ? controller.signal.reason : err;+ this.fail(id, toRenderError(reason, phase), { timings, cache });+ } finally {+ clearTimeout(timer);+ signal.removeEventListener("abort", onAbort);+ await release?.().catch((err) =>+ rendererLogger.warn(+ { jobId: id, err: String(err) },+ "Workspace release failed",+ ),+ );+ if (succeeded || !this.config.keepFailedJobs) {+ await fs.rm(jobDir, { recursive: true, force: true }).catch(() => {});+ }+ this.completedJobs++;+ this.hooks.afterJob?.(this.completedJobs);+ }+ }++ private fail(+ id: string,+ error: RenderError,+ extra: Record<string, unknown> = {},+ ): void {+ const json = redactValue(error.toJSON());+ rendererLogger.warn({ jobId: id, error: json }, "Job failed");+ this.store.update(id, {+ status: error.code === "CANCELLED" ? "CANCELLED" : "FAILED",+ error: json,+ ...(Object.keys(extra).length > 0 && { result: redactValue(extra) }),+ });+ }+}diff --git a/apps/renderer-service/src/jobs/schema.ts b/apps/renderer-service/src/jobs/schema.tsnew file mode 100644index 00000000..7bcae544--- /dev/null+++ b/apps/renderer-service/src/jobs/schema.ts@@ -0,0 +1,152 @@+import z from "zod";+import { isSafeRelativePath } from "../paths.js";++const relPath = (what: string) =>+ z+ .string()+ .min(1)+ .max(1024)+ .refine((p) => p === "." || isSafeRelativePath(p), {+ message: `${what} must be a relative path inside the repo (no "..", no leading "/")`,+ });++const lfsPattern = z.string().min(1).max(512).refine(isSafeRelativePath, {+ message: 'LFS patterns must be relative and must not contain ".."',+});++const sourceSchema = z.object({+ type: z.literal("git").default("git"),+ repo: z.string().min(1).max(2048).optional(),+ ref: z.string().min(1).max(255).optional(),+ commit: z+ .string()+ .regex(/^[0-9a-f]{7,64}$/i, "commit must be a hex SHA (7-64 chars)")+ .optional(),+ workdir: relPath("workdir").default("."),+ lfs: z+ .object({+ enabled: z.boolean().optional(),+ include: z.array(lfsPattern).max(256).default([]),+ exclude: z.array(lfsPattern).max(256).default([]),+ })+ .default({ include: [], exclude: [] }),+ submodules: z.boolean().default(false),+ sparse: z.array(relPath("sparse path")).max(256).optional(),+ auth: z+ .object({+ username: z.string().optional(),+ token: z.string().optional(),+ sshKey: z.string().optional(),+ })+ .optional(),+});++const exportName = z+ .string()+ .regex(/^[A-Za-z_$][\w$]*$/, "export must be a JavaScript identifier");++const targetSchema = z.object({+ entry: relPath("entry").refine(+ (p) => /\.(m?ts|m?js|tsx|jsx|cts|cjs)$/.test(p),+ {+ message: "entry must be a .ts/.mts/.js/.mjs module",+ },+ ),+ export: exportName.default("default"),+ args: z.unknown().optional(),+ select: z.array(z.string().min(1).max(128)).min(1).optional(),+});++export const RENDER_KINDS = [+ "video",+ "frames",+ "frameGrid",+ "lut",+ "audio",+] as const;++export const renderSchema = z.object({+ kind: z.enum(RENDER_KINDS).default("video"),+ codec: z.string().max(32).optional(),+ audioCodec: z.string().max(32).optional(),+ quality: z.string().max(32).optional(),+ frames: z.array(z.number().int().min(0)).min(1).max(10_000).optional(),+ range: z+ .object({ from: z.number().int().min(0), to: z.number().int().min(1) })+ .refine((r) => r.to > r.from, { message: "range.to must be > range.from" })+ .optional(),+ grid: z+ .object({+ from: z.number().int().min(0).optional(),+ to: z.number().int().min(1).optional(),+ step: z.number().int().min(1).optional(),+ columns: z.number().int().min(1).max(64).optional(),+ })+ .optional(),+ qa: z+ .union([+ z.boolean(),+ z.looseObject({ failOn: z.array(z.string()).optional() }),+ ])+ .optional(),+});++export type RenderSettings = z.infer<typeof renderSchema>;++const outputSchema = z.discriminatedUnion("type", [+ z.object({+ type: z.literal("s3"),+ bucket: z.string().min(1).max(255).optional(),+ key: z.string().min(1).max(1024),+ }),+ z.object({+ type: z.literal("presigned"),+ /** One URL per output, in plan order. */+ urls: z+ .array(z.url({ protocol: /^https?$/ }))+ .min(1)+ .max(256),+ headers: z.record(z.string(), z.string()).optional(),+ }),+ z.object({+ type: z.literal("local"),+ path: z.string().min(1).max(1024),+ }),+ z.object({ type: z.literal("inline") }),+]);++const envName = /^[A-Za-z_][A-Za-z0-9_]*$/;++export const jobSchema = z+ .object({+ version: z.literal(2),+ source: sourceSchema.default({+ type: "git",+ workdir: ".",+ lfs: { include: [], exclude: [] },+ submodules: false,+ }),+ target: targetSchema,+ render: renderSchema.default({ kind: "video" }),+ output: outputSchema,+ env: z+ .record(+ z.string().regex(envName, "invalid env var name"),+ z.string().max(32_768),+ )+ .default({}),+ timeoutSec: z.number().int().positive().optional(),+ idempotencyKey: z.string().min(1).max(200).optional(),+ })+ .superRefine((job, ctx) => {+ if (job.render.kind === "frames" && !job.render.frames) {+ ctx.addIssue({+ code: "custom",+ path: ["render", "frames"],+ message: 'required when render.kind is "frames"',+ });+ }+ });++export type JobInput = z.infer<typeof jobSchema>;+export type JobOutputTarget = JobInput["output"];diff --git a/apps/renderer-service/src/jobs/store.ts b/apps/renderer-service/src/jobs/store.tsnew file mode 100644index 00000000..7a2b2176--- /dev/null+++ b/apps/renderer-service/src/jobs/store.ts@@ -0,0 +1,137 @@+import { EventEmitter } from "node:events";+import type { RenderErrorJSON } from "../errors.js";++export type JobStatus =+ | "IN_QUEUE"+ | "IN_PROGRESS"+ | "COMPLETED"+ | "FAILED"+ | "CANCELLED";++export type JobPhaseName =+ | "queued"+ | "resolving"+ | "fetching"+ | "lfs"+ | "installing"+ | "loading"+ | "rendering"+ | "encoding"+ | "verifying"+ | "uploading"+ | "done";++export interface JobProgress {+ phase: JobPhaseName;+ /** Index of the output being rendered, when the plan has several. */+ output?: number;+ frame?: number;+ total?: number;+ fps?: number;+ etaSec?: number;+ message?: string;+}++export interface JobRecord {+ id: string;+ status: JobStatus;+ progress: JobProgress;+ createdAt: string;+ updatedAt: string;+ startedAt?: string;+ finishedAt?: string;+ result?: Record<string, unknown>;+ error?: RenderErrorJSON;+ warnings: string[];+}++const TERMINAL: ReadonlySet<JobStatus> = new Set([+ "COMPLETED",+ "FAILED",+ "CANCELLED",+]);++export function isTerminal(status: JobStatus): boolean {+ return TERMINAL.has(status);+}++/**+ * In-memory job records with TTL and size-bounded eviction. Emits `update`+ * with the record on every change (SSE progress).+ */+export class JobStore extends EventEmitter<{ update: [JobRecord] }> {+ private readonly jobs = new Map<string, JobRecord>();++ constructor(+ private readonly ttlMs: number,+ private readonly maxHistory: number,+ private readonly now: () => number = Date.now,+ ) {+ super();+ this.setMaxListeners(0);+ }++ create(id: string): JobRecord {+ const ts = new Date(this.now()).toISOString();+ const record: JobRecord = {+ id,+ status: "IN_QUEUE",+ progress: { phase: "queued" },+ createdAt: ts,+ updatedAt: ts,+ warnings: [],+ };+ this.jobs.set(id, record);+ this.prune();+ this.emit("update", record);+ return record;+ }++ get(id: string): JobRecord | undefined {+ return this.jobs.get(id);+ }++ update(+ id: string,+ patch: Partial<Omit<JobRecord, "id" | "createdAt">>,+ ): JobRecord | undefined {+ const record = this.jobs.get(id);+ if (!record) return undefined;+ // A finished job never changes again (late progress from a slow abort).+ if (isTerminal(record.status)) return record;+ Object.assign(record, patch, {+ updatedAt: new Date(this.now()).toISOString(),+ });+ if (patch.status && isTerminal(patch.status)) {+ record.finishedAt = record.updatedAt;+ }+ this.emit("update", record);+ return record;+ }++ /** Restores a record written before a crash (crash journal, PLAN.md §8.3). */+ restore(record: JobRecord): void {+ this.jobs.set(record.id, record);+ this.prune();+ }++ prune(): void {+ const cutoff = this.now() - this.ttlMs;+ for (const [id, job] of this.jobs) {+ if (isTerminal(job.status) && Date.parse(job.updatedAt) < cutoff) {+ this.jobs.delete(id);+ }+ }+ // Oldest finished jobs go first; active jobs are never evicted.+ if (this.jobs.size > this.maxHistory) {+ for (const [id, job] of this.jobs) {+ if (this.jobs.size <= this.maxHistory) break;+ if (isTerminal(job.status)) this.jobs.delete(id);+ }+ }+ }++ get size(): number {+ return this.jobs.size;+ }+}diff --git a/apps/renderer-service/src/jobs/validate.ts b/apps/renderer-service/src/jobs/validate.tsnew file mode 100644index 00000000..402111c6--- /dev/null+++ b/apps/renderer-service/src/jobs/validate.ts@@ -0,0 +1,145 @@+import type { RendererConfig } from "../config/index.js";+import { RESERVED_JOB_ENV } from "../config/schema.js";+import { RenderError } from "../errors.js";+import { isRepoAllowed, parseRepoUrl, type RepoRef } from "../git/repo-url.js";+import { hasControlChars } from "../paths.js";+import { checkKeyTemplate } from "./key-template.js";+import { type JobInput, jobSchema } from "./schema.js";++export interface ValidatedJob {+ input: JobInput;+ repo: RepoRef;+ ref: string;+ timeoutSec: number;+}++/**+ * Parses a raw payload and checks it against the worker's security settings+ * (allowlists, buckets, env names, limits). Job input can never widen them.+ */+export function validateJob(+ raw: unknown,+ config: RendererConfig,+): ValidatedJob {+ const parsed = jobSchema.safeParse(raw);+ if (!parsed.success) {+ throw new RenderError(+ "JOB_INVALID",+ "The job does not match the v2 schema.",+ {+ details: {+ issues: parsed.error.issues.map((i) => ({+ path: i.path.join("."),+ message: i.message,+ })),+ },+ hint:+ raw && typeof raw === "object" && "virtualMedia" in raw+ ? "Inline virtualMedia jobs were removed; send a v2 git job."+ : undefined,+ },+ );+ }+ const input = parsed.data;++ const repoUrl = input.source.repo ?? config.source.defaultRepo;+ if (!repoUrl) {+ throw new RenderError(+ "JOB_INVALID",+ "source.repo is required (no FF_DEFAULT_REPO set).",+ );+ }+ const repo = parseRepoUrl(repoUrl, { allowLocal: config.source.allowLocal });+ if (repo.protocol === "http") {+ throw new RenderError(+ "REPO_NOT_ALLOWED",+ "Plain http:// remotes are not allowed; use https.",+ );+ }+ if (+ repo.protocol !== "file" &&+ !isRepoAllowed(repo.id, config.source.allowlist)+ ) {+ throw new RenderError(+ "REPO_NOT_ALLOWED",+ `Repo ${repo.id} is not in FF_REPO_ALLOWLIST.`,+ );+ }++ if (input.source.auth && !config.git.allowJobCredentials) {+ throw new RenderError(+ "JOB_INVALID",+ "source.auth is disabled on this worker (FF_ALLOW_JOB_CREDENTIALS).",+ );+ }++ const ref = input.source.ref ?? config.source.defaultRef;+ if (hasControlChars(ref) || /[ ~^:?*[\\]|\.\.|@\{|^-/.test(ref)) {+ throw new RenderError("JOB_INVALID", `"${ref}" is not a valid git ref.`);+ }++ validateOutput(input, config);++ for (const name of Object.keys(input.env)) {+ if (RESERVED_JOB_ENV.some((re) => re.test(name))) {+ throw new RenderError(+ "JOB_INVALID",+ `env.${name} is reserved and can't be set by a job.`,+ );+ }+ if (!config.jobEnv.allow.some((p) => globMatch(p, name))) {+ throw new RenderError(+ "JOB_INVALID",+ `env.${name} is not allowed by FF_JOB_ENV_ALLOW.`,+ );+ }+ }++ const max = config.limits.jobTimeoutMaxSec;+ const timeoutSec = Math.min(input.timeoutSec ?? max, max);++ return { input, repo, ref, timeoutSec };+}++function validateOutput(input: JobInput, config: RendererConfig): void {+ const out = input.output;+ const driver = config.storage.driver;+ switch (out.type) {+ case "s3": {+ if (driver !== "s3") {+ throw new RenderError(+ "JOB_INVALID",+ 'output.type "s3" needs STORAGE_DRIVER=s3 on the worker.',+ );+ }+ const bucket = out.bucket ?? config.storage.bucket;+ if (!bucket || !config.storage.outputBuckets.includes(bucket)) {+ throw new RenderError(+ "BUCKET_NOT_ALLOWED",+ `Bucket "${bucket}" is not in FF_OUTPUT_BUCKETS.`,+ );+ }+ checkKeyTemplate(out.key);+ return;+ }+ case "local":+ if (!config.storage.outputDir) {+ throw new RenderError(+ "JOB_INVALID",+ 'output.type "local" needs FF_OUTPUT_DIR on the worker.',+ );+ }+ checkKeyTemplate(out.path);+ return;+ case "presigned":+ case "inline":+ return;+ }+}++function globMatch(pattern: string, value: string): boolean {+ const re = new RegExp(+ `^${pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*")}$`,+ );+ return re.test(value);+}diff --git a/apps/renderer-service/src/paths.ts b/apps/renderer-service/src/paths.tsnew file mode 100644index 00000000..915b5fb4--- /dev/null+++ b/apps/renderer-service/src/paths.ts@@ -0,0 +1,46 @@+import fs from "node:fs/promises";+import path from "node:path";+import { RenderError } from "./errors.js";++/** True when `s` contains ASCII control characters (incl. NUL and DEL). */+export function hasControlChars(s: string): boolean {+ for (let i = 0; i < s.length; i++) {+ const c = s.charCodeAt(i);+ if (c < 0x20 || c === 0x7f) return true;+ }+ return false;+}++/**+ * True for a relative POSIX path that stays inside its root: no leading `/`,+ * no `..` segment, no NUL or control characters, no Windows drive letters.+ */+export function isSafeRelativePath(p: string): boolean {+ if (!p || hasControlChars(p)) return false;+ if (p.startsWith("/") || p.startsWith("\\") || /^[a-z]:/i.test(p)) {+ return false;+ }+ return !p.split(/[/\\]/).some((seg) => seg === "..");+}++/**+ * Resolves `rel` under `root` and follows symlinks, failing with PATH_ESCAPE+ * when the real path lands outside the real root.+ */+export async function resolveInside(+ root: string,+ rel: string,+): Promise<string> {+ if (!isSafeRelativePath(rel) && rel !== ".") {+ throw new RenderError("PATH_ESCAPE", `Path "${rel}" leaves the workspace.`);+ }+ const realRoot = await fs.realpath(root);+ const real = await fs.realpath(path.resolve(realRoot, rel));+ if (real !== realRoot && !real.startsWith(realRoot + path.sep)) {+ throw new RenderError(+ "PATH_ESCAPE",+ `Path "${rel}" resolves outside the workspace (symlink?).`,+ );+ }+ return real;+}diff --git a/apps/renderer-service/src/redact.ts b/apps/renderer-service/src/redact.tsnew file mode 100644index 00000000..e13c0287--- /dev/null+++ b/apps/renderer-service/src/redact.ts@@ -0,0 +1,44 @@+/**+ * Strips credentials from anything that leaves the worker: logs, job results+ * and error messages (PLAN.md §9).+ */++const MIN_SECRET_LENGTH = 6;+const secrets = new Set<string>();++/** Registers a value that must never appear in output. */+export function registerSecret(value: string | undefined): void {+ if (value && value.length >= MIN_SECRET_LENGTH) secrets.add(value);+}++export function clearSecrets(): void {+ secrets.clear();+}++// user:pass@ or token@ in any scheme://… URL+const URL_USERINFO = /([a-z][a-z0-9+.-]*:\/\/)[^/\s@]+@/gi;+// Query strings carrying signatures (S3/R2 presigned URLs, SAS tokens, …)+const SIGNED_QUERY =+ /(https?:\/\/[^\s?"']+)\?[^\s"']*(?:X-Amz-Signature|X-Amz-Credential|Signature|sig|token|access_token)=[^\s"']*/gi;++export function redact(text: string): string {+ let out = text+ .replace(URL_USERINFO, "$1***@")+ .replace(SIGNED_QUERY, "$1?***");+ for (const secret of secrets) {+ if (out.includes(secret)) out = out.split(secret).join("***");+ }+ return out;+}++/** Deep-redacts strings inside JSON-like values. */+export function redactValue<T>(value: T): T {+ if (typeof value === "string") return redact(value) as T;+ if (Array.isArray(value)) return value.map(redactValue) as T;+ if (value && typeof value === "object") {+ const out: Record<string, unknown> = {};+ for (const [k, v] of Object.entries(value)) out[k] = redactValue(v);+ return out as T;+ }+ return value;+}diff --git a/apps/renderer-service/src/render-upload.ts b/apps/renderer-service/src/render-upload.tsdeleted file mode 100644index 8d1da807..00000000--- a/apps/renderer-service/src/render-upload.ts+++ /dev/null@@ -1,89 +0,0 @@-import type { VirtualMediaData } from "@framefields/core";-import { HeadlessMediaRenderer } from "@framefields/renderer";-import {- container,- rendererLogger,- type StorageService,- TOKENS,-} from "@framefields/server-utils";--export interface RenderUploadOptions {- virtualMedia: VirtualMediaData;- /** "video" / "gif" render a file, "lut" a LUT, anything else a PNG frame. */- type: string;- frame?: number;- codec?: string;- audioCodec?: string;- quality?: string;- key: string;- bucket: string;- /** Log line prefix, e.g. "[CLI]". */- logPrefix: string;-}--function videoContentType(codec: string | undefined): string {- return codec === "vp8" || codec === "vp9"- ? "video/webm"- : codec === "gif"- ? "image/gif"- : codec === "mp3"- ? "audio/mpeg"- : codec === "aac"- ? "audio/mp4"- : codec === "opus"- ? "audio/webm"- : "video/mp4";-}--/**- * Renders `virtualMedia` and uploads the result to storage under `key`.- * Returns the uploaded content type.- */-export async function renderAndUpload(- options: RenderUploadOptions,-): Promise<string> {- const { virtualMedia, frame, key, bucket, logPrefix } = options;- const normalizedType = options.type.toLowerCase();- const renderer = new HeadlessMediaRenderer();- let buffer: Buffer | undefined;- let filePath: string | undefined;- let contentType: string;- let fileCleanup: (() => Promise<void>) | undefined;-- if (normalizedType === "video" || normalizedType === "gif") {- const renderResult = await renderer.renderVideo(virtualMedia, {- codec: options.codec,- audioCodec: options.audioCodec,- quality: options.quality,- });- filePath = renderResult.filePath;- fileCleanup = renderResult.cleanup;- contentType = videoContentType(options.codec);- } else if (normalizedType === "lut") {- buffer = await renderer.renderLut(virtualMedia, frame);- contentType = "text/plain";- } else {- buffer = await renderer.renderImage(virtualMedia, frame);- contentType = "image/png";- }-- const storage = container.get<StorageService>(TOKENS.STORAGE);- if (filePath) {- rendererLogger.info(- { filePath },- `${logPrefix} Render complete - uploading file to storage.`,- );- await storage.uploadFileToStorage(filePath, key, contentType, bucket);- } else if (buffer) {- rendererLogger.info(- { fileSize: buffer.length },- `${logPrefix} Render complete - uploading buffer to storage.`,- );- await storage.uploadToStorage(buffer, key, contentType, bucket);- }-- if (fileCleanup) {- await fileCleanup().catch(() => {});- }- return contentType;-}diff --git a/apps/renderer-service/src/render.test.ts b/apps/renderer-service/src/render.test.tsdeleted file mode 100644index 97b04eb7..00000000--- a/apps/renderer-service/src/render.test.ts+++ /dev/null@@ -1,56 +0,0 @@-import "reflect-metadata";-import { HeadlessMediaRenderer } from "@framefields/renderer";-import fs from "fs/promises";-import { describe, expect, it } from "vitest";-import { hasActiveJobs, jobs } from "./job-processor.js";--describe("HeadlessMediaRenderer", () => {- it("should render a paint red rect frame", async () => {- const renderer = new HeadlessMediaRenderer();-- const virtualMedia = {- metadata: {- width: 1920,- height: 1080,- fps: 24,- durationMs: 1000,- },- operation: {- op: "Paint",- dataType: "Video",- backgroundColor: "#FF0000",- timeline: {- startFrame: 0,- segments: [{ startSec: 0, endSec: 1 }],- },- },- children: [],- };-- console.log("Rendering frame...");- const buffer = await renderer.renderImage(virtualMedia as any, 5, 24);- await fs.writeFile("test_red_out.png", buffer);- console.log("Done test_red_out.png, size:", buffer.length);- expect(buffer.length).toBeGreaterThan(8159);- });-});--describe("Job Processor API", () => {- it("should manage job status properly", () => {- expect(hasActiveJobs()).toBe(false);-- jobs.set("test-job", {- id: "test-job",- status: "IN_QUEUE",- });-- expect(hasActiveJobs()).toBe(true);-- jobs.set("test-job", {- id: "test-job",- status: "COMPLETED",- });-- expect(hasActiveJobs()).toBe(false);- });-});diff --git a/apps/renderer-service/src/test-helpers.ts b/apps/renderer-service/src/test-helpers.tsnew file mode 100644index 00000000..a133a76b--- /dev/null+++ b/apps/renderer-service/src/test-helpers.ts@@ -0,0 +1,30 @@+import { buildConfig, type RendererConfig } from "./config/schema.js";++/** A valid config for tests; `env` entries override the defaults. */+export function testConfig(env: Record<string, string> = {}): RendererConfig {+ const result = buildConfig(+ {+ FF_MODE: "http",+ FF_HTTP_INSECURE: "true",+ FF_REPO_ALLOWLIST: "github.com/acme/*",+ FF_CACHE_DIR: "/tmp/ff-test-cache",+ FF_WORK_DIR: "/tmp/ff-test-work",+ ...env,+ },+ { exists: () => false, tmpdir: () => "/tmp", homedir: () => "/home/test" },+ );+ if (!result.ok) {+ throw new Error(`testConfig: ${JSON.stringify(result.problems)}`);+ }+ return result.config;+}++export function testJob(overrides: Record<string, unknown> = {}) {+ return {+ version: 2,+ source: { repo: "https://github.com/acme/films.git", ref: "main" },+ target: { entry: "src/film.ts", export: "buildFilm" },+ output: { type: "inline" },+ ...overrides,+ };+}diff --git a/apps/renderer-service/tsconfig.json b/apps/renderer-service/tsconfig.jsonindex f748fe69..8042c289 100644--- a/apps/renderer-service/tsconfig.json+++ b/apps/renderer-service/tsconfig.json@@ -4,8 +4,6 @@"composite": true,"target": "esnext","lib": ["esnext"],- "experimentalDecorators": true,- "emitDecoratorMetadata": true,"module": "nodenext","skipLibCheck": true,"moduleResolution": "nodenext",diff --git a/apps/renderer-service/vitest.config.ts b/apps/renderer-service/vitest.config.tsindex ecf48b64..d84aabcb 100644--- a/apps/renderer-service/vitest.config.ts+++ b/apps/renderer-service/vitest.config.ts@@ -1,17 +1,8 @@-import path from "path";+import path from "node:path";import { defineConfig } from "vitest/config";import "dotenv/config";export default defineConfig({- esbuild: {- target: "es2022",- tsconfigRaw: {- compilerOptions: {- experimentalDecorators: true,- emitDecoratorMetadata: true,- },- },- },test: {globals: true,environment: "node",diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex f349f4d3..10adf779 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -178,15 +178,9 @@ importers:apps/renderer-service:dependencies:- '@framefields/client-utils':- specifier: workspace:*- version: link:../../packages/client-utils'@framefields/core':specifier: workspace:*version: link:../../packages/core- '@framefields/media':- specifier: workspace:*- version: link:../../packages/media'@framefields/renderer':specifier: workspace:*version: link:../../packages/renderer@@ -202,12 +196,6 @@ importers:hono:specifier: ^4.13.7version: 4.13.13- inversify:- specifier: ^7.11.0- version: 7.11.0([email protected])- reflect-metadata:- specifier: ^0.2.2- version: 0.2.2zod:specifier: ^4.3.6version: 4.3.6@@ -413,28 +401,6 @@ importers:specifier: ^5.8.3version: 5.9.3- examples/28_ai_character:- dependencies:- dotenv:- specifier: ^16.4.7- version: 16.6.1- framefields:- specifier: workspace:*- version: link:../../packages/framefields- devDependencies:- '@framefields/tsconfig':- specifier: workspace:*- version: link:../../packages/tsconfig- '@types/node':- specifier: ^24.1.0- version: 24.10.14- tsx:- specifier: ^4.20.3- version: 4.21.0- typescript:- specifier: ^5.8.3- version: 5.9.3-nodes/node-apply-lut:dependencies:'@framefields/core':diff --git a/renderer-entrypoint.sh b/renderer-entrypoint.shindex 77e8d2d5..7b58145c 100755--- a/renderer-entrypoint.sh+++ b/renderer-entrypoint.sh@@ -1,10 +1,31 @@#!/bin/bash-set -e+# Starts the renderer per FF_MODE (PLAN.md §5):+# runpod-serverless (default when RUNPOD_POD_ID is set): Node server on+# localhost + the Python shim in front of it.+# http: the Node server alone.+set -euo pipefail-if command -v xvfb-run >/dev/null 2>&1; then- echo "Starting with xvfb-run..."- exec xvfb-run -a -e /dev/stderr --server-args="-screen 0 4096x2160x24" python3 apps/gatewai-renderer/runpod_worker.py+SERVICE_DIR=apps/renderer-service+MODE="${FF_MODE:-}"+if [ -z "$MODE" ]; then+ if [ -n "${RUNPOD_POD_ID:-}" ]; then MODE=runpod-serverless; else MODE=http; fi+fi+export FF_MODE="$MODE"++# Headless Vulkan/EGL normally needs no X server; FF_XVFB=1 forces one.+NODE_CMD=(node "$SERVICE_DIR/dist/src/index.js")+if [ "${FF_XVFB:-0}" = "1" ] && command -v xvfb-run >/dev/null 2>&1; then+ NODE_CMD=(xvfb-run -a -e /dev/stderr --server-args="-screen 0 4096x2160x24" "${NODE_CMD[@]}")+fi++git lfs install --skip-repo >/dev/null++if [ "$MODE" = "runpod-serverless" ]; then+ "${NODE_CMD[@]}" &+ NODE_PID=$!+ # If either process exits, stop the other so RunPod replaces the worker.+ trap 'kill -TERM "$NODE_PID" 2>/dev/null || true' EXIT TERM INT+ python3 "$SERVICE_DIR/runpod_worker.py"else- echo "xvfb-run not found, starting directly..."- exec python3 apps/gatewai-renderer/runpod_worker.py+ exec "${NODE_CMD[@]}"fi