Code-first video & image, rendered natively on WebGPU.
images
okanasld48a54998cfb
diff --git a/apps/renderer-service/PLAN.md b/apps/renderer-service/PLAN.mdindex d2adc1be..ec2af525 100644--- a/apps/renderer-service/PLAN.md+++ b/apps/renderer-service/PLAN.md@@ -277,12 +277,27 @@ Why the files are split this way:### 6.4 Cache eviction-`FF_CACHE_MAX_GB` and `FF_WORK_MAX_GB` set the size limits (0 = no limit). Eviction runs between jobs, at most every 5 minutes, and again whenever the LFS preflight needs space. Mirrors and workspaces record their last use in a `.ff-last-used` marker.--- In `FF_WORK_DIR`: the least recently used workspaces are evicted. Failed-job dirs kept by `FF_KEEP_FAILED_JOBS` are deleted after a day.-- In `FF_CACHE_DIR`, in this order: LFS objects by last access, then git mirrors by last use, then whole package caches (largest first). Pinned vision models are never evicted.-- The next job's own workspace and mirror are never evicted.-- Another worker's mirror on a shared volume is only evicted once it has been idle for over an hour.+Free disk space decides (`src/worker/janitor.ts`); there is no fixed size budget to configure.++- **When it runs:**+ - Before every job, `statfs` checks the disks of `FF_WORK_DIR` and `FF_CACHE_DIR`. This is cheap, so it never walks the trees.+ - A disk with less free space than max(`FF_DISK_MIN_FREE_PCT` (15), `FF_DISK_MIN_FREE_GB` (10)) triggers eviction, which runs until `FF_DISK_TARGET_FREE_PCT` (25) is free.+ - Eviction also runs at least every 5 minutes, and whenever the LFS preflight or an install needs room for a known number of bytes (`enforce(protect, extraBytes)`).+ - The GB floor is capped at half the disk, so small disks still work.+- **Order:**+ 1. On the work disk: the least recently used workspaces go first. Each takes along the shared module files only it linked (§7.1). Then a sweep removes store files nothing links to.+ 2. On the cache disk: LFS objects by last access, then git mirrors by last use, then whole package caches (largest first).+ 3. When both dirs are on the same disk (same `st_dev`), all of the above run in that order against one free-space figure.+ 4. Free space is re-measured after each removal, so hard links and shared volumes are accounted for exactly.+- **Never evicted:**+ - the next job's own workspace and mirror;+ - the workspace the bound engine was loaded from (always protected, also on the emergency paths);+ - pinned vision models;+ - another worker's mirror on a shared volume, until it has been idle for over an hour.+- **Failed-job dirs** kept by `FF_KEEP_FAILED_JOBS` are deleted after a day.+- **Optional hard caps** `FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` (default 0 = off) still apply on top. They walk the trees, counting each hard-linked file once.+- **Container disks:** inside Docker, `statfs` on the overlay root may report the host's disk rather than the container's allowance, depending on the storage driver's quota support. Where that happens (to be checked on RunPod), set `FF_WORK_MAX_GB` below the container disk size.+- Every eviction is logged ("Cache eviction") with the paths and the disks' free space.### 6.5 Git hosts@@ -323,9 +338,36 @@ Rules that apply to every host:- The janitor never evicts the workspace the bound engine was loaded from.- **Dependencies stay cached by path** for the life of the process, and they are shared with the engine (one copy of `mediabunny` and similar). A new install in a workspace this process already loaded modules from can't take effect in-process. So after installing, the job fails with `ENGINE_SWITCH` and the worker recycles; the retry gets a fresh process, and the install is already done.- **Older engines without worker hooks.** Some framefields versions predate `resetRenderCaches`, `signal` and `onProgress` (2.1.3, the current release, among them). They still render, but stall detection is off, cancellation relies on the grace period, and the worker recycles after each job rather than risk stale path-keyed caches. The warning is `ENGINE_NO_WORKER_HOOKS`.+- **Shared node_modules (§7.1)** — see below.- **npm runs as shipped with Node** unless the project pins `packageManager`. Corepack's unpinned npm can be a different major than the one that wrote the lockfile, which made `npm ci` fail on lockfile sync. pnpm and yarn always go through corepack.- `engines.node` mismatches only produce a warning.+### 7.1 Shared node_modules (`src/install/shared-modules.ts`)++Projects on one worker mostly install the same packages, and framefields with its native binaries is ~800 MB unpacked. npm and yarn extract a full copy per project. pnpm copies too when its store is on another disk (the network volume), because hard links can't cross filesystems. With `FF_SHARED_MODULES=true` (default):++- **Sharing:**+ - After each install, every regular file under any `node_modules` in the slot is hashed (SHA-256, plus the exec bit).+ - Each file is hard-linked to one copy in `<FF_WORK_DIR>/store/v1/<ab>/<hash>`, on the same disk as the workspaces. The swap is atomic: link to a temp name, then rename.+ - Symlinks, empty files and project sources are left alone. Files already linked to each other are hashed once.+ - `EMLINK` keeps the file unshared. If the store is on a different disk, nothing is shared.+- **Read-only.** Shared files are `0444` (`0555` if executable): a write through a hard link would change every project.+ - Installs never write into them: before a reinstall, the installer deletes the slot's `node_modules`.+ - Repo code that writes into `node_modules` while rendering gets `EACCES`, which is classified with a hint. `FF_SHARED_MODULES=false` opts out.+ - This relies on the worker not running as root (the image runs as `framefields`).+- **Bookkeeping:**+ - Each slot lists the keys it links in `ws/<repo>/<slot>.modules`, outside the git worktree so `git clean` keeps it.+ - A reinstall releases keys the new install no longer uses.+ - Evicting a workspace releases its keys. A store file is deleted once its link count drops to 1 (only the store).+ - A sweep removes any leftovers.+- **Install headroom.** Sharing only happens after the install, so an install still needs room for a full extraction.+ - The size each install wrote is recorded per lockfile stamp (`store/install-sizes.json`, last 20).+ - Before installing, the janitor makes room for this lockfile's last size, or the largest seen.+- **Measured** (arm64 container, a project on framefields from npm, 7.8k files):+ - The first project takes 793 MB, of which 48 MB are duplicates within the install.+ - Each further project on the same lockfile adds about 4 MB, since 829 MB is linked.+ - Three projects on a 1.7 GB disk completed; before the third, the janitor evicted the least recently used workspace that wasn't holding the engine.+## 8. Render execution (in-process)### 8.1 Loading the entry@@ -492,7 +534,7 @@ On boot the service validates **all** keys and fails with the complete list of p| Git auth & hosts | `GIT_TOKEN`, `GIT_USERNAME`, `FF_GIT_CREDENTIALS` (`{host: token \| {username, token}}`), `FF_GIT_HOSTS` (`{host: github\|gitlab\|bitbucket\|gitea\|generic}`), `FF_GIT_CA_BUNDLE`, `GIT_SSH_KEY`, `GIT_SSH_KNOWN_HOSTS`, `FF_SSH_ACCEPT_NEW` (false), `FF_ALLOW_JOB_CREDENTIALS` (false) || Install | `FF_INSTALL_SCRIPTS` (true), `FF_ALLOW_UNLOCKED_INSTALL` (false), `FF_NPMRC` || Storage | `STORAGE_DRIVER` (`s3`\|`local`\|`none`), `STORAGE_ENDPOINT`, `STORAGE_REGION` (`auto`), `STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY`, `STORAGE_BUCKET`, `STORAGE_PUBLIC_BASE_URL`, `STORAGE_FORCE_PATH_STYLE` (MinIO), `FF_OUTPUT_BUCKETS`, `FF_OUTPUT_DIR`, `FF_INLINE_MAX_BYTES`, `FF_OUTPUT_MAX_BYTES`. The `R2_*` names are gone, with no aliases. |-| Paths & cache | `FF_CACHE_DIR`, `FF_WORK_DIR`, `FF_CACHE_MAX_GB` (50), `FF_WORK_MAX_GB` (40), `FRAMEFIELDS_MODELS_DIR` (`$FF_CACHE_DIR/models`), `FRAMEFIELDS_FONTS_DIR` |+| Paths & cache | `FF_CACHE_DIR`, `FF_WORK_DIR`, `FF_DISK_MIN_FREE_PCT` (15), `FF_DISK_MIN_FREE_GB` (10), `FF_DISK_TARGET_FREE_PCT` (25), `FF_SHARED_MODULES` (true), optional caps `FF_CACHE_MAX_GB` / `FF_WORK_MAX_GB` (0 = off), `FRAMEFIELDS_MODELS_DIR` (`$FF_CACHE_DIR/models`), `FRAMEFIELDS_FONTS_DIR` || Limits | `FF_QUEUE_MAX` (8), `FF_JOB_TIMEOUT_MAX_SEC` (7200), `FF_TIMEOUT_{GIT,LFS,INSTALL,LOAD,RENDER}_SEC`, `FF_STALL_TIMEOUT_SEC` (300), `FF_KILL_GRACE_SEC` (20), `FF_JOB_TTL_SEC` (3600), `FF_JOB_HISTORY_MAX` (500) || Recycling | `FF_MAX_JOBS_PER_WORKER` (25; 0 = unlimited), `FF_RECYCLE_HEAP_MB` (4096), `FF_RECYCLE_RSS_MB` (0 = off), `FF_EVENT_LOOP_BLOCK_SEC` (120), `FF_STRAY_ERROR_LIMIT` (3), `FF_LEAKED_HANDLE_LIMIT` (50). `NODE_OPTIONS=--max-old-space-size` is set by the image. || GPU | `FF_GPU` (`auto`\|`require`\|`cpu`; default `auto`, RunPod templates `require`). The engine reads `FRAMEFIELDS_HW_CODECS=off` to force software video encoding. |diff --git a/apps/renderer-service/README.md b/apps/renderer-service/README.mdindex ff8ebbd6..8d713c8d 100644--- a/apps/renderer-service/README.md+++ b/apps/renderer-service/README.md@@ -16,6 +16,7 @@ It runs as a **RunPod serverless endpoint**, a **RunPod pod**, or a **Docker con- [Quick start (Docker + MinIO)](#quick-start-docker--minio)- [RunPod](#runpod)- [GPU and CPU rendering](#gpu-and-cpu-rendering)+- [Disk and caches](#disk-and-caches)- [Jobs](#jobs)- [Configuration](#configuration)- [Errors](#errors)@@ -26,7 +27,7 @@ It runs as a **RunPod serverless endpoint**, a **RunPod pod**, or a **Docker con```job ─▶ resolve ref → fetch (mirror cache) → checkout → git LFS pull- → install (skipped when only framefields is needed, or the lockfile is unchanged)+ → install (skipped when the lockfile is unchanged; files shared across projects)→ import your entry → render on the warm GPU → verify (ffprobe) → upload```@@ -34,6 +35,8 @@ job ─▶ resolve ref → fetch (mirror cache) → checkout → git LFS pull- **Your code runs inside the worker process.** The first job loads its project's framefields and initialises the GPU. Later jobs on the same version reuse that warm engine, even from other repos. A job that needs a different version fails with retryable `ENGINE_SWITCH`, and the worker restarts so the retry loads it. Keep a worker (or endpoint) per framefields version if your projects pin different ones.- **Everything is pinned to one commit.** A branch is resolved once per job, and the result reports the SHA it rendered.- **Caches survive between jobs:** git mirrors, LFS objects and package stores live on the cache volume, and checkouts and `node_modules` are reused. A warm job spends almost all its time rendering.+- **Projects share their packages.** Identical `node_modules` files are stored once per worker, as read-only hard links. A second project on the same framefields version adds a few MB of disk instead of about 800 MB.+- **Disk is managed by free space.** Before each job the worker checks how full its disks are, and evicts the least recently used workspaces and caches when free space runs low. There are no size budgets to tune.- **The worker protects itself.** It recycles after N jobs, high memory, GPU device loss, or code that won't stop. If it crashes, the job is still reported after the restart.## Quick start (Docker + MinIO)@@ -154,6 +157,30 @@ Older framefields versions always try the hardware encoder. That fails on GPUs wHere `renderer` is `gpu` or `cpu`. `hardware` is `null` when the machine has no GPU.+## Disk and caches++What the worker keeps on disk:++| Where | What | Disk |+|---|---|---|+| `FF_CACHE_DIR` | Git mirrors, LFS objects, npm/pnpm/yarn download caches | The network volume on RunPod (`/runpod-volume`), shared by workers |+| `FF_WORK_DIR` | One checkout per repo, with its `node_modules`, and the shared module store | Local container disk |++**Shared `node_modules`.** After each install, every `node_modules` file is linked to one copy in a content store on the workspace disk. Projects on the same framefields version share almost all their files: in a test, the first project took 793 MB and each further one about 4 MB.++Shared files are read-only, because changing one would change it for every project:+- Installs are unaffected: the worker deletes `node_modules` before reinstalling.+- Code that writes into `node_modules` while rendering fails with a hint. Write to a temp dir instead, or set `FF_SHARED_MODULES=false`.+- An install still needs room for one full extraction (about 800 MB for framefields), because sharing happens afterwards. The worker makes room for it first.++**Eviction by free space.** Before each job the worker checks both disks:+- **When:** if one has less free space than max(`FF_DISK_MIN_FREE_PCT`, `FF_DISK_MIN_FREE_GB`) (15%, 10 GB), it evicts until `FF_DISK_TARGET_FREE_PCT` (25%) is free. The GB floor never exceeds half of a small disk.+- **Order:** least recently used workspaces (with the shared files only they used), then LFS objects, then git mirrors, then whole package caches.+- **Never evicted:** the current repo, the workspace the loaded engine came from, vision models, and other workers' recently used mirrors.+- **Logging:** every eviction is logged with the disks' free space.++`FF_WORK_MAX_GB` / `FF_CACHE_MAX_GB` add hard caps if you want them. They're only needed where the container reports the host's disk size instead of its own allowance. Some Docker storage drivers do that, and the RunPod smoke test will show whether RunPod does.+## Jobs```jsonc@@ -260,7 +287,10 @@ Everything is set by environment variables. The worker checks them all at startu| `FF_GIT_CA_BUNDLE` | A PEM CA certificate for self-hosted servers with a private CA. || `STORAGE_ENDPOINT`, `STORAGE_BUCKET`, `STORAGE_ACCESS_KEY_ID`, `STORAGE_SECRET_ACCESS_KEY` | S3-compatible output storage. || `STORAGE_PUBLIC_BASE_URL` | Turns output keys into public URLs in results. |-| `FF_CACHE_DIR`, `FF_CACHE_MAX_GB`, `FF_WORK_MAX_GB` | Cache location and size limits. Eviction runs automatically. |+| `FF_CACHE_DIR`, `FF_WORK_DIR` | Where caches (git mirrors, LFS, package downloads) and workspaces live. |+| `FF_DISK_MIN_FREE_PCT`, `FF_DISK_MIN_FREE_GB`, `FF_DISK_TARGET_FREE_PCT` | Eviction starts when a disk has less than max(15%, 10 GB) free, and frees up to 25%. See [Disk and caches](#disk-and-caches). |+| `FF_SHARED_MODULES` | `true` (default): identical `node_modules` files are shared between projects as read-only hard links. |+| `FF_CACHE_MAX_GB`, `FF_WORK_MAX_GB` | Optional hard caps on top of the free-space rule (default off). || `FF_JOB_TIMEOUT_MAX_SEC`, `FF_STALL_TIMEOUT_SEC` | Time limits. || `FF_MAX_JOBS_PER_WORKER` | Recycle after N jobs (default 25). || `FF_GPU` | `auto` (default), `require` or `cpu`. See [GPU and CPU rendering](#gpu-and-cpu-rendering). |diff --git a/apps/renderer-service/deploy/env.example b/apps/renderer-service/deploy/env.exampleindex 8fd170e2..f35cb811 100644--- a/apps/renderer-service/deploy/env.example+++ b/apps/renderer-service/deploy/env.example@@ -4,4 +4,3 @@ FF_REPO_ALLOWLIST=github.com/your-org/*GIT_TOKEN=MINIO_ROOT_USER=framefieldsMINIO_ROOT_PASSWORD=change-me-too-FF_CACHE_MAX_GB=50diff --git a/apps/renderer-service/deploy/runpod/pod.template.json b/apps/renderer-service/deploy/runpod/pod.template.jsonindex 2daa23ea..c900f2b7 100644--- a/apps/renderer-service/deploy/runpod/pod.template.json+++ b/apps/renderer-service/deploy/runpod/pod.template.json@@ -26,7 +26,6 @@"key": "STORAGE_SECRET_ACCESS_KEY","value": "{{ RUNPOD_SECRET_ff_storage_secret }}"},- { "key": "FF_CACHE_MAX_GB", "value": "80" },{ "key": "LOG_LEVEL", "value": "info" }]}diff --git a/apps/renderer-service/deploy/runpod/serverless.template.json b/apps/renderer-service/deploy/runpod/serverless.template.jsonindex 0cfa4e9d..78cd1f70 100644--- a/apps/renderer-service/deploy/runpod/serverless.template.json+++ b/apps/renderer-service/deploy/runpod/serverless.template.json@@ -26,8 +26,6 @@"value": "{{ RUNPOD_SECRET_ff_storage_secret }}"},{ "key": "STORAGE_PUBLIC_BASE_URL", "value": "" },- { "key": "FF_CACHE_MAX_GB", "value": "80" },- { "key": "FF_WORK_MAX_GB", "value": "30" },{ "key": "FF_JOB_TIMEOUT_MAX_SEC", "value": "3600" },{ "key": "LOG_LEVEL", "value": "info" }]diff --git a/apps/renderer-service/src/config/schema.ts b/apps/renderer-service/src/config/schema.tsindex 8b1478cf..02709f21 100644--- a/apps/renderer-service/src/config/schema.ts+++ b/apps/renderer-service/src/config/schema.ts@@ -109,8 +109,16 @@ export const envSchema = z.object({// Paths & cacheFF_CACHE_DIR: z.string().optional(),FF_WORK_DIR: z.string().optional(),- FF_CACHE_MAX_GB: int(50),- FF_WORK_MAX_GB: int(40),+ // Eviction keeps this much of each disk free (the larger of the two)…+ FF_DISK_MIN_FREE_PCT: int(15),+ FF_DISK_MIN_FREE_GB: int(10),+ // …and, once triggered, frees up to this much.+ FF_DISK_TARGET_FREE_PCT: int(25),+ // Optional hard caps; 0 = off (free space alone decides).+ FF_CACHE_MAX_GB: int(0),+ FF_WORK_MAX_GB: int(0),+ // Hard-link identical node_modules files across workspaces.+ FF_SHARED_MODULES: bool(true),// LimitsFF_QUEUE_MAX: int(8),@@ -220,6 +228,19 @@ export function buildConfig(});}+ if (e.FF_DISK_MIN_FREE_PCT > 50) {+ problems.push({ key: "FF_DISK_MIN_FREE_PCT", message: "must be 0–50" });+ }+ if (+ e.FF_DISK_TARGET_FREE_PCT > 60 ||+ e.FF_DISK_TARGET_FREE_PCT < e.FF_DISK_MIN_FREE_PCT+ ) {+ problems.push({+ key: "FF_DISK_TARGET_FREE_PCT",+ message: "must be between FF_DISK_MIN_FREE_PCT and 60",+ });+ }+const allowlist = [...e.FF_REPO_ALLOWLIST];if (e.FF_DEFAULT_REPO) {try {@@ -334,6 +355,10 @@ export function buildConfig(workDir,cacheMaxGb: e.FF_CACHE_MAX_GB,workMaxGb: e.FF_WORK_MAX_GB,+ minFreePct: e.FF_DISK_MIN_FREE_PCT,+ minFreeGb: e.FF_DISK_MIN_FREE_GB,+ targetFreePct: e.FF_DISK_TARGET_FREE_PCT,+ sharedModules: e.FF_SHARED_MODULES,},limits: {queueMax: e.FF_QUEUE_MAX,diff --git a/apps/renderer-service/src/git/checkout.ts b/apps/renderer-service/src/git/checkout.tsindex 40c719b1..6ee460c6 100644--- a/apps/renderer-service/src/git/checkout.ts+++ b/apps/renderer-service/src/git/checkout.ts@@ -80,8 +80,8 @@ export const checkout: Pipeline["checkout"] = async (job, ctx) => {include,exclude: source.lfs.exclude,scope: include.length ? [] : scope,- makeRoom: async () => {- await activeJanitor()?.enforce([repoHash(repo)]);+ makeRoom: async (bytes) => {+ await activeJanitor()?.enforce([repoHash(repo)], bytes);},});const leftover = await findPointers(ws.root, source.workdir);diff --git a/apps/renderer-service/src/git/lfs.ts b/apps/renderer-service/src/git/lfs.tsindex 3c8d7399..e1b4bf86 100644--- a/apps/renderer-service/src/git/lfs.ts+++ b/apps/renderer-service/src/git/lfs.ts@@ -14,7 +14,8 @@ export interface LfsOptions {/** Directories that must be fully usable (workdir, sparse paths). */scope: string[];/** Frees disk (cache eviction) when the preflight finds too little. */- makeRoom?: () => Promise<void>;+ /** Evicts caches so `bytes` more fit (the janitor). */+ makeRoom?: (bytes: number) => Promise<void>;}export interface LfsResult {@@ -129,7 +130,7 @@ export async function pullLfs(!options.makeRoom)throw err;- await options.makeRoom();+ await options.makeRoom(Math.max(needed, total) + DISK_MARGIN_BYTES);await fits();}diff --git a/apps/renderer-service/src/install/installer.ts b/apps/renderer-service/src/install/installer.tsindex 33bb0835..99b9e406 100644--- a/apps/renderer-service/src/install/installer.ts+++ b/apps/renderer-service/src/install/installer.ts@@ -8,15 +8,24 @@ import type { RendererConfig } from "../config/index.js";import { RenderError } from "../errors.js";import { CommandError, exec } from "../git/exec.js";import type { Pipeline, StepContext } from "../jobs/pipeline.js";+import { rendererLogger } from "../logger.js";import { redact } from "../redact.js";import { depsLoadedFrom } from "../render/engine.js";import { workerHealth } from "../worker/health.js";+import { activeJanitor } from "../worker/janitor.js";import {declaredEngineRange,inspectProject,type ProjectInfo,readPackageJson,} from "./project.js";+import {+ expectedInstallSize,+ moduleStoreDir,+ recordInstallSize,+ removeModules,+ shareModules,+} from "./shared-modules.js";export const STAMP_FILE = ".ff-install-stamp";@@ -84,7 +93,39 @@ export const install: Pipeline["install"] = async (checkout, ctx) => {// A half-finished install must never be trusted by the next job.await fs.rm(stampPath, { force: true });+ const shared = ctx.config.paths.sharedModules;+ // Shared files are read-only hard links: a package manager must never+ // write into them, so the install starts from an empty node_modules.+ if (shared) await removeModules(checkout.root);+ // Room for a full extraction: sharing happens only after the install.+ const expected = await expectedInstallSize(ctx.config.paths.workDir, stamp);+ if (expected > 0) {+ await activeJanitor()?.enforce(+ [path.basename(path.dirname(checkout.root))],+ expected,+ );+ }await runInstall(project, ctx, checkout.root);+ if (shared) {+ const result = await shareModules(+ checkout.root,+ moduleStoreDir(ctx.config.paths.workDir),+ );+ await recordInstallSize(+ ctx.config.paths.workDir,+ stamp,+ result.installBytes,+ );+ rendererLogger.info(+ {+ files: result.files,+ linked: result.linked,+ savedMb: Math.round(result.savedBytes / 1024 ** 2),+ skipped: result.skipped,+ },+ "Shared node_modules with other workspaces",+ );+ }await fs.writeFile(stampPath, stamp);// This process holds the old dependencies in its module cache; only a@@ -354,6 +395,7 @@ export function classifyInstallError(if (/ENOSPC|No space left on device/i.test(output)) {return new RenderError("DISK_FULL", "Ran out of disk during the install.", {phase: "install",+ hint: "The install extracts every package before sharing them; give FF_WORK_DIR more disk (containerDiskInGb on RunPod).",});}if (diff --git a/apps/renderer-service/src/install/shared-modules.test.ts b/apps/renderer-service/src/install/shared-modules.test.tsnew file mode 100644index 00000000..15410cd3--- /dev/null+++ b/apps/renderer-service/src/install/shared-modules.test.ts@@ -0,0 +1,172 @@+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { afterAll, beforeAll, describe, expect, it } from "vitest";+import {+ expectedInstallSize,+ manifestPath,+ readManifests,+ recordInstallSize,+ releaseKeys,+ removeModules,+ shareModules,+ sweepStore,+} from "./shared-modules.js";++let tmp: string;+let store: string;++beforeAll(async () => {+ tmp = await fs.mkdtemp(path.join(os.tmpdir(), "ff-shared-"));+ store = path.join(tmp, "work/store/v1");+});+afterAll(async () => {+ // Shared files are read-only; their dirs aren't, so rm still works.+ await fs.rm(tmp, { recursive: true, force: true });+});++const BIG = Buffer.alloc(2 * 1024 * 1024, 7);++/** A workspace slot as an install leaves it. */+async function slot(repo: string, extra: Record<string, string | Buffer> = {}) {+ const root = path.join(tmp, "work/ws", repo, "0");+ const files: Record<string, string | Buffer> = {+ "src/film.ts": "export {}",+ "node_modules/framefields/dist/index.mjs": "export const v = 1;",+ "node_modules/framefields/package.json": '{"name":"framefields"}',+ "node_modules/webgpu/dawn.node": BIG,+ "node_modules/.bin/tool.js": "#!/usr/bin/env node",+ "node_modules/empty.js": "",+ "packages/app/node_modules/left-pad/index.js": "module.exports = 1;",+ ...extra,+ };+ for (const [rel, content] of Object.entries(files)) {+ const p = path.join(root, rel);+ await fs.mkdir(path.dirname(p), { recursive: true });+ await fs.writeFile(p, content);+ }+ await fs.chmod(path.join(root, "node_modules/.bin/tool.js"), 0o755);+ await fs.symlink(+ "../framefields/dist/index.mjs",+ path.join(root, "node_modules/.bin/framefields"),+ );+ return root;+}++const ino = async (p: string) => (await fs.lstat(p)).ino;++describe("shareModules", () => {+ let a: string;+ let b: string;++ it("stores the first project's files once", async () => {+ a = await slot("repo-a");+ const result = await shareModules(a, store);+ expect(result.skipped).toBeUndefined();+ expect(result.files).toBe(5); // the empty file and the symlink are left alone+ expect(result.linked).toBe(0);+ const manifest = (await fs.readFile(manifestPath(a), "utf8")).split("\n");+ expect(manifest).toHaveLength(5);+ expect(manifest.filter((k) => k.endsWith("x"))).toHaveLength(1);+ });++ it("links a second project's identical files to the same copy", async () => {+ b = await slot("repo-b", { "node_modules/only-b.js": "b" });+ const result = await shareModules(b, store);+ expect(result.linked).toBe(5);+ expect(result.savedBytes).toBeGreaterThanOrEqual(BIG.length);++ for (const rel of [+ "node_modules/webgpu/dawn.node",+ "node_modules/framefields/dist/index.mjs",+ "packages/app/node_modules/left-pad/index.js",+ ]) {+ expect(await ino(path.join(b, rel))).toBe(await ino(path.join(a, rel)));+ }+ const dawn = await fs.stat(path.join(b, "node_modules/webgpu/dawn.node"));+ expect(dawn.nlink).toBe(3); // a, b and the store+ expect(dawn.mode & 0o222).toBe(0); // read-only+ const tool = await fs.stat(path.join(b, "node_modules/.bin/tool.js"));+ expect(tool.mode & 0o111).not.toBe(0); // still executable++ // Project sources and symlinks are untouched.+ expect((await fs.stat(path.join(b, "src/film.ts"))).nlink).toBe(1);+ expect(+ (+ await fs.lstat(path.join(b, "node_modules/.bin/framefields"))+ ).isSymbolicLink(),+ ).toBe(true);+ expect(+ await fs.readFile(path.join(b, "node_modules/only-b.js"), "utf8"),+ ).toBe("b");+ });++ it("is a no-op on files that are already shared", async () => {+ const again = await shareModules(b, store);+ expect(again.linked).toBe(0);+ expect(again.files).toBe(6);+ });++ it("refuses in-place writes to shared files", async () => {+ await expect(+ fs.writeFile(path.join(a, "node_modules/framefields/package.json"), "{}"),+ ).rejects.toMatchObject({ code: "EACCES" });+ });++ it("drops store files once no workspace links them", async () => {+ // b reinstalls with a different framefields build.+ await removeModules(b);+ await fs.mkdir(path.join(b, "node_modules/framefields/dist"), {+ recursive: true,+ });+ await fs.writeFile(+ path.join(b, "node_modules/framefields/dist/index.mjs"),+ "export const v = 2;",+ );+ await shareModules(b, store);+ expect(await fs.readdir(path.join(b, "packages/app"))).toEqual([]);++ // The store holds exactly what a and b still link: only-b.js was+ // released; the old framefields build stays because a links it.+ const keysA = await readManifests(path.dirname(a));+ const keysB = await readManifests(path.dirname(b));+ const storeFiles = async () =>+ (await fs.readdir(store, { recursive: true })).filter(+ (f) => f.length > 3,+ );+ expect((await storeFiles()).length).toBe(+ new Set([...keysA, ...keysB]).size,+ );++ // Evicting a: its files used by nobody else go too.+ await fs.rm(path.dirname(a), { recursive: true, force: true });+ await releaseKeys(store, keysA);+ expect((await storeFiles()).length).toBe(keysB.size);+ await expect(+ fs.stat(path.join(b, "node_modules/framefields/dist/index.mjs")),+ ).resolves.toBeTruthy();+ });++ it("sweeps store files nothing links to", async () => {+ await fs.rm(path.dirname(b), { recursive: true, force: true });+ const freed = await sweepStore(store);+ expect(freed).toBeGreaterThan(0);+ const left = (await fs.readdir(store, { recursive: true })).filter(+ (f) => f.length > 3,+ );+ expect(left).toEqual([]);+ });+});++describe("install sizes", () => {+ it("expects this lockfile's last size, else the largest seen", async () => {+ const work = path.join(tmp, "sizes");+ expect(await expectedInstallSize(work, "x")).toBe(0);+ await recordInstallSize(work, "lock-a", 800);+ await recordInstallSize(work, "lock-b", 300);+ expect(await expectedInstallSize(work, "lock-b")).toBe(300);+ expect(await expectedInstallSize(work, "lock-new")).toBe(800);+ await recordInstallSize(work, "lock-a", 500);+ expect(await expectedInstallSize(work, "lock-a")).toBe(500);+ });+});diff --git a/apps/renderer-service/src/install/shared-modules.ts b/apps/renderer-service/src/install/shared-modules.tsnew file mode 100644index 00000000..3172951c--- /dev/null+++ b/apps/renderer-service/src/install/shared-modules.ts@@ -0,0 +1,298 @@+import { createHash, randomBytes } from "node:crypto";+import { createReadStream } from "node:fs";+import fs from "node:fs/promises";+import path from "node:path";+import { rendererLogger } from "../logger.js";++/**+ * Shared node_modules (PLAN.md §7.1). Projects on one worker mostly install+ * the same packages (framefields and its native binaries are hundreds of+ * MB), but npm and yarn extract a full copy per project, and pnpm copies too+ * when its store is on another disk (the network volume). After each install+ * every node_modules file is hashed and hard-linked to one copy in a content+ * store on the workspace disk, so each distinct file is stored once.+ *+ * Shared files are made read-only: a hard link writes through to every+ * project, so an in-place write must fail rather than change other projects.+ * Installs never write in place here (installer.ts removes node_modules+ * before reinstalling), and deleting a link only drops that project's copy.+ *+ * Each workspace slot lists the store keys it links in a manifest next to the+ * slot (outside the git worktree, so `git clean` keeps it). The janitor uses+ * it to drop store files nothing links to any more (nlink 1).+ */++/** Store root, on the same disk as the workspaces (hard links can't cross disks). */+export function moduleStoreDir(workDir: string): string {+ return path.join(workDir, "store", "v1");+}++/** Manifest of store keys a workspace slot links: ws/<repo>/<slot>.modules. */+export function manifestPath(slotRoot: string): string {+ return `${slotRoot}.modules`;+}++interface ShareResult {+ files: number;+ /** Size of the files as the install wrote them (before sharing). */+ installBytes: number;+ /** Files now linked to a copy that already existed (bytes saved). */+ linked: number;+ savedBytes: number;+ /** Why nothing was shared, if so. */+ skipped?: string;+}++const PARALLEL = 32;+const SMALL_FILE = 1024 * 1024;++async function hashFile(file: string, size: number): Promise<string> {+ const hash = createHash("sha256");+ if (size <= SMALL_FILE) {+ hash.update(await fs.readFile(file));+ } else {+ for await (const chunk of createReadStream(file)) hash.update(chunk);+ }+ return hash.digest("hex");+}++/** Regular files under every node_modules dir in `root` (symlinks skipped). */+async function* moduleFiles(+ dir: string,+ inModules = false,+): AsyncGenerator<string> {+ const entries = await fs+ .readdir(dir, { withFileTypes: true })+ .catch(() => []);+ for (const e of entries) {+ const p = path.join(dir, e.name);+ if (e.isDirectory()) {+ if (!inModules && e.name === ".git") continue;+ yield* moduleFiles(p, inModules || e.name === "node_modules");+ } else if (inModules && e.isFile()) {+ yield p;+ }+ }+}++/** Removes every node_modules dir under `root`, before a reinstall. */+export async function removeModules(root: string): Promise<void> {+ const walk = async (dir: string): Promise<void> => {+ const entries = await fs+ .readdir(dir, { withFileTypes: true })+ .catch(() => []);+ await Promise.all(+ entries+ .filter((e) => e.isDirectory() && e.name !== ".git")+ .map((e) => {+ const p = path.join(dir, e.name);+ return e.name === "node_modules"+ ? fs.rm(p, { recursive: true, force: true })+ : walk(p);+ }),+ );+ };+ await walk(root);+}++/** Store path for a key: <store>/ab/<key>. */+function storePath(store: string, key: string): string {+ return path.join(store, key.slice(0, 2), key);+}++async function readManifest(file: string): Promise<Set<string>> {+ const text = await fs.readFile(file, "utf8").catch(() => "");+ return new Set(text.split("\n").filter(Boolean));+}++/** Deletes store files for `keys` that no workspace links any more. */+export async function releaseKeys(+ store: string,+ keys: Iterable<string>,+): Promise<number> {+ let freed = 0;+ for (const key of keys) {+ const file = storePath(store, key);+ const stat = await fs.lstat(file).catch(() => undefined);+ if (stat?.isFile() && stat.nlink <= 1) {+ await fs.rm(file, { force: true });+ freed += stat.size;+ }+ }+ return freed;+}++/** Store keys linked by every slot of a repo's workspace dir. */+export async function readManifests(repoDir: string): Promise<Set<string>> {+ const keys = new Set<string>();+ for (const name of await fs.readdir(repoDir).catch(() => [])) {+ if (!name.endsWith(".modules")) continue;+ for (const key of await readManifest(path.join(repoDir, name))) {+ keys.add(key);+ }+ }+ return keys;+}++/** Deletes every store file nothing links to. Returns bytes freed. */+export async function sweepStore(store: string): Promise<number> {+ let freed = 0;+ for (const shard of await fs.readdir(store).catch(() => [])) {+ const dir = path.join(store, shard);+ for (const name of await fs.readdir(dir).catch(() => [])) {+ const file = path.join(dir, name);+ const stat = await fs.lstat(file).catch(() => undefined);+ if (!stat?.isFile()) {+ // Leftover temp names from an interrupted share.+ if (name.includes(".tmp-")) await fs.rm(file, { force: true });+ continue;+ }+ if (stat.nlink <= 1) {+ await fs.rm(file, { force: true });+ freed += stat.size;+ }+ }+ }+ return freed;+}++/**+ * Links every node_modules file under `slotRoot` to the store, then writes+ * the slot's manifest and releases keys the previous install used but this+ * one doesn't. Never throws: a failure leaves files unshared, not broken.+ */+export async function shareModules(+ slotRoot: string,+ store: string,+): Promise<ShareResult> {+ const result: ShareResult = {+ files: 0,+ installBytes: 0,+ linked: 0,+ savedBytes: 0,+ };+ try {+ await fs.mkdir(store, { recursive: true });+ const [a, b] = await Promise.all([fs.stat(slotRoot), fs.stat(store)]);+ if (a.dev !== b.dev) {+ return { ...result, skipped: "store and workspaces on different disks" };+ }++ const keys = new Set<string>();+ // Files already hard-linked to each other are hashed once.+ const byInode = new Map<string, string>();+ const share = async (file: string): Promise<void> => {+ const stat = await fs.lstat(file);+ if (!stat.isFile() || stat.size === 0) return;+ result.files++;+ result.installBytes += stat.size;+ const inode = `${stat.dev}:${stat.ino}`;+ const exec = (stat.mode & 0o111) !== 0;+ const key =+ byInode.get(inode) ??+ (await hashFile(file, stat.size)) + (exec ? "x" : "");+ byInode.set(inode, key);+ const target = storePath(store, key);++ const existing = await fs.lstat(target).catch(() => undefined);+ if (existing?.ino === stat.ino) {+ keys.add(key);+ return;+ }+ if (existing) {+ if (existing.size !== stat.size) return; // corrupt store entry+ // Swap the file for a link to the shared copy, atomically.+ const tmp = `${file}.ff-tmp-${randomBytes(4).toString("hex")}`;+ try {+ await fs.link(target, tmp);+ } catch (err) {+ // EMLINK: the shared copy has the maximum number of links.+ if ((err as NodeJS.ErrnoException).code === "EMLINK") return;+ throw err;+ }+ await fs.rename(tmp, file);+ result.linked++;+ result.savedBytes += stat.size;+ keys.add(key);+ return;+ }+ // First copy: this file becomes the shared one.+ await fs.mkdir(path.dirname(target), { recursive: true });+ try {+ await fs.link(file, target);+ } catch (err) {+ if ((err as NodeJS.ErrnoException).code === "EEXIST") {+ return share(file); // linked meanwhile: take the other path+ }+ throw err;+ }+ await fs.chmod(target, exec ? 0o555 : 0o444);+ keys.add(key);+ };++ let batch: Promise<void>[] = [];+ for await (const file of moduleFiles(slotRoot)) {+ batch.push(share(file));+ if (batch.length >= PARALLEL) {+ await Promise.all(batch);+ batch = [];+ }+ }+ await Promise.all(batch);++ const manifest = manifestPath(slotRoot);+ const previous = await readManifest(manifest);+ await fs.writeFile(manifest, [...keys].join("\n"));+ await releaseKeys(+ store,+ [...previous].filter((k) => !keys.has(k)),+ );+ return result;+ } catch (err) {+ rendererLogger.warn(+ { err: err instanceof Error ? err.message : String(err) },+ "Sharing node_modules failed; this workspace keeps its own copies",+ );+ return { ...result, skipped: "error" };+ }+}++const SIZES_FILE = "install-sizes.json";+const SIZES_KEPT = 20;++/**+ * Remembers how much an install wrote before sharing: npm and yarn extract a+ * full copy first, so the next install needs that much free disk even when+ * most of it is shared afterwards.+ */+export async function recordInstallSize(+ workDir: string,+ stamp: string,+ bytes: number,+): Promise<void> {+ const file = path.join(workDir, "store", SIZES_FILE);+ const sizes = await readSizes(file);+ delete sizes[stamp];+ sizes[stamp] = bytes;+ const kept = Object.entries(sizes).slice(-SIZES_KEPT);+ await fs.mkdir(path.dirname(file), { recursive: true });+ await fs.writeFile(file, JSON.stringify(Object.fromEntries(kept)));+}++/** Expected install size: this lockfile's last install, else the largest seen. */+export async function expectedInstallSize(+ workDir: string,+ stamp: string,+): Promise<number> {+ const sizes = await readSizes(path.join(workDir, "store", SIZES_FILE));+ return sizes[stamp] ?? Math.max(0, ...Object.values(sizes));+}++async function readSizes(file: string): Promise<Record<string, number>> {+ try {+ const parsed = JSON.parse(await fs.readFile(file, "utf8"));+ return parsed && typeof parsed === "object" ? parsed : {};+ } catch {+ return {};+ }+}diff --git a/apps/renderer-service/src/logger.test.ts b/apps/renderer-service/src/logger.test.tsnew file mode 100644index 00000000..dd3c212a--- /dev/null+++ b/apps/renderer-service/src/logger.test.ts@@ -0,0 +1,24 @@+import { Writable } from "node:stream";+import pino from "pino";+import { describe, expect, it } from "vitest";+import { contextMixin, loggerContext } from "./logger.js";++describe("logger context", () => {+ it("doesn't carry one line's fields into the next", () => {+ const lines: Record<string, unknown>[] = [];+ const sink = new Writable({+ write(chunk, _enc, done) {+ lines.push(JSON.parse(String(chunk)));+ done();+ },+ });+ const log = pino({ mixin: contextMixin }, sink);+ loggerContext.run({ jobId: "j1" }, () => {+ log.info({ removed: 3 }, "first");+ log.info("second");+ });+ expect(lines[0]).toMatchObject({ jobId: "j1", removed: 3 });+ expect(lines[1]).toMatchObject({ jobId: "j1" });+ expect(lines[1]).not.toHaveProperty("removed");+ });+});diff --git a/apps/renderer-service/src/logger.ts b/apps/renderer-service/src/logger.tsindex 7203ea49..fecabf98 100644--- a/apps/renderer-service/src/logger.ts+++ b/apps/renderer-service/src/logger.ts@@ -24,11 +24,20 @@ const version = (() => {/** Per-job / per-request fields merged into every log line. */export const loggerContext = new AsyncLocalStorage<Record<string, unknown>>();+/**+ * Adds the context to a log line. Returns a copy: pino merges each call's+ * fields into the object mixin returns, which would otherwise leak them into+ * the job's later log lines.+ */+export function contextMixin(): Record<string, unknown> {+ return { ...loggerContext.getStore() };+}+export const logger = pino({level: process.env.LOG_LEVEL || "info",timestamp: pino.stdTimeFunctions.isoTime,base: { service: "framefields-renderer", version },- mixin: () => loggerContext.getStore() ?? {},+ mixin: contextMixin,redact: {paths: ["req.headers.authorization",diff --git a/apps/renderer-service/src/output/deliver.ts b/apps/renderer-service/src/output/deliver.tsindex 1c377856..69541612 100644--- a/apps/renderer-service/src/output/deliver.ts+++ b/apps/renderer-service/src/output/deliver.ts@@ -19,18 +19,29 @@ export const deliver: Pipeline["deliver"] = async (files, job, ctx) => {const { input, sha, ref } = job;const config = ctx.config;if (files.length === 0) {- throw new RenderError("OUTPUT_MISSING", "The render produced no files.", { phase: "verify" });+ throw new RenderError("OUTPUT_MISSING", "The render produced no files.", {+ phase: "verify",+ });}ctx.progress({ phase: "verifying" });const verified: VerifiedFile[] = [];for (const file of files) {- verified.push(await verifyFile(file, { maxBytes: config.storage.outputMaxBytes, signal: ctx.signal }));+ verified.push(+ await verifyFile(file, {+ maxBytes: config.storage.outputMaxBytes,+ signal: ctx.signal,+ }),+ );}- const failOn = typeof input.render.qa === "object" ? input.render.qa.failOn : undefined;+ const failOn =+ typeof input.render.qa === "object" ? input.render.qa.failOn : undefined;const qaIssues = verified.flatMap((f) =>- qaFailures(f.qa, failOn as string[] | undefined).map((i) => ({ output: f.name, ...i })),+ qaFailures(f.qa, failOn as string[] | undefined).map((i) => ({+ output: f.name,+ ...i,+ })),);if (qaIssues.length > 0 && config.qaBlockUpload) {throw qaError(qaIssues, []);@@ -81,7 +92,9 @@ export const deliver: Pipeline["deliver"] = async (files, job, ctx) => {case "local": {const paths = renderKeys(out.path, named, vars);for (const [i, d] of deliveries.entries()) {- delivered.push(await deliverLocal(d, config.storage.outputDir as string, paths[i]));+ delivered.push(+ await deliverLocal(d, config.storage.outputDir as string, paths[i]),+ );}break;}@@ -92,7 +105,10 @@ export const deliver: Pipeline["deliver"] = async (files, job, ctx) => {throw new RenderError("OUTPUT_TOO_LARGE",`Outputs total ${total} bytes, too large to return inline (FF_INLINE_MAX_BYTES=${config.storage.inlineMaxBytes}).`,- { phase: "upload", hint: 'Use an "s3", "presigned" or "local" output.' },+ {+ phase: "upload",+ hint: 'Use an "s3", "presigned" or "local" output.',+ },);}delivered = await Promise.all(deliveries.map(deliverInline));@@ -104,24 +120,40 @@ export const deliver: Pipeline["deliver"] = async (files, job, ctx) => {return delivered;};-function qaError(issues: Record<string, unknown>[], outputs: DeliveredFile[]): RenderError {- return new RenderError("QA_FAILED", `QA found ${issues.length} issue(s) listed in render.qa.failOn.`, {- phase: "verify",- details: {- issues: issues.slice(0, 50),- // Delivered anyway (unless FF_QA_BLOCK_UPLOAD), so callers can inspect them.- outputs: outputs.map(({ base64: _, ...rest }) => rest),+function qaError(+ issues: Record<string, unknown>[],+ outputs: DeliveredFile[],+): RenderError {+ return new RenderError(+ "QA_FAILED",+ `QA found ${issues.length} issue(s) listed in render.qa.failOn.`,+ {+ phase: "verify",+ details: {+ issues: issues.slice(0, 50),+ // Delivered anyway (unless FF_QA_BLOCK_UPLOAD), so callers can inspect them.+ outputs: outputs.map(({ base64: _, ...rest }) => rest),+ },},- });+ );}/*** Pipeline step: finds outputs an earlier attempt of the same job already* delivered for this commit (s3 + idempotencyKey only).*/-export const findExisting: NonNullable<Pipeline["findExisting"]> = async (input, sha, ctx) => {+export const findExisting: NonNullable<Pipeline["findExisting"]> = async (+ input,+ sha,+ ctx,+) => {if (!input.idempotencyKey || input.output.type !== "s3") return undefined;const bucket = input.output.bucket ?? (ctx.config.storage.bucket as string);- const manifest = await readManifest(ctx.config, bucket, input.idempotencyKey, sha);+ const manifest = await readManifest(+ ctx.config,+ bucket,+ input.idempotencyKey,+ sha,+ );return manifest?.outputs;};diff --git a/apps/renderer-service/src/output/output.test.ts b/apps/renderer-service/src/output/output.test.tsindex 144da8f0..9ea607d1 100644--- a/apps/renderer-service/src/output/output.test.ts+++ b/apps/renderer-service/src/output/output.test.ts@@ -15,7 +15,10 @@ import { qaFailures, verifyFile } from "./verify.js";let tmp: string;let server: http.Server;let base: string;-const objects = new Map<string, { body: Buffer; headers: http.IncomingHttpHeaders }>();+const objects = new Map<+ string,+ { body: Buffer; headers: http.IncomingHttpHeaders }+>();/** Scripted responses for presigned PUTs, consumed in order. */const presignedPlan: number[] = [];@@ -29,8 +32,15 @@ beforeAll(async () => {const key = decodeURIComponent(url.pathname);if (key.startsWith("/presigned/")) {const status = presignedPlan.shift() ?? 200;- if (status === 200) objects.set(key, { body: Buffer.concat(chunks), headers: req.headers });- res.writeHead(status).end(status === 403 ? "<Error><Message>Request has expired</Message></Error>" : "");+ if (status === 200)+ objects.set(key, { body: Buffer.concat(chunks), headers: req.headers });+ res+ .writeHead(status)+ .end(+ status === 403+ ? "<Error><Message>Request has expired</Message></Error>"+ : "",+ );return;}if (req.method === "PUT") {@@ -38,11 +48,18 @@ beforeAll(async () => {res.writeHead(200, { etag: '"x"' }).end();} else if (objects.has(key)) {const o = objects.get(key)!;- res.writeHead(200, { "content-length": o.body.length, "content-type": "application/octet-stream" });+ res.writeHead(200, {+ "content-length": o.body.length,+ "content-type": "application/octet-stream",+ });res.end(req.method === "HEAD" ? undefined : o.body);} else {res.writeHead(404, { "content-type": "application/xml" });- res.end(req.method === "HEAD" ? undefined : "<Error><Code>NoSuchKey</Code><Message>nope</Message></Error>");+ res.end(+ req.method === "HEAD"+ ? undefined+ : "<Error><Code>NoSuchKey</Code><Message>nope</Message></Error>",+ );}});await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));@@ -62,7 +79,11 @@ const hasFfmpeg = (() => {}})();-async function file(name: string, content: string | Buffer, ext = "bin"): Promise<RenderedFile> {+async function file(+ name: string,+ content: string | Buffer,+ ext = "bin",+): Promise<RenderedFile> {const p = path.join(tmp, `${name}.${ext}`);await fs.writeFile(p, content);return { name, path: p, ext, contentType: "application/octet-stream" };@@ -70,8 +91,27 @@ async function file(name: string, content: string | Buffer, ext = "bin"): Promisfunction mp4(name: string, frames: number): RenderedFile {const p = path.join(tmp, `${name}.mp4`);- execFileSync("ffmpeg", ["-y", "-v", "error", "-f", "lavfi", "-i", "color=c=red:s=64x64:r=24", "-frames:v", String(frames), "-pix_fmt", "yuv420p", p]);- return { name, path: p, ext: "mp4", contentType: "video/mp4", expectedFrames: frames };+ execFileSync("ffmpeg", [+ "-y",+ "-v",+ "error",+ "-f",+ "lavfi",+ "-i",+ "color=c=red:s=64x64:r=24",+ "-frames:v",+ String(frames),+ "-pix_fmt",+ "yuv420p",+ p,+ ]);+ return {+ name,+ path: p,+ ext: "mp4",+ contentType: "video/mp4",+ expectedFrames: frames,+ };}async function codeOf(promise: Promise<unknown>) {@@ -86,23 +126,44 @@ async function codeOf(promise: Promise<unknown>) {describe("verifyFile", () => {it("hashes and sizes a file", async () => {const f = await verifyFile(await file("a", "hello"), { maxBytes: 100 });- expect(f).toMatchObject({ bytes: 5, sha256: "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824" });+ expect(f).toMatchObject({+ bytes: 5,+ sha256:+ "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824",+ });});it.each([- ["missing", async () => ({ name: "x", path: path.join(tmp, "nope"), ext: "bin", contentType: "" }), "OUTPUT_MISSING"],+ [+ "missing",+ async () => ({+ name: "x",+ path: path.join(tmp, "nope"),+ ext: "bin",+ contentType: "",+ }),+ "OUTPUT_MISSING",+ ],["empty", () => file("empty", ""), "OUTPUT_INVALID"],["too large", () => file("big", "0123456789"), "OUTPUT_TOO_LARGE"],["not media", () => file("fake", "not a video", "mp4"), "OUTPUT_INVALID"],])("rejects %s output", async (_, make, code) => {- expect(await codeOf(verifyFile(await make(), { maxBytes: 5 + (code === "OUTPUT_TOO_LARGE" ? 0 : 100) }))).toBe(code);+ expect(+ await codeOf(+ verifyFile(await make(), {+ maxBytes: 5 + (code === "OUTPUT_TOO_LARGE" ? 0 : 100),+ }),+ ),+ ).toBe(code);});it.skipIf(!hasFfmpeg)("probes media and checks the frame count", async () => {const ok = await verifyFile(mp4("clip", 24), { maxBytes: 0 });expect(ok).toMatchObject({ frames: 24, durationMs: 1000 });const wrong = { ...mp4("short", 10), expectedFrames: 24 };- expect(await codeOf(verifyFile(wrong, { maxBytes: 0 }))).toBe("OUTPUT_INVALID");+ expect(await codeOf(verifyFile(wrong, { maxBytes: 0 }))).toBe(+ "OUTPUT_INVALID",+ );});it("matches QA issues by code or severity", () => {@@ -130,7 +191,12 @@ describe("deliver", () => {FF_OUTPUT_DIR: path.join(tmp, "out"),});- function run(output: unknown, files: RenderedFile[], extra: Record<string, unknown> = {}, env: Record<string, string> = {}) {+ function run(+ output: unknown,+ files: RenderedFile[],+ extra: Record<string, unknown> = {},+ env: Record<string, string> = {},+ ) {const ctx: StepContext = {jobId: "job1",config: testConfig({ ...storageEnv(), ...env }),@@ -146,13 +212,30 @@ describe("deliver", () => {output,...extra,});- return { promise: deliver(files, { input, sha: "9f2c1e4abcdef", ref: "main" }, ctx), input, ctx };+ return {+ promise: deliver(+ files,+ { input, sha: "9f2c1e4abcdef", ref: "main" },+ ctx,+ ),+ input,+ ctx,+ };}it("uploads to S3 with templated keys, metadata and public URLs", async () => {- const files = [await file("main", "AAA", "mp4b"), await file("teaser", "BB", "mp4b")];- const out = await run({ type: "s3", key: "acme/{shortSha}/{name}.{ext}" }, files).promise;- expect(out.map((o) => o.key)).toEqual(["acme/9f2c1e4/main.mp4b", "acme/9f2c1e4/teaser.mp4b"]);+ const files = [+ await file("main", "AAA", "mp4b"),+ await file("teaser", "BB", "mp4b"),+ ];+ const out = await run(+ { type: "s3", key: "acme/{shortSha}/{name}.{ext}" },+ files,+ ).promise;+ expect(out.map((o) => o.key)).toEqual([+ "acme/9f2c1e4/main.mp4b",+ "acme/9f2c1e4/teaser.mp4b",+ ]);expect(out[0].url).toBe("https://cdn.example.com/acme/9f2c1e4/main.mp4b");const stored = objects.get("/renders/acme/9f2c1e4/main.mp4b");expect(stored?.body.toString()).toBe("AAA");@@ -161,10 +244,18 @@ describe("deliver", () => {it("writes an idempotency manifest that findExisting returns for the same commit", async () => {const files = [await file("main", "CCC", "mp4b")];- const { promise, input, ctx } = run({ type: "s3", key: "idem/{name}.{ext}" }, files, { idempotencyKey: "launch-v12" });+ const { promise, input, ctx } = run(+ { type: "s3", key: "idem/{name}.{ext}" },+ files,+ { idempotencyKey: "launch-v12" },+ );const delivered = await promise;- expect(await findExisting?.(input, "9f2c1e4abcdef", ctx)).toEqual(delivered);- expect(await findExisting?.(input, "0000000different", ctx)).toBeUndefined();+ expect(await findExisting?.(input, "9f2c1e4abcdef", ctx)).toEqual(+ delivered,+ );+ expect(+ await findExisting?.(input, "0000000different", ctx),+ ).toBeUndefined();// An output deleted from the bucket invalidates the manifest.objects.delete("/renders/idem/main.mp4b");expect(await findExisting?.(input, "9f2c1e4abcdef", ctx)).toBeUndefined();@@ -172,25 +263,46 @@ describe("deliver", () => {it("PUTs presigned URLs, retrying 5xx and failing fast on expiry", async () => {presignedPlan.push(503, 200);- const ok = await run({ type: "presigned", urls: [`${base}/presigned/a?X-Amz-Signature=s`] }, [await file("p", "PPP")]).promise;+ const ok = await run(+ { type: "presigned", urls: [`${base}/presigned/a?X-Amz-Signature=s`] },+ [await file("p", "PPP")],+ ).promise;expect(ok[0].url).toBe(`${base}/presigned/a`);expect(objects.get("/presigned/a")?.body.toString()).toBe("PPP");presignedPlan.push(403);- expect(await codeOf(run({ type: "presigned", urls: [`${base}/presigned/b`] }, [await file("p2", "x")]).promise)).toBe(- "UPLOAD_URL_EXPIRED",- );expect(- await codeOf(run({ type: "presigned", urls: [`${base}/presigned/c`] }, [await file("q1", "x"), await file("q2", "y")]).promise),+ await codeOf(+ run({ type: "presigned", urls: [`${base}/presigned/b`] }, [+ await file("p2", "x"),+ ]).promise,+ ),+ ).toBe("UPLOAD_URL_EXPIRED");+ expect(+ await codeOf(+ run({ type: "presigned", urls: [`${base}/presigned/c`] }, [+ await file("q1", "x"),+ await file("q2", "y"),+ ]).promise,+ ),).toBe("JOB_INVALID");});it("copies to FF_OUTPUT_DIR and refuses to collide or escape", async () => {- const out = await run({ type: "local", path: "{ref}/{name}.{ext}" }, [await file("loc", "LLL")]).promise;- expect(await fs.readFile(path.join(tmp, "out/main/loc.bin"), "utf8")).toBe("LLL");+ const out = await run({ type: "local", path: "{ref}/{name}.{ext}" }, [+ await file("loc", "LLL"),+ ]).promise;+ expect(await fs.readFile(path.join(tmp, "out/main/loc.bin"), "utf8")).toBe(+ "LLL",+ );expect(out[0].path).toBe("main/loc.bin");expect(- await codeOf(run({ type: "local", path: "same.{ext}" }, [await file("l1", "1"), await file("l2", "2")]).promise),+ await codeOf(+ run({ type: "local", path: "same.{ext}" }, [+ await file("l1", "1"),+ await file("l2", "2"),+ ]).promise,+ ),).toBe("JOB_INVALID");});@@ -198,12 +310,23 @@ describe("deliver", () => {const out = await run({ type: "inline" }, [await file("i", "hi")]).promise;expect(Buffer.from(out[0].base64 ?? "", "base64").toString()).toBe("hi");expect(- await codeOf(run({ type: "inline" }, [await file("big", "x".repeat(100))], {}, { FF_INLINE_MAX_BYTES: "50" }).promise),+ await codeOf(+ run(+ { type: "inline" },+ [await file("big", "x".repeat(100))],+ {},+ { FF_INLINE_MAX_BYTES: "50" },+ ).promise,+ ),).toBe("OUTPUT_TOO_LARGE");});it("fails with QA_FAILED after delivering, unless uploads are blocked", async () => {- const qa = { issues: [{ severity: "warning", code: "frozen_video", message: "frozen" }] };+ const qa = {+ issues: [+ { severity: "warning", code: "frozen_video", message: "frozen" },+ ],+ };const files = [{ ...(await file("qa", "QQQ")), qa }];const render = { render: { qa: { failOn: ["frozen_video"] } } };try {@@ -214,7 +337,9 @@ describe("deliver", () => {expect((err as RenderError).details?.outputs).toHaveLength(1);}try {- await run({ type: "inline" }, files, render, { FF_QA_BLOCK_UPLOAD: "true" }).promise;+ await run({ type: "inline" }, files, render, {+ FF_QA_BLOCK_UPLOAD: "true",+ }).promise;} catch (err) {expect((err as RenderError).details?.outputs).toEqual([]);}diff --git a/apps/renderer-service/src/output/sinks.ts b/apps/renderer-service/src/output/sinks.tsindex dfb9e288..3d4be55a 100644--- a/apps/renderer-service/src/output/sinks.ts+++ b/apps/renderer-service/src/output/sinks.ts@@ -18,13 +18,17 @@ import type { VerifiedFile } from "./verify.js";const ATTEMPTS = 3;/** Retries transient storage failures with backoff. */-async function withRetries<T>(signal: AbortSignal, fn: () => Promise<T>): Promise<T> {+async function withRetries<T>(+ signal: AbortSignal,+ fn: () => Promise<T>,+): Promise<T> {for (let attempt = 1; ; attempt++) {try {return await fn();} catch (err) {const classified = classifyUploadError(err);- if (!classified.retryable || attempt >= ATTEMPTS || signal.aborted) throw classified;+ if (!classified.retryable || attempt >= ATTEMPTS || signal.aborted)+ throw classified;await new Promise((r) => setTimeout(r, 500 * 2 ** attempt));}}@@ -33,9 +37,16 @@ async function withRetries<T>(signal: AbortSignal, fn: () => Promise<T>): Promisexport function classifyUploadError(err: unknown): RenderError {if (err instanceof RenderError) return err;const name = err instanceof S3ServiceException ? err.name : "";- const status = err instanceof S3ServiceException ? err.$metadata?.httpStatusCode : undefined;+ const status =+ err instanceof S3ServiceException+ ? err.$metadata?.httpStatusCode+ : undefined;const message = redact(err instanceof Error ? err.message : String(err));- if (/AccessDenied|InvalidAccessKeyId|SignatureDoesNotMatch|InvalidToken|ExpiredToken|AllAccessDisabled/.test(name)) {+ if (+ /AccessDenied|InvalidAccessKeyId|SignatureDoesNotMatch|InvalidToken|ExpiredToken|AllAccessDisabled/.test(+ name,+ )+ ) {return new RenderError("UPLOAD_AUTH_FAILED", `${name}: ${message}`, {phase: "upload",hint: "Check STORAGE_ACCESS_KEY_ID / STORAGE_SECRET_ACCESS_KEY and that they may write to the bucket.",@@ -43,13 +54,22 @@ export function classifyUploadError(err: unknown): RenderError {});}if (/NoSuchBucket/.test(name)) {- return new RenderError("UPLOAD_AUTH_FAILED", `Bucket does not exist: ${message}`, { phase: "upload", cause: err });+ return new RenderError(+ "UPLOAD_AUTH_FAILED",+ `Bucket does not exist: ${message}`,+ { phase: "upload", cause: err },+ );}- if (/ENOSPC/.test(message)) return new RenderError("DISK_FULL", message, { phase: "upload" });- return new RenderError("UPLOAD_FAILED", status ? `HTTP ${status}: ${message}` : message, {- phase: "upload",- cause: err,- });+ if (/ENOSPC/.test(message))+ return new RenderError("DISK_FULL", message, { phase: "upload" });+ return new RenderError(+ "UPLOAD_FAILED",+ status ? `HTTP ${status}: ${message}` : message,+ {+ phase: "upload",+ cause: err,+ },+ );}let client: S3Client | undefined;@@ -67,7 +87,10 @@ export function s3Client(config: RendererConfig): S3Client {return client;}-export function publicUrl(config: RendererConfig, key: string): string | undefined {+export function publicUrl(+ config: RendererConfig,+ key: string,+): string | undefined {const base = config.storage.publicBaseUrl;if (!base) return undefined;return `${base.replace(/\/+$/, "")}/${key.split("/").map(encodeURIComponent).join("/")}`;@@ -131,18 +154,32 @@ export async function deliverPresigned(} as RequestInit);if (res.ok) return;const text = await res.text().catch(() => "");- if ((res.status === 403 || res.status === 400) && /expired|Request has expired|AccessDenied/i.test(text)) {- throw new RenderError("UPLOAD_URL_EXPIRED", `The presigned URL was rejected (${res.status}).`, {- phase: "upload",- hint: "Sign upload URLs with an expiry at least as long as the job timeout.",- });+ if (+ (res.status === 403 || res.status === 400) &&+ /expired|Request has expired|AccessDenied/i.test(text)+ ) {+ throw new RenderError(+ "UPLOAD_URL_EXPIRED",+ `The presigned URL was rejected (${res.status}).`,+ {+ phase: "upload",+ hint: "Sign upload URLs with an expiry at least as long as the job timeout.",+ },+ );}- const error = new RenderError("UPLOAD_FAILED", `Presigned PUT failed with HTTP ${res.status}.`, {- phase: "upload",- details: { body: redact(text.slice(0, 500)) },- });+ const error = new RenderError(+ "UPLOAD_FAILED",+ `Presigned PUT failed with HTTP ${res.status}.`,+ {+ phase: "upload",+ details: { body: redact(text.slice(0, 500)) },+ },+ );if (res.status < 500 && res.status !== 429) {- throw new RenderError("UPLOAD_AUTH_FAILED", error.message, { phase: "upload", details: error.details });+ throw new RenderError("UPLOAD_AUTH_FAILED", error.message, {+ phase: "upload",+ details: error.details,+ });}throw error;});@@ -150,11 +187,19 @@ export async function deliverPresigned(}/** Copies into FF_OUTPUT_DIR atomically (temp file + rename). */-export async function deliverLocal({ file }: Delivery, outputDir: string, relPath: string): Promise<DeliveredFile> {+export async function deliverLocal(+ { file }: Delivery,+ outputDir: string,+ relPath: string,+): Promise<DeliveredFile> {const root = path.resolve(outputDir);const target = path.resolve(root, relPath);if (!target.startsWith(root + path.sep)) {- throw new RenderError("PATH_ESCAPE", `Output path ${relPath} leaves FF_OUTPUT_DIR.`, { phase: "upload" });+ throw new RenderError(+ "PATH_ESCAPE",+ `Output path ${relPath} leaves FF_OUTPUT_DIR.`,+ { phase: "upload" },+ );}await fs.mkdir(path.dirname(target), { recursive: true });const tmp = `${target}.partial-${process.pid}`;@@ -168,11 +213,18 @@ export async function deliverLocal({ file }: Delivery, outputDir: string, relPatreturn summary(file, { path: relPath });}-export async function deliverInline({ file }: Delivery): Promise<DeliveredFile> {- return summary(file, { base64: (await fs.readFile(file.path)).toString("base64") });+export async function deliverInline({+ file,+}: Delivery): Promise<DeliveredFile> {+ return summary(file, {+ base64: (await fs.readFile(file.path)).toString("base64"),+ });}-function summary(file: VerifiedFile, extra: Partial<DeliveredFile>): DeliveredFile {+function summary(+ file: VerifiedFile,+ extra: Partial<DeliveredFile>,+): DeliveredFile {return {name: file.name,bytes: file.bytes,@@ -207,21 +259,37 @@ export async function readManifest(): Promise<Manifest | undefined> {const s3 = s3Client(config);try {- const res = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: manifestKey(idempotencyKey) }));- const manifest = JSON.parse((await res.Body?.transformToString()) ?? "{}") as Manifest;- if (manifest.sha !== sha || manifest.idempotencyKey !== idempotencyKey) return undefined;+ const res = await s3.send(+ new GetObjectCommand({+ Bucket: bucket,+ Key: manifestKey(idempotencyKey),+ }),+ );+ const manifest = JSON.parse(+ (await res.Body?.transformToString()) ?? "{}",+ ) as Manifest;+ if (manifest.sha !== sha || manifest.idempotencyKey !== idempotencyKey)+ return undefined;for (const output of manifest.outputs) {if (!output.key) return undefined;await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: output.key }));}return manifest;} catch (err) {- if (err instanceof S3ServiceException && (err.name === "NoSuchKey" || err.name === "NotFound")) return undefined;+ if (+ err instanceof S3ServiceException &&+ (err.name === "NoSuchKey" || err.name === "NotFound")+ )+ return undefined;throw classifyUploadError(err);}}-export async function writeManifest(config: RendererConfig, bucket: string, manifest: Manifest): Promise<void> {+export async function writeManifest(+ config: RendererConfig,+ bucket: string,+ manifest: Manifest,+): Promise<void> {await s3Client(config).send(new PutObjectCommand({Bucket: bucket,diff --git a/apps/renderer-service/src/output/verify.ts b/apps/renderer-service/src/output/verify.tsindex f400fd89..674f3bf7 100644--- a/apps/renderer-service/src/output/verify.ts+++ b/apps/renderer-service/src/output/verify.ts@@ -25,10 +25,16 @@ export async function verifyFile(): Promise<VerifiedFile> {const stat = await fs.stat(file.path).catch(() => undefined);if (!stat?.isFile()) {- throw new RenderError("OUTPUT_MISSING", `Output ${file.name} was not written.`, { phase: "verify" });+ throw new RenderError(+ "OUTPUT_MISSING",+ `Output ${file.name} was not written.`,+ { phase: "verify" },+ );}if (stat.size === 0) {- throw new RenderError("OUTPUT_INVALID", `Output ${file.name} is empty.`, { phase: "verify" });+ throw new RenderError("OUTPUT_INVALID", `Output ${file.name} is empty.`, {+ phase: "verify",+ });}if (options.maxBytes > 0 && stat.size > options.maxBytes) {throw new RenderError(@@ -38,7 +44,11 @@ export async function verifyFile();}- const verified: VerifiedFile = { ...file, bytes: stat.size, sha256: await sha256File(file.path) };+ const verified: VerifiedFile = {+ ...file,+ bytes: stat.size,+ sha256: await sha256File(file.path),+ };if (options.probe !== false && PROBED.has(file.ext)) {Object.assign(verified, await probe(file, options.signal));}@@ -47,29 +57,49 @@ export async function verifyFile(export async function sha256File(file: string): Promise<string> {const hash = createHash("sha256");- for await (const chunk of createReadStream(file)) hash.update(chunk as Buffer);+ for await (const chunk of createReadStream(file))+ hash.update(chunk as Buffer);return hash.digest("hex");}interface ProbeOutput {format?: { duration?: string };- streams?: { codec_type?: string; nb_frames?: string; nb_read_frames?: string }[];+ streams?: {+ codec_type?: string;+ nb_frames?: string;+ nb_read_frames?: string;+ }[];}-async function probe(file: RenderedFile, signal?: AbortSignal): Promise<{ durationMs?: number; frames?: number }> {+async function probe(+ file: RenderedFile,+ signal?: AbortSignal,+): Promise<{ durationMs?: number; frames?: number }> {let out: ProbeOutput;try {const { stdout } = await exec("ffprobe",- ["-v", "error", "-print_format", "json", "-show_format", "-show_streams", file.path],+ [+ "-v",+ "error",+ "-print_format",+ "json",+ "-show_format",+ "-show_streams",+ file.path,+ ],{ timeoutMs: 120_000, signal },);out = JSON.parse(stdout) as ProbeOutput;} catch (err) {if (err instanceof CommandError) {- throw new RenderError("OUTPUT_INVALID", `Output ${file.name} is not readable media: ${err.message}`, {- phase: "verify",- });+ throw new RenderError(+ "OUTPUT_INVALID",+ `Output ${file.name} is not readable media: ${err.message}`,+ {+ phase: "verify",+ },+ );}if (/ENOENT/.test(String(err))) {// No ffprobe on this host: size and hash checks still ran.@@ -82,13 +112,23 @@ async function probe(file: RenderedFile, signal?: AbortSignal): Promise<{ duraticonst video = out.streams?.find((s) => s.codec_type === "video");const frames = Number(video?.nb_frames ?? video?.nb_read_frames);const result = {- ...(Number.isFinite(duration) && { durationMs: Math.round(duration * 1000) }),+ ...(Number.isFinite(duration) && {+ durationMs: Math.round(duration * 1000),+ }),...(Number.isFinite(frames) && frames > 0 && { frames }),};if (!out.streams?.length) {- throw new RenderError("OUTPUT_INVALID", `Output ${file.name} has no streams.`, { phase: "verify" });+ throw new RenderError(+ "OUTPUT_INVALID",+ `Output ${file.name} has no streams.`,+ { phase: "verify" },+ );}- if (file.expectedFrames && result.frames !== undefined && Math.abs(result.frames - file.expectedFrames) > 1) {+ if (+ file.expectedFrames &&+ result.frames !== undefined &&+ Math.abs(result.frames - file.expectedFrames) > 1+ ) {throw new RenderError("OUTPUT_INVALID",`Output ${file.name} has ${result.frames} frames; the engine rendered ${file.expectedFrames}.`,@@ -109,8 +149,13 @@ interface QaIssueLike {* issue code (`black_frames`, `frozen_video`, `audio_clipping`, …) or a* severity (`error`, `warning`).*/-export function qaFailures(report: unknown, failOn: readonly string[] | undefined): QaIssueLike[] {+export function qaFailures(+ report: unknown,+ failOn: readonly string[] | undefined,+): QaIssueLike[] {if (!failOn?.length || !report || typeof report !== "object") return [];const issues = (report as { issues?: QaIssueLike[] }).issues ?? [];- return issues.filter((i) => failOn.includes(i.code ?? "") || failOn.includes(i.severity ?? ""));+ return issues.filter(+ (i) => failOn.includes(i.code ?? "") || failOn.includes(i.severity ?? ""),+ );}diff --git a/apps/renderer-service/src/render/render-step.ts b/apps/renderer-service/src/render/render-step.tsindex acec12f6..9dd4cda6 100644--- a/apps/renderer-service/src/render/render-step.ts+++ b/apps/renderer-service/src/render/render-step.ts@@ -456,6 +456,17 @@ export function classifyRenderError(cause: err,});}+ if (+ /EACCES|EPERM|permission denied/i.test(message) &&+ /node_modules/.test(message)+ ) {+ return new RenderError("RENDER_FAILED", scrub(message), {+ phase: "render",+ hint: "node_modules files are shared read-only between projects (FF_SHARED_MODULES); write to a temp dir instead, or set FF_SHARED_MODULES=false.",+ details,+ cause: err,+ });+ }if (/Cannot open local media file|ENOENT|no such file/i.test(message)) {return new RenderError("ASSET_NOT_FOUND", scrub(message), {phase: "render",diff --git a/apps/renderer-service/src/render/render.test.ts b/apps/renderer-service/src/render/render.test.tsindex 1a8d7187..eef5539c 100644--- a/apps/renderer-service/src/render/render.test.ts+++ b/apps/renderer-service/src/render/render.test.ts@@ -301,6 +301,18 @@ describe("render error classification", () => {expect(classifyRenderError(new Error(message), "/ws").code).toBe(code);});+ it("explains writes into shared node_modules", () => {+ const err = classifyRenderError(+ new Error(+ "EACCES: permission denied, open '/ws/node_modules/pkg/cache.json'",+ ),+ "/ws",+ );+ expect(err.code).toBe("RENDER_FAILED");+ expect(err.hint).toMatch(/FF_SHARED_MODULES/);+ expect(err.message).toContain("<repo>/node_modules");+ });+it("keeps only project stack frames, repo-relative", () => {const err = new Error("x");err.stack =diff --git a/apps/renderer-service/src/worker/boot.ts b/apps/renderer-service/src/worker/boot.tsindex 9f70abdd..98e23cc3 100644--- a/apps/renderer-service/src/worker/boot.ts+++ b/apps/renderer-service/src/worker/boot.ts@@ -86,7 +86,18 @@ export function bootWorker();const readiness = new ReadinessTracker();- const janitor = new CacheJanitor(config, workerId());+ // The bound engine is loaded from a workspace: never evict that one.+ const engineRepo = () => {+ const engine = boundEngine();+ return engine ? [path.basename(path.dirname(engine.workspace))] : [];+ };+ const janitor = new CacheJanitor(+ config,+ workerId(),+ undefined,+ undefined,+ engineRepo,+ );const journal = new CrashJournal(path.join(config.paths.workDir, "inflight.json"),);@@ -103,13 +114,7 @@ export function bootWorker(const runner = new JobRunner(config, pipeline, {journal,- beforeJob: (job) => {- // The bound engine is loaded from a workspace: never evict that one.- const engine = boundEngine();- const keep = [repoHash(job.repo)];- if (engine) keep.push(path.basename(path.dirname(engine.workspace)));- return janitor.maybeRun(keep);- },+ beforeJob: (job) => janitor.maybeRun([repoHash(job.repo)]),afterJob: (completed) => {const engine = boundEngine();if (engine && boundVersion !== engine.version) {diff --git a/apps/renderer-service/src/worker/janitor.test.ts b/apps/renderer-service/src/worker/janitor.test.tsindex 6afe6713..8f2350d6 100644--- a/apps/renderer-service/src/worker/janitor.test.ts+++ b/apps/renderer-service/src/worker/janitor.test.ts@@ -2,10 +2,13 @@ import fs from "node:fs/promises";import os from "node:os";import path from "node:path";import { afterAll, beforeAll, describe, expect, it } from "vitest";+import { moduleStoreDir, shareModules } from "../install/shared-modules.js";import { testConfig } from "../test-helpers.js";-import { CacheJanitor, dirSize, LAST_USED } from "./janitor.js";+import { CacheJanitor, type DiskProbe, LAST_USED, usage } from "./janitor.js";-const MB = 1024 * 1024;+/** Test sizes are in "MB" of 64 KiB, so the scenarios stay fast. */+const MB = 64 * 1024;+let fill = 0;let tmp: string;beforeAll(async () => {@@ -15,25 +18,64 @@ afterAll(async () => {await fs.rm(tmp, { recursive: true, force: true });});-async function blob(p: string, mb: number, ageMin = 0) {+async function blob(p: string, mb: number, ageMin = 0, byte?: number) {await fs.mkdir(path.dirname(p), { recursive: true });- await fs.writeFile(p, Buffer.alloc(mb * MB));+ // Distinct contents, unless deliberately shared via hard links.+ await fs.writeFile(p, Buffer.alloc(mb * MB, byte ?? ++fill % 256));const t = new Date(Date.now() - ageMin * 60_000);await fs.utimes(p, t, t);}-/** Limits are whole GB, so tests scale sizes via a fractional limit hack: 1 GB = 1024 MB. */-function janitor(root: string, env: Record<string, string>) {+/**+ * Fake disks of `capacityMb`: free space is the capacity minus what is+ * actually stored under the dirs on that disk (hard links count once).+ */+function fakeDisks(+ root: string,+ capacityMb: number,+ sameDisk = true,+): DiskProbe {+ return async (dir) => {+ const work = path.join(root, "work");+ const cache = path.join(root, "cache");+ const onWork = dir.startsWith(work);+ const dirs = sameDisk ? [work, cache] : [onWork ? work : cache];+ return {+ id: sameDisk || onWork ? "work" : "cache",+ totalBytes: capacityMb * MB,+ freeBytes: capacityMb * MB - (await usage(dirs)),+ };+ };+}++function janitor(+ root: string,+ disk: DiskProbe,+ env: Record<string, string> = {},+ intervalMs = 0,+ alwaysProtect: () => string[] = () => [],+) {const config = testConfig({FF_WORK_DIR: path.join(root, "work"),FF_CACHE_DIR: path.join(root, "cache"),+ // 15% free triggers eviction, which then frees up to 25%.+ FF_DISK_MIN_FREE_GB: "0",...env,});- return { j: new CacheJanitor(config, "me", 0), config };+ return new CacheJanitor(config, "me", intervalMs, disk, alwaysProtect);}-describe("CacheJanitor", () => {- it("evicts least-recently-used workspaces, sparing the protected one", async () => {+const names = (paths: string[]) => paths.map((p) => path.basename(p));++describe("CacheJanitor (free space)", () => {+ it("leaves everything while the disk has room", async () => {+ const root = path.join(tmp, "roomy");+ await blob(path.join(root, "work/ws/old/0/big.bin"), 600, 120);+ const j = janitor(root, fakeDisks(root, 4000));+ expect(await j.enforce([])).toEqual([]);+ });++ it("evicts least-recently-used workspaces until the target is free", async () => {const root = path.join(tmp, "ws-case");const ws = path.join(root, "work/ws");await blob(path.join(ws, "old/0/big.bin"), 600);@@ -42,14 +84,58 @@ describe("CacheJanitor", () => {await blob(path.join(ws, "mid", LAST_USED), 0, 60);await blob(path.join(ws, "protected/0/big.bin"), 600);await blob(path.join(ws, "protected", LAST_USED), 0, 500);- const { j } = janitor(root, { FF_WORK_MAX_GB: "1", FF_CACHE_MAX_GB: "0" });- const removed = await j.enforce(["protected"]);- expect(removed.map((p) => path.basename(p))).toEqual(["old"]);- expect(await dirSize(ws)).toBeLessThanOrEqual(1024 * MB);+ // 1700 MB disk, 1500 used: 200 free < 255 (15%); target 425.+ // Removing "old" (the least recently used) leaves 800 free.+ const j = janitor(root, fakeDisks(root, 1700));+ expect(names(await j.enforce(["protected"]))).toEqual(["old"]);});- it("evicts LFS objects before mirrors, and leaves other workers' busy mirrors", async () => {+ it(+ "frees shared module files only when their last workspace goes",+ {+ timeout: 30_000,+ },+ async () => {+ const root = path.join(tmp, "shared-case");+ const store = moduleStoreDir(path.join(root, "work"));+ const slot = async (repo: string, ageMin: number) => {+ const dir = path.join(root, "work/ws", repo);+ await blob(+ path.join(dir, "0/node_modules/webgpu/dawn.node"),+ 400,+ 0,+ 7,+ );+ await blob(path.join(dir, "0/node_modules", `${repo}.js`), 100);+ await shareModules(path.join(dir, "0"), store);+ await blob(path.join(dir, LAST_USED), 0, ageMin);+ };+ await slot("a", 60);+ await slot("b", 30);+ await slot("c", 1);+ // Stored once: 400 shared + 3 × 100 own = 700 MB.+ const used = async () =>+ Math.round((await usage([path.join(root, "work")])) / MB);+ expect(await used()).toBe(700);++ // 810 MB disk: 110 free < 121 (15%); target 202. Removing "a" frees+ // only its own 100 MB (b and c still link dawn.node): 210 free.+ const j = janitor(root, fakeDisks(root, 810));+ expect(names(await j.enforce(["c"]))).toEqual(["a"]);+ expect(await used()).toBe(600);++ // Tighter disk: b goes too, but dawn.node stays for c.+ const tight = janitor(root, fakeDisks(root, 650));+ expect(names(await tight.enforce(["c"]))).toEqual(["b"]);+ await expect(+ fs.stat(path.join(root, "work/ws/c/0/node_modules/webgpu/dawn.node")),+ ).resolves.toBeTruthy();+ expect(await used()).toBe(500);+ },+ );++ it("on a separate cache disk: LFS objects, then mirrors, then package caches", async () => {const root = path.join(tmp, "cache-case");const cache = path.join(root, "cache");await blob(path.join(cache, "lfs/objects/aa/bb/old-oid"), 500, 300);@@ -59,23 +145,21 @@ describe("CacheJanitor", () => {await blob(path.join(cache, "git/other/repo2.git/pack.bin"), 300);await blob(path.join(cache, "git/other/repo2.git", LAST_USED), 0, 5);await blob(path.join(cache, "models/vision.onnx"), 100);- const { j } = janitor(root, { FF_CACHE_MAX_GB: "1", FF_WORK_MAX_GB: "0" });+ // A full work disk is not the cache disk's problem.+ await blob(path.join(root, "work/ws/x/0/big.bin"), 10);+ const disk = fakeDisks(root, 1500, false);+ const j = janitor(root, disk);- // 1300 MB total, limit 1024: the old LFS object (500) alone is enough.- const removed = await j.enforce([]);- expect(removed.map((p) => path.basename(p))).toEqual(["old-oid"]);+ // 1300 used of 1500: 200 free < 225. The oldest LFS object suffices.+ expect(names(await j.enforce([]))).toEqual(["old-oid"]);- // Over again: the oldest LFS object covers it on its own.await blob(path.join(cache, "lfs/objects/ee/ff/more"), 700, 2);- expect((await j.enforce([])).map((p) => path.basename(p))).toEqual([- "more",- ]);+ expect(names(await j.enforce([]))).toEqual(["more"]);// Far over: LFS first, then our own idle mirror, then the package store.// The other worker's recently used mirror and the models stay.await blob(path.join(cache, "pnpm-store/v10/blob"), 1200);- const again = await j.enforce([]);- expect(again.map((p) => path.basename(p))).toEqual([+ expect(names(await j.enforce([]))).toEqual(["new-oid","repo1.git","pnpm-store",@@ -86,6 +170,65 @@ describe("CacheJanitor", () => {await expect(fs.stat(path.join(cache, "models/vision.onnx")),).resolves.toBeTruthy();+ await expect(+ fs.stat(path.join(root, "work/ws/x/0/big.bin")),+ ).resolves.toBeTruthy();+ });++ it("never evicts the workspace the engine was loaded from", async () => {+ const root = path.join(tmp, "engine");+ await blob(path.join(root, "work/ws/engine/0/big.bin"), 500);+ await blob(path.join(root, "work/ws/engine", LAST_USED), 0, 500);+ await blob(path.join(root, "work/ws/other/0/big.bin"), 100);+ const j = janitor(root, fakeDisks(root, 650), {}, 0, () => ["engine"]);+ // Low on space, but the oldest workspace holds the engine.+ expect(names(await j.enforce([]))).toEqual(["other"]);+ });++ it("on one shared disk, evicts workspaces before caches", async () => {+ const root = path.join(tmp, "one-disk");+ await blob(path.join(root, "cache/lfs/objects/aa/bb/oid"), 300, 500);+ await blob(path.join(root, "work/ws/r/0/big.bin"), 300);+ await blob(path.join(root, "work/ws/r", LAST_USED), 0, 1);+ const j = janitor(root, fakeDisks(root, 650));+ // 50 free < 97; target 162: the workspace goes, the LFS object stays.+ expect(names(await j.enforce([]))).toEqual(["r"]);+ });++ it("makes extra room for a download that is about to start", async () => {+ const root = path.join(tmp, "extra");+ await blob(path.join(root, "cache/lfs/objects/aa/bb/oid"), 300, 500);+ const j = janitor(root, fakeDisks(root, 2000));+ expect(await j.enforce([])).toEqual([]);+ // 1700 free, but 1600 more are needed on top of the 300 floor.+ expect(names(await j.enforce([], 1600 * MB))).toEqual(["oid"]);+ });++ it("runs between intervals as soon as a disk runs low", async () => {+ const root = path.join(tmp, "interval");+ let capacity = 4000;+ const disk: DiskProbe = (dir) => fakeDisks(root, capacity)(dir);+ await blob(path.join(root, "work/ws/r/0/big.bin"), 500);+ const j = janitor(root, disk, {}, 60 * 60 * 1000);+ await j.maybeRun([]); // first run: nothing to do+ capacity = 550;+ await j.maybeRun([]);+ await expect(fs.stat(path.join(root, "work/ws/r"))).rejects.toThrow();+ });++ it("still honours optional size caps", async () => {+ const root = path.join(tmp, "caps");+ // Caps are whole GB: sparse files report real sizes without the writes.+ const sparse = async (p: string, bytes: number) => {+ await fs.mkdir(path.dirname(p), { recursive: true });+ await fs.writeFile(p, "");+ await fs.truncate(p, bytes);+ };+ await sparse(path.join(root, "work/ws/a/0/big.bin"), 700 * 1024 ** 2);+ await blob(path.join(root, "work/ws/a", LAST_USED), 0, 60);+ await sparse(path.join(root, "work/ws/b/0/big.bin"), 700 * 1024 ** 2);+ const j = janitor(root, fakeDisks(root, 100_000), { FF_WORK_MAX_GB: "1" });+ expect(names(await j.enforce([]))).toEqual(["a"]);});it("drops failed-job dirs older than a day", async () => {@@ -95,8 +238,7 @@ describe("CacheJanitor", () => {const t = new Date(Date.now() - 2 * 24 * 3600_000);await fs.utimes(old, t, t);await blob(path.join(root, "work/jobs/new-job/out/x.mp4"), 1);- const { j } = janitor(root, {});- const removed = await j.enforce([]);- expect(removed.map((p) => path.basename(p))).toEqual(["old-job"]);+ const j = janitor(root, fakeDisks(root, 100_000));+ expect(names(await j.enforce([]))).toEqual(["old-job"]);});});diff --git a/apps/renderer-service/src/worker/janitor.ts b/apps/renderer-service/src/worker/janitor.tsindex b6eee8a2..e19423fb 100644--- a/apps/renderer-service/src/worker/janitor.ts+++ b/apps/renderer-service/src/worker/janitor.ts@@ -2,6 +2,12 @@ import { existsSync } from "node:fs";import fs from "node:fs/promises";import path from "node:path";import type { RendererConfig } from "../config/index.js";+import {+ moduleStoreDir,+ readManifests,+ releaseKeys,+ sweepStore,+} from "../install/shared-modules.js";import { rendererLogger } from "../logger.js";/** Marker whose mtime records when a mirror or workspace was last used. */@@ -33,6 +39,25 @@ export async function markUsed(dir: string): Promise<void> {await fs.utimes(marker, now, now).catch(() => fs.writeFile(marker, ""));}+/** A filesystem as statfs sees it. */+export interface DiskInfo {+ /** Identifies the filesystem: dirs with equal ids share free space. */+ id: string;+ totalBytes: number;+ freeBytes: number;+}++export type DiskProbe = (dir: string) => Promise<DiskInfo>;++const statDisk: DiskProbe = async (dir) => {+ const [fsStat, stat] = await Promise.all([fs.statfs(dir), fs.stat(dir)]);+ return {+ id: String(stat.dev),+ totalBytes: Number(fsStat.blocks) * Number(fsStat.bsize),+ freeBytes: Number(fsStat.bavail) * Number(fsStat.bsize),+ };+};+interface Candidate {path: string;bytes: number;@@ -40,11 +65,24 @@ interface Candidate {kind: string;}+/** One eviction stage on one disk: removes items until `enough()` says stop. */+type Stage = (+ enough: () => Promise<boolean>,+ removed: string[],+) => Promise<void>;+/**- * Keeps FF_WORK_DIR and FF_CACHE_DIR under their size limits (PLAN.md §6.4).- * Runs between jobs, so nothing it deletes is in use. Eviction order:- * workspaces (LRU), then LFS objects (LRU), then git mirrors (LRU), then- * whole package caches. Pinned vision models are never evicted.+ * Keeps the workspace and cache disks from filling up (PLAN.md §6.4). Runs+ * between jobs, so nothing it deletes is in use.+ *+ * Free space decides: when a disk has less free than+ * max(FF_DISK_MIN_FREE_PCT, FF_DISK_MIN_FREE_GB), the janitor evicts until+ * FF_DISK_TARGET_FREE_PCT is free. Order: workspaces (LRU) and the shared+ * module files only they used, then LFS objects (LRU), then git mirrors (LRU),+ * then whole package caches. When FF_WORK_DIR and FF_CACHE_DIR are on the+ * same disk the stages run in that order against one free-space figure.+ * FF_WORK_MAX_GB / FF_CACHE_MAX_GB optionally cap usage on top. Pinned vision+ * models are never evicted.*/export class CacheJanitor {private lastRun = 0;@@ -53,34 +91,84 @@ export class CacheJanitor {private readonly config: RendererConfig,private readonly workerId: string,private readonly intervalMs = 5 * 60 * 1000,+ private readonly disk: DiskProbe = statDisk,+ /** Repo hashes never evicted, whoever asks (e.g. the bound engine's). */+ private readonly alwaysProtect: () => readonly string[] = () => [],) {current = this;}- /** Runs at most once per interval unless forced. Never throws. */+ /**+ * Before each job: evicts right away when a disk is low on space (cheap+ * statfs check), otherwise at most once per interval. Never throws.+ */async maybeRun(protect: readonly string[], force = false): Promise<void> {- if (!force && Date.now() - this.lastRun < this.intervalMs) return;+ const due = force || Date.now() - this.lastRun >= this.intervalMs;+ if (!due && !(await this.lowOnSpace().catch(() => false))) return;this.lastRun = Date.now();try {- const removed = await this.enforce(protect);- if (removed.length) {- rendererLogger.info(- { removed: removed.length, paths: removed.slice(0, 20) },- "Cache eviction",- );- }+ await this.enforce(protect);} catch (err) {rendererLogger.warn({ err: String(err) }, "Cache eviction failed");}}+ /** Free space per disk, for logs and /health. */+ async describeDisks(): Promise<+ Record<string, { freeGb: number; totalGb: number }>+ > {+ const { workDir, cacheDir } = this.config.paths;+ const out: Record<string, { freeGb: number; totalGb: number }> = {};+ for (const [name, dir] of [+ ["work", workDir],+ ["cache", cacheDir],+ ] as const) {+ const d = await this.disk(dir);+ out[name] = {+ freeGb: Math.round((d.freeBytes / GB) * 10) / 10,+ totalGb: Math.round((d.totalBytes / GB) * 10) / 10,+ };+ }+ return out;+ }++ private thresholds(d: DiskInfo, extra = 0) {+ const { minFreePct, minFreeGb, targetFreePct } = this.config.paths;+ // A fixed GB floor can't demand most of a small disk.+ const floor = Math.min(minFreeGb * GB, d.totalBytes * 0.5);+ const low = Math.max((d.totalBytes * minFreePct) / 100, floor) + extra;+ const target = Math.max(+ Math.min((d.totalBytes * targetFreePct) / 100, d.totalBytes * 0.6),+ low,+ );+ return { low, target };+ }++ private async lowOnSpace(): Promise<boolean> {+ const { workDir, cacheDir } = this.config.paths;+ for (const dir of [workDir, cacheDir]) {+ if (!existsSync(dir)) continue;+ const d = await this.disk(dir);+ if (d.freeBytes < this.thresholds(d).low) return true;+ }+ return false;+ }+/**- * Evicts until both limits hold. `protect` lists repo hashes the next job- * needs (its workspace and mirror stay).+ * Evicts until each disk has its target free space and the optional caps+ * hold. `protect` lists repo hashes the next job needs (its workspace and+ * mirror stay). `extraBytes` raises the bar, for a download that is about+ * to need that much (LFS).*/- async enforce(protect: readonly string[]): Promise<string[]> {+ async enforce(+ protectOnly: readonly string[],+ extraBytes = 0,+ ): Promise<string[]> {+ const protect = [...protectOnly, ...this.alwaysProtect()];const removed: string[] = [];const { workDir, cacheDir, workMaxGb, cacheMaxGb } = this.config.paths;+ await fs.mkdir(workDir, { recursive: true });+ await fs.mkdir(cacheDir, { recursive: true });// Failed-job dirs kept for debugging expire after a day.for (const dir of await children(path.join(workDir, "jobs"))) {@@ -89,59 +177,108 @@ export class CacheJanitor {}}- if (workMaxGb > 0) {- const workspaces = (await children(path.join(workDir, "ws"))).filter(- (d) => !protect.includes(path.basename(d)),- );- const candidates = await Promise.all(- workspaces.map(async (d) => ({- path: d,- bytes: await dirSize(d),- lastUsed: await mtime(path.join(d, LAST_USED), d),- kind: "workspace",- })),- );- const total =- (await dirSize(path.join(workDir, "ws"))) +- (await dirSize(path.join(workDir, "jobs")));- await evict(candidates, total - workMaxGb * GB, removed);+ const workStages: Stage[] = [+ (enough, out) => this.evictWorkspaces(protect, enough, out),+ async () => {+ await sweepStore(moduleStoreDir(workDir));+ },+ ];+ const cacheStages: Stage[] = [+ (enough, out) =>+ evictEach(lfsObjects(path.join(cacheDir, "lfs")), enough, out),+ async (enough, out) => evictEach(this.mirrors(protect), enough, out),+ (enough, out) => evictEach(this.stores(), enough, out, false),+ ];++ // Free space, per disk.+ const [work, cache] = await Promise.all([+ this.disk(workDir),+ this.disk(cacheDir),+ ]);+ const disks =+ work.id === cache.id+ ? [{ dir: workDir, stages: [...workStages, ...cacheStages] }]+ : [+ { dir: workDir, stages: workStages },+ { dir: cacheDir, stages: cacheStages },+ ];+ for (const { dir, stages } of disks) {+ const first = await this.disk(dir);+ const { low, target } = this.thresholds(first, extraBytes);+ if (first.freeBytes >= low) continue;+ const enough = async () => (await this.disk(dir)).freeBytes >= target;+ for (const stage of stages) {+ if (await enough()) break;+ await stage(enough, removed);+ }}- if (cacheMaxGb > 0) {- let total = await dirSize(cacheDir);- const over = () => total - cacheMaxGb * GB;- if (over() > 0) {- const lfs = await lfsObjects(path.join(cacheDir, "lfs"));- total -= await evict(lfs, over(), removed);+ // Optional caps on top, measured by walking the trees.+ if (workMaxGb > 0) {+ const over = async () =>+ (await usage([+ path.join(workDir, "ws"),+ path.join(workDir, "jobs"),+ path.join(workDir, "store"),+ ])) >+ workMaxGb * GB;+ const enough = async () => !(await over());+ if (await over()) {+ for (const stage of workStages) {+ if (await enough()) break;+ await stage(enough, removed);+ }}- if (over() > 0) {- total -= await evict(await this.mirrors(protect), over(), removed);+ }+ if (cacheMaxGb > 0) {+ const enough = async () => (await usage([cacheDir])) <= cacheMaxGb * GB;+ for (const stage of cacheStages) {+ if (await enough()) break;+ await stage(enough, removed);}- if (over() > 0) {- const stores = await Promise.all(- STORES.map(async (s) => {- const p = path.join(cacheDir, s);- return {- path: p,- bytes: await dirSize(p),- lastUsed: 0,- kind: "store",- };+ }+ if (removed.length) {+ rendererLogger.info(+ {+ removed: removed.length,+ paths: removed.slice(0, 20),+ ...(extraBytes > 0 && {+ neededMb: Math.round(extraBytes / 1024 ** 2),}),- );- // Largest first: one eviction usually suffices.- stores.sort((a, b) => b.bytes - a.bytes);- total -= await evict(- stores.filter((s) => s.bytes > 0),- over(),- removed,- false,- );- }+ disks: await this.describeDisks().catch(() => undefined),+ },+ "Cache eviction",+ );}return removed;}+ /** Oldest workspace first; each takes its now-unused shared module files along. */+ private async evictWorkspaces(+ protect: readonly string[],+ enough: () => Promise<boolean>,+ removed: string[],+ ): Promise<void> {+ const { workDir } = this.config.paths;+ const store = moduleStoreDir(workDir);+ const workspaces = await Promise.all(+ (await children(path.join(workDir, "ws")))+ .filter((d) => !protect.includes(path.basename(d)))+ .map(async (d) => ({+ path: d,+ lastUsed: await mtime(path.join(d, LAST_USED), d),+ })),+ );+ workspaces.sort((a, b) => a.lastUsed - b.lastUsed);+ for (const ws of workspaces) {+ if (await enough()) return;+ // Remove the workspace first, so its store files drop to one link.+ const keys = await readManifests(ws.path);+ await remove(ws.path, removed);+ await releaseKeys(store, keys);+ }+ }+private async mirrors(protect: readonly string[]): Promise<Candidate[]> {const out: Candidate[] = [];for (const workerDir of await children(@@ -154,34 +291,37 @@ export class CacheJanitor {const lastUsed = await mtime(path.join(mirror, LAST_USED), mirror);// Another worker may be using its mirror right now.if (!own && Date.now() - lastUsed < FOREIGN_MIRROR_IDLE_MS) continue;- out.push({- path: mirror,- bytes: await dirSize(mirror),- lastUsed,- kind: "mirror",- });+ out.push({ path: mirror, bytes: 0, lastUsed, kind: "mirror" });}}return out;}++ /** Package caches, largest first: one eviction usually suffices. */+ private async stores(): Promise<Candidate[]> {+ const stores = await Promise.all(+ STORES.map(async (s) => {+ const p = path.join(this.config.paths.cacheDir, s);+ return { path: p, bytes: await usage([p]), lastUsed: 0, kind: "store" };+ }),+ );+ return stores.filter((s) => s.bytes > 0).sort((a, b) => b.bytes - a.bytes);+ }}-/** Removes candidates (oldest first unless `sort` is false) until `bytes` are freed. */-async function evict(- candidates: Candidate[],- bytes: number,+/** Removes candidates (oldest first unless `sort` is false) until `enough()`. */+async function evictEach(+ candidates: Promise<Candidate[]>,+ enough: () => Promise<boolean>,removed: string[],sort = true,-): Promise<number> {- if (bytes <= 0) return 0;- if (sort) candidates.sort((a, b) => a.lastUsed - b.lastUsed);- let freed = 0;- for (const c of candidates) {- if (freed >= bytes) break;+): Promise<void> {+ const list = await candidates;+ if (sort) list.sort((a, b) => a.lastUsed - b.lastUsed);+ for (const c of list) {+ if (await enough()) return;await remove(c.path, removed);- freed += c.bytes;}- return freed;}async function remove(p: string, removed: string[]): Promise<void> {@@ -231,9 +371,13 @@ async function lfsObjects(root: string): Promise<Candidate[]> {return out;}-/** Bytes on disk under `dir` (symlinks not followed). */-export async function dirSize(dir: string): Promise<number> {+/**+ * Bytes on disk under `dirs` (symlinks not followed). A file hard-linked in+ * several places (shared node_modules) counts once.+ */+export async function usage(dirs: string[]): Promise<number> {let total = 0;+ const seen = new Set<string>();const walk = async (d: string): Promise<void> => {const entries = await fs.readdir(d, { withFileTypes: true })@@ -243,13 +387,16 @@ export async function dirSize(dir: string): Promise<number> {const p = path.join(d, e.name);if (e.isDirectory()) await walk(p);else if (e.isFile()) {- // Read first: `total += await …` would add to a stale total.- const size = (await fs.lstat(p).catch(() => undefined))?.size ?? 0;- total += size;+ const stat = await fs.lstat(p).catch(() => undefined);+ if (!stat) return;+ const id = `${stat.dev}:${stat.ino}`;+ if (seen.has(id)) return;+ seen.add(id);+ total += stat.size;}}),);};- await walk(dir);+ for (const dir of dirs) await walk(dir);return total;}