Code-first video & image, rendered natively on WebGPU.
feat(storage): pin assets on your own S3-compatible bucket
okanasla1bc911411e0
diff --git a/.changeset/asset-storage.md b/.changeset/asset-storage.mdnew file mode 100644index 00000000..11f285ed--- /dev/null+++ b/.changeset/asset-storage.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++Keep media out of git on your own bucket. `framefields assets add|push|pull|status` stores files by sha256 on any S3-compatible bucket (Cloudflare R2, S3, B2, MinIO) and pins them in `framefields.json`; a remote's `publicUrl` (a custom domain or CDN) serves reads without keys, and uploads are marked immutable so CDNs cache them. Cloud renders upload what the bucket lacks, sign reads of private remotes, and workers download and cache them by hash. `framefields/storage` exposes the client.diff --git a/CLAUDE.md b/CLAUDE.mdindex 32b4a8ea..b93594d4 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -13,12 +13,13 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb- Keep this file current when structure or dev patterns change but do not bloat it.## Repo map-- `packages/framefields`: public SDK. `Composition`, `Layer.*`, `LayerAnimation`, `Signal`, effects (`src/effects/classes.ts`), `FontManager`, `renderFrameGrid`. `src/index.ts` is the export surface. CLI in `src/cli/` (`init`, `add`, `list`, `fonts add|list`, `check`, `renders`, `render --cloud`); `framefields.json` (`cli/config.ts`) holds the components folder, fonts and named `renders` that the CLI, `renderCloud` and the dashboard pick by name; `framefields/cloud` (`renderCloud`) is the GPU-free client for api.framefields.dev; `init` never prompts or overwrites, and its file templates are strings in `templates.ts` (a test parses them).+- `packages/framefields`: public SDK. `Composition`, `Layer.*`, `LayerAnimation`, `Signal`, effects (`src/effects/classes.ts`), `FontManager`, `renderFrameGrid`. `src/index.ts` is the export surface. CLI in `src/cli/` (`init`, `add`, `list`, `fonts add|list`, `assets add|push|pull|status`, `check`, `renders`, `render --cloud`); `framefields.json` (`cli/config.ts`) holds the components folder, fonts, named `renders` that the CLI, `renderCloud` and the dashboard pick by name, and `storage` remotes plus sha256-pinned `assets`; `framefields/cloud` (`renderCloud`) is the GPU-free client for api.framefields.dev; `init` never prompts or overwrites, and its file templates are strings in `templates.ts` (a test parses them).- `packages/components`: the component registry `framefields add` copies into user projects (shadcn-style). `defineComponent`/`bind` contract in `src/core/contract.ts`, shared helpers in `src/core/` (keys, color, layout, surface, theme). Components import only `framefields` and `../core`. Registry items declare font needs (e.g. emoji); CLI fonts are pinned by URL, size and SHA-256 in `cli/fonts.ts`. Prop docs in `components.json` are generated (drift test). Tree-shake size budgets are tested.- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), private; `framefields` bundles it as a lazy chunk (`clip-vision.ts` imports it dynamically for `comp.vision(..., { sam2 | sam3 })`; `index.ts` re-exports types only) and exposes the trackers as `framefields/sam`. No weights ship; `packages/framefields/NOTICE.md` covers their licenses. Pins in `src/models/registry.ts`.+- `packages/storage`: content-addressed assets on any S3-compatible bucket (R2 first): manifest format, SigV4 client (aws4fetch), verified downloads. Shared by the CLI (`framefields/storage`) and the render worker, which fetches pinned assets after LFS.- `packages/core`: shared types, `Effect` base, signals, `emojiPresentation` (the one UTS #51 rule used by layout, preload and lint).- Others: `renderers`, `media`, `node-sdk`, `server-utils`, `client-utils`, `tensor-webgpu`, `vision`, `tsconfig`.- `nodes/node-*`: one VFX/audio/layout node each (`src/shared/`, `src/renderers/`, `src/index.ts`). Only `node-compositor` has a `SKILL.md`; update it when Compositor behavior changes (guarded by `skillmd-check.test.ts`). Effect renderers declare `reach` and `maxPixelScale` (node-sdk) when they can: without `maxPixelScale` a scaled-up leaf using the effect stays at layout resolution.diff --git a/apps/renderer-service/Dockerfile b/apps/renderer-service/Dockerfileindex e6ea40cb..6a54dbcc 100644--- a/apps/renderer-service/Dockerfile+++ b/apps/renderer-service/Dockerfile@@ -36,7 +36,7 @@ COPY --from=pruner /repo/out/pnpm-lock.yaml ./pnpm-lock.yaml# test-only devDependencies (framefields) are installed but never built.RUN pnpm install --frozen-lockfile --ignore-scriptsCOPY --from=pruner /repo/out/full/ .-RUN pnpm --filter @framefields/renderer-service build \+RUN pnpm --filter @framefields/storage --filter @framefields/renderer-service build \&& pnpm --filter @framefields/renderer-service deploy --prod --legacy /out \# Pinned SSH host keys for github.com / gitlab.com / bitbucket.org (§6.5).&& /out/deploy/known-hosts.shdiff --git a/apps/renderer-service/package.json b/apps/renderer-service/package.jsonindex 7bd94d22..b8e6ad7a 100644--- a/apps/renderer-service/package.json+++ b/apps/renderer-service/package.json@@ -27,6 +27,7 @@"dependencies": {"@aws-sdk/client-s3": "^3.1151.0","@aws-sdk/lib-storage": "^3.1151.0",+ "@framefields/storage": "workspace:*","@hono/node-server": "^1.19.11","dotenv": "^17.0.1","hono": "^4.13.7",diff --git a/apps/renderer-service/src/errors.ts b/apps/renderer-service/src/errors.tsindex 41db8b6b..818122f0 100644--- a/apps/renderer-service/src/errors.ts+++ b/apps/renderer-service/src/errors.ts@@ -9,6 +9,7 @@ export type JobPhase =| "resolve"| "fetch"| "lfs"+ | "assets"| "install"| "load"| "render"@@ -49,6 +50,13 @@ const CODES = {LFS_POINTER_UNRESOLVED: { phase: "lfs", retryable: false },UNSUPPORTED_GIT_FILTER: { phase: "lfs", retryable: false },+ ASSET_MANIFEST_INVALID: { phase: "assets", retryable: false },+ ASSET_PRIVATE: { phase: "assets", retryable: false },+ ASSET_NOT_UPLOADED: { phase: "assets", retryable: false },+ ASSET_FORBIDDEN: { phase: "assets", retryable: false },+ ASSET_DOWNLOAD_FAILED: { phase: "assets", retryable: true },+ ASSET_HASH_MISMATCH: { phase: "assets", retryable: false },+DISK_FULL: { phase: "any", retryable: true },LOCKFILE_MISSING: { phase: "install", retryable: false },diff --git a/apps/renderer-service/src/git/assets.test.ts b/apps/renderer-service/src/git/assets.test.tsnew file mode 100644index 00000000..95ad509d--- /dev/null+++ b/apps/renderer-service/src/git/assets.test.ts@@ -0,0 +1,145 @@+import { createHash } from "node:crypto";+import {+ mkdtempSync,+ readFileSync,+ rmSync,+ symlinkSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import path from "node:path";+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import { fetchAssets } from "./assets.js";++let workdir: string;+let cacheDir: string;+beforeEach(() => {+ workdir = mkdtempSync(path.join(tmpdir(), "ff-ws-"));+ cacheDir = mkdtempSync(path.join(tmpdir(), "ff-cache-"));+});+afterEach(() => {+ rmSync(workdir, { recursive: true, force: true });+ rmSync(cacheDir, { recursive: true, force: true });+});++const sha = (s: string) => createHash("sha256").update(s).digest("hex");+const INTRO = "intro frames";+const RAW = "raw interview";++function project(assets: Record<string, unknown>) {+ writeFileSync(+ path.join(workdir, "framefields.json"),+ JSON.stringify({+ storage: {+ default: "media",+ remotes: {+ media: {+ endpoint: "https://acct.r2.cloudflarestorage.com",+ bucket: "films",+ publicUrl: "https://media.example.com",+ },+ raw: {+ endpoint: "https://acct.r2.cloudflarestorage.com",+ bucket: "raw",+ },+ },+ },+ assets,+ }),+ );+}++function served(files: Record<string, string>) {+ const requests: string[] = [];+ const fetch = (async (input: string | URL | Request) => {+ const url = String(input instanceof Request ? input.url : input);+ requests.push(url);+ const body = files[url];+ return body === undefined+ ? new Response(null, { status: 404 })+ : new Response(body);+ }) as typeof globalThis.fetch;+ return { fetch, requests };+}++const introUrl = `https://media.example.com/sha256/${sha(INTRO).slice(0, 2)}/${sha(INTRO)}`;++describe("fetchAssets", () => {+ it("downloads from the custom domain once, then copies from the cache", async () => {+ project({ "assets/intro.mp4": { sha256: sha(INTRO), size: INTRO.length } });+ const cdn = served({ [introUrl]: INTRO });+ expect(await fetchAssets(workdir, { cacheDir, fetch: cdn.fetch })).toEqual({+ files: 1,+ bytes: INTRO.length,+ cache: "miss",+ });+ expect(readFileSync(path.join(workdir, "assets/intro.mp4"), "utf8")).toBe(+ INTRO,+ );++ rmSync(path.join(workdir, "assets"), { recursive: true });+ expect(+ (await fetchAssets(workdir, { cacheDir, fetch: cdn.fetch })).cache,+ ).toBe("hit");+ expect(cdn.requests).toEqual([introUrl]);+ });++ it("reads private remotes through the job's signed URLs", async () => {+ project({+ "assets/raw.mov": { sha256: sha(RAW), size: RAW.length, remote: "raw" },+ });+ await expect(+ fetchAssets(workdir, { cacheDir, fetch: served({}).fetch }),+ ).rejects.toMatchObject({+ code: "ASSET_PRIVATE",+ });+ const signedUrl =+ "https://acct.r2.cloudflarestorage.com/raw/x?X-Amz-Signature=s";+ const cdn = served({ [signedUrl]: RAW });+ await fetchAssets(workdir, {+ cacheDir,+ fetch: cdn.fetch,+ signed: { [sha(RAW)]: signedUrl },+ });+ expect(readFileSync(path.join(workdir, "assets/raw.mov"), "utf8")).toBe(+ RAW,+ );+ });++ it("names missing uploads and wrong bytes", async () => {+ project({ "assets/intro.mp4": { sha256: sha(INTRO), size: INTRO.length } });+ await expect(+ fetchAssets(workdir, { cacheDir, fetch: served({}).fetch }),+ ).rejects.toMatchObject({+ code: "ASSET_NOT_UPLOADED",+ hint: "Run `npx framefields assets push` and render again.",+ });+ await expect(+ fetchAssets(workdir, {+ cacheDir,+ fetch: served({ [introUrl]: "intro framez" }).fetch,+ }),+ ).rejects.toMatchObject({ code: "ASSET_HASH_MISMATCH" });+ });++ it("does nothing without assets, and refuses paths outside the project", async () => {+ expect((await fetchAssets(workdir, { cacheDir })).cache).toBe("off");+ project({ "../escape.mp4": { sha256: sha(INTRO), size: 1 } });+ await expect(fetchAssets(workdir, { cacheDir })).rejects.toMatchObject({+ code: "ASSET_MANIFEST_INVALID",+ });+ });++ it("won't write through a folder that links out of the checkout", async () => {+ const outside = mkdtempSync(path.join(tmpdir(), "ff-outside-"));+ symlinkSync(outside, path.join(workdir, "assets"));+ project({ "assets/intro.mp4": { sha256: sha(INTRO), size: INTRO.length } });+ await expect(+ fetchAssets(workdir, {+ cacheDir,+ fetch: served({ [introUrl]: INTRO }).fetch,+ }),+ ).rejects.toMatchObject({ code: "PATH_ESCAPE" });+ rmSync(outside, { recursive: true, force: true });+ });+});diff --git a/apps/renderer-service/src/git/assets.ts b/apps/renderer-service/src/git/assets.tsnew file mode 100644index 00000000..af6af8a5--- /dev/null+++ b/apps/renderer-service/src/git/assets.ts@@ -0,0 +1,192 @@+import { constants, existsSync } from "node:fs";+import fs from "node:fs/promises";+import path from "node:path";+import {+ type AssetEntry,+ AssetError,+ type AssetManifest,+ downloadVerified,+ hashFile,+ ManifestError,+ parseAssets,+ parseStorage,+ readUrl,+ type StorageConfig,+} from "@framefields/storage";+import { RenderError } from "../errors.js";+import { resolveInside } from "../paths.js";++/** Under the cache dir: verified asset bytes at assets/aa/<sha256>. */+export const ASSET_CACHE = "assets";+const CONCURRENCY = 4;++export interface AssetOptions {+ cacheDir: string;+ /** Signed GET URLs for private assets, by sha256 (`source.assets`). */+ signed?: Readonly<Record<string, string>>;+ signal?: AbortSignal;+ /** Evicts caches so `bytes` more fit (the janitor). */+ makeRoom?: (bytes: number) => Promise<void>;+ fetch?: typeof fetch;+}++export interface AssetResult {+ files: number;+ bytes: number;+ cache: "hit" | "partial" | "miss" | "off";+}++/**+ * Pipeline step after LFS: puts every asset the project's framefields.json+ * pins in place, from the worker's cache or its URL (the remote's public+ * domain, the asset's own URL, or a signature the job carries), checked+ * against its sha256.+ */+export async function fetchAssets(+ workdir: string,+ options: AssetOptions,+): Promise<AssetResult> {+ const manifest = await readManifest(workdir);+ const entries = Object.entries(manifest?.assets ?? {});+ if (!manifest || entries.length === 0)+ return { files: 0, bytes: 0, cache: "off" };++ const plan = await Promise.all(+ entries.map(async ([rel, entry]) => ({+ rel,+ entry,+ dest: path.join(workdir, rel),+ cached: cachePath(options.cacheDir, entry.sha256),+ })),+ );+ const todo: typeof plan = [];+ for (const item of plan)+ if (!(await present(item.dest, item.entry))) todo.push(item);+ const misses = todo.filter((t) => !existsSync(t.cached));+ const missBytes = misses.reduce((n, t) => n + t.entry.size, 0);+ if (missBytes > 0) await options.makeRoom?.(missBytes);++ let next = 0;+ const worker = async () => {+ while (next < todo.length) {+ const item = todo[next++];+ if (!existsSync(item.cached))+ await download(+ item.rel,+ item.entry,+ item.cached,+ manifest.storage,+ options,+ );+ await place(workdir, item.rel, item.cached);+ }+ };+ await Promise.all(+ Array.from({ length: Math.min(CONCURRENCY, todo.length) }, worker),+ );+ return {+ files: entries.length,+ bytes: missBytes,+ cache:+ misses.length === 0+ ? "hit"+ : misses.length === entries.length+ ? "miss"+ : "partial",+ };+}++async function readManifest(+ workdir: string,+): Promise<{ storage?: StorageConfig; assets?: AssetManifest } | undefined> {+ const file = path.join(workdir, "framefields.json");+ if (!existsSync(file)) return undefined;+ try {+ const raw = JSON.parse(await fs.readFile(file, "utf8")) as {+ storage?: unknown;+ assets?: unknown;+ };+ if (raw.assets === undefined) return undefined;+ const storage =+ raw.storage === undefined ? undefined : parseStorage(raw.storage);+ return { storage, assets: parseAssets(raw.assets, storage) };+ } catch (err) {+ if (err instanceof ManifestError || err instanceof SyntaxError)+ throw new RenderError("ASSET_MANIFEST_INVALID", err.message, {+ hint: "`npx framefields assets status` reads the same file locally.",+ });+ throw err;+ }+}++/**+ * Copies the cached bytes to `rel`, refusing folders that symlink out of+ * the checkout and replacing (not writing through) a symlink at `rel`.+ */+async function place(workdir: string, rel: string, cached: string) {+ let existing = path.posix.dirname(rel);+ while (existing !== "." && !existsSync(path.join(workdir, existing)))+ existing = path.posix.dirname(existing);+ await resolveInside(workdir, existing);+ const dest = path.join(workdir, rel);+ await fs.mkdir(path.dirname(dest), { recursive: true });+ await fs.rm(dest, { force: true });+ await fs.copyFile(cached, dest, constants.COPYFILE_FICLONE);+}++/** A file already in the checkout counts when its bytes match. */+async function present(dest: string, entry: AssetEntry): Promise<boolean> {+ const stat = await fs.lstat(dest).catch(() => undefined);+ if (!stat?.isFile() || stat.size !== entry.size) return false;+ return (await hashFile(dest)).sha256 === entry.sha256;+}++async function download(+ rel: string,+ entry: AssetEntry,+ cached: string,+ storage: StorageConfig | undefined,+ options: AssetOptions,+): Promise<void> {+ const url = readUrl(entry, storage, options.signed);+ if (!url)+ throw new RenderError(+ "ASSET_PRIVATE",+ `${rel} is on a private remote and the job carries no signed URL for it.`,+ {+ hint: "Render from the CLI or renderCloud (they sign reads with your keys), or give the remote a publicUrl.",+ },+ );+ try {+ await downloadVerified(url, cached, entry, {+ fetch: options.fetch,+ signal: options.signal,+ });+ } catch (err) {+ if (!(err instanceof AssetError)) throw err;+ throw new RenderError(ERROR_CODES[err.code], `${rel}: ${err.message}`, {+ hint: HINTS[err.code],+ cause: err,+ });+ }+}++const ERROR_CODES = {+ ASSET_NOT_FOUND: "ASSET_NOT_UPLOADED",+ ASSET_FORBIDDEN: "ASSET_FORBIDDEN",+ ASSET_DOWNLOAD_FAILED: "ASSET_DOWNLOAD_FAILED",+ ASSET_HASH_MISMATCH: "ASSET_HASH_MISMATCH",+} as const;++const HINTS: Record<AssetError["code"], string> = {+ ASSET_NOT_FOUND: "Run `npx framefields assets push` and render again.",+ ASSET_FORBIDDEN:+ "The publicUrl doesn't serve the bucket, or a signed URL expired: render again from the CLI.",+ ASSET_DOWNLOAD_FAILED: "The bucket or CDN didn't answer; retried.",+ ASSET_HASH_MISMATCH:+ "The bucket holds other bytes than framefields.json pins: `npx framefields assets push` from the machine with the right file.",+};++function cachePath(cacheDir: string, sha256: string): string {+ return path.join(cacheDir, ASSET_CACHE, sha256.slice(0, 2), sha256);+}diff --git a/apps/renderer-service/src/git/checkout.ts b/apps/renderer-service/src/git/checkout.tsindex 6ee460c6..7044c25e 100644--- a/apps/renderer-service/src/git/checkout.ts+++ b/apps/renderer-service/src/git/checkout.ts@@ -4,6 +4,7 @@ import { RenderError } from "../errors.js";import type { CheckoutResult, Pipeline } from "../jobs/pipeline.js";import { resolveInside } from "../paths.js";import { activeJanitor, markUsed } from "../worker/janitor.js";+import { fetchAssets } from "./assets.js";import { openGitSession } from "./auth.js";import { hostProfile } from "./hosts.js";import { findPointers, pullLfs } from "./lfs.js";@@ -92,6 +93,16 @@ export const checkout: Pipeline["checkout"] = async (job, ctx) => {);}+ ctx.progress({ phase: "assets" });+ const assets = await fetchAssets(workdir, {+ cacheDir: ctx.config.paths.cacheDir,+ signed: source.assets,+ signal: ctx.signal,+ makeRoom: async (bytes) => {+ await activeJanitor()?.enforce([repoHash(repo)], bytes);+ },+ });+return {root: ws.root,workdir,@@ -100,6 +111,7 @@ export const checkout: Pipeline["checkout"] = async (job, ctx) => {mirror: mirror.cache,workspace: ws.warm ? "warm" : "cold",lfs: lfs.cache,+ assets: assets.cache,},release: async () => {},} satisfies CheckoutResult;diff --git a/apps/renderer-service/src/jobs/schema.ts b/apps/renderer-service/src/jobs/schema.tsindex 01ec78ad..41a4bd4e 100644--- a/apps/renderer-service/src/jobs/schema.ts+++ b/apps/renderer-service/src/jobs/schema.ts@@ -31,6 +31,16 @@ const sourceSchema = z.object({}).default({ include: [], exclude: [] }),submodules: z.boolean().default(false),+ /**+ * Signed GET URLs for assets framefields.json pins on private remotes,+ * by sha256 (the CLI signs them with the user's keys).+ */+ assets: z+ .record(+ z.string().regex(/^[0-9a-f]{64}$/, "keys are sha256 hashes"),+ z.url({ protocol: /^https$/ }).max(8192),+ )+ .optional(),sparse: z.array(relPath("sparse path")).max(256).optional(),auth: z.object({diff --git a/apps/renderer-service/src/jobs/store.ts b/apps/renderer-service/src/jobs/store.tsindex 2d86c486..a52532ec 100644--- a/apps/renderer-service/src/jobs/store.ts+++ b/apps/renderer-service/src/jobs/store.ts@@ -13,6 +13,7 @@ export type JobPhaseName =| "resolving"| "fetching"| "lfs"+ | "assets"| "installing"| "loading"| "rendering"diff --git a/apps/renderer-service/src/worker/janitor.ts b/apps/renderer-service/src/worker/janitor.tsindex cba664b5..af5902b5 100644--- a/apps/renderer-service/src/worker/janitor.ts+++ b/apps/renderer-service/src/worker/janitor.ts@@ -2,6 +2,7 @@ import { existsSync } from "node:fs";import fs from "node:fs/promises";import path from "node:path";import type { RendererConfig } from "../config/index.js";+import { ASSET_CACHE } from "../git/assets.js";import {moduleStoreDir,readManifests,@@ -269,6 +270,8 @@ export class CacheJanitor {const cacheStages: Stage[] = [(enough, out) =>evictEach(lfsObjects(path.join(cacheDir, "lfs")), enough, out),+ (enough, out) =>+ evictEach(assetObjects(path.join(cacheDir, ASSET_CACHE)), enough, out),async (enough, out) => evictEach(this.mirrors(protect), enough, out),(enough, out) => evictEach(this.stores(), enough, out, false),];@@ -437,27 +440,36 @@ async function mtime(p: string, fallback?: string): Promise<number> {/** git-lfs stores objects at lfs/objects/aa/bb/<oid>; LRU by access time. */async function lfsObjects(root: string): Promise<Candidate[]> {const out: Candidate[] = [];- const objects = path.join(root, "objects");- if (!existsSync(objects)) return out;- for (const a of await children(objects)) {- for (const b of await children(a)) {- for (const name of await fs.readdir(b).catch(() => [])) {- const p = path.join(b, name);- const stat = await fs.stat(p).catch(() => undefined);- if (stat?.isFile()) {- out.push({- path: p,- bytes: stat.size,- lastUsed: Math.max(stat.atimeMs, stat.mtimeMs),- kind: "lfs",- });- }- }- }- }+ for (const a of await children(path.join(root, "objects")))+ for (const b of await children(a)) await cachedFiles(b, "lfs", out);return out;}+/** framefields.json assets are cached at assets/aa/<sha256>. */+async function assetObjects(root: string): Promise<Candidate[]> {+ const out: Candidate[] = [];+ for (const a of await children(root)) await cachedFiles(a, "asset", out);+ return out;+}++async function cachedFiles(+ dir: string,+ kind: string,+ out: Candidate[],+): Promise<void> {+ for (const name of await fs.readdir(dir).catch(() => [])) {+ const p = path.join(dir, name);+ const stat = await fs.stat(p).catch(() => undefined);+ if (stat?.isFile())+ out.push({+ path: p,+ bytes: stat.size,+ lastUsed: Math.max(stat.atimeMs, stat.mtimeMs),+ kind,+ });+ }+}+/*** Bytes on disk under `dirs` (symlinks not followed). A file hard-linked in* several places (shared node_modules) counts once.diff --git a/packages/framefields/package.json b/packages/framefields/package.jsonindex 3202f8e1..bd374e3c 100644--- a/packages/framefields/package.json+++ b/packages/framefields/package.json@@ -87,6 +87,12 @@"import": "./dist/cloud/index.mjs","default": "./dist/cloud/index.mjs"},+ "./storage": {+ "types": "./dist/storage/index.d.mts",+ "development": "./src/storage/index.ts",+ "import": "./dist/storage/index.mjs",+ "default": "./dist/storage/index.mjs"+ },"./sam": {"types": "./dist/sam/index.d.mts","development": "./src/sam/index.ts",@@ -113,6 +119,7 @@"@mediabunny/server": "^1.61.6","@napi-rs/webcodecs": "^1.3.1","@resvg/resvg-js": "^2.6.2",+ "aws4fetch": "^1.0.20","d3-format": "^3.1.2","d3-scale": "^4.0.2","d3-shape": "^3.2.0",@@ -191,6 +198,7 @@"@framefields/node-vision": "workspace:*","@framefields/renderer": "workspace:*","@framefields/renderers": "workspace:*",+ "@framefields/storage": "workspace:*","@framefields/tensor-webgpu": "workspace:*","@framefields/tsconfig": "workspace:*","@framefields/vision": "workspace:*",diff --git a/packages/framefields/src/cli.ts b/packages/framefields/src/cli.tsindex afb6147a..4e96ec20 100644--- a/packages/framefields/src/cli.ts+++ b/packages/framefields/src/cli.ts@@ -7,6 +7,7 @@* npx framefields list the components you can add* npx framefields fonts add <role...> install fonts (emoji)* npx framefields fonts list the fonts you can install+ * npx framefields assets add|push|pull|status media on your own bucket* npx framefields renders the renders named in framefields.json* npx framefields render <name> --cloud render on api.framefields.dev*/@@ -15,6 +16,7 @@ import { join, relative, resolve } from "node:path";import { fileURLToPath } from "node:url";import { parseArgs } from "node:util";import { type AddReport, add } from "./cli/add.js";+import { runAssets } from "./cli/assets.js";import { runCheck } from "./cli/check.js";import { runCloudRender } from "./cli/cloud.js";import { readConfig } from "./cli/config.js";@@ -57,6 +59,13 @@ Commands:Install fonts (pinned, hash-checked, with their license)and register them in the generated <components>/fonts.ts.fonts list List the fonts you can install.+ assets add <path...>+ Keep media out of git: hash files (or every file in a+ folder), upload them to the project's bucket (R2, S3, any+ S3-compatible storage) and pin them in framefields.json.+ assets push Upload recorded assets the bucket lacks.+ assets pull Download missing assets, checked against their sha256.+ assets status Which assets are here, missing or changed.renders List the renders named in framefields.json.render [name|entry] --cloudRender on api.framefields.dev from the pushed commit of@@ -88,6 +97,14 @@ Options for render --cloud:--json Print the final status as JSON.Ctrl-C cancels the render; GPU time already used is charged.+Options for assets:+ --remote <name> The remote in framefields.json "storage" (default: its+ default, or its only one).+ --overwrite pull: replace files changed locally.+ Keys: FRAMEFIELDS_STORAGE_<REMOTE>_ACCESS_KEY_ID and _SECRET_ACCESS_KEY,+ or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Reads need none when the+ remote has a publicUrl (a custom domain or CDN in front of the bucket).+Options for init:--name <name> Package name (default: the folder's name).--pm <manager> npm, pnpm, yarn or bun (default: the one running this).@@ -115,6 +132,7 @@ Examples:npx framefields init my-film --installnpx framefields add mouse-pointer focus-zoomnpx framefields fonts add emoji+ npx framefields assets add assets/footagenpx framefields render film --cloud --plan`;@@ -148,6 +166,7 @@ async function main(argv: string[]): Promise<number> {codec: { type: "string" },quality: { type: "string" },select: { type: "string" },+ remote: { type: "string" },help: { type: "boolean", short: "h" },},});@@ -175,6 +194,14 @@ async function main(argv: string[]): Promise<number> {}if (command === "check") return check(names, cwd, values, log);if (command === "renders") return listRenders(cwd, values.json, log);+ if (command === "assets")+ return runAssets(names, {+ cwd,+ log,+ remote: values.remote,+ overwrite: values.overwrite,+ json: values.json,+ });if (command === "render") {if (names.length > 1 || !values.cloud) {process.stderr.write(diff --git a/packages/framefields/src/cli/assets.test.ts b/packages/framefields/src/cli/assets.test.tsnew file mode 100644index 00000000..f88a61f4--- /dev/null+++ b/packages/framefields/src/cli/assets.test.ts@@ -0,0 +1,218 @@+import { createHash } from "node:crypto";+import {+ mkdirSync,+ mkdtempSync,+ readFileSync,+ rmSync,+ writeFileSync,+} from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import {+ addAssets,+ assetStatus,+ prepareCloudAssets,+ pullAssets,+ pushAssets,+} from "./assets.js";+import { CONFIG_FILE, readConfig } from "./config.js";++let cwd: string;+beforeEach(() => {+ cwd = mkdtempSync(join(tmpdir(), "ff-assets-"));+ mkdirSync(join(cwd, "assets/footage"), { recursive: true });+ writeFileSync(join(cwd, "assets/footage/intro.mp4"), "intro frames");+ writeFileSync(join(cwd, "assets/footage/outro.mov"), "outro frames");+ writeFileSync(join(cwd, ".gitignore"), "node_modules/");+});+afterEach(() => rmSync(cwd, { recursive: true, force: true }));++const sha = (s: string) => createHash("sha256").update(s).digest("hex");+const env = {+ FRAMEFIELDS_STORAGE_MEDIA_ACCESS_KEY_ID: "id",+ FRAMEFIELDS_STORAGE_MEDIA_SECRET_ACCESS_KEY: "secret",+};++/**+ * One bucket behind two hosts: the S3 API (signed) and its custom domain+ * (public reads), as an R2 bucket with a custom domain is.+ */+function fakeR2() {+ const objects = new Map<string, Buffer>();+ const log: string[] = [];+ const handler = async (input: Request | string | URL, init?: RequestInit) => {+ const req = input instanceof Request ? input : new Request(input, init);+ const url = new URL(req.url);+ log.push(`${req.method} ${url.host}${url.pathname}`);+ const key =+ url.host === "media.example.com"+ ? url.pathname.slice(1)+ : url.pathname.replace(/^\/films\//, "");+ if (+ url.host !== "media.example.com" &&+ !url.searchParams.has("X-Amz-Signature")+ )+ expect(req.headers.get("authorization")).toMatch(/^AWS4-HMAC-SHA256/);+ const body = objects.get(key);+ if (req.method === "PUT") {+ objects.set(key, Buffer.from(await req.arrayBuffer()));+ return new Response(null);+ }+ if (!body) return new Response(null, { status: 404 });+ if (req.method === "HEAD")+ return new Response(null, {+ headers: { "content-length": String(body.length) },+ });+ return new Response(body);+ };+ return { fetch: handler as typeof fetch, objects, log };+}++const manifest = (storage: object) =>+ writeFileSync(join(cwd, CONFIG_FILE), JSON.stringify({ storage }));++const media = {+ endpoint: "https://acct.r2.cloudflarestorage.com",+ bucket: "films",+ publicUrl: "https://media.example.com",+};++describe("framefields assets", () => {+ it("adds a folder: uploads, pins by sha256, gitignores", async () => {+ manifest({ remotes: { media } });+ const r2 = fakeR2();+ const added = await addAssets(["assets/footage"], {+ cwd,+ env,+ fetch: r2.fetch,+ });+ expect(added.map((a) => [a.path, a.uploaded])).toEqual([+ ["assets/footage/intro.mp4", true],+ ["assets/footage/outro.mov", true],+ ]);+ expect(readConfig(cwd).config.assets).toEqual({+ "assets/footage/intro.mp4": { sha256: sha("intro frames"), size: 12 },+ "assets/footage/outro.mov": { sha256: sha("outro frames"), size: 12 },+ });+ expect(+ r2.objects+ .get(`sha256/${sha("intro frames").slice(0, 2)}/${sha("intro frames")}`)+ ?.toString(),+ ).toBe("intro frames");+ expect(readFileSync(join(cwd, ".gitignore"), "utf8")).toBe(+ "node_modules/\n# framefields assets: on the bucket, pinned in framefields.json\n/assets/footage/intro.mp4\n/assets/footage/outro.mov\n",+ );+ // Adding again uploads nothing and ignores nothing twice.+ const again = await addAssets(["assets/footage/intro.mp4"], {+ cwd,+ env,+ fetch: r2.fetch,+ });+ expect(again[0].uploaded).toBe(false);+ expect(+ readFileSync(join(cwd, ".gitignore"), "utf8").match(/intro/g),+ ).toHaveLength(1);+ });++ it("pulls from the custom domain without keys, and verifies", async () => {+ manifest({ remotes: { media } });+ const r2 = fakeR2();+ await addAssets(["assets/footage"], { cwd, env, fetch: r2.fetch });+ rmSync(join(cwd, "assets/footage/intro.mp4"));+ writeFileSync(join(cwd, "assets/footage/outro.mov"), "edited");+ r2.log.length = 0;+ const pulled = await pullAssets({ cwd, env: {}, fetch: r2.fetch });+ expect(pulled).toEqual({+ downloaded: ["assets/footage/intro.mp4"],+ present: [],+ changed: ["assets/footage/outro.mov"],+ });+ expect(r2.log).toEqual([+ `GET media.example.com/sha256/${sha("intro frames").slice(0, 2)}/${sha("intro frames")}`,+ ]);+ expect(readFileSync(join(cwd, "assets/footage/intro.mp4"), "utf8")).toBe(+ "intro frames",+ );+ expect((await assetStatus(cwd)).map((a) => a.state)).toEqual([+ "ok",+ "changed",+ ]);+ await pullAssets({ cwd, env: {}, fetch: r2.fetch, overwrite: true });+ expect(readFileSync(join(cwd, "assets/footage/outro.mov"), "utf8")).toBe(+ "outro frames",+ );+ });++ it("needs keys for a private bucket, and names them", async () => {+ manifest({ remotes: { media: { ...media, publicUrl: undefined } } });+ const r2 = fakeR2();+ await addAssets(["assets/footage/intro.mp4"], {+ cwd,+ env,+ fetch: r2.fetch,+ });+ rmSync(join(cwd, "assets/footage/intro.mp4"));+ await expect(pullAssets({ cwd, env: {}, fetch: r2.fetch })).rejects.toThrow(+ "set FRAMEFIELDS_STORAGE_MEDIA_ACCESS_KEY_ID and FRAMEFIELDS_STORAGE_MEDIA_SECRET_ACCESS_KEY",+ );+ await pullAssets({ cwd, env, fetch: r2.fetch });+ expect(readFileSync(join(cwd, "assets/footage/intro.mp4"), "utf8")).toBe(+ "intro frames",+ );+ });++ it("push uploads what the bucket lost, and refuses files that changed", async () => {+ manifest({ remotes: { media } });+ const r2 = fakeR2();+ await addAssets(["assets/footage"], { cwd, env, fetch: r2.fetch });+ r2.objects.clear();+ expect(await pushAssets({ cwd, env, fetch: r2.fetch })).toEqual({+ uploaded: ["assets/footage/intro.mp4", "assets/footage/outro.mov"],+ present: [],+ });+ r2.objects.clear();+ writeFileSync(join(cwd, "assets/footage/intro.mp4"), "edited");+ await expect(pushAssets({ cwd, env, fetch: r2.fetch })).rejects.toThrow(+ "assets/footage/intro.mp4 (changed here)",+ );+ });++ it("signs private assets for cloud renders; public ones need no signature", async () => {+ manifest({+ default: "media",+ remotes: { media, raw: { endpoint: media.endpoint, bucket: "films" } },+ });+ const r2 = fakeR2();+ const keys = {+ ...env,+ FRAMEFIELDS_STORAGE_RAW_ACCESS_KEY_ID: "id",+ FRAMEFIELDS_STORAGE_RAW_SECRET_ACCESS_KEY: "secret",+ };+ await addAssets(["assets/footage/intro.mp4"], {+ cwd,+ env: keys,+ fetch: r2.fetch,+ });+ await addAssets(["assets/footage/outro.mov"], {+ cwd,+ env: keys,+ fetch: r2.fetch,+ remote: "raw",+ });+ expect(+ readConfig(cwd).config.assets?.["assets/footage/outro.mov"]?.remote,+ ).toBe("raw");+ const signed = await prepareCloudAssets({+ cwd,+ env: keys,+ fetch: r2.fetch,+ });+ expect(Object.keys(signed)).toEqual([sha("outro frames")]);+ expect(signed[sha("outro frames")]).toContain("X-Amz-Signature=");+ // Without keys, a private remote can't be signed.+ await expect(+ prepareCloudAssets({ cwd, env: {}, fetch: r2.fetch }),+ ).rejects.toThrow("assets/footage/outro.mov is on raw, which needs keys");+ });+});diff --git a/packages/framefields/src/cli/assets.ts b/packages/framefields/src/cli/assets.tsnew file mode 100644index 00000000..edcb9a68--- /dev/null+++ b/packages/framefields/src/cli/assets.ts@@ -0,0 +1,397 @@+/**+ * `framefields assets`: media kept out of git, on an S3-compatible bucket+ * (R2, S3, B2, MinIO), pinned by content in `framefields.json`.+ *+ * add <path...> hash, upload what the bucket lacks, record, gitignore+ * push upload recorded assets the bucket lacks+ * pull download missing assets, verified byte for byte+ * status which assets are here, missing, or changed+ *+ * Reads need no keys when the remote has a `publicUrl` (a custom domain or+ * CDN in front of the bucket); otherwise they're signed with the remote's+ * keys. Cloud renders get signed URLs for private assets from the CLI, so+ * the keys never leave the machine.+ */+import { execFileSync } from "node:child_process";+import {+ appendFileSync,+ existsSync,+ readdirSync,+ readFileSync,+ statSync,+} from "node:fs";+import { join, relative, resolve, sep } from "node:path";+import {+ type AssetEntry,+ type AssetManifest,+ Bucket,+ credentialEnvNames,+ credentialsFromEnv,+ downloadVerified,+ hashFile,+ isAssetPath,+ pickRemote,+ readUrl,+ type StorageConfig,+} from "@framefields/storage";+import { CONFIG_FILE, readConfig, writeConfig } from "./config.js";++export interface AssetOptions {+ /** Project folder (where framefields.json is). */+ cwd: string;+ /** Where remote keys come from. Default `process.env`. */+ env?: Readonly<Record<string, string | undefined>>;+ fetch?: typeof fetch;+ log?: (line: string) => void;+ signal?: AbortSignal;+}++export type AssetsErrorCode =+ | "no_storage"+ | "no_credentials"+ | "not_uploaded"+ | "outside_project"+ | "not_found";++export class AssetsError extends Error {+ override name = "AssetsError";+ constructor(+ message: string,+ readonly code: AssetsErrorCode,+ ) {+ super(message);+ }+}++export type AssetState = "ok" | "missing" | "changed";++/** Hashes and uploads files (folders: every file in them), and records them. */+export async function addAssets(+ paths: string[],+ options: AssetOptions & { remote?: string },+): Promise<{ path: string; size: number; uploaded: boolean }[]> {+ const { cwd } = options;+ const { config } = readConfig(cwd);+ const { name, remote } = pickRemote(config.storage, options.remote);+ const bucket = openBucket(config.storage, name, options);+ const files = paths.flatMap((p) => expand(cwd, p));+ if (files.length === 0)+ throw new AssetsError(`No files in ${paths.join(", ")}.`, "not_found");+ const added: { path: string; size: number; uploaded: boolean }[] = [];+ const assets: AssetManifest = { ...config.assets };+ for (const file of files) {+ const digest = await hashFile(join(cwd, file));+ const uploaded = !(await bucket.has(digest.sha256, digest.size));+ if (uploaded) await upload(bucket, cwd, file, digest, options.log);+ const isDefault = name === pickRemote(config.storage).name;+ assets[file] = { ...digest, ...(!isDefault && { remote: name }) };+ added.push({ path: file, size: digest.size, uploaded });+ }+ writeConfig(cwd, { ...config, assets: sortKeys(assets) });+ ignore(cwd, files);+ const tracked = trackedByGit(cwd, files);+ if (tracked.length)+ options.log?.(+ `Already committed to git: ${tracked.join(", ")}. Stop tracking them with \`git rm --cached ${tracked.join(" ")}\` (the files stay here).`,+ );+ if (remote.publicUrl === undefined)+ options.log?.(+ `${name} has no publicUrl: cloud renders from the CLI sign reads, dashboard renders can't read it.`,+ );+ return added;+}++/** Uploads every recorded asset its bucket lacks, from the local file. */+export async function pushAssets(+ options: AssetOptions,+): Promise<{ uploaded: string[]; present: string[] }> {+ const { config } = readConfig(options.cwd);+ const uploaded: string[] = [];+ const present: string[] = [];+ const problems: string[] = [];+ for (const [path, entry] of storedAssets(config.assets)) {+ const bucket = openBucket(config.storage, entry.remote, options);+ if (await bucket.has(entry.sha256, entry.size)) {+ present.push(path);+ continue;+ }+ const state = await localState(options.cwd, path, entry);+ if (state !== "ok") {+ problems.push(`${path} (${state} here)`);+ continue;+ }+ await upload(bucket, options.cwd, path, entry, options.log);+ uploaded.push(path);+ }+ if (problems.length)+ throw new AssetsError(+ `Not on the bucket, and the local file doesn't match ${CONFIG_FILE}: ${problems.join(", ")}. Restore the file, or record the new version with \`framefields assets add\`.`,+ "not_uploaded",+ );+ return { uploaded, present };+}++/**+ * Downloads every asset that's missing here. Files changed locally are+ * left alone (and reported) unless `overwrite`.+ */+export async function pullAssets(+ options: AssetOptions & { overwrite?: boolean },+): Promise<{ downloaded: string[]; present: string[]; changed: string[] }> {+ const { config } = readConfig(options.cwd);+ const result = {+ downloaded: [] as string[],+ present: [] as string[],+ changed: [] as string[],+ };+ for (const [path, entry] of Object.entries(config.assets ?? {})) {+ const state = await localState(options.cwd, path, entry);+ if (state === "ok") {+ result.present.push(path);+ continue;+ }+ if (state === "changed" && !options.overwrite) {+ result.changed.push(path);+ continue;+ }+ const url = await readableUrl(entry, config.storage, options);+ options.log?.(`Downloading ${path} (${megabytes(entry.size)})`);+ await downloadVerified(url, join(options.cwd, path), entry, {+ fetch: options.fetch,+ signal: options.signal,+ });+ result.downloaded.push(path);+ }+ return result;+}++/** Each recorded asset's state here: present and matching, missing, or changed. */+export async function assetStatus(+ cwd: string,+): Promise<{ path: string; size: number; state: AssetState }[]> {+ const { config } = readConfig(cwd);+ return Promise.all(+ Object.entries(config.assets ?? {}).map(async ([path, entry]) => ({+ path,+ size: entry.size,+ state: await localState(cwd, path, entry),+ })),+ );+}++/**+ * Before a cloud render: uploads assets the bucket lacks (when there are+ * keys), checks public ones are reachable, and signs reads of private ones.+ * Returns the signed URLs by sha256, for the render request.+ */+export async function prepareCloudAssets(+ options: AssetOptions & { expiresSec?: number },+): Promise<Record<string, string>> {+ const { config } = readConfig(options.cwd);+ const signed: Record<string, string> = {};+ const missing: string[] = [];+ for (const [path, entry] of storedAssets(config.assets)) {+ const { name, remote } = pickRemote(config.storage, entry.remote);+ const keys = credentialsFromEnv(name, options.env);+ if (!keys) {+ if (!remote.publicUrl) throw noCredentials(name, path);+ const url = readUrl(entry, config.storage) as string;+ if (!(await reachable(url, options.fetch))) missing.push(path);+ continue;+ }+ const bucket = new Bucket(remote, keys, { fetch: options.fetch });+ if (!(await bucket.has(entry.sha256, entry.size))) {+ if ((await localState(options.cwd, path, entry)) !== "ok") {+ missing.push(path);+ continue;+ }+ await upload(bucket, options.cwd, path, entry, options.log);+ }+ if (!remote.publicUrl)+ signed[entry.sha256] = await bucket.presign(+ entry.sha256,+ options.expiresSec ?? 86_400,+ );+ }+ if (missing.length)+ throw new AssetsError(+ `Not on the bucket: ${missing.join(", ")}. Run \`framefields assets push\` from a machine that has them (and the remote's keys).`,+ "not_uploaded",+ );+ return signed;+}++/** Assets kept on a remote (not pinned to a fixed `url`). */+function storedAssets(+ assets: AssetManifest | undefined,+): [string, AssetEntry][] {+ return Object.entries(assets ?? {}).filter(([, entry]) => !entry.url);+}++async function localState(+ cwd: string,+ path: string,+ entry: AssetEntry,+): Promise<AssetState> {+ const file = join(cwd, path);+ if (!existsSync(file)) return "missing";+ if (statSync(file).size !== entry.size) return "changed";+ return (await hashFile(file)).sha256 === entry.sha256 ? "ok" : "changed";+}++async function readableUrl(+ entry: AssetEntry,+ storage: StorageConfig | undefined,+ options: AssetOptions,+): Promise<string> {+ const url = readUrl(entry, storage);+ if (url) return url;+ return openBucket(storage, entry.remote, options).presign(entry.sha256, 3600);+}++function openBucket(+ storage: StorageConfig | undefined,+ remoteName: string | undefined,+ options: AssetOptions,+): Bucket {+ const { name, remote } = pickRemote(storage, remoteName);+ const keys = credentialsFromEnv(name, options.env);+ if (!keys) throw noCredentials(name);+ return new Bucket(remote, keys, { fetch: options.fetch });+}++function noCredentials(remote: string, path?: string): AssetsError {+ const [id, secret] = credentialEnvNames(remote);+ return new AssetsError(+ `${path ? `${path} is on ${remote}, which` : `Remote ${remote}`} needs keys: set ${id} and ${secret} (or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)${path ? ", or give it a publicUrl" : ""}.`,+ "no_credentials",+ );+}++async function upload(+ bucket: Bucket,+ cwd: string,+ path: string,+ digest: { sha256: string; size: number },+ log?: (line: string) => void,+): Promise<void> {+ log?.(`Uploading ${path} (${megabytes(digest.size)})`);+ let shown = 0;+ await bucket.upload(join(cwd, path), digest, ({ sent, total }) => {+ const percent = Math.floor((sent / total) * 10) * 10;+ if (total > 64e6 && percent > shown && percent < 100) {+ shown = percent;+ log?.(` ${percent}%`);+ }+ });+}++async function reachable(url: string, fetchImpl = fetch): Promise<boolean> {+ const res = await fetchImpl(url, { method: "HEAD" }).catch(() => undefined);+ return res?.ok ?? false;+}++/** Project-relative POSIX paths of `path`, or of every file under it. */+function expand(cwd: string, path: string): string[] {+ const abs = resolve(cwd, path);+ const rel = relative(cwd, abs).split(sep).join("/");+ if (!isAssetPath(rel) || rel.startsWith("../"))+ throw new AssetsError(`${path} is outside the project.`, "outside_project");+ if (!existsSync(abs))+ throw new AssetsError(`${path} does not exist.`, "not_found");+ if (!statSync(abs).isDirectory()) return [rel];+ return readdirSync(abs, { withFileTypes: true })+ .filter((d) => !d.name.startsWith("."))+ .flatMap((d) => expand(cwd, `${rel}/${d.name}`))+ .sort();+}++const IGNORE_HEADER =+ "# framefields assets: on the bucket, pinned in framefields.json";++/** Adds the files to .gitignore, under one marked block. */+function ignore(cwd: string, files: string[]): void {+ const file = join(cwd, ".gitignore");+ const text = existsSync(file) ? readFileSync(file, "utf8") : "";+ const lines = new Set(text.split("\n"));+ const fresh = files.map((f) => `/${f}`).filter((l) => !lines.has(l));+ if (fresh.length === 0) return;+ const header = lines.has(IGNORE_HEADER) ? [] : [IGNORE_HEADER];+ const lead = text && !text.endsWith("\n") ? "\n" : "";+ appendFileSync(file, `${lead}${[...header, ...fresh].join("\n")}\n`);+}++function trackedByGit(cwd: string, files: string[]): string[] {+ try {+ return execFileSync("git", ["ls-files", "--", ...files], {+ cwd,+ encoding: "utf8",+ stdio: ["ignore", "pipe", "ignore"],+ })+ .split("\n")+ .filter(Boolean);+ } catch {+ return [];+ }+}++function sortKeys(assets: AssetManifest): AssetManifest {+ return Object.fromEntries(+ Object.entries(assets).sort(([a], [b]) => a.localeCompare(b)),+ );+}++export function megabytes(bytes: number): string {+ return `${(bytes / 1e6).toFixed(bytes < 1e7 ? 1 : 0)} MB`;+}++const ASSETS_USAGE = `Usage: framefields assets add <path...> [--remote <name>]+ framefields assets push | pull [--overwrite] | status [--json]+`;++/** The `framefields assets` command. */+export async function runAssets(+ [sub, ...paths]: string[],+ options: AssetOptions & {+ remote?: string;+ overwrite?: boolean;+ json?: boolean;+ },+): Promise<number> {+ const log = options.log ?? (() => {});+ if (sub === "add" && paths.length > 0) {+ const added = await addAssets(paths, options);+ const sent = added.filter((a) => a.uploaded).length;+ log(+ `Recorded ${added.length} asset${added.length === 1 ? "" : "s"} in ${CONFIG_FILE} (${sent} uploaded, ${added.length - sent} already on the bucket). Commit ${CONFIG_FILE} and .gitignore.`,+ );+ return 0;+ }+ if (sub === "push") {+ const { uploaded, present } = await pushAssets(options);+ log(+ `Uploaded ${uploaded.length}; ${present.length} already on the bucket.`,+ );+ return 0;+ }+ if (sub === "pull") {+ const { downloaded, present, changed } = await pullAssets(options);+ log(`Downloaded ${downloaded.length}; ${present.length} already here.`);+ if (changed.length)+ log(+ `Changed here, left alone: ${changed.join(", ")}. \`framefields assets add\` records them, --overwrite restores them.`,+ );+ return changed.length ? 1 : 0;+ }+ if (sub === "status") {+ const entries = await assetStatus(options.cwd);+ if (options.json) log(JSON.stringify({ assets: entries }));+ else if (entries.length === 0) log(`No assets in ${CONFIG_FILE}.`);+ else+ for (const e of entries)+ log(`${e.state.padEnd(8)} ${e.path} (${megabytes(e.size)})`);+ return entries.some((e) => e.state !== "ok") ? 1 : 0;+ }+ process.stderr.write(ASSETS_USAGE);+ return 2;+}diff --git a/packages/framefields/src/cli/check-assets.ts b/packages/framefields/src/cli/check-assets.tsindex 8a0cb238..0370b9a6 100644--- a/packages/framefields/src/cli/check-assets.ts+++ b/packages/framefields/src/cli/check-assets.ts@@ -22,7 +22,7 @@ export function checkLocalAssets(stage: "assets",path,...(layerId && { layerId }),- message: `Local asset '${source}' does not exist. Restore it or correct the source path relative to the project working directory.`,+ message: `Local asset '${source}' does not exist. Restore it (\`npx framefields assets pull\` when framefields.json pins it) or correct the source path relative to the project working directory.`,});};for (const [i, font] of (spec.fonts ?? []).entries())diff --git a/packages/framefields/src/cli/cloud.ts b/packages/framefields/src/cli/cloud.tsindex 8ac8e6a8..d81b1767 100644--- a/packages/framefields/src/cli/cloud.ts+++ b/packages/framefields/src/cli/cloud.ts@@ -93,6 +93,7 @@ export async function runCloudRender(signal: controller.signal,onProgress: narrate,onWarning: (m) => log(`warning: ${m}`),+ onLog: log,});if (o.json) {log(JSON.stringify(result));diff --git a/packages/framefields/src/cli/config.ts b/packages/framefields/src/cli/config.tsindex 206091d5..d9eeb412 100644--- a/packages/framefields/src/cli/config.ts+++ b/packages/framefields/src/cli/config.ts@@ -20,9 +20,18 @@* A named render is what `framefields render film --cloud`, `framefields* check film`, `renderCloud({ render: "film" })` and the dashboard's New* render dialog pick, so nobody chooses files and exports by hand.+ *+ * `storage` and `assets` pin media by content on an S3-compatible bucket+ * (`framefields assets`; the format lives in `@framefields/storage`).*/import { existsSync, readFileSync, writeFileSync } from "node:fs";import { join } from "node:path";+import {+ type AssetManifest,+ parseAssets,+ parseStorage,+ type StorageConfig,+} from "@framefields/storage";export const CONFIG_FILE = "framefields.json";@@ -58,6 +67,10 @@ export interface ProjectConfig {fonts?: Partial<Record<string, string>>;/** Named renders, by name (letters, digits, `-`, `_`, `.`). */renders?: Record<string, RenderSpec>;+ /** Buckets holding the project's assets. */+ storage?: StorageConfig;+ /** Files the project needs, by path, pinned by sha256 and size. */+ assets?: AssetManifest;}export const RENDER_NAME = /^[A-Za-z0-9][\w.-]{0,63}$/;@@ -87,7 +100,7 @@ export function readConfig(cwd: string): {);}if (!isRecord(raw)) throw new Error(`${CONFIG_FILE} must be a JSON object`);- const { components, fonts, renders } = raw;+ const { components, fonts, renders, storage, assets } = raw;if (components !== undefined && typeof components !== "string")throw new Error(`${CONFIG_FILE}: "components" must be a directory path`);if (@@ -98,11 +111,17 @@ export function readConfig(cwd: string): {throw new Error(`${CONFIG_FILE}: "fonts" must map roles to font file paths, e.g. { "emoji": "assets/fonts/NotoColorEmoji.ttf" }`,);+ const parsedStorage =+ storage === undefined ? undefined : parseStorage(storage);// Other keys (a `$schema`, say) survive a rewrite.const config = {...raw,components: components ?? fallback,...(renders !== undefined && { renders: parseRenders(renders) }),+ ...(parsedStorage && { storage: parsedStorage }),+ ...(assets !== undefined && {+ assets: parseAssets(assets, parsedStorage),+ }),} as ProjectConfig;return { config, saved: components !== undefined };}diff --git a/packages/framefields/src/cloud/index.ts b/packages/framefields/src/cloud/index.tsindex ddf81ead..01f54528 100644--- a/packages/framefields/src/cloud/index.ts+++ b/packages/framefields/src/cloud/index.ts@@ -28,6 +28,7 @@ import path from "node:path";import { Readable } from "node:stream";import { pipeline } from "node:stream/promises";import type { ReadableStream as WebReadableStream } from "node:stream/web";+import { AssetsError, prepareCloudAssets } from "../cli/assets.js";import { CONFIG_FILE, findRender, readConfig } from "../cli/config.js";import { type CloudSource, resolveGitSource } from "./git-source.js";@@ -68,6 +69,8 @@ export interface CloudRenderOptions {onProgress?: (status: CloudRenderStatus) => void;/** Called for each warning (uncommitted changes). Default: stderr. */onWarning?: (message: string) => void;+ /** Called as assets upload before the render. Default: stderr. */+ onLog?: (message: string) => void;/** Aborting cancels the cloud render. */signal?: AbortSignal;/** Status poll interval, ms. Default 3000. */@@ -152,9 +155,12 @@ export async function renderCloud(options.onWarning ??((m: string) => process.stderr.write(`framefields: ${m}\n`));for (const w of resolved.warnings) warn(w);+ const assets = await cloudAssets(cwd, options);let status = await api.start({- source: resolved.source,+ source: Object.keys(assets).length+ ? { ...resolved.source, assets }+ : resolved.source,target: {entry: resolved.entry,export: o.export ?? "buildFilm",@@ -206,6 +212,29 @@ export async function renderCloud(return status;}+/**+ * Uploads assets the bucket lacks and signs reads of private ones: the+ * workers download each asset framefields.json pins at the commit.+ */+async function cloudAssets(+ cwd: string,+ options: CloudRenderOptions,+): Promise<Record<string, string>> {+ try {+ return await prepareCloudAssets({+ cwd,+ signal: options.signal,+ log:+ options.onLog ??+ ((m: string) => process.stderr.write(`framefields: ${m}\n`)),+ });+ } catch (err) {+ if (err instanceof AssetsError)+ throw new CloudRenderError(err.message, `assets_${err.code}`);+ throw err;+ }+}+/*** The options with a named render's fields filled in where the caller left* them out, plus the name and title the dashboard shows.diff --git a/packages/framefields/src/storage/index.ts b/packages/framefields/src/storage/index.tsnew file mode 100644index 00000000..7fb051fd--- /dev/null+++ b/packages/framefields/src/storage/index.ts@@ -0,0 +1,6 @@+/**+ * `framefields/storage`: a project's assets on any S3-compatible bucket+ * (Cloudflare R2, S3, B2, MinIO), stored by content and pinned in+ * `framefields.json`. The `framefields assets` commands are built on it.+ */+export * from "@framefields/storage";diff --git a/packages/framefields/tsdown.config.ts b/packages/framefields/tsdown.config.tsindex 68c0716a..c1f0b272 100644--- a/packages/framefields/tsdown.config.ts+++ b/packages/framefields/tsdown.config.ts@@ -16,6 +16,7 @@ export default defineConfig({"src/components/index.ts","src/sam/index.ts","src/cloud/index.ts",+ "src/storage/index.ts","src/cli.ts","src/cli/check-worker.ts",],@@ -61,6 +62,7 @@ export default defineConfig({"pino","file-type","dotenv",+ "aws4fetch","@huggingface/tokenizers",],});diff --git a/packages/storage/package.json b/packages/storage/package.jsonnew file mode 100644index 00000000..7e6ca33c--- /dev/null+++ b/packages/storage/package.json@@ -0,0 +1,39 @@+{+ "name": "@framefields/storage",+ "version": "2.12.0",+ "private": true,+ "description": "Content-addressed asset storage for framefields projects on any S3-compatible bucket (Cloudflare R2, S3, B2, MinIO).",+ "type": "module",+ "sideEffects": false,+ "license": "Apache-2.0",+ "module": "./dist/index.mjs",+ "types": "./dist/index.d.mts",+ "exports": {+ ".": {+ "types": "./dist/index.d.mts",+ "development": "./src/index.ts",+ "import": "./dist/index.mjs",+ "default": "./dist/index.mjs"+ }+ },+ "files": [+ "dist/**"+ ],+ "scripts": {+ "build": "tsdown",+ "dev": "tsdown --watch",+ "typecheck": "tsc --noEmit -p .",+ "test": "vitest run",+ "clean": "rm -rf .turbo && rm -rf node_modules && rm -rf dist"+ },+ "devDependencies": {+ "@framefields/tsconfig": "workspace:*",+ "@types/node": "^24.1.0",+ "tsdown": "^0.19.0",+ "typescript": "^5.8.3",+ "vitest": "^4.1.5"+ },+ "dependencies": {+ "aws4fetch": "^1.0.20"+ }+}diff --git a/packages/storage/src/bucket.ts b/packages/storage/src/bucket.tsnew file mode 100644index 00000000..bac8986e--- /dev/null+++ b/packages/storage/src/bucket.ts@@ -0,0 +1,302 @@+/**+ * The few S3 calls asset storage needs (HEAD, PUT, multipart upload,+ * presigned GET), signed with SigV4 by aws4fetch. Works with any+ * S3-compatible service; R2 is the one to pick, since it charges no egress.+ */+import { createHash } from "node:crypto";+import { open } from "node:fs/promises";+import { AwsClient } from "aws4fetch";+import { contentType } from "./files.js";+import { objectKey, type StorageRemote } from "./manifest.js";++export interface Credentials {+ accessKeyId: string;+ secretAccessKey: string;+ sessionToken?: string;+}++/**+ * Keys for `remote` from the environment:+ * `FRAMEFIELDS_STORAGE_<REMOTE>_ACCESS_KEY_ID` / `…_SECRET_ACCESS_KEY`+ * (the remote's name upper-cased, `-` as `_`), else the standard+ * `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`.+ */+export function credentialsFromEnv(+ remote: string,+ env: Readonly<Record<string, string | undefined>> = process.env,+): Credentials | undefined {+ const scope = `FRAMEFIELDS_STORAGE_${remote.toUpperCase().replace(/-/g, "_")}`;+ const own = env[`${scope}_ACCESS_KEY_ID`];+ const ownSecret = env[`${scope}_SECRET_ACCESS_KEY`];+ if (own && ownSecret) return { accessKeyId: own, secretAccessKey: ownSecret };+ const id = env.AWS_ACCESS_KEY_ID;+ const secret = env.AWS_SECRET_ACCESS_KEY;+ if (!id || !secret) return undefined;+ return {+ accessKeyId: id,+ secretAccessKey: secret,+ ...(env.AWS_SESSION_TOKEN && { sessionToken: env.AWS_SESSION_TOKEN }),+ };+}++/** The env vars `credentialsFromEnv` reads for `remote`, for messages. */+export function credentialEnvNames(remote: string): [string, string] {+ const scope = `FRAMEFIELDS_STORAGE_${remote.toUpperCase().replace(/-/g, "_")}`;+ return [`${scope}_ACCESS_KEY_ID`, `${scope}_SECRET_ACCESS_KEY`];+}++export class StorageError extends Error {+ override name = "StorageError";+ constructor(+ message: string,+ readonly status?: number,+ ) {+ super(message);+ }+}++export interface UploadProgress {+ sent: number;+ total: number;+}++export interface BucketOptions {+ fetch?: typeof fetch;+ /** Files above this go up in parts. Default 64 MiB. */+ multipartThreshold?: number;+ /** Default 16 MiB; grows so a file never needs more than 10,000 parts. */+ partSize?: number;+ /** Parts in flight at once. Default 4. */+ concurrency?: number;+}++const MiB = 1024 * 1024;+/**+ * Content-addressed objects never change: CDNs and browsers may keep them+ * forever, which is what makes a custom domain in front of the bucket cheap.+ */+const IMMUTABLE = "public, max-age=31536000, immutable";++/** One remote's bucket, signed with `credentials`. */+export class Bucket {+ private readonly aws: AwsClient;+ private readonly fetch: typeof fetch;++ constructor(+ readonly remote: StorageRemote,+ credentials: Credentials,+ private readonly options: BucketOptions = {},+ ) {+ this.aws = new AwsClient({+ ...credentials,+ service: "s3",+ region: remote.region ?? "auto",+ retries: 3,+ });+ this.fetch = options.fetch ?? fetch;+ }++ keyFor(sha256: string): string {+ return objectKey(sha256, this.remote.prefix);+ }++ /** Whether the bucket holds `sha256` at the expected size. */+ async has(sha256: string, size: number): Promise<boolean> {+ const res = await this.send(this.keyFor(sha256), { method: "HEAD" });+ if (res.status === 404) return false;+ if (!res.ok) throw await this.failure("HEAD", res);+ return Number(res.headers.get("content-length")) === size;+ }++ /**+ * Uploads `file` (already hashed) under its content key. The bucket+ * checks every byte against the sha256 sent with it.+ */+ async upload(+ file: string,+ digest: { sha256: string; size: number },+ onProgress?: (p: UploadProgress) => void,+ ): Promise<void> {+ const key = this.keyFor(digest.sha256);+ const headers = {+ "content-type": contentType(file),+ "cache-control": IMMUTABLE,+ };+ const threshold = this.options.multipartThreshold ?? 64 * MiB;+ if (digest.size <= threshold) {+ const handle = await open(file);+ try {+ const body = await handle.readFile();+ await this.put(key, body, digest.sha256, headers);+ } finally {+ await handle.close();+ }+ onProgress?.({ sent: digest.size, total: digest.size });+ return;+ }+ await this.uploadParts(file, key, digest.size, headers, onProgress);+ }++ /** A GET URL anyone can use until it expires (at most 7 days). */+ async presign(sha256: string, expiresSec = 86_400): Promise<string> {+ const url = new URL(this.url(this.keyFor(sha256)));+ url.searchParams.set(+ "X-Amz-Expires",+ String(Math.min(expiresSec, 604_800)),+ );+ const signed = await this.aws.sign(url, {+ method: "GET",+ aws: { signQuery: true },+ });+ return signed.url;+ }++ private async put(+ key: string,+ body: Uint8Array,+ sha256: string,+ headers: Record<string, string>,+ query = "",+ ): Promise<Response> {+ const res = await this.send(key, {+ method: "PUT",+ body,+ headers: { ...headers, "x-amz-content-sha256": sha256 },+ query,+ });+ if (!res.ok) throw await this.failure("PUT", res);+ return res;+ }++ private async uploadParts(+ file: string,+ key: string,+ size: number,+ headers: Record<string, string>,+ onProgress?: (p: UploadProgress) => void,+ ): Promise<void> {+ const partSize = Math.max(+ this.options.partSize ?? 16 * MiB,+ Math.ceil(size / 10_000),+ 5 * MiB,+ );+ const created = await this.send(key, {+ method: "POST",+ query: "uploads=",+ headers,+ });+ if (!created.ok) throw await this.failure("CreateMultipartUpload", created);+ const uploadId = /<UploadId>([^<]+)<\/UploadId>/.exec(+ await created.text(),+ )?.[1];+ if (!uploadId)+ throw new StorageError("CreateMultipartUpload returned no UploadId");+ const id = encodeURIComponent(uploadId);+ const count = Math.ceil(size / partSize);+ const etags: string[] = new Array(count);+ const handle = await open(file);+ let next = 0;+ let sent = 0;+ const worker = async () => {+ while (next < count) {+ const index = next++;+ const length = Math.min(partSize, size - index * partSize);+ const chunk = Buffer.alloc(length);+ await handle.read(chunk, 0, length, index * partSize);+ const sha = createHash("sha256").update(chunk).digest("hex");+ const res = await this.put(+ key,+ chunk,+ sha,+ {},+ `partNumber=${index + 1}&uploadId=${id}`,+ );+ etags[index] = res.headers.get("etag") ?? "";+ sent += length;+ onProgress?.({ sent, total: size });+ }+ };+ try {+ await Promise.all(+ Array.from(+ { length: Math.min(this.options.concurrency ?? 4, count) },+ worker,+ ),+ );+ const body = `<CompleteMultipartUpload>${etags+ .map(+ (etag, i) =>+ `<Part><PartNumber>${i + 1}</PartNumber><ETag>${escapeXml(etag)}</ETag></Part>`,+ )+ .join("")}</CompleteMultipartUpload>`;+ const done = await this.send(key, {+ method: "POST",+ query: `uploadId=${id}`,+ body,+ headers: { "content-type": "application/xml" },+ });+ // S3 can answer 200 with an <Error> body.+ const text = await done.text();+ if (!done.ok || text.includes("<Error>"))+ throw new StorageError(+ `CompleteMultipartUpload failed (${done.status}): ${errorCode(text)}`,+ done.status,+ );+ } catch (err) {+ await this.send(key, { method: "DELETE", query: `uploadId=${id}` }).catch(+ () => undefined,+ );+ throw err;+ } finally {+ await handle.close();+ }+ }++ private url(key: string, query = ""): string {+ const base = this.remote.endpoint.replace(/\/+$/, "");+ const path = key.split("/").map(encodeURIComponent).join("/");+ return `${base}/${this.remote.bucket}/${path}${query ? `?${query}` : ""}`;+ }++ private async send(+ key: string,+ init: {+ method: string;+ query?: string;+ body?: Uint8Array | string;+ headers?: Record<string, string>;+ },+ ): Promise<Response> {+ const signed = await this.aws.sign(this.url(key, init.query), {+ method: init.method,+ headers: init.headers,+ body: init.body,+ });+ return this.fetch(signed);+ }++ private async failure(what: string, res: Response): Promise<StorageError> {+ const text = await res.text().catch(() => "");+ const hint =+ res.status === 403+ ? ": check the remote's keys and that they may write to the bucket"+ : res.status === 404+ ? ": check the remote's endpoint and bucket"+ : "";+ return new StorageError(+ `${what} ${this.remote.bucket} failed (${res.status} ${errorCode(text)})${hint}`,+ res.status,+ );+ }+}++function errorCode(xml: string): string {+ return /<Code>([^<]+)<\/Code>/.exec(xml)?.[1] ?? "no error code";+}++function escapeXml(text: string): string {+ return text+ .replace(/&/g, "&")+ .replace(/</g, "<")+ .replace(/"/g, """);+}diff --git a/packages/storage/src/files.ts b/packages/storage/src/files.tsnew file mode 100644index 00000000..23be148b--- /dev/null+++ b/packages/storage/src/files.ts@@ -0,0 +1,185 @@+/**+ * Hashing and verified downloads: every byte an asset contributes is checked+ * against the sha256 its manifest entry pins.+ */+import { createHash, randomBytes } from "node:crypto";+import { createReadStream, createWriteStream } from "node:fs";+import { mkdir, rename, rm } from "node:fs/promises";+import { dirname } from "node:path";+import { Readable, Transform } from "node:stream";+import { pipeline } from "node:stream/promises";+import type { ReadableStream as WebReadableStream } from "node:stream/web";++export type AssetErrorCode =+ | "ASSET_NOT_FOUND"+ | "ASSET_FORBIDDEN"+ | "ASSET_DOWNLOAD_FAILED"+ | "ASSET_HASH_MISMATCH";++export class AssetError extends Error {+ override name = "AssetError";+ constructor(+ readonly code: AssetErrorCode,+ message: string,+ options?: { cause?: unknown },+ ) {+ super(message, options);+ }+}++export interface FileDigest {+ sha256: string;+ size: number;+}++export async function hashFile(path: string): Promise<FileDigest> {+ const hash = createHash("sha256");+ let size = 0;+ for await (const chunk of createReadStream(path)) {+ hash.update(chunk as Buffer);+ size += (chunk as Buffer).length;+ }+ return { sha256: hash.digest("hex"), size };+}++export interface DownloadOptions {+ fetch?: typeof fetch;+ signal?: AbortSignal;+ /** Attempts for network errors and 5xx/429 responses. Default 3. */+ attempts?: number;+}++/**+ * Downloads `url` to `dest` through a temporary file, and only renames it+ * into place once its size and sha256 match `expected`.+ */+export async function downloadVerified(+ url: string,+ dest: string,+ expected: FileDigest,+ options: DownloadOptions = {},+): Promise<void> {+ const attempts = options.attempts ?? 3;+ for (let attempt = 1; ; attempt++) {+ try {+ await downloadOnce(url, dest, expected, options);+ return;+ } catch (err) {+ const transient =+ err instanceof AssetError+ ? err.code === "ASSET_DOWNLOAD_FAILED"+ : !options.signal?.aborted;+ if (!transient || attempt >= attempts) throw err;+ await new Promise((r) => setTimeout(r, 500 * 2 ** attempt));+ }+ }+}++async function downloadOnce(+ url: string,+ dest: string,+ expected: FileDigest,+ options: DownloadOptions,+): Promise<void> {+ const shown = redactQuery(url);+ const res = await (options.fetch ?? fetch)(url, { signal: options.signal });+ if (!res.ok || !res.body) {+ await res.body?.cancel();+ if (res.status === 404)+ throw new AssetError("ASSET_NOT_FOUND", `${shown} returned 404`);+ if (res.status === 401 || res.status === 403)+ throw new AssetError(+ "ASSET_FORBIDDEN",+ `${shown} returned ${res.status}: the bucket isn't public there, or the signed URL expired`,+ );+ throw new AssetError(+ "ASSET_DOWNLOAD_FAILED",+ `${shown} returned ${res.status}`,+ );+ }+ await mkdir(dirname(dest), { recursive: true });+ const temp = `${dest}.${randomBytes(4).toString("hex")}.part`;+ const hash = createHash("sha256");+ let size = 0;+ const tap = new Transform({+ transform(chunk: Buffer, _encoding, done) {+ size += chunk.length;+ if (size > expected.size)+ return done(+ new AssetError(+ "ASSET_HASH_MISMATCH",+ `${shown} is larger than the ${expected.size} bytes framefields.json pins`,+ ),+ );+ hash.update(chunk);+ done(null, chunk);+ },+ });+ try {+ await pipeline(+ Readable.fromWeb(res.body as WebReadableStream<Uint8Array>),+ tap,+ createWriteStream(temp),+ { signal: options.signal },+ ).catch((err: unknown) => {+ if (err instanceof AssetError || options.signal?.aborted) throw err;+ throw new AssetError(+ "ASSET_DOWNLOAD_FAILED",+ `${shown}: ${(err as Error).message}`,+ { cause: err },+ );+ });+ const sha256 = hash.digest("hex");+ if (size !== expected.size || sha256 !== expected.sha256)+ throw new AssetError(+ "ASSET_HASH_MISMATCH",+ `${shown} is not the file framefields.json pins (got ${size} bytes, sha256 ${sha256.slice(0, 12)}…; expected ${expected.size} bytes, ${expected.sha256.slice(0, 12)}…)`,+ );+ await rename(temp, dest);+ } finally {+ await rm(temp, { force: true });+ }+}++/** Signed URLs carry credentials in the query: never print them. */+export function redactQuery(url: string): string {+ const q = url.indexOf("?");+ return q < 0 ? url : `${url.slice(0, q)}?…`;+}++const TYPES: Record<string, string> = {+ mp4: "video/mp4",+ m4v: "video/mp4",+ mov: "video/quicktime",+ webm: "video/webm",+ mkv: "video/x-matroska",+ mp3: "audio/mpeg",+ m4a: "audio/mp4",+ aac: "audio/aac",+ wav: "audio/wav",+ flac: "audio/flac",+ ogg: "audio/ogg",+ png: "image/png",+ jpg: "image/jpeg",+ jpeg: "image/jpeg",+ webp: "image/webp",+ avif: "image/avif",+ gif: "image/gif",+ svg: "image/svg+xml",+ json: "application/json",+ lottie: "application/zip",+ glb: "model/gltf-binary",+ gltf: "model/gltf+json",+ ttf: "font/ttf",+ otf: "font/otf",+ woff2: "font/woff2",+ cube: "text/plain",+ srt: "text/plain",+ vtt: "text/vtt",+};++/** The Content-Type a CDN serves the file with. */+export function contentType(path: string): string {+ const ext = path.slice(path.lastIndexOf(".") + 1).toLowerCase();+ return TYPES[ext] ?? "application/octet-stream";+}diff --git a/packages/storage/src/index.ts b/packages/storage/src/index.tsnew file mode 100644index 00000000..202bc420--- /dev/null+++ b/packages/storage/src/index.ts@@ -0,0 +1,35 @@+export {+ Bucket,+ type BucketOptions,+ type Credentials,+ credentialEnvNames,+ credentialsFromEnv,+ StorageError,+ type UploadProgress,+} from "./bucket.js";+export {+ AssetError,+ type AssetErrorCode,+ contentType,+ type DownloadOptions,+ downloadVerified,+ type FileDigest,+ hashFile,+ redactQuery,+} from "./files.js";+export {+ type AssetEntry,+ type AssetManifest,+ isAssetPath,+ ManifestError,+ objectKey,+ parseAssets,+ parseStorage,+ pickRemote,+ publicUrl,+ REMOTE_NAME,+ readUrl,+ SHA256,+ type StorageConfig,+ type StorageRemote,+} from "./manifest.js";diff --git a/packages/storage/src/manifest.ts b/packages/storage/src/manifest.tsnew file mode 100644index 00000000..1b39467a--- /dev/null+++ b/packages/storage/src/manifest.ts@@ -0,0 +1,257 @@+/**+ * The `storage` and `assets` sections of a project's `framefields.json`.+ *+ * ```json+ * {+ * "storage": {+ * "remotes": {+ * "media": {+ * "endpoint": "https://<account>.r2.cloudflarestorage.com",+ * "bucket": "films",+ * "publicUrl": "https://media.example.com"+ * }+ * }+ * },+ * "assets": {+ * "assets/footage/intro.mp4": { "sha256": "…", "size": 48213 }+ * }+ * }+ * ```+ *+ * Assets are stored by content (`sha256/ab/<hash>`), so the manifest pins+ * exact bytes the way a lockfile pins packages, and the files themselves+ * stay out of git.+ */++/** An S3-compatible bucket: Cloudflare R2, AWS S3, Backblaze B2, MinIO, … */+export interface StorageRemote {+ /** The S3 API endpoint, e.g. `https://<account>.r2.cloudflarestorage.com`. */+ endpoint: string;+ bucket: string;+ /** SigV4 region. Default `auto` (R2); AWS needs the bucket's region. */+ region?: string;+ /** Key prefix inside the bucket, e.g. `films/launch/`. */+ prefix?: string;+ /**+ * Where the bucket is publicly readable: a custom domain (an R2 custom+ * domain or any CDN in front of the bucket) or the `r2.dev` URL. Reads use+ * it instead of signed URLs, so anyone can pull and render without keys.+ */+ publicUrl?: string;+}++export interface StorageConfig {+ /** The remote new assets go to. Default: the only remote. */+ default?: string;+ remotes: Record<string, StorageRemote>;+}++/** One file the project needs, pinned by content. */+export interface AssetEntry {+ sha256: string;+ size: number;+ /** The remote holding it. Default: `storage.default`. */+ remote?: string;+ /** A fixed https URL to fetch it from instead of a remote (public media). */+ url?: string;+}++export type AssetManifest = Record<string, AssetEntry>;++export const REMOTE_NAME = /^[A-Za-z0-9][\w-]{0,63}$/;+export const SHA256 = /^[0-9a-f]{64}$/;+const FILE = "framefields.json";++export class ManifestError extends Error {+ override name = "ManifestError";+}++/** Validates `storage`; errors name the remote and field at fault. */+export function parseStorage(value: unknown): StorageConfig {+ if (!isRecord(value) || !isRecord(value.remotes))+ throw new ManifestError(+ `${FILE}: "storage" needs "remotes", e.g. { "remotes": { "media": { "endpoint": "https://<account>.r2.cloudflarestorage.com", "bucket": "films" } } }`,+ );+ const remotes: Record<string, StorageRemote> = {};+ for (const [name, spec] of Object.entries(value.remotes))+ remotes[name] = parseRemote(name, spec);+ const names = Object.keys(remotes);+ if (names.length === 0)+ throw new ManifestError(`${FILE}: "storage.remotes" is empty`);+ if (value.default !== undefined && !names.includes(value.default as string))+ throw new ManifestError(+ `${FILE}: "storage.default" must name a remote (${names.join(", ")})`,+ );+ return {+ ...(typeof value.default === "string" && { default: value.default }),+ remotes,+ };+}++function parseRemote(name: string, spec: unknown): StorageRemote {+ const fail = (message: string): never => {+ throw new ManifestError(`${FILE}: storage.remotes.${name} ${message}`);+ };+ if (!REMOTE_NAME.test(name))+ fail("has an invalid name: use letters, digits, - and _ (at most 64)");+ if (!isRecord(spec)) fail("must be an object with an endpoint and a bucket");+ const s = spec as Record<string, unknown>;+ if (!isHttpUrl(s.endpoint))+ fail(+ 'needs "endpoint": the S3 API URL, e.g. "https://<account>.r2.cloudflarestorage.com"',+ );+ if (+ typeof s.bucket !== "string" ||+ !/^[a-z0-9][a-z0-9.-]{1,62}$/.test(s.bucket)+ )+ fail('needs "bucket": the bucket name');+ if (s.region !== undefined && typeof s.region !== "string")+ fail('"region" must be a string');+ if (+ s.prefix !== undefined &&+ (typeof s.prefix !== "string" ||+ s.prefix.startsWith("/") ||+ s.prefix.split("/").includes(".."))+ )+ fail('"prefix" must be a key prefix such as "films/launch/"');+ if (s.publicUrl !== undefined && !isHttpUrl(s.publicUrl))+ fail(+ '"publicUrl" must be an http(s) URL, e.g. "https://media.example.com"',+ );+ return {+ endpoint: s.endpoint as string,+ bucket: s.bucket as string,+ ...(s.region !== undefined && { region: s.region as string }),+ ...(typeof s.prefix === "string" &&+ s.prefix && { prefix: normalizePrefix(s.prefix) }),+ ...(s.publicUrl !== undefined && { publicUrl: s.publicUrl as string }),+ };+}++/** Validates `assets`; errors name the path and field at fault. */+export function parseAssets(+ value: unknown,+ storage?: StorageConfig,+): AssetManifest {+ if (!isRecord(value))+ throw new ManifestError(+ `${FILE}: "assets" must map project paths to { "sha256", "size" }`,+ );+ const out: AssetManifest = {};+ for (const [path, entry] of Object.entries(value)) {+ const fail = (message: string): never => {+ throw new ManifestError(`${FILE}: assets["${path}"] ${message}`);+ };+ if (!isAssetPath(path))+ fail("must be a path inside the project, e.g. assets/footage/intro.mp4");+ if (!isRecord(entry)) fail('must be { "sha256", "size" }');+ const e = entry as Record<string, unknown>;+ if (typeof e.sha256 !== "string" || !SHA256.test(e.sha256))+ fail('needs "sha256": 64 lowercase hex digits');+ if (+ typeof e.size !== "number" ||+ !Number.isSafeInteger(e.size) ||+ e.size < 0+ )+ fail('needs "size" in bytes');+ if (e.url !== undefined && !isHttpUrl(e.url))+ fail('"url" must be an http(s) URL');+ if (e.remote !== undefined) {+ if (typeof e.remote !== "string" || !storage?.remotes[e.remote])+ fail(`"remote" must name a remote in "storage.remotes"`);+ } else if (e.url === undefined && !storage)+ fail('needs a "url", or a "storage" section with a remote to hold it');+ out[path] = {+ sha256: e.sha256 as string,+ size: e.size as number,+ ...(e.remote !== undefined && { remote: e.remote as string }),+ ...(e.url !== undefined && { url: e.url as string }),+ };+ }+ return out;+}++/** The remote an asset (or a new one, without `name`) belongs to. */+export function pickRemote(+ storage: StorageConfig | undefined,+ name?: string,+): { name: string; remote: StorageRemote } {+ const names = Object.keys(storage?.remotes ?? {});+ const chosen =+ name ?? storage?.default ?? (names.length === 1 ? names[0] : undefined);+ if (!storage || names.length === 0)+ throw new ManifestError(+ `${FILE} has no "storage" remotes: add one (an R2 or S3 bucket) first`,+ );+ if (!chosen)+ throw new ManifestError(+ `${FILE}: name the remote (${names.join(", ")}), or set "storage.default"`,+ );+ const remote = storage.remotes[chosen];+ if (!remote)+ throw new ManifestError(+ `${FILE}: no remote named "${chosen}" (${names.join(", ")})`,+ );+ return { name: chosen, remote };+}++/** The object key holding `sha256`: `<prefix>sha256/ab/<hash>`. */+export function objectKey(sha256: string, prefix = ""): string {+ return `${prefix}sha256/${sha256.slice(0, 2)}/${sha256}`;+}++/** The asset's URL on the remote's public domain, if it has one. */+export function publicUrl(+ remote: StorageRemote,+ sha256: string,+): string | undefined {+ if (!remote.publicUrl) return undefined;+ return `${remote.publicUrl.replace(/\/+$/, "")}/${objectKey(sha256, remote.prefix)}`;+}++/**+ * Where to download an asset without credentials: a signed URL handed in+ * (`signed`, by sha256), its own `url`, or its remote's public domain.+ * Undefined means the asset is on a private remote and needs a signature.+ */+export function readUrl(+ entry: AssetEntry,+ storage: StorageConfig | undefined,+ signed?: Readonly<Record<string, string>>,+): string | undefined {+ const given = signed?.[entry.sha256];+ if (given) return given;+ if (entry.url) return entry.url;+ if (!storage) return undefined;+ return publicUrl(pickRemote(storage, entry.remote).remote, entry.sha256);+}++export function isAssetPath(path: string): boolean {+ return (+ path.length > 0 &&+ path.length <= 1024 &&+ !path.startsWith("/") &&+ !path.includes("\\") &&+ !path+ .split("/")+ .some((part) => part === "" || part === "." || part === "..")+ );+}++function normalizePrefix(prefix: string): string {+ return prefix.endsWith("/") ? prefix : `${prefix}/`;+}++function isHttpUrl(value: unknown): boolean {+ if (typeof value !== "string") return false;+ try {+ const { protocol } = new URL(value);+ return protocol === "https:" || protocol === "http:";+ } catch {+ return false;+ }+}++function isRecord(value: unknown): value is Record<string, unknown> {+ return typeof value === "object" && value !== null && !Array.isArray(value);+}diff --git a/packages/storage/src/storage.test.ts b/packages/storage/src/storage.test.tsnew file mode 100644index 00000000..f84a4517--- /dev/null+++ b/packages/storage/src/storage.test.ts@@ -0,0 +1,281 @@+import { createHash } from "node:crypto";+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";+import { tmpdir } from "node:os";+import { join } from "node:path";+import { afterEach, beforeEach, describe, expect, it } from "vitest";+import {+ AssetError,+ Bucket,+ credentialsFromEnv,+ downloadVerified,+ hashFile,+ parseAssets,+ parseStorage,+ readUrl,+ type StorageRemote,+} from "./index.js";++let dir: string;+beforeEach(() => {+ dir = mkdtempSync(join(tmpdir(), "ff-storage-"));+});+afterEach(() => rmSync(dir, { recursive: true, force: true }));++const sha = (data: Uint8Array) =>+ createHash("sha256").update(data).digest("hex");++const r2: StorageRemote = {+ endpoint: "https://acct.r2.cloudflarestorage.com",+ bucket: "films",+ prefix: "launch/",+ publicUrl: "https://media.example.com/",+};++/** An in-memory S3 that verifies payload hashes the way S3 and R2 do. */+function fakeS3() {+ const objects = new Map<+ string,+ { body: Buffer; type: string | null; cache: string | null }+ >();+ const uploads = new Map<string, Map<number, Buffer>>();+ const requests: string[] = [];+ const handler = async (input: Request | string | URL): Promise<Response> => {+ const req = input as Request;+ const url = new URL(req.url);+ const key = decodeURIComponent(url.pathname);+ requests.push(+ `${req.method} ${url.pathname}${url.search ? `?${[...url.searchParams.keys()].join("&")}` : ""}`,+ );+ expect(req.headers.get("authorization")).toMatch(/^AWS4-HMAC-SHA256 /);+ const body = Buffer.from(await req.arrayBuffer());+ if (req.method === "HEAD") {+ const o = objects.get(key);+ return o+ ? new Response(null, {+ headers: { "content-length": String(o.body.length) },+ })+ : new Response(null, { status: 404 });+ }+ if (req.method === "PUT") {+ if (req.headers.get("x-amz-content-sha256") !== sha(body))+ return new Response(+ "<Error><Code>XAmzContentSHA256Mismatch</Code></Error>",+ { status: 400 },+ );+ const uploadId = url.searchParams.get("uploadId");+ if (uploadId) {+ uploads+ .get(uploadId)+ ?.set(Number(url.searchParams.get("partNumber")), body);+ return new Response(null, {+ headers: { etag: `"${sha(body).slice(0, 8)}"` },+ });+ }+ objects.set(key, {+ body,+ type: req.headers.get("content-type"),+ cache: req.headers.get("cache-control"),+ });+ return new Response(null);+ }+ if (req.method === "POST" && url.searchParams.has("uploads")) {+ uploads.set("u1", new Map());+ objects.set(`${key}#meta`, {+ body: Buffer.alloc(0),+ type: req.headers.get("content-type"),+ cache: req.headers.get("cache-control"),+ });+ return new Response(+ "<InitiateMultipartUploadResult><UploadId>u1</UploadId></InitiateMultipartUploadResult>",+ );+ }+ if (req.method === "POST") {+ const parts = uploads.get(url.searchParams.get("uploadId") ?? "");+ const numbers = [+ ...body.toString().matchAll(/<PartNumber>(\d+)<\/PartNumber>/g),+ ].map((m) => Number(m[1]));+ const joined = Buffer.concat(+ numbers.map((n) => parts?.get(n) ?? Buffer.alloc(0)),+ );+ const meta = objects.get(`${key}#meta`);+ objects.set(key, {+ body: joined,+ type: meta?.type ?? null,+ cache: meta?.cache ?? null,+ });+ return new Response("<CompleteMultipartUploadResult/>");+ }+ return new Response(null, { status: 405 });+ };+ return { fetch: handler as typeof fetch, objects, requests };+}++const creds = { accessKeyId: "AKID", secretAccessKey: "secret" };++describe("Bucket", () => {+ it("uploads by content, immutable and typed for a CDN, and skips what it has", async () => {+ const s3 = fakeS3();+ const bucket = new Bucket(r2, creds, { fetch: s3.fetch });+ const file = join(dir, "clip.mp4");+ writeFileSync(file, "frames");+ const digest = await hashFile(file);+ expect(await bucket.has(digest.sha256, digest.size)).toBe(false);+ await bucket.upload(file, digest);+ const key = `/films/launch/sha256/${digest.sha256.slice(0, 2)}/${digest.sha256}`;+ expect(s3.objects.get(key)).toEqual({+ body: Buffer.from("frames"),+ type: "video/mp4",+ cache: "public, max-age=31536000, immutable",+ });+ expect(await bucket.has(digest.sha256, digest.size)).toBe(true);+ expect(await bucket.has(digest.sha256, digest.size + 1)).toBe(false);+ });++ it("uploads large files in hash-checked parts", async () => {+ const s3 = fakeS3();+ const bucket = new Bucket(r2, creds, {+ fetch: s3.fetch,+ multipartThreshold: 10,+ partSize: 1,+ });+ const data = Buffer.alloc(12 * 1024 * 1024, 7);+ data.write("head", 0);+ data.write("tail", data.length - 4);+ const file = join(dir, "long.mov");+ writeFileSync(file, data);+ const digest = await hashFile(file);+ const seen: number[] = [];+ await bucket.upload(file, digest, (p) => seen.push(p.sent));+ const stored = s3.objects.get(`/films/${bucket.keyFor(digest.sha256)}`);+ expect(stored?.body.equals(data)).toBe(true);+ expect(stored?.type).toBe("video/quicktime");+ // 5 MiB minimum parts: 5 + 5 + 2.+ expect(s3.requests.filter((r) => r.includes("partNumber"))).toHaveLength(3);+ expect(seen.at(-1)).toBe(data.length);+ });++ it("presigns GETs without exposing keys and caps them at 7 days", async () => {+ const bucket = new Bucket(r2, creds);+ const url = new URL(await bucket.presign("ab".repeat(32), 30 * 86_400));+ expect(url.origin + url.pathname).toBe(+ `https://acct.r2.cloudflarestorage.com/films/launch/sha256/ab/${"ab".repeat(32)}`,+ );+ expect(url.searchParams.get("X-Amz-Expires")).toBe("604800");+ expect(url.searchParams.get("X-Amz-Signature")).toMatch(/^[0-9a-f]{64}$/);+ expect(url.search).not.toContain("secret");+ });+});++describe("manifest", () => {+ const storage = parseStorage({+ remotes: { media: r2, private: { endpoint: r2.endpoint, bucket: "raw" } },+ default: "media",+ });++ it("reads from the custom domain, a pinned url, or a signature", () => {+ const hash = "cd".repeat(32);+ expect(readUrl({ sha256: hash, size: 1 }, storage)).toBe(+ `https://media.example.com/launch/sha256/cd/${hash}`,+ );+ expect(+ readUrl({ sha256: hash, size: 1, url: "https://x.dev/a.mp4" }, storage),+ ).toBe("https://x.dev/a.mp4");+ expect(+ readUrl({ sha256: hash, size: 1, remote: "private" }, storage),+ ).toBeUndefined();+ expect(+ readUrl({ sha256: hash, size: 1, remote: "private" }, storage, {+ [hash]: "https://signed",+ }),+ ).toBe("https://signed");+ });++ it("names the field at fault", () => {+ expect(() =>+ parseStorage({ remotes: { media: { bucket: "films" } } }),+ ).toThrow('storage.remotes.media needs "endpoint"');+ expect(() =>+ parseStorage({+ remotes: { media: { ...r2, publicUrl: "media.example.com" } },+ }),+ ).toThrow('"publicUrl" must be an http(s) URL');+ expect(() =>+ parseAssets(+ { "../x.mp4": { sha256: "ab".repeat(32), size: 1 } },+ storage,+ ),+ ).toThrow("must be a path inside the project");+ expect(() =>+ parseAssets({ "a.mp4": { sha256: "AB", size: 1 } }, storage),+ ).toThrow('needs "sha256"');+ expect(() =>+ parseAssets(+ { "a.mp4": { sha256: "ab".repeat(32), size: 1, remote: "nope" } },+ storage,+ ),+ ).toThrow('"remote" must name a remote');+ expect(() =>+ parseAssets({ "a.mp4": { sha256: "ab".repeat(32), size: 1 } }),+ ).toThrow('needs a "url"');+ });++ it("takes keys per remote, then the AWS variables", () => {+ expect(+ credentialsFromEnv("raw-footage", {+ FRAMEFIELDS_STORAGE_RAW_FOOTAGE_ACCESS_KEY_ID: "a",+ FRAMEFIELDS_STORAGE_RAW_FOOTAGE_SECRET_ACCESS_KEY: "b",+ AWS_ACCESS_KEY_ID: "c",+ AWS_SECRET_ACCESS_KEY: "d",+ }),+ ).toEqual({ accessKeyId: "a", secretAccessKey: "b" });+ expect(+ credentialsFromEnv("media", {+ AWS_ACCESS_KEY_ID: "c",+ AWS_SECRET_ACCESS_KEY: "d",+ }),+ ).toEqual({+ accessKeyId: "c",+ secretAccessKey: "d",+ });+ expect(credentialsFromEnv("media", {})).toBeUndefined();+ });+});++describe("downloadVerified", () => {+ const serve = (body: string, status = 200) =>+ (async () =>+ new Response(status === 200 ? body : null, {+ status,+ })) as unknown as typeof fetch;++ it("writes the file only when it matches", async () => {+ const data = Buffer.from("pinned bytes");+ const dest = join(dir, "a/b.wav");+ await downloadVerified(+ "https://cdn/x",+ dest,+ { sha256: sha(data), size: data.length },+ { fetch: serve("pinned bytes") },+ );+ expect(readFileSync(dest, "utf8")).toBe("pinned bytes");+ const bad = downloadVerified(+ "https://cdn/x",+ join(dir, "c.wav"),+ { sha256: sha(data), size: data.length },+ { fetch: serve("pinned bytez") },+ );+ await expect(bad).rejects.toMatchObject({ code: "ASSET_HASH_MISMATCH" });+ });++ it("explains 403s and never prints signatures", async () => {+ const err = await downloadVerified(+ "https://acct.r2.cloudflarestorage.com/films/k?X-Amz-Signature=abc",+ join(dir, "d"),+ { sha256: "ab".repeat(32), size: 1 },+ { fetch: serve("", 403) },+ ).catch((e: unknown) => e);+ expect(err).toBeInstanceOf(AssetError);+ expect((err as AssetError).code).toBe("ASSET_FORBIDDEN");+ expect((err as AssetError).message).not.toContain("abc");+ });+});diff --git a/packages/storage/tsconfig.json b/packages/storage/tsconfig.jsonnew file mode 100644index 00000000..c00e3e04--- /dev/null+++ b/packages/storage/tsconfig.json@@ -0,0 +1,9 @@+{+ "extends": "@framefields/tsconfig/base.json",+ "compilerOptions": {+ "lib": ["ES2022", "ESNext"],+ "types": ["node"]+ },+ "include": ["src"],+ "exclude": ["dist", "node_modules"]+}diff --git a/packages/storage/tsdown.config.ts b/packages/storage/tsdown.config.tsnew file mode 100644index 00000000..94257a01--- /dev/null+++ b/packages/storage/tsdown.config.ts@@ -0,0 +1,10 @@+import { defineConfig } from "tsdown";++export default defineConfig({+ entry: ["src/index.ts"],+ format: ["esm"],+ dts: true,+ clean: true,+ sourcemap: true,+ treeshake: true,+});diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 7ea8f960..4ce3327b 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -226,7 +226,8 @@ api.framefields.dev renders the project on framefields' GPUs and splits a long f- Commit and push first. Uncommitted changes are not rendered (the CLI warns), a commit not on the remote is refused, and so is a folder outside a git repo or without a remote.- Run it from the project folder (where `framefields.json` and `package.json` are). In a monorepo, the CLI sends that folder as the workdir.-- The worker installs the project from its lockfile and renders with **the project's own framefields version**, so commit the lockfile. Footage, audio and fonts must be in Git LFS (or committed); `.gitignore`d files don't exist on the worker.+- The worker installs the project from its lockfile and renders with **the project's own framefields version**, so commit the lockfile. Footage, audio and fonts must be pinned in `framefields.json` with `framefields assets add`, in Git LFS, or committed; other `.gitignore`d files don't exist on the worker.+- Pinned assets: before starting, the CLI (and `renderCloud`) uploads any the bucket lacks and signs reads of assets on remotes without a `publicUrl` (valid 24 h; the keys stay on the machine). Each worker downloads them (cached by hash for its next job); on R2 those downloads are free, unlike LFS bandwidth. Dashboard renders have nobody to sign, so they read only remotes with a `publicUrl`.- Private repos: the user connects GitHub once at https://framefields.dev/dashboard/settings#connections (the framefields GitHub App, ideally on all repositories, so new repos render without another step). Public repos need nothing.- It bills the account behind `FRAMEFIELDS_API_KEY` (keys at https://framefields.dev/dashboard/keys). If authentication is required, ask the user to set it in the process environment; never print, write or embed it.@@ -278,6 +279,9 @@ The error carries the worker's code. Fix the project, commit, push and render ag| `ENTRY_NOT_FOUND`, `EXPORT_NOT_FOUND`, `EXPORT_INVALID` | `entry`/`export` in framefields.json or flags are wrong | `framefields check <name>` locally. || `COMPOSITION_BUILD_FAILED`, `EXPORT_SIDE_EFFECT` | The builder throws, or renders/writes while building | Keep builders pure; `framefields check` reproduces it. || `ASSET_NOT_FOUND`, `LFS_OBJECT_MISSING`, `LFS_POINTER_UNRESOLVED` | An asset isn't in the repo or its LFS object wasn't pushed | `git lfs ls-files`, `git lfs push --all origin`; un-ignore assets the film needs. |+| `ASSET_NOT_UPLOADED`, `ASSET_HASH_MISMATCH` | A pinned asset isn't on the bucket, or the bucket holds other bytes | `npx framefields assets push` from the machine with the right file. |+| `ASSET_PRIVATE`, `ASSET_FORBIDDEN` | The worker can't read the bucket: no `publicUrl` and no signature (a dashboard render), or the custom domain doesn't serve it | Render from the CLI, or give the remote a working `publicUrl`. |+| `ASSET_MANIFEST_INVALID` | `storage` or `assets` in framefields.json is malformed | `npx framefields assets status` shows the error locally. || `ENGINE_UNSUPPORTED` | The project's framefields is too old to render chunks | Upgrade framefields in the project (2.11+), or render with `--workers 1`. || `RENDER_TIMEOUT`, `RENDER_OOM` | One worker can't finish its part | Use `--workers auto` (or more workers) to split it. |diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 2d30aa8f..6503656f 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -243,34 +243,33 @@ Add scripts to `package.json`:`render:cloud` renders on framefields' GPUs (api.framefields.dev) from the pushed commit, with the same `FRAMEFIELDS_API_KEY` as the AI gateway, and splits long films across several GPUs; `framefields-render` covers it. When the user wants another version of the film (a teaser, a vertical cut, a language), add an export and a named render for it rather than a flag in `render.ts`.-### Video assets & Git LFS--Video inputs (`Layer.video("assets/footage/clip.mp4")`), Lottie animations (`Layer.lottie(...)`), audio tracks (`Layer.audio(...)`), LUTs, and custom fonts belong in **Git LFS**, while render outputs in `output/` belong in `.gitignore`.--1. **Initialize Git LFS:**- ```bash- git lfs install+### Video assets: a bucket, or Git LFS++Footage, audio, Lottie files, LUTs and generated media are too big for plain git. Render outputs in `output/` stay in `.gitignore`. Small files (logos, short sound effects, a LUT) can simply be committed.++**Preferred: `framefields assets` on the project's own bucket.** Any S3-compatible bucket works; Cloudflare R2 is the one to suggest, since it charges no egress. The files stay out of git, `framefields.json` pins each one by sha256, and anyone (or any worker) gets exactly those bytes back.++1. Add a remote to `framefields.json`. `publicUrl` is optional: a custom domain (or CDN, or the `r2.dev` URL) in front of the bucket, so reads need no keys. Content-addressed objects are uploaded `immutable`, so a CDN caches them forever.+ ```json+ "storage": {+ "remotes": {+ "media": {+ "endpoint": "https://<account-id>.r2.cloudflarestorage.com",+ "bucket": "films",+ "prefix": "launch/",+ "publicUrl": "https://media.example.com"+ }+ }+ }```+ AWS needs `"region"`; R2 uses `auto`. Several remotes are fine (a public one for media you may share, a private one for raw footage): `"default"` names the one new assets go to, `--remote` picks another.+2. Keys come from the environment, never the file: `FRAMEFIELDS_STORAGE_MEDIA_ACCESS_KEY_ID` and `FRAMEFIELDS_STORAGE_MEDIA_SECRET_ACCESS_KEY` (the remote's name upper-cased), or `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. Ask the user to set them; never print or write them.+3. `npx framefields assets add assets/footage` hashes the files (a folder means every file in it), uploads what the bucket lacks, records them under `assets` and adds them to `.gitignore`. Commit `framefields.json` and `.gitignore`. Adding a changed file again records the new version.+4. In a fresh clone or container, `npx framefields assets pull` downloads what's missing and checks every byte; it leaves files changed locally alone unless `--overwrite`. `assets status` lists each asset as `ok`, `missing` or `changed`; `assets push` re-uploads what the bucket lacks.-2. **Configure `.gitattributes` at the project root:**- ```gitattributes- assets/**/*.mp4 filter=lfs diff=lfs merge=lfs -text- assets/**/*.mov filter=lfs diff=lfs merge=lfs -text- assets/**/*.webm filter=lfs diff=lfs merge=lfs -text- assets/**/*.mp3 filter=lfs diff=lfs merge=lfs -text- assets/**/*.wav filter=lfs diff=lfs merge=lfs -text- assets/**/*.cube filter=lfs diff=lfs merge=lfs -text- assets/**/*.lottie filter=lfs diff=lfs merge=lfs -text- assets/fonts/**/*.ttf filter=lfs diff=lfs merge=lfs -text- assets/fonts/**/*.otf filter=lfs diff=lfs merge=lfs -text- ```--3. **Hydrate pointers before rendering:** In freshly cloned or containerized agent environments, LFS media exist as ~130-byte text pointer files. Framefields' native WebGPU and WebCodecs demuxers cannot decode pointers and will fail or produce blank frames. Always ensure pointers are hydrated:- ```bash- git lfs pull- ```+Code references assets by path as usual (`Layer.video("assets/footage/intro.mp4")`); nothing in the film changes. A missing pinned file makes `framefields check` say to run `assets pull`.-4. **Keep `.gitignore` targeted:** Never add `*.mp4` to `.gitignore` when tracking source footage with LFS. Ignore output directories (`output/`, `scratch/`) instead.+**Or Git LFS**, when the project already uses it or there's no bucket. `npx framefields init` writes the LFS `.gitattributes`. In fresh clones LFS media are ~130-byte pointer files that decode as blank frames, so run `git lfs pull` first, and never add `*.mp4` to `.gitignore` for LFS-tracked footage. Every clone and every cloud render worker downloads LFS objects from the git host, which bills that bandwidth to the repo owner; a bucket with no egress fees avoids it.## 4. Verify, cheapest step first@@ -297,10 +296,10 @@ Show the user the frame PNGs or the grid before the full render. `framefields-re## 5. Render on a server-The framefields render worker (Docker, RunPod pod or serverless) renders a pushed commit: it pulls LFS assets, installs from the lockfile, imports `src/film.ts`, calls `buildFilm`, and uploads the result. A project built as above works unchanged if:+The framefields render worker (Docker, RunPod pod or serverless) renders a pushed commit: it pulls LFS assets, downloads the assets `framefields.json` pins, installs from the lockfile, imports `src/film.ts`, calls `buildFilm`, and uploads the result. A project built as above works unchanged if:- `framefields` is 2.2+, the lockfile is committed and matches `package.json`, and `packageManager` is pinned;-- every input is under `assets/`, tracked by Git LFS and pushed (`git lfs ls-files` lists it);+- every input is under `assets/`, and either pinned with `framefields assets add` (and on the bucket) or tracked by Git LFS and pushed (`git lfs ls-files` lists it);- asset paths come from `import.meta.url`, never `process.cwd()`;- nothing writes into `node_modules` while rendering (files are shared read-only).diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex c163355a..c42ed31b 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -184,6 +184,9 @@ importers:'@aws-sdk/lib-storage':specifier: ^3.1151.0version: 3.1151.0(@aws-sdk/[email protected])+ '@framefields/storage':+ specifier: workspace:*+ version: link:../../packages/storage'@hono/node-server':specifier: '>=2.0.5'version: 2.0.12([email protected])@@ -2832,6 +2835,9 @@ importers:'@resvg/resvg-js':specifier: ^2.6.2version: 2.6.2+ aws4fetch:+ specifier: ^1.0.20+ version: 1.0.20d3-format:specifier: ^3.1.2version: 3.1.2@@ -3046,6 +3052,9 @@ importers:'@framefields/renderers':specifier: workspace:*version: link:../renderers+ '@framefields/storage':+ specifier: workspace:*+ version: link:../storage'@framefields/tensor-webgpu':specifier: workspace:*version: link:../tensor-webgpu@@ -3423,6 +3432,28 @@ importers:specifier: ^5.0.1version: 5.0.1(@opentelemetry/[email protected])(@types/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))+ packages/storage:+ dependencies:+ aws4fetch:+ specifier: ^1.0.20+ version: 1.0.20+ devDependencies:+ '@framefields/tsconfig':+ specifier: workspace:*+ version: link:../tsconfig+ '@types/node':+ specifier: ^24.1.0+ version: 24.10.14+ tsdown:+ specifier: ^0.19.0+ version: 0.19.0([email protected])([email protected])+ typescript:+ specifier: ^5.8.3+ version: 5.9.3+ vitest:+ specifier: ^4.1.11+ version: 4.1.11(@opentelemetry/[email protected])(@types/[email protected])(@vitest/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))+packages/tensor-webgpu:dependencies:'@framefields/core':@@ -5367,6 +5398,9 @@ packages:resolution: {integrity: sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==}engines: {node: '>=8.0.0'}+ resolution: {integrity: sha512-/djoAN709iY65ETD6LKCtyyEI04XIBP5xVvfmNxsEP0uJB5tyaGBztSryRr4HqMStr9R06PisQE7m9zDTXKu6g==}+resolution: {integrity: sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==}@@ -8485,6 +8519,8 @@ snapshots:+ [email protected]: {}+