Code-first video & image, rendered natively on WebGPU.
use gatewya
okanasldf0af48474b8
diff --git a/README.md b/README.mdindex 1d297769..ba124be2 100644--- a/README.md+++ b/README.md@@ -4,7 +4,7 @@### Compositing, motion graphics, and 3D for code-first video — one npm package that AI agents drive with code.-[**Documentation**](https://framefields.dev/docs)  ·  [**Website**](https://framefields.dev)  ·  [**Discord**](https://discord.gg/cbqMGGme5)  ·  [**Examples**](examples)+[**Documentation**](https://framefields.dev/docs)  ·  [**Website**](https://framefields.dev)  ·  [**Models**](https://framefields.dev/models)  ·  [**Discord**](https://discord.gg/cbqMGGme5)  ·  [**Examples**](examples)[](https://www.npmjs.com/package/framefields)[](https://framefields.dev/docs)@@ -163,10 +163,33 @@ process.exit(0);---+## AI Gateway++Framefields includes optional, built-in access to the [**framefields.dev AI Gateway**](https://framefields.dev/models), letting agents and developers generate creative media—cinematic images, video footage, background music, voiceovers, and sound effects—directly from code. Authenticated via `FRAMEFIELDS_API_KEY` (an `ff_live_…` key from your [dashboard](https://framefields.dev/dashboard/keys)), the built-in `ai` client provides a unified interface (`run`, `subscribe`, `stream`, and `queue`) across hundreds of creative foundation models like Flux, Seedance, OpenAI, Nano Banana, Kling, Minimax, Suno, and ElevenLabs without configuring separate provider accounts or billing.++To get started, store your API key in `.env` and create a dedicated script (such as `src/generate.ts`) that calls `ai.subscribe()` to render media into `assets/`. Because AI generation is asynchronous and billed per execution, assets are generated once, saved locally, and tracked with Git LFS—ready to be loaded cleanly into `Layer.image()`, `Layer.video()`, or `Layer.audio()` inside your frame-accurate compositions without recurring cost or latency during previews and renders. Browse the full model catalog and schemas at [**framefields.dev/models**](https://framefields.dev/models).++```typescript+import { ai } from "framefields";++// 1. Generate an image asset with Flux Schnell+const still = await ai.subscribe<{ images: { url: string }[] }>("flux/schnell", {+ input: { prompt: "Cyberpunk alley in the rain, volumetric neon, 35mm", image_size: "landscape_16_9" },+});++// 2. Animate it into video with Kling or generate music & voiceovers+const clip = await ai.subscribe<{ video: { url: string } }>("kling-video/v2.1/standard/image-to-video", {+ input: { prompt: "Slow forward tracking shot through neon rain", image_url: still.data.images[0].url, duration: "5" },+});+```++---+## DocumentationEverything else—including full API references, interactive tutorials, and guides—is available at [**framefields.dev/docs**](https://framefields.dev/docs):+- **[AI Gateway & Asset Generation](https://framefields.dev/models)** — Unified SDK for generating images, video clips, music, and voice via hundreds of hosted models- **[Typography & Animators](https://framefields.dev/docs)** — Slug analytic vector text, After Effects-style range animators, and typewriter cadence- **[WebGPU 2D VFX](https://framefields.dev/docs)** — 50+ shaders: 3D LUTs, Curves, Film Grain, Glassmorphism, and Blur- **[Unified 3D Scenes](https://framefields.dev/docs)** — LookAt cameras, PBR materials, studio lighting, shadows, and glTF/GLB modelsdiff --git a/examples/19_framefields_film/src/generate-assets.ts b/examples/19_framefields_film/src/generate-assets.tsindex cd9ae3c1..937d707a 100644--- a/examples/19_framefields_film/src/generate-assets.ts+++ b/examples/19_framefields_film/src/generate-assets.ts@@ -12,6 +12,7 @@ import fs from "node:fs/promises";import path from "node:path";import { fileURLToPath } from "node:url";import dotenv from "dotenv";+import { ai } from "framefields";import { SCORE, SHOTS } from "./shots.js";const HERE = path.dirname(fileURLToPath(import.meta.url));@@ -19,8 +20,10 @@ const ROOT = path.resolve(HERE, "../../..");const ASSETS = path.resolve(HERE, "../assets");dotenv.config({ path: path.join(ROOT, ".env"), quiet: true });-const FAL_KEY = process.env.FAL_API_KEY ?? process.env.FAL_KEY;-if (!FAL_KEY) throw new Error("FAL_API_KEY missing from .env");+function checkApiKey() {+ const key = process.env.FRAMEFIELDS_API_KEY;+ if (!key) throw new Error("FRAMEFIELDS_API_KEY missing from .env");+}const FONTS: Record<string, string> = {"InstrumentSerif-Regular.ttf":@@ -58,37 +61,6 @@ async function download(url: string, dest: string): Promise<void> {await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));}-/** Submits to the fal queue and polls until the result is ready. */-async function fal<T>(model: string, input: Record<string, unknown>): Promise<T> {- const headers = {- Authorization: `Key ${FAL_KEY}`,- "Content-Type": "application/json",- };- const submit = await request(`https://queue.fal.run/${model}`, {- method: "POST",- headers,- body: JSON.stringify(input),- });- if (!submit.ok) throw new Error(`${model} submit → ${submit.status} ${await submit.text()}`);- const { status_url, response_url } = (await submit.json()) as {- status_url: string;- response_url: string;- };-- for (;;) {- await sleep(4000);- const res = await request(status_url, { headers });- const { status } = (await res.json()) as { status: string };- if (status === "COMPLETED") break;- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS") {- throw new Error(`${model} → ${status}`);- }- }- const result = await request(response_url, { headers });- if (!result.ok) throw new Error(`${model} result → ${result.status} ${await result.text()}`);- return (await result.json()) as T;-}-async function fonts() {const dir = path.join(ROOT, "assets/fonts");for (const [file, url] of Object.entries(FONTS)) {@@ -100,55 +72,74 @@ async function fonts() {}async function images() {+ checkApiKey();await Promise.all(SHOTS.map(async (shot) => {const dest = path.join(ASSETS, `${shot.id}.png`);if (await exists(dest)) return;const t0 = Date.now();- const out = await fal<{ images: { url: string }[] }>("openai/gpt-image-2", {- prompt: shot.image,- image_size: { width: 2048, height: 1152 },- quality: "high",- output_format: "png",- });+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>(+ "openai/gpt-image-2",+ {+ input: {+ prompt: shot.image,+ image_size: { width: 2048, height: 1152 },+ quality: "high",+ output_format: "png",+ },+ },+ );await download(out.images[0].url, dest);- console.log(`image ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`);+ console.log(+ `image ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`,+ );}),);}-/** fal accepts data URIs for image inputs, so the local plate is sent as-is. */+/** The gateway accepts data URIs for image inputs, so the local plate is sent as-is. */async function videos() {+ checkApiKey();await Promise.all(SHOTS.map(async (shot) => {const dest = path.join(ASSETS, `${shot.id}.mp4`);if (await exists(dest)) return;const plate = await fs.readFile(path.join(ASSETS, `${shot.id}.png`));const t0 = Date.now();- const out = await fal<{ video: { url: string } }>(+ const { data: out } = await ai.subscribe<{ video: { url: string } }>(shot.model ?? "minimax/h3-max-turbo/image-to-video",{- prompt: shot.motion,- image_url: `data:image/png;base64,${plate.toString("base64")}`,- duration: shot.seconds,- resolution: "1080P",- prompt_expansion_mode: "disabled",+ input: {+ prompt: shot.motion,+ image_url: `data:image/png;base64,${plate.toString("base64")}`,+ duration: shot.seconds,+ resolution: "1080P",+ prompt_expansion_mode: "disabled",+ },},);await download(out.video.url, dest);- console.log(`video ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`);+ console.log(+ `video ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`,+ );}),);}async function music() {+ checkApiKey();const dest = path.join(ASSETS, "score.mp3");if (await exists(dest)) return;const t0 = Date.now();- const out = await fal<{ audio: { url: string } }>("fal-ai/elevenlabs/music", {- composition_plan: SCORE,- respect_sections_durations: true,- });+ const { data: out } = await ai.subscribe<{ audio: { url: string } }>(+ "fal-ai/elevenlabs/music",+ {+ input: {+ composition_plan: SCORE,+ respect_sections_durations: true,+ },+ },+ );await download(out.audio.url, dest);console.log(`music score (${((Date.now() - t0) / 1000).toFixed(0)}s)`);}diff --git a/examples/21_full_circle/README.md b/examples/21_full_circle/README.mdindex 7debe16c..8a45c4b7 100644--- a/examples/21_full_circle/README.md+++ b/examples/21_full_circle/README.md@@ -22,7 +22,7 @@ pnpm test| Beat grid → `assets/score.json` | `src/beat-grid.ts` (onset flux, autocorrelation, comb refinement, drop-anchored downbeat) | free || Circle tracks → `assets/circles.json` | `src/circles.ts` (ffmpeg grey frames, seeded flood fill per frame) | free |-`FAL_API_KEY` is read from the repo root `.env`. Every step is idempotent; delete a file to regenerate it.+`FRAMEFIELDS_API_KEY` is read from the repo root `.env`. Every step is idempotent; delete a file to regenerate it.Generated media is gitignored; the two JSON files are tracked so tests and the edit math run without it.## How the film is builtdiff --git a/examples/21_full_circle/src/fal.ts b/examples/21_full_circle/src/fal.tsdeleted file mode 100644index d9971a9a..00000000--- a/examples/21_full_circle/src/fal.ts+++ /dev/null@@ -1,98 +0,0 @@-/**- * Minimal fal.ai queue client: submit, poll, fetch the result, download files.- * Every call is priced up front from the model's published unit price so the- * pipeline can print what a run cost.- */-import fs from "node:fs/promises";-import path from "node:path";--const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));--/** USD per unit, from https://api.fal.ai/v1/models/pricing (October 2026). */-export const PRICING = {- "fal-ai/z-image/turbo": { usd: 0.005, per: "megapixel" },- "minimax/h3-max-turbo/image-to-video": { usd: 0.015, per: "second" },- "minimax/h3-max/image-to-video": { usd: 0.03, per: "second" },- "minimax/music-3": { usd: 0.002, per: "second" },-} as const;--export type FalModel = keyof typeof PRICING;--export const estimate = (model: FalModel, units: number) =>- PRICING[model].usd * units;--export class FalClient {- private spent = 0;-- constructor(private readonly key: string) {}-- get totalUsd(): number {- return this.spent;- }-- /** Submits to the fal queue, polls until done, returns the JSON result. */- async run<T>(- model: FalModel,- input: Record<string, unknown>,- units: number,- ): Promise<T> {- const headers = {- Authorization: `Key ${this.key}`,- "Content-Type": "application/json",- };- const submit = await request(`https://queue.fal.run/${model}`, {- method: "POST",- headers,- body: JSON.stringify(input),- });- if (!submit.ok)- throw new Error(- `${model} submit → ${submit.status} ${await submit.text()}`,- );- const { status_url, response_url } = (await submit.json()) as {- status_url: string;- response_url: string;- };-- for (;;) {- await sleep(3000);- const res = await request(status_url, { headers });- const { status } = (await res.json()) as { status: string };- if (status === "COMPLETED") break;- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS")- throw new Error(`${model} → ${status}`);- }- const result = await request(response_url, { headers });- if (!result.ok)- throw new Error(- `${model} result → ${result.status} ${await result.text()}`,- );- this.spent += estimate(model, units);- return (await result.json()) as T;- }-}--/** fetch that rides out transient network failures (long polls drop connections). */-async function request(url: string, init?: RequestInit): Promise<Response> {- for (let attempt = 1; ; attempt++) {- try {- return await fetch(url, init);- } catch (err) {- if (attempt === 5) throw err;- await sleep(2000 * attempt);- }- }-}--export async function download(url: string, dest: string): Promise<void> {- const res = await request(url);- if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);- await fs.mkdir(path.dirname(dest), { recursive: true });- await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));-}--export const exists = (p: string) =>- fs.access(p).then(- () => true,- () => false,- );diff --git a/examples/21_full_circle/src/gateway.ts b/examples/21_full_circle/src/gateway.tsnew file mode 100644index 00000000..7138f15f--- /dev/null+++ b/examples/21_full_circle/src/gateway.ts@@ -0,0 +1,45 @@+/**+ * AI gateway helpers: model pricing estimates, download, and file existence.+ */+import fs from "node:fs/promises";+import path from "node:path";++const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));++/** USD per unit, from https://framefields.dev/models (October 2026). */+export const PRICING = {+ "fal-ai/z-image/turbo": { usd: 0.005, per: "megapixel" },+ "minimax/h3-max-turbo/image-to-video": { usd: 0.015, per: "second" },+ "minimax/h3-max/image-to-video": { usd: 0.03, per: "second" },+ "minimax/music-3": { usd: 0.002, per: "second" },+} as const;++export type GatewayModel = keyof typeof PRICING;++export const estimate = (model: GatewayModel, units: number) =>+ PRICING[model].usd * units;++/** fetch that rides out transient network failures. */+async function request(url: string, init?: RequestInit): Promise<Response> {+ for (let attempt = 1; ; attempt++) {+ try {+ return await fetch(url, init);+ } catch (err) {+ if (attempt === 5) throw err;+ await sleep(2000 * attempt);+ }+ }+}++export async function download(url: string, dest: string): Promise<void> {+ const res = await request(url);+ if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);+ await fs.mkdir(path.dirname(dest), { recursive: true });+ await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));+}++export const exists = (p: string) =>+ fs.access(p).then(+ () => true,+ () => false,+ );diff --git a/examples/21_full_circle/src/generate-assets.ts b/examples/21_full_circle/src/generate-assets.tsindex 4a772323..a828f544 100644--- a/examples/21_full_circle/src/generate-assets.ts+++ b/examples/21_full_circle/src/generate-assets.ts@@ -16,6 +16,7 @@ import { execFile } from "node:child_process";import fs from "node:fs/promises";import path from "node:path";import dotenv from "dotenv";+import { ai } from "framefields";import { analyzeBeatGrid } from "./beat-grid.js";import {type Circle,@@ -23,7 +24,7 @@ import {cleanTrack,measureCircle,} from "./circles.js";-import { download, estimate, exists, FalClient } from "./fal.js";+import { download, estimate, exists } from "./gateway.js";import { ASSETS, ROOT } from "./paths.js";import { SCORE, SCORE_SECONDS, SHOTS } from "./shots.js";import { readWav } from "./wav.js";@@ -47,10 +48,11 @@ const STILL_MP = (STILL.width * STILL.height) / 1e6;const elapsed = (t0: number) => `${((Date.now() - t0) / 1000).toFixed(0)}s`;-function client(): FalClient {- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;- if (!key) throw new Error("FAL_API_KEY missing from .env");- return new FalClient(key);+let spent = 0;++function checkApiKey(): void {+ const key = process.env.FRAMEFIELDS_API_KEY;+ if (!key) throw new Error("FRAMEFIELDS_API_KEY missing from .env");}async function fonts() {@@ -62,30 +64,34 @@ async function fonts() {}}-async function images(fal: FalClient) {+async function images() {+ checkApiKey();await Promise.all(SHOTS.map(async (shot) => {const dest = path.join(ASSETS, `${shot.id}.png`);if (await exists(dest)) return;const t0 = Date.now();- const out = await fal.run<{ images: { url: string }[] }>(+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>("fal-ai/z-image/turbo",{- prompt: shot.image,- image_size: STILL,- num_inference_steps: 8,- output_format: "png",+ input: {+ prompt: shot.image,+ image_size: STILL,+ num_inference_steps: 8,+ output_format: "png",+ },},- STILL_MP,);await download(out.images[0].url, dest);+ spent += estimate("fal-ai/z-image/turbo", STILL_MP);console.log(`image ${shot.id} (${elapsed(t0)})`);}),);}-/** fal accepts data URIs for image inputs, so the local still is sent as-is. */-async function videos(fal: FalClient, target?: string) {+/** The gateway accepts data URIs for image inputs, so the local still is sent as-is. */+async function videos(target?: string) {+ checkApiKey();const shotList = target ? SHOTS.filter((s) => s.id === target) : SHOTS;await Promise.all(shotList.map(async (shot) => {@@ -93,37 +99,43 @@ async function videos(fal: FalClient, target?: string) {if (await exists(dest)) return;const still = await fs.readFile(path.join(ASSETS, `${shot.id}.png`));const t0 = Date.now();- const out = await fal.run<{ video: { url: string } }>(+ const { data: out } = await ai.subscribe<{ video: { url: string } }>(shot.model,{- prompt: shot.motion,- image_url: `data:image/png;base64,${still.toString("base64")}`,- duration: shot.seconds,- resolution: "1080P",- prompt_expansion_mode: "disabled",+ input: {+ prompt: shot.motion,+ image_url: `data:image/png;base64,${still.toString("base64")}`,+ duration: shot.seconds,+ resolution: "1080P",+ prompt_expansion_mode: "disabled",+ },},- shot.seconds,);await download(out.video.url, dest);+ spent += estimate(shot.model, shot.seconds);console.log(`video ${shot.id} via ${shot.model} (${elapsed(t0)})`);}),);}-async function music(fal: FalClient) {+async function music() {+ checkApiKey();const dest = path.join(ASSETS, "score.wav");if (await exists(dest)) return;const t0 = Date.now();- const out = await fal.run<{+ const { data: out } = await ai.subscribe<{audio: { url: string };duration: number;seed: number;- }>(- "minimax/music-3",- { prompt: SCORE.prompt, lyrics: SCORE.lyrics, duration: SCORE_SECONDS },- SCORE_SECONDS,- );+ }>("minimax/music-3", {+ input: {+ prompt: SCORE.prompt,+ lyrics: SCORE.lyrics,+ duration: SCORE_SECONDS,+ },+ });await download(out.audio.url, dest);+ spent += estimate("minimax/music-3", SCORE_SECONDS);console.log(`music score ${out.duration.toFixed(1)}s seed ${out.seed} (${elapsed(t0)})`,);@@ -224,18 +236,15 @@ await fs.mkdir(ASSETS, { recursive: true });if (step === "cost") {cost();} else {- // The analysis steps are local; only generation needs a fal key.- const fal = runs("fonts", "images", "videos", "music") ? client() : null;if (runs("fonts")) await fonts();- if (fal && runs("images")) await images(fal);+ if (runs("images")) await images();// Music and motion are independent; run them side by side.await Promise.all([- fal && runs("videos") ? videos(fal, target) : undefined,- fal && runs("music") ? music(fal) : undefined,+ runs("videos") ? videos(target) : undefined,+ runs("music") ? music() : undefined,]);if (runs("music", "grid")) await grid();if (runs("videos", "circles")) await circles(target);- if (fal && fal.totalUsd > 0)- console.log(`spent ~$${fal.totalUsd.toFixed(3)}`);+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);}process.exit(0);diff --git a/examples/22_framefields_launch/README.md b/examples/22_framefields_launch/README.mdindex 6dd42bb6..6108737c 100644--- a/examples/22_framefields_launch/README.md+++ b/examples/22_framefields_launch/README.md@@ -31,7 +31,7 @@ pnpm test| Images | `fal-ai/flux-2-pro`: a backdrop, a camera on a green screen, a light leak on black, a paper grain (`images.ts`) | `assets/images/` |The soundtrack is `assets/song.mp3` exactly as generated: nothing stretches, shifts or re-masters it, so-the film's clock is the take's clock. `FAL_API_KEY` is read from the repo root `.env`. The music model+the film's clock is the take's clock. `FRAMEFIELDS_API_KEY` is read from the repo root `.env`. The music modeltreats section lengths as guidance, so nothing in the picture assumes the plan held: regenerate the songand the film re-cuts itself.diff --git a/examples/22_framefields_launch/src/fal.ts b/examples/22_framefields_launch/src/fal.tsdeleted file mode 100644index e1e67ba5..00000000--- a/examples/22_framefields_launch/src/fal.ts+++ /dev/null@@ -1,92 +0,0 @@-/**- * Minimal fal.ai queue client (after examples/21_full_circle): submit, poll,- * fetch the result. Each call is priced from the model's published unit price- * so a run reports what it cost.- */-import fs from "node:fs/promises";-import path from "node:path";--const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));--/** USD per unit, from https://api.fal.ai/v1/models/pricing (October 2026). */-export const PRICING = {- "fal-ai/elevenlabs/music/v2.5": { usd: 0.00007, per: "compute second" },- "fal-ai/elevenlabs/speech-to-text/scribe-v2": { usd: 0.008, per: "minute" },- "fal-ai/demucs": { usd: 0.0007, per: "second" },- "fal-ai/flux-2-pro": { usd: 0.03, per: "megapixel" },-} as const;--export type FalModel = keyof typeof PRICING;--export class FalClient {- private spent = 0;-- constructor(private readonly key: string) {}-- get totalUsd(): number {- return this.spent;- }-- async run<T>(- model: FalModel,- input: Record<string, unknown>,- units: number,- ): Promise<T> {- const headers = {- Authorization: `Key ${this.key}`,- "Content-Type": "application/json",- };- const submit = await request(`https://queue.fal.run/${model}`, {- method: "POST",- headers,- body: JSON.stringify(input),- });- if (!submit.ok)- throw new Error(- `${model} submit → ${submit.status} ${await submit.text()}`,- );- const { status_url, response_url } = (await submit.json()) as {- status_url: string;- response_url: string;- };- for (;;) {- await sleep(3000);- const res = await request(status_url, { headers });- const { status } = (await res.json()) as { status: string };- if (status === "COMPLETED") break;- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS")- throw new Error(`${model} → ${status}`);- }- const result = await request(response_url, { headers });- if (!result.ok)- throw new Error(- `${model} result → ${result.status} ${await result.text()}`,- );- this.spent += PRICING[model].usd * units;- return (await result.json()) as T;- }-}--async function request(url: string, init?: RequestInit): Promise<Response> {- for (let attempt = 1; ; attempt++) {- try {- return await fetch(url, init);- } catch (err) {- if (attempt === 5) throw err;- await sleep(2000 * attempt);- }- }-}--export async function download(url: string, dest: string): Promise<void> {- const res = await request(url);- if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);- await fs.mkdir(path.dirname(dest), { recursive: true });- await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));-}--export const exists = (p: string) =>- fs.access(p).then(- () => true,- () => false,- );diff --git a/examples/22_framefields_launch/src/gateway.ts b/examples/22_framefields_launch/src/gateway.tsnew file mode 100644index 00000000..9f5c15c1--- /dev/null+++ b/examples/22_framefields_launch/src/gateway.ts@@ -0,0 +1,41 @@+/**+ * AI gateway helpers: model pricing estimates, download, and file existence.+ */+import fs from "node:fs/promises";+import path from "node:path";++const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));++/** USD per unit, from https://framefields.dev/models (October 2026). */+export const PRICING = {+ "fal-ai/elevenlabs/music/v2.5": { usd: 0.00007, per: "compute second" },+ "fal-ai/elevenlabs/speech-to-text/scribe-v2": { usd: 0.008, per: "minute" },+ "fal-ai/demucs": { usd: 0.0007, per: "second" },+ "fal-ai/flux-2-pro": { usd: 0.03, per: "megapixel" },+} as const;++export type GatewayModel = keyof typeof PRICING;++async function request(url: string, init?: RequestInit): Promise<Response> {+ for (let attempt = 1; ; attempt++) {+ try {+ return await fetch(url, init);+ } catch (err) {+ if (attempt === 5) throw err;+ await sleep(2000 * attempt);+ }+ }+}++export async function download(url: string, dest: string): Promise<void> {+ const res = await request(url);+ if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);+ await fs.mkdir(path.dirname(dest), { recursive: true });+ await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));+}++export const exists = (p: string) =>+ fs.access(p).then(+ () => true,+ () => false,+ );diff --git a/examples/22_framefields_launch/src/generate-images.ts b/examples/22_framefields_launch/src/generate-images.tsindex 59fa3c5a..19aea8f9 100644--- a/examples/22_framefields_launch/src/generate-images.ts+++ b/examples/22_framefields_launch/src/generate-images.ts@@ -1,6 +1,6 @@/*** pnpm images → assets/images/*.png: the layers the compositing chapter- * stacks, generated with FLUX.2 [pro] on fal. Each is made for one+ * stacks, generated with FLUX.2 [pro] on the AI gateway. Each is made for one* compositing operation:* backdrop the plate everything sits on* subject a camera on a flat green screen, for ColorKey to pull@@ -9,30 +9,39 @@* Idempotent: delete an image to regenerate it.*/import path from "node:path";-import { download, exists, FalClient } from "./fal.js";+import { ai } from "framefields";+import { download, exists, PRICING } from "./gateway.js";import { IMAGE_SIZE, IMAGES, type ImageName, image } from "./images.js";import { ROOT } from "./paths.js";async function main() {process.loadEnvFile(path.join(ROOT, ".env"));- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;- if (!key) throw new Error("FAL_API_KEY missing from the repo root .env");- const fal = new FalClient(key);+ const key = process.env.FRAMEFIELDS_API_KEY;+ if (!key)+ throw new Error("FRAMEFIELDS_API_KEY missing from the repo root .env");+ let spent = 0;const megapixels = (IMAGE_SIZE.width * IMAGE_SIZE.height) / 1e6;for (const [name, prompt] of Object.entries(IMAGES) as [ImageName,string,][]) {if (await exists(image(name))) continue;- const out = await fal.run<{ images: { url: string }[] }>(+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>("fal-ai/flux-2-pro",- { prompt, image_size: IMAGE_SIZE, output_format: "png", seed: 22 },- megapixels,+ {+ input: {+ prompt,+ image_size: IMAGE_SIZE,+ output_format: "png",+ seed: 22,+ },+ },);await download(out.images[0].url, image(name));+ spent += PRICING["fal-ai/flux-2-pro"].usd * megapixels;console.log(`image ${name}`);}- console.log(`fal $${fal.totalUsd.toFixed(3)}`);+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);}await main();diff --git a/examples/22_framefields_launch/src/generate-song.ts b/examples/22_framefields_launch/src/generate-song.tsindex cc6d539a..6cdb1441 100644--- a/examples/22_framefields_launch/src/generate-song.ts+++ b/examples/22_framefields_launch/src/generate-song.ts@@ -17,7 +17,8 @@import { execFileSync } from "node:child_process";import fs from "node:fs/promises";import path from "node:path";-import { download, exists, FalClient, PRICING } from "./fal.js";+import { ai } from "framefields";+import { download, exists, PRICING } from "./gateway.js";import { BAR_SEC, BPM, PLAN_SEC, SAMPLE_RATE } from "./grid.js";import { alignLyrics, type HeardWord } from "./music/align.js";import { analyzeBeatGrid } from "./music/beat-grid.js";@@ -41,17 +42,19 @@ const stem = (name: string) => asset(`stems/${name}.wav`);const step = process.argv[2] ?? "all";-function client(): FalClient {+let spent = 0;++function checkApiKey(): void {process.loadEnvFile(path.join(ROOT, ".env"));- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;- if (!key) throw new Error("FAL_API_KEY missing from the repo root .env");- return new FalClient(key);+ const key = process.env.FRAMEFIELDS_API_KEY;+ if (!key)+ throw new Error("FRAMEFIELDS_API_KEY missing from the repo root .env");}/** Generous upper bound on the music model's compute time, for the estimate. */const SONG_COMPUTE_SEC = 120;-async function song(fal: FalClient, seed: number): Promise<void> {+async function song(seed: number): Promise<void> {if (await exists(SONG)) return;const durations = sectionDurationsMs();// v2.5 has no global styles: the first chunk's styles set the whole song's@@ -66,16 +69,18 @@ async function song(fal: FalClient, seed: number): Promise<void> {})),};const t0 = Date.now();- const out = await fal.run<{ audio: { url: string } }>(+ const { data: out } = await ai.subscribe<{ audio: { url: string } }>("fal-ai/elevenlabs/music/v2.5",- { composition_plan: plan, seed, output_format: "mp3_48000_192" },- SONG_COMPUTE_SEC,+ {+ input: { composition_plan: plan, seed, output_format: "mp3_48000_192" },+ },);await download(out.audio.url, SONG);await fs.writeFile(SONG_META,JSON.stringify({ url: out.audio.url, seed, plan }, null, 2),);+ spent += PRICING["fal-ai/elevenlabs/music/v2.5"].usd * SONG_COMPUTE_SEC;console.log(`song ${SONG} seed ${seed} (${((Date.now() - t0) / 1000).toFixed(0)} s)`,);@@ -85,21 +90,21 @@ async function song(fal: FalClient, seed: number): Promise<void> {type StemName = "vocals" | "drums";const STEM_NAMES: StemName[] = ["vocals", "drums"];-async function stems(fal: FalClient): Promise<void> {+async function stems(): Promise<void> {if (await exists(STEMS)) return;const { url } = JSON.parse(await fs.readFile(SONG_META, "utf8")) as {url: string;};- const out = await fal.run<Record<StemName, { url: string } | null>>(- "fal-ai/demucs",- {+ const { data: out } = await ai.subscribe<+ Record<StemName, { url: string } | null>+ >("fal-ai/demucs", {+ input: {audio_url: url,model: "htdemucs_ft",stems: STEM_NAMES,output_format: "wav",},- PLAN_SEC,- );+ });const urls: Partial<Record<StemName, string>> = {};for (const name of STEM_NAMES) {const file = out[name];@@ -108,6 +113,7 @@ async function stems(fal: FalClient): Promise<void> {urls[name] = file.url;}await fs.writeFile(STEMS, JSON.stringify(urls, null, 2));+ spent += PRICING["fal-ai/demucs"].usd * PLAN_SEC;console.log(`stems ${STEM_NAMES.join(", ")}`);}@@ -132,25 +138,28 @@ const KEYTERMS = ["Illustrator",];-async function words(fal: FalClient): Promise<void> {+async function words(): Promise<void> {if (await exists(WORDS)) return;const { vocals: url } = JSON.parse(await fs.readFile(STEMS, "utf8"),) as Record<StemName, string>;- const out = await fal.run<{ words: ScribeWord[] }>(+ const { data: out } = await ai.subscribe<{ words: ScribeWord[] }>("fal-ai/elevenlabs/speech-to-text/scribe-v2",{- audio_url: url,- language_code: "eng",- keyterms: KEYTERMS,- tag_audio_events: false,+ input: {+ audio_url: url,+ language_code: "eng",+ keyterms: KEYTERMS,+ tag_audio_events: false,+ },},- PLAN_SEC / 60,);const heard: HeardWord[] = out.words.filter((w) => w.type === "word").map(({ text, start, end }) => ({ text: text.trim(), start, end }));await fs.writeFile(WORDS, JSON.stringify(heard, null, 1));+ spent +=+ PRICING["fal-ai/elevenlabs/speech-to-text/scribe-v2"].usd * (PLAN_SEC / 60);console.log(`words ${heard.length} heard: ${heard.map((h) => h.text).join(" ")}`,);@@ -292,11 +301,11 @@ async function skippedLines(): Promise<string[]> {const MAX_TAKES = 4;/** Generates takes until one sings every line, setting rejected ones aside. */-async function generate(fal: FalClient): Promise<void> {+async function generate(): Promise<void> {for (let take = 1; ; take++) {- await song(fal, take);- await stems(fal);- await words(fal);+ await song(take);+ await stems();+ await words();const skipped = await skippedLines();if (!skipped.length) return;const rejected = path.resolve(@@ -324,9 +333,9 @@ if (step === "cost") {);} else {if (step === "all") {- const fal = client();- await generate(fal);- console.log(`fal $${fal.totalUsd.toFixed(3)}`);+ checkApiKey();+ await generate();+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);}await measure();}diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 3a22c7bb..38a735ce 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -16,6 +16,7 @@ This skill takes a project from nothing to a verified first render. For anything| Post-processing, color grading, VFX, on-device vision (tracking, cutouts, pose) | `framefields-effects` || Frame inspection, frame grids, pixel tests, video export and QA | `framefields-render` || Generating images, video, music, voice with AI models (`ai`) | this skill, section 6 |+| Reporting bugs or engine defects to GitHub | this skill, section 8 |---@@ -255,7 +256,7 @@ Never go straight to a full video render. On a machine without a GPU every rende4. `npm run preview` when a person is watching: they play the film with sound before you export (see below).5. `npm run render` once frames and motion look right.-Show the user the frame PNGs or the grid before the full render. `framefields-render` covers pixel-probe tests, render QA (`renderVideo({ qa })`), and diagnosing blank frames. If you suspect an issue or defect in the engine itself, file an issue at https://github.com/gatewai-dev/framefields.+Show the user the frame PNGs or the grid before the full render. `framefields-render` covers pixel-probe tests, render QA (`renderVideo({ qa })`), and diagnosing blank frames. If you suspect an issue or defect in the engine itself or ai gateway, see [section 8](#8-report-engine-issues).### Live preview@@ -339,3 +340,20 @@ Then `Layer.image("assets/generated/street.jpg")` / `Layer.video("assets/generat## 7. Grow the filmOnce the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading, grain, and vision-driven effects, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.++## 8. Report engine issues++When you encounter an unexpected crash, native WebGPU / Dawn failure, WGSL shader compilation error, compositor schema validation defect, or behavior bug in `framefields` itself (distinguishable from project-level configuration or asset errors):++1. **Ask the user first:** Never open an issue autonomously without explicit confirmation. Explain what went wrong, summarize the reproduction, and ask if they would like an issue reported to `gatewai-dev/framefields`.+2. **Collect diagnostic details:**+ - Operating system and architecture (`process.platform`, `process.arch`).+ - Node.js version (`node --version`) and `framefields` version.+ - GPU backend and device (run `describeGpu()` or note Metal, Vulkan, DX12, or CPU lavapipe/WARP).+ - Minimal reproduction code (the smallest composition or script that triggers it).+ - Exact error message, full stack trace, or validation failure log.+3. **Open the issue:** Once the user confirms, submit the issue via GitHub CLI:+ ```bash+ gh issue create --repo gatewai-dev/framefields --title "<concise summary>" --body "<diagnostic details and repro>"+ ```+ If `gh` is unavailable or not authenticated in the environment, output the title, formatted body, and a direct link for the user: https://github.com/gatewai-dev/framefields/issues/new.