Skip to content

Code-first video & image, rendered natively on WebGPU.

use gatewya

okanasldf0af48474b8
diff --git a/README.md b/README.md
index 1d297769..ba124be2 100644
--- a/README.md
+++ b/README.md
@@ -4,7 +4,7 @@
### Compositing, motion graphics, and 3D for code-first video — one npm package that AI agents drive with code.
-[**Documentation**](https://framefields.dev/docs)  ·  [**Website**](https://framefields.dev)  ·  [**Discord**](https://discord.gg/cbqMGGme5)  ·  [**Examples**](examples)
+[**Documentation**](https://framefields.dev/docs)  ·  [**Website**](https://framefields.dev)  ·  [**Models**](https://framefields.dev/models)  ·  [**Discord**](https://discord.gg/cbqMGGme5)  ·  [**Examples**](examples)
[![npm](https://img.shields.io/badge/npm-framefields-cb3837?logo=npm&logoColor=white)](https://www.npmjs.com/package/framefields)
[![docs](https://img.shields.io/badge/docs-framefields.dev%2Fdocs-6366f1)](https://framefields.dev/docs)
@@ -163,10 +163,33 @@ process.exit(0);
---
+## AI Gateway
+
+Framefields includes optional, built-in access to the [**framefields.dev AI Gateway**](https://framefields.dev/models), letting agents and developers generate creative media—cinematic images, video footage, background music, voiceovers, and sound effects—directly from code. Authenticated via `FRAMEFIELDS_API_KEY` (an `ff_live_…` key from your [dashboard](https://framefields.dev/dashboard/keys)), the built-in `ai` client provides a unified interface (`run`, `subscribe`, `stream`, and `queue`) across hundreds of creative foundation models like Flux, Seedance, OpenAI, Nano Banana, Kling, Minimax, Suno, and ElevenLabs without configuring separate provider accounts or billing.
+
+To get started, store your API key in `.env` and create a dedicated script (such as `src/generate.ts`) that calls `ai.subscribe()` to render media into `assets/`. Because AI generation is asynchronous and billed per execution, assets are generated once, saved locally, and tracked with Git LFS—ready to be loaded cleanly into `Layer.image()`, `Layer.video()`, or `Layer.audio()` inside your frame-accurate compositions without recurring cost or latency during previews and renders. Browse the full model catalog and schemas at [**framefields.dev/models**](https://framefields.dev/models).
+
+```typescript
+import { ai } from "framefields";
+
+// 1. Generate an image asset with Flux Schnell
+const still = await ai.subscribe<{ images: { url: string }[] }>("flux/schnell", {
+ input: { prompt: "Cyberpunk alley in the rain, volumetric neon, 35mm", image_size: "landscape_16_9" },
+});
+
+// 2. Animate it into video with Kling or generate music & voiceovers
+const clip = await ai.subscribe<{ video: { url: string } }>("kling-video/v2.1/standard/image-to-video", {
+ input: { prompt: "Slow forward tracking shot through neon rain", image_url: still.data.images[0].url, duration: "5" },
+});
+```
+
+---
+
## Documentation
Everything else—including full API references, interactive tutorials, and guides—is available at [**framefields.dev/docs**](https://framefields.dev/docs):
+- **[AI Gateway & Asset Generation](https://framefields.dev/models)** — Unified SDK for generating images, video clips, music, and voice via hundreds of hosted models
- **[Typography & Animators](https://framefields.dev/docs)** — Slug analytic vector text, After Effects-style range animators, and typewriter cadence
- **[WebGPU 2D VFX](https://framefields.dev/docs)** — 50+ shaders: 3D LUTs, Curves, Film Grain, Glassmorphism, and Blur
- **[Unified 3D Scenes](https://framefields.dev/docs)** — LookAt cameras, PBR materials, studio lighting, shadows, and glTF/GLB models
diff --git a/examples/19_framefields_film/src/generate-assets.ts b/examples/19_framefields_film/src/generate-assets.ts
index cd9ae3c1..937d707a 100644
--- a/examples/19_framefields_film/src/generate-assets.ts
+++ b/examples/19_framefields_film/src/generate-assets.ts
@@ -12,6 +12,7 @@ import fs from "node:fs/promises";
import path from "node:path";
import { fileURLToPath } from "node:url";
import dotenv from "dotenv";
+import { ai } from "framefields";
import { SCORE, SHOTS } from "./shots.js";
const HERE = path.dirname(fileURLToPath(import.meta.url));
@@ -19,8 +20,10 @@ const ROOT = path.resolve(HERE, "../../..");
const ASSETS = path.resolve(HERE, "../assets");
dotenv.config({ path: path.join(ROOT, ".env"), quiet: true });
-const FAL_KEY = process.env.FAL_API_KEY ?? process.env.FAL_KEY;
-if (!FAL_KEY) throw new Error("FAL_API_KEY missing from .env");
+function checkApiKey() {
+ const key = process.env.FRAMEFIELDS_API_KEY;
+ if (!key) throw new Error("FRAMEFIELDS_API_KEY missing from .env");
+}
const FONTS: Record<string, string> = {
"InstrumentSerif-Regular.ttf":
@@ -58,37 +61,6 @@ async function download(url: string, dest: string): Promise<void> {
await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));
}
-/** Submits to the fal queue and polls until the result is ready. */
-async function fal<T>(model: string, input: Record<string, unknown>): Promise<T> {
- const headers = {
- Authorization: `Key ${FAL_KEY}`,
- "Content-Type": "application/json",
- };
- const submit = await request(`https://queue.fal.run/${model}`, {
- method: "POST",
- headers,
- body: JSON.stringify(input),
- });
- if (!submit.ok) throw new Error(`${model} submit → ${submit.status} ${await submit.text()}`);
- const { status_url, response_url } = (await submit.json()) as {
- status_url: string;
- response_url: string;
- };
-
- for (;;) {
- await sleep(4000);
- const res = await request(status_url, { headers });
- const { status } = (await res.json()) as { status: string };
- if (status === "COMPLETED") break;
- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS") {
- throw new Error(`${model} → ${status}`);
- }
- }
- const result = await request(response_url, { headers });
- if (!result.ok) throw new Error(`${model} result → ${result.status} ${await result.text()}`);
- return (await result.json()) as T;
-}
-
async function fonts() {
const dir = path.join(ROOT, "assets/fonts");
for (const [file, url] of Object.entries(FONTS)) {
@@ -100,55 +72,74 @@ async function fonts() {
}
async function images() {
+ checkApiKey();
await Promise.all(
SHOTS.map(async (shot) => {
const dest = path.join(ASSETS, `${shot.id}.png`);
if (await exists(dest)) return;
const t0 = Date.now();
- const out = await fal<{ images: { url: string }[] }>("openai/gpt-image-2", {
- prompt: shot.image,
- image_size: { width: 2048, height: 1152 },
- quality: "high",
- output_format: "png",
- });
+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>(
+ "openai/gpt-image-2",
+ {
+ input: {
+ prompt: shot.image,
+ image_size: { width: 2048, height: 1152 },
+ quality: "high",
+ output_format: "png",
+ },
+ },
+ );
await download(out.images[0].url, dest);
- console.log(`image ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`);
+ console.log(
+ `image ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`,
+ );
}),
);
}
-/** fal accepts data URIs for image inputs, so the local plate is sent as-is. */
+/** The gateway accepts data URIs for image inputs, so the local plate is sent as-is. */
async function videos() {
+ checkApiKey();
await Promise.all(
SHOTS.map(async (shot) => {
const dest = path.join(ASSETS, `${shot.id}.mp4`);
if (await exists(dest)) return;
const plate = await fs.readFile(path.join(ASSETS, `${shot.id}.png`));
const t0 = Date.now();
- const out = await fal<{ video: { url: string } }>(
+ const { data: out } = await ai.subscribe<{ video: { url: string } }>(
shot.model ?? "minimax/h3-max-turbo/image-to-video",
{
- prompt: shot.motion,
- image_url: `data:image/png;base64,${plate.toString("base64")}`,
- duration: shot.seconds,
- resolution: "1080P",
- prompt_expansion_mode: "disabled",
+ input: {
+ prompt: shot.motion,
+ image_url: `data:image/png;base64,${plate.toString("base64")}`,
+ duration: shot.seconds,
+ resolution: "1080P",
+ prompt_expansion_mode: "disabled",
+ },
},
);
await download(out.video.url, dest);
- console.log(`video ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`);
+ console.log(
+ `video ${shot.id} (${((Date.now() - t0) / 1000).toFixed(0)}s)`,
+ );
}),
);
}
async function music() {
+ checkApiKey();
const dest = path.join(ASSETS, "score.mp3");
if (await exists(dest)) return;
const t0 = Date.now();
- const out = await fal<{ audio: { url: string } }>("fal-ai/elevenlabs/music", {
- composition_plan: SCORE,
- respect_sections_durations: true,
- });
+ const { data: out } = await ai.subscribe<{ audio: { url: string } }>(
+ "fal-ai/elevenlabs/music",
+ {
+ input: {
+ composition_plan: SCORE,
+ respect_sections_durations: true,
+ },
+ },
+ );
await download(out.audio.url, dest);
console.log(`music score (${((Date.now() - t0) / 1000).toFixed(0)}s)`);
}
diff --git a/examples/21_full_circle/README.md b/examples/21_full_circle/README.md
index 7debe16c..8a45c4b7 100644
--- a/examples/21_full_circle/README.md
+++ b/examples/21_full_circle/README.md
@@ -22,7 +22,7 @@ pnpm test
| Beat grid → `assets/score.json` | `src/beat-grid.ts` (onset flux, autocorrelation, comb refinement, drop-anchored downbeat) | free |
| Circle tracks → `assets/circles.json` | `src/circles.ts` (ffmpeg grey frames, seeded flood fill per frame) | free |
-`FAL_API_KEY` is read from the repo root `.env`. Every step is idempotent; delete a file to regenerate it.
+`FRAMEFIELDS_API_KEY` is read from the repo root `.env`. Every step is idempotent; delete a file to regenerate it.
Generated media is gitignored; the two JSON files are tracked so tests and the edit math run without it.
## How the film is built
diff --git a/examples/21_full_circle/src/fal.ts b/examples/21_full_circle/src/fal.ts
deleted file mode 100644
index d9971a9a..00000000
--- a/examples/21_full_circle/src/fal.ts
+++ /dev/null
@@ -1,98 +0,0 @@
-/**
- * Minimal fal.ai queue client: submit, poll, fetch the result, download files.
- * Every call is priced up front from the model's published unit price so the
- * pipeline can print what a run cost.
- */
-import fs from "node:fs/promises";
-import path from "node:path";
-
-const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
-
-/** USD per unit, from https://api.fal.ai/v1/models/pricing (October 2026). */
-export const PRICING = {
- "fal-ai/z-image/turbo": { usd: 0.005, per: "megapixel" },
- "minimax/h3-max-turbo/image-to-video": { usd: 0.015, per: "second" },
- "minimax/h3-max/image-to-video": { usd: 0.03, per: "second" },
- "minimax/music-3": { usd: 0.002, per: "second" },
-} as const;
-
-export type FalModel = keyof typeof PRICING;
-
-export const estimate = (model: FalModel, units: number) =>
- PRICING[model].usd * units;
-
-export class FalClient {
- private spent = 0;
-
- constructor(private readonly key: string) {}
-
- get totalUsd(): number {
- return this.spent;
- }
-
- /** Submits to the fal queue, polls until done, returns the JSON result. */
- async run<T>(
- model: FalModel,
- input: Record<string, unknown>,
- units: number,
- ): Promise<T> {
- const headers = {
- Authorization: `Key ${this.key}`,
- "Content-Type": "application/json",
- };
- const submit = await request(`https://queue.fal.run/${model}`, {
- method: "POST",
- headers,
- body: JSON.stringify(input),
- });
- if (!submit.ok)
- throw new Error(
- `${model} submit → ${submit.status} ${await submit.text()}`,
- );
- const { status_url, response_url } = (await submit.json()) as {
- status_url: string;
- response_url: string;
- };
-
- for (;;) {
- await sleep(3000);
- const res = await request(status_url, { headers });
- const { status } = (await res.json()) as { status: string };
- if (status === "COMPLETED") break;
- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS")
- throw new Error(`${model} → ${status}`);
- }
- const result = await request(response_url, { headers });
- if (!result.ok)
- throw new Error(
- `${model} result → ${result.status} ${await result.text()}`,
- );
- this.spent += estimate(model, units);
- return (await result.json()) as T;
- }
-}
-
-/** fetch that rides out transient network failures (long polls drop connections). */
-async function request(url: string, init?: RequestInit): Promise<Response> {
- for (let attempt = 1; ; attempt++) {
- try {
- return await fetch(url, init);
- } catch (err) {
- if (attempt === 5) throw err;
- await sleep(2000 * attempt);
- }
- }
-}
-
-export async function download(url: string, dest: string): Promise<void> {
- const res = await request(url);
- if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);
- await fs.mkdir(path.dirname(dest), { recursive: true });
- await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));
-}
-
-export const exists = (p: string) =>
- fs.access(p).then(
- () => true,
- () => false,
- );
diff --git a/examples/21_full_circle/src/gateway.ts b/examples/21_full_circle/src/gateway.ts
new file mode 100644
index 00000000..7138f15f
--- /dev/null
+++ b/examples/21_full_circle/src/gateway.ts
@@ -0,0 +1,45 @@
+/**
+ * AI gateway helpers: model pricing estimates, download, and file existence.
+ */
+import fs from "node:fs/promises";
+import path from "node:path";
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+/** USD per unit, from https://framefields.dev/models (October 2026). */
+export const PRICING = {
+ "fal-ai/z-image/turbo": { usd: 0.005, per: "megapixel" },
+ "minimax/h3-max-turbo/image-to-video": { usd: 0.015, per: "second" },
+ "minimax/h3-max/image-to-video": { usd: 0.03, per: "second" },
+ "minimax/music-3": { usd: 0.002, per: "second" },
+} as const;
+
+export type GatewayModel = keyof typeof PRICING;
+
+export const estimate = (model: GatewayModel, units: number) =>
+ PRICING[model].usd * units;
+
+/** fetch that rides out transient network failures. */
+async function request(url: string, init?: RequestInit): Promise<Response> {
+ for (let attempt = 1; ; attempt++) {
+ try {
+ return await fetch(url, init);
+ } catch (err) {
+ if (attempt === 5) throw err;
+ await sleep(2000 * attempt);
+ }
+ }
+}
+
+export async function download(url: string, dest: string): Promise<void> {
+ const res = await request(url);
+ if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);
+ await fs.mkdir(path.dirname(dest), { recursive: true });
+ await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));
+}
+
+export const exists = (p: string) =>
+ fs.access(p).then(
+ () => true,
+ () => false,
+ );
diff --git a/examples/21_full_circle/src/generate-assets.ts b/examples/21_full_circle/src/generate-assets.ts
index 4a772323..a828f544 100644
--- a/examples/21_full_circle/src/generate-assets.ts
+++ b/examples/21_full_circle/src/generate-assets.ts
@@ -16,6 +16,7 @@ import { execFile } from "node:child_process";
import fs from "node:fs/promises";
import path from "node:path";
import dotenv from "dotenv";
+import { ai } from "framefields";
import { analyzeBeatGrid } from "./beat-grid.js";
import {
type Circle,
@@ -23,7 +24,7 @@ import {
cleanTrack,
measureCircle,
} from "./circles.js";
-import { download, estimate, exists, FalClient } from "./fal.js";
+import { download, estimate, exists } from "./gateway.js";
import { ASSETS, ROOT } from "./paths.js";
import { SCORE, SCORE_SECONDS, SHOTS } from "./shots.js";
import { readWav } from "./wav.js";
@@ -47,10 +48,11 @@ const STILL_MP = (STILL.width * STILL.height) / 1e6;
const elapsed = (t0: number) => `${((Date.now() - t0) / 1000).toFixed(0)}s`;
-function client(): FalClient {
- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;
- if (!key) throw new Error("FAL_API_KEY missing from .env");
- return new FalClient(key);
+let spent = 0;
+
+function checkApiKey(): void {
+ const key = process.env.FRAMEFIELDS_API_KEY;
+ if (!key) throw new Error("FRAMEFIELDS_API_KEY missing from .env");
}
async function fonts() {
@@ -62,30 +64,34 @@ async function fonts() {
}
}
-async function images(fal: FalClient) {
+async function images() {
+ checkApiKey();
await Promise.all(
SHOTS.map(async (shot) => {
const dest = path.join(ASSETS, `${shot.id}.png`);
if (await exists(dest)) return;
const t0 = Date.now();
- const out = await fal.run<{ images: { url: string }[] }>(
+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>(
"fal-ai/z-image/turbo",
{
- prompt: shot.image,
- image_size: STILL,
- num_inference_steps: 8,
- output_format: "png",
+ input: {
+ prompt: shot.image,
+ image_size: STILL,
+ num_inference_steps: 8,
+ output_format: "png",
+ },
},
- STILL_MP,
);
await download(out.images[0].url, dest);
+ spent += estimate("fal-ai/z-image/turbo", STILL_MP);
console.log(`image ${shot.id} (${elapsed(t0)})`);
}),
);
}
-/** fal accepts data URIs for image inputs, so the local still is sent as-is. */
-async function videos(fal: FalClient, target?: string) {
+/** The gateway accepts data URIs for image inputs, so the local still is sent as-is. */
+async function videos(target?: string) {
+ checkApiKey();
const shotList = target ? SHOTS.filter((s) => s.id === target) : SHOTS;
await Promise.all(
shotList.map(async (shot) => {
@@ -93,37 +99,43 @@ async function videos(fal: FalClient, target?: string) {
if (await exists(dest)) return;
const still = await fs.readFile(path.join(ASSETS, `${shot.id}.png`));
const t0 = Date.now();
- const out = await fal.run<{ video: { url: string } }>(
+ const { data: out } = await ai.subscribe<{ video: { url: string } }>(
shot.model,
{
- prompt: shot.motion,
- image_url: `data:image/png;base64,${still.toString("base64")}`,
- duration: shot.seconds,
- resolution: "1080P",
- prompt_expansion_mode: "disabled",
+ input: {
+ prompt: shot.motion,
+ image_url: `data:image/png;base64,${still.toString("base64")}`,
+ duration: shot.seconds,
+ resolution: "1080P",
+ prompt_expansion_mode: "disabled",
+ },
},
- shot.seconds,
);
await download(out.video.url, dest);
+ spent += estimate(shot.model, shot.seconds);
console.log(`video ${shot.id} via ${shot.model} (${elapsed(t0)})`);
}),
);
}
-async function music(fal: FalClient) {
+async function music() {
+ checkApiKey();
const dest = path.join(ASSETS, "score.wav");
if (await exists(dest)) return;
const t0 = Date.now();
- const out = await fal.run<{
+ const { data: out } = await ai.subscribe<{
audio: { url: string };
duration: number;
seed: number;
- }>(
- "minimax/music-3",
- { prompt: SCORE.prompt, lyrics: SCORE.lyrics, duration: SCORE_SECONDS },
- SCORE_SECONDS,
- );
+ }>("minimax/music-3", {
+ input: {
+ prompt: SCORE.prompt,
+ lyrics: SCORE.lyrics,
+ duration: SCORE_SECONDS,
+ },
+ });
await download(out.audio.url, dest);
+ spent += estimate("minimax/music-3", SCORE_SECONDS);
console.log(
`music score ${out.duration.toFixed(1)}s seed ${out.seed} (${elapsed(t0)})`,
);
@@ -224,18 +236,15 @@ await fs.mkdir(ASSETS, { recursive: true });
if (step === "cost") {
cost();
} else {
- // The analysis steps are local; only generation needs a fal key.
- const fal = runs("fonts", "images", "videos", "music") ? client() : null;
if (runs("fonts")) await fonts();
- if (fal && runs("images")) await images(fal);
+ if (runs("images")) await images();
// Music and motion are independent; run them side by side.
await Promise.all([
- fal && runs("videos") ? videos(fal, target) : undefined,
- fal && runs("music") ? music(fal) : undefined,
+ runs("videos") ? videos(target) : undefined,
+ runs("music") ? music() : undefined,
]);
if (runs("music", "grid")) await grid();
if (runs("videos", "circles")) await circles(target);
- if (fal && fal.totalUsd > 0)
- console.log(`spent ~$${fal.totalUsd.toFixed(3)}`);
+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);
}
process.exit(0);
diff --git a/examples/22_framefields_launch/README.md b/examples/22_framefields_launch/README.md
index 6dd42bb6..6108737c 100644
--- a/examples/22_framefields_launch/README.md
+++ b/examples/22_framefields_launch/README.md
@@ -31,7 +31,7 @@ pnpm test
| Images | `fal-ai/flux-2-pro`: a backdrop, a camera on a green screen, a light leak on black, a paper grain (`images.ts`) | `assets/images/` |
The soundtrack is `assets/song.mp3` exactly as generated: nothing stretches, shifts or re-masters it, so
-the film's clock is the take's clock. `FAL_API_KEY` is read from the repo root `.env`. The music model
+the film's clock is the take's clock. `FRAMEFIELDS_API_KEY` is read from the repo root `.env`. The music model
treats section lengths as guidance, so nothing in the picture assumes the plan held: regenerate the song
and the film re-cuts itself.
diff --git a/examples/22_framefields_launch/src/fal.ts b/examples/22_framefields_launch/src/fal.ts
deleted file mode 100644
index e1e67ba5..00000000
--- a/examples/22_framefields_launch/src/fal.ts
+++ /dev/null
@@ -1,92 +0,0 @@
-/**
- * Minimal fal.ai queue client (after examples/21_full_circle): submit, poll,
- * fetch the result. Each call is priced from the model's published unit price
- * so a run reports what it cost.
- */
-import fs from "node:fs/promises";
-import path from "node:path";
-
-const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
-
-/** USD per unit, from https://api.fal.ai/v1/models/pricing (October 2026). */
-export const PRICING = {
- "fal-ai/elevenlabs/music/v2.5": { usd: 0.00007, per: "compute second" },
- "fal-ai/elevenlabs/speech-to-text/scribe-v2": { usd: 0.008, per: "minute" },
- "fal-ai/demucs": { usd: 0.0007, per: "second" },
- "fal-ai/flux-2-pro": { usd: 0.03, per: "megapixel" },
-} as const;
-
-export type FalModel = keyof typeof PRICING;
-
-export class FalClient {
- private spent = 0;
-
- constructor(private readonly key: string) {}
-
- get totalUsd(): number {
- return this.spent;
- }
-
- async run<T>(
- model: FalModel,
- input: Record<string, unknown>,
- units: number,
- ): Promise<T> {
- const headers = {
- Authorization: `Key ${this.key}`,
- "Content-Type": "application/json",
- };
- const submit = await request(`https://queue.fal.run/${model}`, {
- method: "POST",
- headers,
- body: JSON.stringify(input),
- });
- if (!submit.ok)
- throw new Error(
- `${model} submit → ${submit.status} ${await submit.text()}`,
- );
- const { status_url, response_url } = (await submit.json()) as {
- status_url: string;
- response_url: string;
- };
- for (;;) {
- await sleep(3000);
- const res = await request(status_url, { headers });
- const { status } = (await res.json()) as { status: string };
- if (status === "COMPLETED") break;
- if (status !== "IN_QUEUE" && status !== "IN_PROGRESS")
- throw new Error(`${model} → ${status}`);
- }
- const result = await request(response_url, { headers });
- if (!result.ok)
- throw new Error(
- `${model} result → ${result.status} ${await result.text()}`,
- );
- this.spent += PRICING[model].usd * units;
- return (await result.json()) as T;
- }
-}
-
-async function request(url: string, init?: RequestInit): Promise<Response> {
- for (let attempt = 1; ; attempt++) {
- try {
- return await fetch(url, init);
- } catch (err) {
- if (attempt === 5) throw err;
- await sleep(2000 * attempt);
- }
- }
-}
-
-export async function download(url: string, dest: string): Promise<void> {
- const res = await request(url);
- if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);
- await fs.mkdir(path.dirname(dest), { recursive: true });
- await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));
-}
-
-export const exists = (p: string) =>
- fs.access(p).then(
- () => true,
- () => false,
- );
diff --git a/examples/22_framefields_launch/src/gateway.ts b/examples/22_framefields_launch/src/gateway.ts
new file mode 100644
index 00000000..9f5c15c1
--- /dev/null
+++ b/examples/22_framefields_launch/src/gateway.ts
@@ -0,0 +1,41 @@
+/**
+ * AI gateway helpers: model pricing estimates, download, and file existence.
+ */
+import fs from "node:fs/promises";
+import path from "node:path";
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
+
+/** USD per unit, from https://framefields.dev/models (October 2026). */
+export const PRICING = {
+ "fal-ai/elevenlabs/music/v2.5": { usd: 0.00007, per: "compute second" },
+ "fal-ai/elevenlabs/speech-to-text/scribe-v2": { usd: 0.008, per: "minute" },
+ "fal-ai/demucs": { usd: 0.0007, per: "second" },
+ "fal-ai/flux-2-pro": { usd: 0.03, per: "megapixel" },
+} as const;
+
+export type GatewayModel = keyof typeof PRICING;
+
+async function request(url: string, init?: RequestInit): Promise<Response> {
+ for (let attempt = 1; ; attempt++) {
+ try {
+ return await fetch(url, init);
+ } catch (err) {
+ if (attempt === 5) throw err;
+ await sleep(2000 * attempt);
+ }
+ }
+}
+
+export async function download(url: string, dest: string): Promise<void> {
+ const res = await request(url);
+ if (!res.ok) throw new Error(`GET ${url} → ${res.status}`);
+ await fs.mkdir(path.dirname(dest), { recursive: true });
+ await fs.writeFile(dest, Buffer.from(await res.arrayBuffer()));
+}
+
+export const exists = (p: string) =>
+ fs.access(p).then(
+ () => true,
+ () => false,
+ );
diff --git a/examples/22_framefields_launch/src/generate-images.ts b/examples/22_framefields_launch/src/generate-images.ts
index 59fa3c5a..19aea8f9 100644
--- a/examples/22_framefields_launch/src/generate-images.ts
+++ b/examples/22_framefields_launch/src/generate-images.ts
@@ -1,6 +1,6 @@
/**
* pnpm images → assets/images/*.png: the layers the compositing chapter
- * stacks, generated with FLUX.2 [pro] on fal. Each is made for one
+ * stacks, generated with FLUX.2 [pro] on the AI gateway. Each is made for one
* compositing operation:
* backdrop the plate everything sits on
* subject a camera on a flat green screen, for ColorKey to pull
@@ -9,30 +9,39 @@
* Idempotent: delete an image to regenerate it.
*/
import path from "node:path";
-import { download, exists, FalClient } from "./fal.js";
+import { ai } from "framefields";
+import { download, exists, PRICING } from "./gateway.js";
import { IMAGE_SIZE, IMAGES, type ImageName, image } from "./images.js";
import { ROOT } from "./paths.js";
async function main() {
process.loadEnvFile(path.join(ROOT, ".env"));
- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;
- if (!key) throw new Error("FAL_API_KEY missing from the repo root .env");
- const fal = new FalClient(key);
+ const key = process.env.FRAMEFIELDS_API_KEY;
+ if (!key)
+ throw new Error("FRAMEFIELDS_API_KEY missing from the repo root .env");
+ let spent = 0;
const megapixels = (IMAGE_SIZE.width * IMAGE_SIZE.height) / 1e6;
for (const [name, prompt] of Object.entries(IMAGES) as [
ImageName,
string,
][]) {
if (await exists(image(name))) continue;
- const out = await fal.run<{ images: { url: string }[] }>(
+ const { data: out } = await ai.subscribe<{ images: { url: string }[] }>(
"fal-ai/flux-2-pro",
- { prompt, image_size: IMAGE_SIZE, output_format: "png", seed: 22 },
- megapixels,
+ {
+ input: {
+ prompt,
+ image_size: IMAGE_SIZE,
+ output_format: "png",
+ seed: 22,
+ },
+ },
);
await download(out.images[0].url, image(name));
+ spent += PRICING["fal-ai/flux-2-pro"].usd * megapixels;
console.log(`image ${name}`);
}
- console.log(`fal $${fal.totalUsd.toFixed(3)}`);
+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);
}
await main();
diff --git a/examples/22_framefields_launch/src/generate-song.ts b/examples/22_framefields_launch/src/generate-song.ts
index cc6d539a..6cdb1441 100644
--- a/examples/22_framefields_launch/src/generate-song.ts
+++ b/examples/22_framefields_launch/src/generate-song.ts
@@ -17,7 +17,8 @@
import { execFileSync } from "node:child_process";
import fs from "node:fs/promises";
import path from "node:path";
-import { download, exists, FalClient, PRICING } from "./fal.js";
+import { ai } from "framefields";
+import { download, exists, PRICING } from "./gateway.js";
import { BAR_SEC, BPM, PLAN_SEC, SAMPLE_RATE } from "./grid.js";
import { alignLyrics, type HeardWord } from "./music/align.js";
import { analyzeBeatGrid } from "./music/beat-grid.js";
@@ -41,17 +42,19 @@ const stem = (name: string) => asset(`stems/${name}.wav`);
const step = process.argv[2] ?? "all";
-function client(): FalClient {
+let spent = 0;
+
+function checkApiKey(): void {
process.loadEnvFile(path.join(ROOT, ".env"));
- const key = process.env.FAL_API_KEY ?? process.env.FAL_KEY;
- if (!key) throw new Error("FAL_API_KEY missing from the repo root .env");
- return new FalClient(key);
+ const key = process.env.FRAMEFIELDS_API_KEY;
+ if (!key)
+ throw new Error("FRAMEFIELDS_API_KEY missing from the repo root .env");
}
/** Generous upper bound on the music model's compute time, for the estimate. */
const SONG_COMPUTE_SEC = 120;
-async function song(fal: FalClient, seed: number): Promise<void> {
+async function song(seed: number): Promise<void> {
if (await exists(SONG)) return;
const durations = sectionDurationsMs();
// v2.5 has no global styles: the first chunk's styles set the whole song's
@@ -66,16 +69,18 @@ async function song(fal: FalClient, seed: number): Promise<void> {
})),
};
const t0 = Date.now();
- const out = await fal.run<{ audio: { url: string } }>(
+ const { data: out } = await ai.subscribe<{ audio: { url: string } }>(
"fal-ai/elevenlabs/music/v2.5",
- { composition_plan: plan, seed, output_format: "mp3_48000_192" },
- SONG_COMPUTE_SEC,
+ {
+ input: { composition_plan: plan, seed, output_format: "mp3_48000_192" },
+ },
);
await download(out.audio.url, SONG);
await fs.writeFile(
SONG_META,
JSON.stringify({ url: out.audio.url, seed, plan }, null, 2),
);
+ spent += PRICING["fal-ai/elevenlabs/music/v2.5"].usd * SONG_COMPUTE_SEC;
console.log(
`song ${SONG} seed ${seed} (${((Date.now() - t0) / 1000).toFixed(0)} s)`,
);
@@ -85,21 +90,21 @@ async function song(fal: FalClient, seed: number): Promise<void> {
type StemName = "vocals" | "drums";
const STEM_NAMES: StemName[] = ["vocals", "drums"];
-async function stems(fal: FalClient): Promise<void> {
+async function stems(): Promise<void> {
if (await exists(STEMS)) return;
const { url } = JSON.parse(await fs.readFile(SONG_META, "utf8")) as {
url: string;
};
- const out = await fal.run<Record<StemName, { url: string } | null>>(
- "fal-ai/demucs",
- {
+ const { data: out } = await ai.subscribe<
+ Record<StemName, { url: string } | null>
+ >("fal-ai/demucs", {
+ input: {
audio_url: url,
model: "htdemucs_ft",
stems: STEM_NAMES,
output_format: "wav",
},
- PLAN_SEC,
- );
+ });
const urls: Partial<Record<StemName, string>> = {};
for (const name of STEM_NAMES) {
const file = out[name];
@@ -108,6 +113,7 @@ async function stems(fal: FalClient): Promise<void> {
urls[name] = file.url;
}
await fs.writeFile(STEMS, JSON.stringify(urls, null, 2));
+ spent += PRICING["fal-ai/demucs"].usd * PLAN_SEC;
console.log(`stems ${STEM_NAMES.join(", ")}`);
}
@@ -132,25 +138,28 @@ const KEYTERMS = [
"Illustrator",
];
-async function words(fal: FalClient): Promise<void> {
+async function words(): Promise<void> {
if (await exists(WORDS)) return;
const { vocals: url } = JSON.parse(
await fs.readFile(STEMS, "utf8"),
) as Record<StemName, string>;
- const out = await fal.run<{ words: ScribeWord[] }>(
+ const { data: out } = await ai.subscribe<{ words: ScribeWord[] }>(
"fal-ai/elevenlabs/speech-to-text/scribe-v2",
{
- audio_url: url,
- language_code: "eng",
- keyterms: KEYTERMS,
- tag_audio_events: false,
+ input: {
+ audio_url: url,
+ language_code: "eng",
+ keyterms: KEYTERMS,
+ tag_audio_events: false,
+ },
},
- PLAN_SEC / 60,
);
const heard: HeardWord[] = out.words
.filter((w) => w.type === "word")
.map(({ text, start, end }) => ({ text: text.trim(), start, end }));
await fs.writeFile(WORDS, JSON.stringify(heard, null, 1));
+ spent +=
+ PRICING["fal-ai/elevenlabs/speech-to-text/scribe-v2"].usd * (PLAN_SEC / 60);
console.log(
`words ${heard.length} heard: ${heard.map((h) => h.text).join(" ")}`,
);
@@ -292,11 +301,11 @@ async function skippedLines(): Promise<string[]> {
const MAX_TAKES = 4;
/** Generates takes until one sings every line, setting rejected ones aside. */
-async function generate(fal: FalClient): Promise<void> {
+async function generate(): Promise<void> {
for (let take = 1; ; take++) {
- await song(fal, take);
- await stems(fal);
- await words(fal);
+ await song(take);
+ await stems();
+ await words();
const skipped = await skippedLines();
if (!skipped.length) return;
const rejected = path.resolve(
@@ -324,9 +333,9 @@ if (step === "cost") {
);
} else {
if (step === "all") {
- const fal = client();
- await generate(fal);
- console.log(`fal $${fal.totalUsd.toFixed(3)}`);
+ checkApiKey();
+ await generate();
+ if (spent > 0) console.log(`spent ~$${spent.toFixed(3)}`);
}
await measure();
}
diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.md
index 3a22c7bb..38a735ce 100644
--- a/plugins/framefields/skills/framefields/SKILL.md
+++ b/plugins/framefields/skills/framefields/SKILL.md
@@ -16,6 +16,7 @@ This skill takes a project from nothing to a verified first render. For anything
| Post-processing, color grading, VFX, on-device vision (tracking, cutouts, pose) | `framefields-effects` |
| Frame inspection, frame grids, pixel tests, video export and QA | `framefields-render` |
| Generating images, video, music, voice with AI models (`ai`) | this skill, section 6 |
+| Reporting bugs or engine defects to GitHub | this skill, section 8 |
---
@@ -255,7 +256,7 @@ Never go straight to a full video render. On a machine without a GPU every rende
4. `npm run preview` when a person is watching: they play the film with sound before you export (see below).
5. `npm run render` once frames and motion look right.
-Show the user the frame PNGs or the grid before the full render. `framefields-render` covers pixel-probe tests, render QA (`renderVideo({ qa })`), and diagnosing blank frames. If you suspect an issue or defect in the engine itself, file an issue at https://github.com/gatewai-dev/framefields.
+Show the user the frame PNGs or the grid before the full render. `framefields-render` covers pixel-probe tests, render QA (`renderVideo({ qa })`), and diagnosing blank frames. If you suspect an issue or defect in the engine itself or ai gateway, see [section 8](#8-report-engine-issues).
### Live preview
@@ -339,3 +340,20 @@ Then `Layer.image("assets/generated/street.jpg")` / `Layer.video("assets/generat
## 7. Grow the film
Once the first render works, move to the specialist skills: `framefields-compose` for splitting the film into scene modules on a beat grid, `framefields-effects` for grading, grain, and vision-driven effects, and `framefields-render` for tests and delivery checks. Keep `theme.ts` as the single source of sizes, colors, fonts, and timing as the project grows.
+
+## 8. Report engine issues
+
+When you encounter an unexpected crash, native WebGPU / Dawn failure, WGSL shader compilation error, compositor schema validation defect, or behavior bug in `framefields` itself (distinguishable from project-level configuration or asset errors):
+
+1. **Ask the user first:** Never open an issue autonomously without explicit confirmation. Explain what went wrong, summarize the reproduction, and ask if they would like an issue reported to `gatewai-dev/framefields`.
+2. **Collect diagnostic details:**
+ - Operating system and architecture (`process.platform`, `process.arch`).
+ - Node.js version (`node --version`) and `framefields` version.
+ - GPU backend and device (run `describeGpu()` or note Metal, Vulkan, DX12, or CPU lavapipe/WARP).
+ - Minimal reproduction code (the smallest composition or script that triggers it).
+ - Exact error message, full stack trace, or validation failure log.
+3. **Open the issue:** Once the user confirms, submit the issue via GitHub CLI:
+ ```bash
+ gh issue create --repo gatewai-dev/framefields --title "<concise summary>" --body "<diagnostic details and repro>"
+ ```
+ If `gh` is unavailable or not authenticated in the environment, output the title, formatted body, and a direct link for the user: https://github.com/gatewai-dev/framefields/issues/new.