Code-first video & image, rendered natively on WebGPU.
Connect Framefields plugin to MCP gateway (#54)
Okan Aslankan0297339246df
diff --git a/plugins/framefields/.claude-plugin/plugin.json b/plugins/framefields/.claude-plugin/plugin.jsonindex 1550d239..ea06c266 100644--- a/plugins/framefields/.claude-plugin/plugin.json+++ b/plugins/framefields/.claude-plugin/plugin.json@@ -18,13 +18,5 @@"skills"],"homepage": "https://github.com/gatewai-dev/framefields#agent-skills--plugins",- "repository": "https://github.com/gatewai-dev/framefields",- "userConfig": {- "framefields_api_key": {- "type": "string",- "title": "Framefields API key",- "description": "API key for the Framefields AI gateway (https://framefields.dev/dashboard/keys)",- "sensitive": true- }- }+ "repository": "https://github.com/gatewai-dev/framefields"}diff --git a/plugins/framefields/.mcp.json b/plugins/framefields/.mcp.jsonnew file mode 100644index 00000000..c46e6403--- /dev/null+++ b/plugins/framefields/.mcp.json@@ -0,0 +1,9 @@+{+ "mcpServers": {+ "framefields": {+ "type": "http",+ "url": "https://framefields.dev/api/mcp",+ "oauth": { "scopes": "framefields:media" }+ }+ }+}diff --git a/plugins/framefields/README.md b/plugins/framefields/README.mdindex 29c849e8..520ccb6c 100644--- a/plugins/framefields/README.md+++ b/plugins/framefields/README.md@@ -13,11 +13,11 @@ Skills load automatically when a task matches, for example *"add a film-grain pa## How it works-This plugin contains only skills: Markdown instructions for your agent. It bundles no executables, MCP servers, hooks, or package launchers. The engine itself is the [`framefields`](https://www.npmjs.com/package/framefields) npm package, which the skills tell your agent to add to your project (`npm install framefields`, Node.js 22 or later).+This plugin contains skills and a connection to the hosted Framefields MCP server. It bundles no executables, hooks, or package launchers. The engine itself is the [`framefields`](https://www.npmjs.com/package/framefields) npm package, which the skills tell your agent to add to your project (`npm install framefields`, Node.js 22 or later).## Data and privacy-The plugin sends no data anywhere. When code that uses framefields' on-device vision features runs, the SDK downloads pinned, Apache-2.0 model weights from Hugging Face the first time each model is used and verifies them by SHA-256. When code calls `ai` (AI generation), the request, its inputs and any files uploaded with `ai.storage` go to the framefields gateway at `api.framefields.dev` and on to the model provider that runs the model; the gateway records usage for billing but does not store prompts or outputs. Nothing else leaves your machine.+When you connect the MCP server and request generation, prompts and other inputs are sent to the Framefields gateway and the provider running the selected model. The gateway stores generated outputs and usage records. Standalone code using `ai` sends generation requests and uploaded files through the gateway as well. On-device vision downloads model weights on first use and verifies them by SHA-256. See the [privacy policy](https://framefields.dev/privacy/) for details.## Licensediff --git a/plugins/framefields/mcp.json b/plugins/framefields/mcp.jsonnew file mode 100644index 00000000..e846fc7b--- /dev/null+++ b/plugins/framefields/mcp.json@@ -0,0 +1,9 @@+{+ "$schema": "https://agent-plugins.org/schemas/1.0.0/mcp.schema.json",+ "mcpServers": {+ "framefields": {+ "type": "streamable-http",+ "url": "https://framefields.dev/api/mcp"+ }+ }+}diff --git a/plugins/framefields/plugin.json b/plugins/framefields/plugin.jsonindex 42a7cd3a..93df4524 100644--- a/plugins/framefields/plugin.json+++ b/plugins/framefields/plugin.json@@ -4,8 +4,8 @@"version": "2.6.2","description": "Author, render and verify video compositions with the framefields WebGPU SDK.","author": {- "name": "Framefields",- "url": "https://github.com/gatewai-dev"+ "name": "Gatewai Studio",+ "url": "https://framefields.dev"},"homepage": "https://github.com/gatewai-dev/framefields","repository": "https://github.com/gatewai-dev/framefields",@@ -22,16 +22,16 @@"interface": {"displayName": "Framefields","shortDescription": "Code-first video on WebGPU",- "longDescription": "Framefields teaches your coding agent to make videos with code. The skills cover setting up a project with the framefields npm package, building compositions (layout, typography, keyframed animation, 3D scenes, audio), applying effects and color grading, on-device vision such as subject cutouts and tracking, and rendering: single frames, contact sheets, and MP4 export with built-in QA. Everything renders natively on WebGPU in Node.js, without a browser or desktop creative suite.\n\nIntended for developers and agents producing motion graphics, product videos, and programmatic films. Limitations: the plugin contains skills only; rendering needs Node.js 22 or later and a WebGPU-capable GPU (Metal or Vulkan), or a CPU renderer (Mesa lavapipe, Windows WARP) for slower rendering without one, and the vision features download open model weights from Hugging Face on first use. framefields is in beta, so APIs may change.",- "developerName": "Framefields",- "category": "Design",+ "longDescription": "Framefields helps coding agents create videos with code. Set up projects, compose scenes with layout, typography, animation, 3D and audio, apply visual effects, and inspect or export rendered frames and video.\n\nDesigned for developers making motion graphics, product videos and programmatic films. The SDK runs in Node.js and uses WebGPU or a supported software renderer. On-device vision features download model weights on first use. Framefields is in beta and APIs may change.\n\nThe Framefields gateway also supports generating image, video and audio assets for use in a project.",+ "developerName": "Gatewai Studio",+ "category": "Creativity","capabilities": ["Set up a framefields project","Compose and animate video","Apply effects and color grading","Render frames, grids, and MP4"],- "websiteURL": "https://github.com/gatewai-dev/framefields",+ "websiteURL": "https://framefields.dev","supportURL": "https://github.com/gatewai-dev/framefields/issues","defaultPrompt": ["Set up framefields in this project and render a 3-second title card.",@@ -41,12 +41,25 @@"brandColor": "#3B82F6","brandColorDark": "#7DD3FC","composerIcon": "./assets/logo.png",- "logo": "./assets/logo.png"+ "logo": "./assets/logo.png",+ "privacyPolicyURL": "https://framefields.dev/privacy/",+ "termsOfServiceURL": "https://framefields.dev/terms/"},"onboardingSkill": "./skills/framefields/SKILL.md",+ "review": {+ "prompts": [+ "Set up framefields in this project and render a 3-second title card.",+ "Add film grain and a vignette to my framefields composition.",+ "Render a frame grid of my intro so I can check the motion."+ ],+ "commerce": true,+ "commerce_description": "Optional AI asset generation uses prepaid credits purchased on framefields.dev. Local composition and rendering do not require a purchase."+ },"publication": {- "release_notes": "Adds the framefields onboarding skill for setting up a project and a first verified render, and fixes the frame-grid example in framefields-render."- }+ "release_notes": "Updates the skills for Codex credential handling and adds public listing and review metadata.",+ "countries": []+ },+ "requires_local_executor": true}}}diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 78b38562..c3114c06 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -134,7 +134,7 @@ await session.closed; // keep the process alive while the user watches- **Notes pinned on the picture are the user's precise feedback.** Paused, the user clicks the frame (or drags an area) and writes what should change; the note keeps the frame, the spot as fractions of the frame (`x`, `y`, and `w`, `h` for an area) and the text. Each note prints in the output (`note 3 at 0:12.50 (frame 750), spot 42%,31%: …`) with the path of `note-<id>.jpg`, the frame with the spot marked, and all of them are in `.framefields/preview-notes/notes.json` (`.framefields` ignores itself in git; `notesDir` moves it, `false` turns notes off). To act on them: read `notes.json`, open each open note's picture to see exactly what it points at, find the layer in the code from the moment and the spot, make the change, and run the preview again. The tab comes back with the notes, so the user can check them and tick them off. You can also set `"done": true` in `notes.json` yourself. A chat that can't read files gets the same list from the **Copy** button in the Notes panel.- **Point at a moment.** `#t=12.5` in the URL opens the player at 12.5 s (`#frame=750` by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.- **Drive it from a script, not with clicks.** If you can run JavaScript in the page (an agent browser), use `window.framefieldsPreview`: `await seek(seconds)` or `await seekFrame(n)` resolve once that frame is drawn (take the screenshot then), `play()` and `pause()` control playback, `notes()` lists the notes, and `state()` returns `{ state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }`. `<html data-preview="ready">` marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.-- **Show it where the user is.** `startPreview` only serves the page and returns the URL. If your app has a built-in browser (Claude Code's browser pane, Codex's), open the URL there. Otherwise give the user the link, or pass `open: true` to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).+- **Show it where the user is.** `startPreview` only serves the page and returns the URL. If your app has a built-in browser, open the URL there. Otherwise give the user the link, or pass `open: true` to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).- **Playback runs in real time, on any device.** Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Next to the timecode the page shows the frame rate playback reaches, apart from the film's own `fps`; `state()` reports it as `playbackFps`, and `renderMs` is how long the last frame took to draw. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.- Audio is the same mix `renderVideo` muxes (`audio: false` skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the picture skips ahead to catch up with the sound.@@ -171,7 +171,7 @@ Agents often run in sandboxes or CI runners with no GPU. framefields then render- **macOS:** always has a Metal GPU, VMs included; framefields' WebGPU build has no Vulkan, so lavapipe isn't used there.- **Confirm it:** `await describeGpu()` returns `software: true`.- **Budget for speed:** about 12× slower than an Apple M1 Pro's GPU in our Linux test (the smoke project's three videos: 9.7 min against 49 s); it scales with CPU cores. Frames, grids and short exports are fine. Vision layers are the slowest part.-- **Long films:** render on a GPU machine, or push the project and submit it to the framefields render worker (Docker or RunPod, renders a git commit on a GPU; authenticate job submissions with `${user_config.framefields_api_key}` if required, rather than reading credentials from the host machine). Its project checklist and job format: https://framefields.dev/docs/guides/render-service/+- **Long films:** render on a GPU machine, or push the project and submit it to the framefields render worker (Docker or RunPod, renders a git commit on a GPU; if authentication is required, ask the user to set `FRAMEFIELDS_API_KEY` in the process environment and use it without printing or embedding it). Its project checklist and job format: https://framefields.dev/docs/guides/render-service/## Production Render Script Patterndiff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex dabf5066..0dc505b7 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -274,7 +274,7 @@ Show the user the frame PNGs or the grid before the full render. `framefields-re`npm run preview` serves a localhost player (picture, sound, timeline) and prints its URL. The page runs `film.ts` itself and renders with WebGPU in real time, so the user sees the real film; a slow device skips frames to keep up with the sound.- Run it as a background command and read its output: it says when the page opened, prints errors from the page (a failed build, a frame that threw, a missing font), and says why it stopped. It stops 5 s after its last tab closes; just run it again.-- It doesn't open anything. If your app has a built-in browser (Claude Code, Codex), open the URL there; otherwise give the user the link. `startPreview(..., { open: true })` opens their default browser.+- It doesn't open anything. If your app has a built-in browser, open the URL there; otherwise give the user the link. `startPreview(..., { open: true })` opens their default browser.- **Reference assets from the file's URL.** The page runs `film.ts` itself, so asset paths must resolve in the browser too: build them from the module's own URL, `path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../assets/fonts")`, rather than `process.cwd()` or a bare relative path. The preview serves any file inside the project (the folder with `package.json`); pass `root` (one or more paths) to `startPreview` for fonts or media kept elsewhere.- After changing the film, run it again. It takes over the same URL, and the open tab reloads by itself at the same moment.- Link to a moment with `#t=12.5` (seconds). In a browser you can script, `window.framefieldsPreview` has `seek(seconds)`, `play()`, `pause()` and `state()`; `seek` resolves once the frame is drawn, so screenshot after it.@@ -293,18 +293,24 @@ A failed job returns `{ code, hint, retryable }`; resubmit unchanged only when `## 6. Generate assets with AI+When the connected Framefields MCP server is available, use its model search and schema tools to choose inputs, then its generation and status/result tools to produce assets. The connection uses account sign-in; do not ask for or expose an API key. Generation may consume credits, so tell the user the intended model, number of calls, and applicable cost and obtain their approval before a paid request. Save results to the project before using them in a film. The code-based gateway workflow below is an alternative for standalone scripts.+`framefields` exports `ai`: hundreds of AI models (images, video, music, sound effects, voice, upscaling, background removal) through the framefields AI gateway, each named by its catalog id and billed to the user's framefields account. `run`, `subscribe`, `stream`, `queue` and `storage` cover every model.```typescriptimport { ai } from "framefields";+const apiKey = process.env.FRAMEFIELDS_API_KEY;+if (!apiKey) throw new Error("Set FRAMEFIELDS_API_KEY before generating assets.");+ai.config({ apiKey });+const { data, requestId } = await ai.subscribe<{ images: { url: string }[] }>("flux/schnell", {input: { prompt: "Neon street at night, rain, 35mm", image_size: "landscape_16_9" },});console.log(data.images[0].url);```-**The key.** Requests authenticate with an `ff_live_…` key from https://framefields.dev/dashboard/keys. Take the value as `${user_config.framefields_api_key}` that the user chose to share via the plugin's `userConfig` option (`sensitive: true`), instead of reading credentials from the user's environment or files. Configure it with `ai.config({ apiKey: "${user_config.framefields_api_key}" })`. Never read credentials from the user's machine or environment files without authorization, never write the key into source files, and never ask the user to paste keys into chat. Without a key, every call throws before any request is sent. Failures throw `ApiError` with the gateway's `type`: on `insufficient_credits` (status `402`) tell the user to top up at https://framefields.dev/dashboard/billing rather than retrying; `key_limit_reached` means the key's monthly cap; `rate_limited` and `too_many_inflight_requests` mean wait and retry. A `ValidationError` (`422`) lists the bad input fields in `fieldErrors`.+**The key.** Requests authenticate with an `ff_live_…` key from https://framefields.dev/dashboard/keys. For local execution, ask the user to set `FRAMEFIELDS_API_KEY` in the process environment themselves; never ask them to paste a key into chat. The script reads `process.env.FRAMEFIELDS_API_KEY` and fails with an actionable message when it is missing. Do not inspect or print environment variables or credential files, embed the key in source, or commit it. AI generation is optional and does not affect local composition and rendering. Without a key, gateway calls cannot run. Failures throw `ApiError` with the gateway's `type`: on `insufficient_credits` (status `402`) tell the user to top up at https://framefields.dev/dashboard/billing rather than retrying; `key_limit_reached` means the key's monthly cap; `rate_limited` and `too_many_inflight_requests` mean wait and retry. A `ValidationError` (`422`) lists the bad input fields in `fieldErrors`.**Generate once, into `assets/`, in its own script.** Generation costs money and is not deterministic, so it never runs inside `buildFilm()`, the preview, tests or the render worker. Write `src/generate.ts`, run it on purpose, download every output into `assets/generated/` (tracked in Git LFS, as above), and reference the local file from the film. Generated output URLs expire, and the worker renders from the repository.@@ -314,7 +320,9 @@ import fs from "node:fs/promises";import path from "node:path";import { ai } from "framefields";-ai.config({ apiKey: "${user_config.framefields_api_key}" });+const apiKey = process.env.FRAMEFIELDS_API_KEY;+if (!apiKey) throw new Error("Set FRAMEFIELDS_API_KEY before generating assets.");+ai.config({ apiKey });const OUT = path.resolve(import.meta.dirname, "../assets/generated");await fs.mkdir(OUT, { recursive: true });@@ -355,7 +363,7 @@ Then `Layer.image("assets/generated/street.jpg")` / `Layer.video("assets/generatCoding agents interacting with the AI gateway can discover endpoints, inspect credit runway, and handle top-ups programmatically:- **Manifest discovery:** Read `GET https://framefields.dev/.well-known/agent-payment.json` for live auth schemes, pack amounts, and endpoint paths.-- **Runway check:** `GET https://framefields.dev/api/v1/agent/balance` with `Authorization: Bearer ${user_config.framefields_api_key}` returns `{ balance_usd, held_usd, available_usd }`. Query this before expensive generation batches.+- **Runway check:** `GET https://framefields.dev/api/v1/agent/balance` using the `FRAMEFIELDS_API_KEY` process environment variable as the bearer credential (never print it) returns `{ balance_usd, held_usd, available_usd }`. Query this before expensive generation batches.- **Autonomous 402 flow:** When credits are exhausted, the gateway returns HTTP `402 Payment Required` with a dynamic Dodo checkout link:```json{diff --git a/scripts/validate-agent-plugin.mjs b/scripts/validate-agent-plugin.mjsindex cf20e054..e444a90f 100644--- a/scripts/validate-agent-plugin.mjs+++ b/scripts/validate-agent-plugin.mjs@@ -161,6 +161,27 @@ function checkManifests() {}}+function checkMcpManifests() {+ const portableFile = join(PLUGIN_DIR, "mcp.json");+ const claudeFile = join(PLUGIN_DIR, ".mcp.json");+ if (!existsSync(portableFile) || !existsSync(claudeFile)) {+ fail("both portable and Claude MCP configurations are required");+ return;+ }+ const portable = readJson(portableFile).mcpServers?.framefields;+ const claude = readJson(claudeFile).mcpServers?.framefields;+ if (portable?.type !== "streamable-http" || claude?.type !== "http")+ fail("MCP configurations must use their respective HTTP transports");+ if (+ portable?.url !== "https://framefields.dev/api/mcp" ||+ claude?.url !== portable?.url+ )+ fail("MCP configurations must point to the same Framefields endpoint");+ if (claude?.oauth?.scopes !== "framefields:media")+ fail("Claude MCP configuration must request the media scope");+ ok("portable and Claude MCP configurations");+}+function checkSkills() {const skillsDir = join(PLUGIN_DIR, "skills");if (!existsSync(skillsDir)) {@@ -339,6 +360,7 @@ function checkGeneratedEffects() {}checkManifests();+checkMcpManifests();checkSkills();checkSymlinks();checkGeneratedEffects();