Code-first video & image, rendered natively on WebGPU.
Use SDK for Framefields media generation and configure MCP for Claude Code (#56)
Okan Aslankanf728bc097b1a
diff --git a/plugins/framefields/.claude-plugin/plugin.json b/plugins/framefields/.claude-plugin/plugin.jsonindex ea06c266..a42c9b83 100644--- a/plugins/framefields/.claude-plugin/plugin.json+++ b/plugins/framefields/.claude-plugin/plugin.json@@ -18,5 +18,10 @@"skills"],"homepage": "https://github.com/gatewai-dev/framefields#agent-skills--plugins",- "repository": "https://github.com/gatewai-dev/framefields"+ "repository": "https://github.com/gatewai-dev/framefields",+ "icon": "./assets/logo.png",+ "documentationUrl": "https://framefields.dev/docs/",+ "supportUrl": "https://github.com/gatewai-dev/framefields/issues",+ "privacyPolicyUrl": "https://framefields.dev/privacy/",+ "termsOfServiceUrl": "https://framefields.dev/terms/"}diff --git a/plugins/framefields/README.md b/plugins/framefields/README.mdindex 520ccb6c..fd922b1b 100644--- a/plugins/framefields/README.md+++ b/plugins/framefields/README.md@@ -11,13 +11,26 @@ Agent skills for [framefields](https://github.com/gatewai-dev/framefields), a coSkills load automatically when a task matches, for example *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*.+## Installation++Install into Claude Code via the marketplace:++```bash+/plugin install framefields --marketplace gatewai-dev/framefields+```+## How it works-This plugin contains skills and a connection to the hosted Framefields MCP server. It bundles no executables, hooks, or package launchers. The engine itself is the [`framefields`](https://www.npmjs.com/package/framefields) npm package, which the skills tell your agent to add to your project (`npm install framefields`, Node.js 22 or later).+This plugin contains skills and a read-only connection to the hosted Framefields MCP server for model information and price estimates. Generate media through the Framefields SDK using a separately configured gateway API key. It bundles no executables, hooks, or package launchers. The engine itself is the [`framefields`](https://www.npmjs.com/package/framefields) npm package, which the skills tell your agent to add to your project (`npm install framefields`, Node.js 22 or later).++The connected MCP server exposes read-only tools to assist in selecting models and inputs:+- `list_models`: List supported AI models, identifiers, and tasks+- `get_model_schema`: Get the input and output JSON schema for any catalog model+- `estimate_generation_cost`: Check published rates and estimated costs before running a script## Data and privacy-When you connect the MCP server and request generation, prompts and other inputs are sent to the Framefields gateway and the provider running the selected model. The gateway stores generated outputs and usage records. Standalone code using `ai` sends generation requests and uploaded files through the gateway as well. On-device vision downloads model weights on first use and verifies them by SHA-256. See the [privacy policy](https://framefields.dev/privacy/) for details.+When you run code using `ai`, prompts, inputs and uploaded files are sent to the Framefields gateway and the provider running the selected model. The gateway stores generated outputs and usage records. The MCP connection provides model information and price estimates; it does not generate media. On-device vision downloads model weights on first use and verifies them by SHA-256. See the [privacy policy](https://framefields.dev/privacy/) for details.## Licensediff --git a/plugins/framefields/plugin.json b/plugins/framefields/plugin.jsonindex 718a352b..08136cd3 100644--- a/plugins/framefields/plugin.json+++ b/plugins/framefields/plugin.json@@ -22,7 +22,7 @@"interface": {"displayName": "Framefields","shortDescription": "Turn ideas into videos",- "longDescription": "Framefields helps coding agents create videos with code. Set up projects, compose scenes with layout, typography, animation, 3D and audio, apply visual effects, and inspect or export rendered frames and video.\n\nDesigned for developers making motion graphics, product videos and programmatic films. The SDK runs in Node.js and uses WebGPU or a supported software renderer. On-device vision features download model weights on first use.\n\nThe Framefields gateway also supports generating image, video and audio assets for use in a project.",+ "longDescription": "Framefields helps coding agents create videos with code. Set up projects, compose scenes with layout, typography, animation, 3D and audio, apply visual effects, and inspect or export rendered frames and video.\n\nDesigned for developers making motion graphics, product videos and programmatic films. The SDK runs in Node.js and uses WebGPU or a supported software renderer. On-device vision features download model weights on first use.\n\nGenerate image, video and audio assets through the Framefields SDK. The connected server provides model information and price estimates.","developerName": "Gatewai Studio","category": "Creativity","capabilities": [@@ -56,7 +56,7 @@"commerce_description": "Optional AI asset generation uses prepaid credits purchased on framefields.dev. Local composition and rendering do not require a purchase."},"publication": {- "release_notes": "Updates the skills for Codex credential handling and adds public listing and review metadata.",+ "release_notes": "Uses the SDK for media creation and keeps the connected server focused on model information and cost estimates.","countries": []},"requires_local_executor": truediff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 0dc505b7..374b04ac 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -293,9 +293,13 @@ A failed job returns `{ code, hint, retryable }`; resubmit unchanged only when `## 6. Generate assets with AI-When the connected Framefields MCP server is available, use its model search and schema tools to choose inputs, then its generation and status/result tools to produce assets. The connection uses account sign-in; do not ask for or expose an API key. Generation may consume credits, so tell the user the intended model, number of calls, and applicable cost and obtain their approval before a paid request. Save results to the project before using them in a film. The code-based gateway workflow below is an alternative for standalone scripts.+Use the connected Framefields MCP server for `list_models`, `get_model_schema`, and `estimate_generation_cost` when available (in Claude Code sessions, these are exposed as `mcp__plugin_framefields_framefields__list_models`, `mcp__plugin_framefields_framefields__get_model_schema`, and `mcp__plugin_framefields_framefields__estimate_generation_cost`). Use `import { ai } from "framefields"` in a local script to generate, edit, or process media; MCP does not submit or retrieve generation jobs. The SDK authenticates with a separate gateway API key (`FRAMEFIELDS_API_KEY`), not the MCP OAuth connection. Use only the currently supported catalog models and the input fields from the selected model's schema.-`framefields` exports `ai`: hundreds of AI models (images, video, music, sound effects, voice, upscaling, background removal) through the framefields AI gateway, each named by its catalog id and billed to the user's framefields account. `run`, `subscribe`, `stream`, `queue` and `storage` cover every model.+- `list_models`: inspect catalog models and category tasks (`text-to-image`, `text-to-video`, `speech-to-text`, etc.).+- `get_model_schema`: check required and optional inputs for the model before writing the script.+- `estimate_generation_cost`: check published rates or request cost estimates before running paid generation.++`framefields` exports `ai` for image, video and audio operations through the Framefields gateway. Each request uses a model ID and account credits. `run`, `subscribe`, `stream`, `queue`, and `storage` support the gateway workflow.```typescriptimport { ai } from "framefields";@@ -304,7 +308,7 @@ const apiKey = process.env.FRAMEFIELDS_API_KEY;if (!apiKey) throw new Error("Set FRAMEFIELDS_API_KEY before generating assets.");ai.config({ apiKey });-const { data, requestId } = await ai.subscribe<{ images: { url: string }[] }>("flux/schnell", {+const { data, requestId } = await ai.subscribe<{ images: { url: string }[] }>("seedream/4.5", {input: { prompt: "Neon street at night, rain, 35mm", image_size: "landscape_16_9" },});console.log(data.images[0].url);@@ -333,7 +337,7 @@ async function save(url: string, name: string): Promise<string> {return file;}-const still = await ai.subscribe<{ images: { url: string }[] }>("flux/schnell", {+const still = await ai.subscribe<{ images: { url: string }[] }>("seedream/4.5", {input: { prompt: "Neon street at night, rain, 35mm", image_size: "landscape_16_9", seed: 7 },});await save(still.data.images[0].url, "street.jpg");@@ -341,7 +345,7 @@ await save(still.data.images[0].url, "street.jpg");// Feed a local file to a model: put a Blob in the input and it is uploaded for you// (or call ai.storage.upload(blob) yourself and pass the URL).const image_url = new Blob([await fs.readFile(path.join(OUT, "street.jpg"))], { type: "image/jpeg" });-const clip = await ai.subscribe<{ video: { url: string } }>("kling-video/v2.1/standard/image-to-video", {+const clip = await ai.subscribe<{ video: { url: string } }>("seedance/2.0/image-to-video", {input: { prompt: "slow push in, rain falling", image_url, duration: "5" },logs: true,onQueueUpdate: (u) => u.status === "IN_PROGRESS" && u.logs.forEach((l) => console.log(l.message)),@@ -354,7 +358,7 @@ Then `Layer.image("assets/generated/street.jpg")` / `Layer.video("assets/generat- **Pick the model and its input from the catalog.** https://framefields.dev/models lists every model with its id, category, price per unit and input schema. Each model has its own input and output fields (`images[0].url`, `video.url`, `audio.url`, `audio_file.url` …): read the schema and pass the output shape as the type argument, or log `data` once, instead of guessing.- **Long jobs** (video, music): `subscribe` waits and polls. For several at once, `ai.queue.submit(model, { input })` each, keep every `request_id`, then `ai.queue.result(model, { requestId })`; an interrupted script can collect the results instead of paying again. Reading a status or result is free and retried on transient errors; submitting is never retried, so it can't bill twice.-- **Ask before spending.** Before a batch, or before any video model, tell the user which models you will call, how many times, and the catalog price; generate a single draft first and show it.+- **Costs:** Generation consumes credits. When useful, use the MCP price-estimate tool or the model catalog to report an estimate or unit rate; actual charges can vary. Do not invent a total when only a unit rate is known.- Pass a fixed `seed` where the model accepts one, and keep the prompts in `generate.ts`, so a result can be regenerated.- Frame-rate and size: generated clips rarely match the composition's `fps` or aspect. Use `fit: "cover"` on the layer, and check the clip's real duration before cutting to it.