Skip to content

Code-first video & image, rendered natively on WebGPU.

rm local test mock refs

okanasl3b7dcab6f860
diff --git a/packages/cli/ROADMAP.md b/packages/cli/ROADMAP.md
new file mode 100644
index 00000000..3d2c70ee
--- /dev/null
+++ b/packages/cli/ROADMAP.md
@@ -0,0 +1,79 @@
+# `ff` roadmap
+
+Status: **Draft** · Date: 2026-10-10 · Companion: `framefields-cloud/docs/ROADMAP.md`
+
+## Direction
+
+The cloud is moving to GitHub-hosted repositories with media in Git LFS (see the cloud roadmap).
+That splits `ff` in two:
+
+- **The local half is the product, and it stays.** `ls`, `check`, `frames`, `grid`, `preview`, `render --local` and `init` are the verification ladder agents and humans use on every change. They don't touch the cloud, and they're open source.
+- **The cloud half shrinks to what Framefields uniquely does:** cloud renders and preview links. Git, auth for git, assets, issues and PRs belong to `git`, `git lfs` and `gh`.
+
+Nothing in the cloud flow may *require* `ff`. A user with only `git` and a browser must be able to
+connect, push, get PR previews and render. `ff` makes it faster, and makes it scriptable for agents.
+
+| Command | Fate |
+|---|---|
+| `ls`, `check`, `frames`, `grid`, `preview`, `render --local`, `config` | Keep |
+| `init` | Keep; also writes `.gitattributes` for LFS (phase 2) |
+| `auth login/status/logout` | Keep (API key); add browser login (phase 1) |
+| `auth setup-git`, hidden `git-credential` | **Remove** (GitHub credentials do this) |
+| `repo create`, `repo fork`, `repo delete` | **Remove** → `repo connect` / `repo disconnect` |
+| `repo view`, `repo list` | Keep, re-pointed at connected repos |
+| `clone` | **Remove** (`git clone` + `git lfs` do it) |
+| `push`, `pull` (`sync.ts`) | **Remove** (plain `git`) |
+| `asset sync/pull/status`, `assets.ts`, `framefields.assets.json` | **Remove**; replaced by `lfs` checks (phase 2) |
+| `issue *`, `pr *` | **Remove** (`gh`) |
+| `render` (cloud), `render status/list/download/cancel` | Keep and extend |
+| `api` | Keep |
+
+Roughly 2,000 of the CLI's ~5,100 source lines go away (`issue.ts`, `pr.ts`, `sync.ts`,
+`clone.ts`, `assets.ts`, `asset.ts`, most of `repo.ts`, the credential parts of `auth.ts` and
+`git/index.ts`), with the matching parts of `test/cloud.test.ts` and `SPEC.md` §cloud.
+
+---
+
+## Phase 1 — Align with GitHub-hosted repos (ships with cloud phase 1)
+
+- [ ] `ff repo connect`: resolve the GitHub remote of the current directory; if the App isn't installed on it, open the install/connect page and wait for it to finish; then write `git config framefields.repo`. `--json` and non-TTY paths print the URL to open instead of prompting.
+- [ ] `ff repo disconnect`, and `ff repo view`/`list` against the new repo shape (GitHub full name, default branch, last push).
+- [ ] `ff auth login` without `--with-token` opens a browser device-code flow (sign in with GitHub on the web, the CLI receives an API key). `FF_TOKEN` stays the CI/agent path.
+- [ ] Remove `auth setup-git`, `git-credential`, `repo create/fork/delete`, `clone`, `push`, `pull`, `issue`, `pr`. In this release they print a one-line pointer ("use `git push`", "use `gh pr create`") and exit 2; delete them one release later.
+- [ ] `render` (cloud) checks the commit is on GitHub (`git ls-remote`) instead of on the Framefields remote.
+- [ ] Bump to 3.0.0: it's a breaking change for anyone scripting the cloud commands.
+
+## Phase 2 — Assets through Git LFS (ships with cloud phase 2)
+
+- [ ] `ff init` writes `.gitattributes` with LFS rules for the manifest's `assets.directory` and common media extensions (`*.mp4 *.mov *.wav *.mp3 *.png *.jpg *.exr *.ttf *.otf *.glb *.onnx`), and warns if `git lfs` isn't installed.
+- [ ] `ff check` adds an **assets** step:
+ - media files over a size threshold that aren't LFS-tracked → error, with the `git lfs track`/`git lfs migrate` command to run;
+ - LFS objects referenced at `HEAD` but missing locally (not pulled) → error naming `git lfs pull`.
+- [ ] `ff lfs setup` for existing projects: write the rules, and print (not run) the `git lfs migrate import` command, since rewriting history is the user's call.
+- [ ] Remove `asset *`, `assets.ts` and every read/write of `framefields.assets.json`. Local rendering already reads files from disk, so nothing in the engine changes.
+
+## Phase 3 — Cloud rendering (ships with cloud phase 3)
+
+- [ ] `ff render --cloud` options matching the plans: `--resolution 4k`, `--fps`, `--format mp4|prores|webm-alpha`, `--chunks auto`.
+- [ ] Clear errors for plan limits (exit 3 with the upgrade URL as the hint) and for compositions that fail on the server ("passes `ff check` locally?" hint).
+- [ ] `ff render --wait` streams the server's progress events (already in the API) to a progress bar on stderr.
+- [ ] Parity test in CI: one example rendered locally and in the cloud must match frame hashes (or a PSNR threshold for encoded output).
+
+## Phase 4 — Preview links (ships with cloud phase 4)
+
+- [ ] `ff open [<composition>]`: open the share page for the current branch's latest preview render; `--json` prints the URL. Agents use it to hand a human a link.
+- [ ] `ff previews` (or `render list --previews`): preview renders for the current branch/PR with their check status.
+- [ ] Engine-side: add an optional `previews` section to `framefields.json` (which compositions and quality to render on push/PR). The format lives in `framefields/project` (SPEC §3.4) so `ff`, the cloud and the engine agree; `ff check` validates it.
+
+## Phase 5 — Agent-native (any time after phase 1; local parts can start now)
+
+- [ ] `ff mcp`: an MCP server over stdio exposing `ls`, `check`, `frames`, `grid`, `render` (local and cloud), `open`. Frames and grids come back as images, so an agent can *see* its change and iterate. This is the differentiator in the pitch, and the local half needs no cloud at all.
+- [ ] Ship a `framefields-render` skill/instructions file that teaches agents the verification ladder through `ff`, replacing per-project `render.ts` notes.
+
+## Not planned
+
+- Re-implementing `gh` (issues, PRs, reviews) or git transport. If a cloud feature needs a git operation, `ff` shells out to `git`.
+- Any asset storage protocol of our own. If Framefields-hosted LFS ships (cloud phase 6), it's a standard LFS endpoint configured in `.lfsconfig`; `ff` only helps write that file.
+
+## Related cleanup spotted while planning
+- [x] `packages/webgpu-renderers/src/slug/slug-font-cache.ts`: removed hardcoded machine-specific font paths in favor of resolving against project font directories and upward traversal.
diff --git a/packages/webgpu-renderers/src/slug/slug-font-cache.ts b/packages/webgpu-renderers/src/slug/slug-font-cache.ts
index 88049e79..ae5c677d 100644
--- a/packages/webgpu-renderers/src/slug/slug-font-cache.ts
+++ b/packages/webgpu-renderers/src/slug/slug-font-cache.ts
@@ -712,10 +712,37 @@ export class SlugFontCache {
if (isNode && !isHttp) {
// Node.js local file environment
const fs = await import(
- /* webpackIgnore: true */ "fs/promises"
+ /* webpackIgnore: true */ "node:fs/promises"
);
- const path = await import(/* webpackIgnore: true */ "path");
+ const path = await import(
+ /* webpackIgnore: true */ "node:path"
+ );
+ const fontDirs: string[] = [];
+ if (process.env.FONTS_DIR) {
+ fontDirs.push(path.resolve(process.env.FONTS_DIR));
+ }
+ if (process.env.FRAMEFIELDS_FONTS_DIR) {
+ fontDirs.push(
+ path.resolve(process.env.FRAMEFIELDS_FONTS_DIR),
+ );
+ }
+
+ // Upward traversal to locate assets/fonts in parent monorepo roots
+ let currentDir = process.cwd();
+ for (let i = 0; i < 5; i++) {
+ fontDirs.push(path.resolve(currentDir, "assets/fonts"));
+ const parent = path.dirname(currentDir);
+ if (parent === currentDir) break;
+ currentDir = parent;
+ }
+
const candidates: string[] = [];
+ const addCandidate = (candidatePath: string) => {
+ if (!candidates.includes(candidatePath)) {
+ candidates.push(candidatePath);
+ }
+ };
+
if (fontUrl && !fontUrl.startsWith("//api/")) {
let fontPath = fontUrl;
if (fontUrl.startsWith("file://")) {
@@ -725,8 +752,11 @@ export class SlugFontCache {
fontPath = fontUrl.replace(/^file:\/\//, "");
}
}
- candidates.push(fontPath);
- candidates.push(path.resolve(process.cwd(), fontPath));
+ addCandidate(fontPath);
+ addCandidate(path.resolve(process.cwd(), fontPath));
+ for (const dir of fontDirs) {
+ addCandidate(path.resolve(dir, path.basename(fontPath)));
+ }
}
const cleanFamily = fontFamily.replace(/['"]/g, "").trim();
const variations = [
@@ -734,29 +764,16 @@ export class SlugFontCache {
cleanFamily.replace(/ /g, ""),
cleanFamily.replace(/ /g, "_"),
];
- for (const v of variations) {
- candidates.push(
- path.resolve(process.cwd(), "assets/fonts", v + ".ttf"),
- );
- candidates.push(
- path.resolve(process.cwd(), "assets/fonts", v + ".otf"),
- );
- candidates.push(
- `/Users/okanaslankan/framefields/assets/fonts/${v}.ttf`,
- );
- candidates.push(
- `/Users/okanaslankan/Gatewai/apps/gatewai-artifex/assets/fonts/${v}.ttf`,
- );
- candidates.push(
- `/Users/okanaslankan/Gatewai/apps/gatewai-backend/src/assets/fonts/${v}/font_file.ttf`,
- );
+ for (const dir of fontDirs) {
+ for (const v of variations) {
+ addCandidate(path.resolve(dir, `${v}.ttf`));
+ addCandidate(path.resolve(dir, `${v}.otf`));
+ }
+ }
+ for (const dir of fontDirs) {
+ addCandidate(path.resolve(dir, "Inter.ttf"));
+ addCandidate(path.resolve(dir, "Inter.otf"));
}
- candidates.push(
- "/Users/okanaslankan/framefields/assets/fonts/Inter.ttf",
- );
- candidates.push(
- "/Users/okanaslankan/Gatewai/apps/gatewai-artifex/assets/fonts/Inter.ttf",
- );
let nodeBuffer: Buffer | null = null;
for (const c of candidates) {
diff --git a/scripts/render-3d-demo.mjs b/scripts/render-3d-demo.mjs
index b31356a6..dc235c64 100644
--- a/scripts/render-3d-demo.mjs
+++ b/scripts/render-3d-demo.mjs
@@ -1,17 +1,18 @@
+import fs from "node:fs/promises";
+import path from "node:path";
import {
CameraAnimation,
Composition,
FontManager,
Layer,
} from "../packages/framefields/dist/index.mjs";
-import fs from "node:fs/promises";
-import path from "node:path";
async function main() {
console.log("=== WebGPU 3D Camera & Multiplane Render Demonstration ===");
- const artifactDir = "/Users/okanaslankan/.gemini/antigravity-ide/brain/18faf2d5-3c9b-408e-8ba7-bcbf855a7dc5";
- const assetsDir = "/Users/okanaslankan/framefields/assets/renders-3d";
+ const assetsDir =
+ process.env.ASSETS_DIR || path.resolve(process.cwd(), "assets/renders-3d");
+ const artifactDir = process.env.ARTIFACT_DIR || assetsDir;
await fs.mkdir(artifactDir, { recursive: true });
await fs.mkdir(assetsDir, { recursive: true });
@@ -99,7 +100,10 @@ async function main() {
);
const dollyBuffer = await compDolly.renderFrame({ frame: 20 });
- await fs.writeFile(path.join(artifactDir, "3d_camera_dolly.png"), dollyBuffer);
+ await fs.writeFile(
+ path.join(artifactDir, "3d_camera_dolly.png"),
+ dollyBuffer,
+ );
await fs.writeFile(path.join(assetsDir, "3d_camera_dolly.png"), dollyBuffer);
console.log(" Saved: 3d_camera_dolly.png");
@@ -150,7 +154,10 @@ async function main() {
);
const orbitBuffer = await compOrbit.renderFrame({ frame: 22 });
- await fs.writeFile(path.join(artifactDir, "3d_camera_orbit.png"), orbitBuffer);
+ await fs.writeFile(
+ path.join(artifactDir, "3d_camera_orbit.png"),
+ orbitBuffer,
+ );
await fs.writeFile(path.join(assetsDir, "3d_camera_orbit.png"), orbitBuffer);
console.log(" Saved: 3d_camera_orbit.png");
@@ -232,7 +239,9 @@ async function main() {
await fs.writeFile(path.join(assetsDir, "3d_camera_dof.png"), dofBuffer);
console.log(" Saved: 3d_camera_dof.png");
- console.log("=== All 3D Camera Render Demonstrations Generated Successfully ===");
+ console.log(
+ "=== All 3D Camera Render Demonstrations Generated Successfully ===",
+ );
}
main().catch((err) => {
diff --git a/scripts/render-3d-text-demo.mjs b/scripts/render-3d-text-demo.mjs
index 32dab420..59b4fa49 100644
--- a/scripts/render-3d-text-demo.mjs
+++ b/scripts/render-3d-text-demo.mjs
@@ -1,3 +1,5 @@
+import fs from "node:fs/promises";
+import path from "node:path";
import {
CameraAnimation,
Composition,
@@ -6,14 +8,14 @@ import {
Layer3D,
LayerAnimation,
} from "../packages/framefields/dist/index.mjs";
-import fs from "node:fs/promises";
-import path from "node:path";
async function main() {
console.log("=== WebGPU 3D Typography Render Demonstration ===");
- const artifactDir = "/Users/okanaslankan/.gemini/antigravity-ide/brain/18faf2d5-3c9b-408e-8ba7-bcbf855a7dc5";
- const outputDir = "/Users/okanaslankan/framefields/examples/output/3d_text_demo";
+ const outputDir =
+ process.env.OUTPUT_DIR ||
+ path.resolve(process.cwd(), "examples/output/3d_text_demo");
+ const artifactDir = process.env.ARTIFACT_DIR || outputDir;
await fs.mkdir(artifactDir, { recursive: true });
await fs.mkdir(outputDir, { recursive: true });
diff --git a/scripts/render-demo.mjs b/scripts/render-demo.mjs
index fd9fd2b3..97a7b41e 100644
--- a/scripts/render-demo.mjs
+++ b/scripts/render-demo.mjs
@@ -1,321 +1,339 @@
-import {
- Composition,
- FontManager,
- Layer,
- LayerAnimation,
- Media,
- Vignette,
- Blur,
- ColorBalance,
- Signal,
- signal,
- computed,
- AudioSignal,
- AudioSignalExtractor
-} from "/Users/okanaslankan/framefields/packages/framefields/dist/index.mjs";
-import { HeadlessMediaRenderer } from "/Users/okanaslankan/framefields/packages/framefields/dist/renderer/index.mjs";
import fs from "node:fs/promises";
+import path from "node:path";
+import {
+ AudioSignal,
+ AudioSignalExtractor,
+ Blur,
+ ColorBalance,
+ Composition,
+ computed,
+ FontManager,
+ Layer,
+ LayerAnimation,
+ Media,
+ Signal,
+ signal,
+ Vignette,
+} from "../packages/framefields/dist/index.mjs";
+import { HeadlessMediaRenderer } from "../packages/framefields/dist/renderer/index.mjs";
async function main() {
- console.log("=== Framefields Signal-Driven Animation Pipeline ===");
- const inputVideo = "/Users/okanaslankan/framefields/export (5).mp4";
- const outputPng = "/Users/okanaslankan/framefields/rendered_preview.png";
- const outputMp4 = "/Users/okanaslankan/framefields/rendered_composition.mp4";
-
- console.log("1. Registering TrueType fonts via FontManager singleton...");
- const font = await FontManager.register("assets/fonts/Inter.ttf");
- console.log(` FontManager registered: "${font.family}" (${font.postscriptName}), unitsPerEm: ${font.unitsPerEm}`);
-
- console.log("2. Initializing AudioSignalExtractor node & extracting WebGPU audio features...");
- const audioExtractor = new AudioSignalExtractor({
- nodeId: "speech_extractor",
- extractionMode: "rms_envelope",
- attackMs: 12,
- releaseMs: 140,
- sensitivity: 1.25,
- noiseFloorDb: -42,
- dynamicRangeDb: 40,
- smoothing: 0.18,
- curve: "smoothstep"
- });
-
- const audio = await audioExtractor.extract(inputVideo, { fps: 24 });
- console.log(` Extracted WebGPU audio speech signals (${audio.numFrames} frames) via AudioSignalExtractor`);
-
- console.log("3. Initializing composition and defining reactive signals...");
- const comp = new Composition({
- width: 1280,
- height: 720,
- fps: 24,
- backgroundColor: "#0d1117"
- });
-
- // Define reactive signals
- // 1. Rhythmic breathing pulse signal (1.4 Hz sinusoidal LFO)
- comp.addSignal("pulse_lfo", (frame, fps) => {
- const t = frame / fps;
- const wave = Math.sin(t * Math.PI * 2 * 1.4);
- return (wave + 1) * 0.5; // normalized 0..1
- });
-
- // 2. Real-time speech energy signal extracted via AudioSignalExtractor
- comp.addSignal("energy_signal", (frame) => {
- audio.seekFrame(frame);
- return audio.energy.value;
- });
-
- // Base video layer
- comp.add(Layer.video(inputVideo, {
- x: 0,
- y: 0,
- width: 1280,
- height: 720,
- fit: "cover"
- }));
-
- // Audio track (mixed directly from video source)
- comp.addAudio(Layer.audio(inputVideo, {
- volume: 1.0
- }));
-
- // 1. Frosted dark card container behind typography
- comp.add(
- Layer.shape("rect", {
- id: "card-container",
- x: 70,
- y: 490,
- width: 1140,
- height: 165,
- borderRadius: 18,
- fillColor: "rgba(15, 23, 42, 0.88)"
- }).animate(
- new LayerAnimation()
- .slideInY(590, 490, 0, 22, "power3.out")
- .fadeIn(0, 16)
- )
- );
-
- // 2. Pulsing glowing indicator circle (scale & color modulated by pulse_lfo signal)
- comp.add(
- Layer.shape("circle", {
- id: "pulse-dot",
- x: 100,
- y: 520,
- width: 14,
- height: 14,
- borderRadius: 7,
- fillColor: "#38bdf8"
- }).animate(
- new LayerAnimation()
- .signal("scale", "pulse_lfo", { multiplier: 0.35, offset: 0.85 })
- .colorSignal("fillColor", "pulse_lfo", { colorA: "#38bdf8", colorB: "#ec4899" })
- .fadeIn(0, 18)
- )
- );
-
- // 3. Accent pill badge (scale modulated by signal, organic rotation wiggle)
- comp.add(
- Layer.shape("rect", {
- id: "vfx-badge",
- x: 126,
- y: 514,
- width: 185,
- height: 26,
- borderRadius: 6,
- fillColor: "#38bdf8"
- }).animate(
- new LayerAnimation()
- .signal("scale", "pulse_lfo", { multiplier: 0.08, offset: 1.0 })
- .wiggle("rotation", { frequency: 1.2, amplitude: 3.0, seed: 101 })
- .slideInY(536, 514, 6, 22)
- .fadeIn(6, 18)
- )
- );
-
- // 4. Badge label
- comp.add(
- Layer.text("AUDIO SIGNAL EXTRACTOR", {
- id: "badge-label",
- x: 136,
- y: 520,
- fontSize: 11,
- fill: "#0f172a",
- fontFamily: "Inter",
- fontWeight: "bold"
- }).animate(
- new LayerAnimation()
- .slideInY(542, 520, 6, 22)
- .fadeIn(6, 18)
- )
- );
-
- // 5. Signal reactive level track (meter background)
- comp.add(
- Layer.shape("rect", {
- id: "meter-bg",
- x: 330,
- y: 524,
- width: 220,
- height: 6,
- borderRadius: 3,
- fillColor: "rgba(255, 255, 255, 0.12)"
- }).animate(
- new LayerAnimation()
- .fadeIn(10, 24)
- )
- );
-
- // 6. Active signal meter bar (width & color dynamically modulated by speech energy_signal)
- comp.add(
- Layer.shape("rect", {
- id: "meter-active",
- x: 330,
- y: 524,
- width: 60,
- height: 6,
- borderRadius: 3,
- fillColor: "#38bdf8"
- }).animate(
- new LayerAnimation()
- .signal("width", "energy_signal", { multiplier: 190, offset: 25 })
- .colorSignal("fillColor", "energy_signal", { colorA: "#38bdf8", colorB: "#f43f5e" })
- .fadeIn(10, 24)
- )
- );
-
- // 7. Signal meter label
- comp.add(
- Layer.text("SIGNAL MODULATION", {
- id: "meter-label",
- x: 566,
- y: 522,
- fontSize: 11,
- fill: "#94a3b8",
- fontFamily: "Inter",
- fontWeight: "bold"
- }).animate(
- new LayerAnimation()
- .fadeIn(12, 26)
- )
- );
-
- // 8. Main title (entrance slide-up & fade-in)
- comp.add(
- Layer.text("Framefields Render Engine", {
- id: "main-title",
- x: 100,
- y: 554,
- fontSize: 32,
- fill: "#ffffff",
- fontFamily: "Inter"
- }).animate(
- new LayerAnimation()
- .slideInY(580, 554, 8, 26)
- .fadeIn(8, 22)
- )
- );
-
- // 9. Subtitle (entrance slide-up & fade-in)
- comp.add(
- Layer.text("node-audio-signal-extractor • WebGPU Audio DSP • Reactive Speech Blur", {
- id: "subtitle",
- x: 100,
- y: 605,
- fontSize: 15,
- fill: "#94a3b8",
- fontFamily: "Inter"
- }).animate(
- new LayerAnimation()
- .slideInY(625, 605, 14, 32)
- .fadeIn(14, 28)
- )
- );
-
- console.log("3. Dynamically computing composition duration & configuring reactive effects...");
- const durationMs = await comp.computeDuration();
- console.log(` Dynamically computed duration: ${durationMs}ms (${(durationMs / 1000).toFixed(2)}s)`);
-
- console.log("4. Instantiating reactive effect classes with game-engine per-frame lifecycle hooks...");
-
- // 1. First-class ColorBalance effect
- const colorBalance = new ColorBalance({
- shadows: { cyanRed: 12, magentaGreen: -4, yellowBlue: -12 },
- midtones: { cyanRed: 4, magentaGreen: 6, yellowBlue: 2 },
- highlights: { cyanRed: -4, magentaGreen: 0, yellowBlue: 8 },
- preserveLuminosity: true
- });
-
- // 2. Procedural breathing LFO for subtle vignette modulation
- const vignetteBreathing = Signal.builder({
- type: "sine",
- frequency: 0.8,
- amplitude: 8.0,
- offset: 50.0
- });
-
- // 3. First-class Vignette effect bound to reactive signals with per-frame mutation
- const vignette = new Vignette({
- strength: vignetteBreathing,
- radius: 0.85,
- softness: 0.55
- });
-
- // Per-frame game engine hook: dynamically adjust radius over frames
- vignette.onRequestFrame(({ frame }) => {
- if (frame < 50) {
- vignette.radius = 0.75 + frame * 0.002;
- }
- });
-
- // 4. Reactive Speech Blur: Screen blurs dynamically when the man is talking!
- // When quiet/silent: 0 blur (crystal clear).
- // When speech detected: blur smoothly scales with voice energy (up to 20px).
- const speechBlurStrength = computed(() => {
- const energy = audio.energy.value;
- if (energy < 0.15) return 0;
- return Math.min(22, Math.max(0, (energy - 0.15) * 32 + 2));
- });
-
- const blur = new Blur({
- blurType: "Gaussian",
- strength: speechBlurStrength
- });
-
- blur.onRequestFrame(({ frame }) => {
- audio.seekFrame(frame);
- });
-
- const compVM = await comp.toVirtualMediaAsync();
- const styledMedia = new Media(compVM)
- .apply(colorBalance)
- .apply(vignette)
- .apply(blur);
-
- const finalVM = styledMedia.toVirtualMedia();
- const renderer = new HeadlessMediaRenderer();
-
- console.log("5. Rendering high-quality preview frame at frame 48 (2.0s)...");
- const t0 = Date.now();
- const pngBuffer = await renderer.renderImage(finalVM, 48, 24);
- await fs.writeFile(outputPng, pngBuffer);
- console.log(` Saved preview PNG (${(pngBuffer.length / 1024).toFixed(1)} KB) in ${Date.now() - t0}ms`);
-
- console.log(`6. Rendering and encoding full-duration MP4 video (${(durationMs / 1000).toFixed(2)}s, ${Math.round(durationMs / 1000 * 24)} frames @ 24fps)...`);
- const t1 = Date.now();
- const { filePath: tempVideoPath, cleanup } = await renderer.renderVideo(finalVM, {
- codec: "h264",
- quality: "medium"
- });
-
- await fs.copyFile(tempVideoPath, outputMp4);
- await cleanup();
- const stat = await fs.stat(outputMp4);
- console.log(` Saved full-length MP4 video (${(stat.size / 1024 / 1024).toFixed(2)} MB) in ${Date.now() - t1}ms`);
-
- console.log("=== Render Completed Successfully ===");
+ console.log("=== Framefields Signal-Driven Animation Pipeline ===");
+ const inputVideo =
+ process.env.INPUT_VIDEO || path.resolve(process.cwd(), "export.mp4");
+ const outputPng = path.resolve(process.cwd(), "rendered_preview.png");
+ const outputMp4 = path.resolve(process.cwd(), "rendered_composition.mp4");
+
+ console.log("1. Registering TrueType fonts via FontManager singleton...");
+ const font = await FontManager.register("assets/fonts/Inter.ttf");
+ console.log(
+ ` FontManager registered: "${font.family}" (${font.postscriptName}), unitsPerEm: ${font.unitsPerEm}`,
+ );
+
+ console.log(
+ "2. Initializing AudioSignalExtractor node & extracting WebGPU audio features...",
+ );
+ const audioExtractor = new AudioSignalExtractor({
+ nodeId: "speech_extractor",
+ extractionMode: "rms_envelope",
+ attackMs: 12,
+ releaseMs: 140,
+ sensitivity: 1.25,
+ noiseFloorDb: -42,
+ dynamicRangeDb: 40,
+ smoothing: 0.18,
+ curve: "smoothstep",
+ });
+
+ const audio = await audioExtractor.extract(inputVideo, { fps: 24 });
+ console.log(
+ ` Extracted WebGPU audio speech signals (${audio.numFrames} frames) via AudioSignalExtractor`,
+ );
+
+ console.log("3. Initializing composition and defining reactive signals...");
+ const comp = new Composition({
+ width: 1280,
+ height: 720,
+ fps: 24,
+ backgroundColor: "#0d1117",
+ });
+
+ // Define reactive signals
+ // 1. Rhythmic breathing pulse signal (1.4 Hz sinusoidal LFO)
+ comp.addSignal("pulse_lfo", (frame, fps) => {
+ const t = frame / fps;
+ const wave = Math.sin(t * Math.PI * 2 * 1.4);
+ return (wave + 1) * 0.5; // normalized 0..1
+ });
+
+ // 2. Real-time speech energy signal extracted via AudioSignalExtractor
+ comp.addSignal("energy_signal", (frame) => {
+ audio.seekFrame(frame);
+ return audio.energy.value;
+ });
+
+ // Base video layer
+ comp.add(
+ Layer.video(inputVideo, {
+ x: 0,
+ y: 0,
+ width: 1280,
+ height: 720,
+ fit: "cover",
+ }),
+ );
+
+ // Audio track (mixed directly from video source)
+ comp.addAudio(
+ Layer.audio(inputVideo, {
+ volume: 1.0,
+ }),
+ );
+
+ // 1. Frosted dark card container behind typography
+ comp.add(
+ Layer.shape("rect", {
+ id: "card-container",
+ x: 70,
+ y: 490,
+ width: 1140,
+ height: 165,
+ borderRadius: 18,
+ fillColor: "rgba(15, 23, 42, 0.88)",
+ }).animate(
+ new LayerAnimation()
+ .slideInY(590, 490, 0, 22, "power3.out")
+ .fadeIn(0, 16),
+ ),
+ );
+
+ // 2. Pulsing glowing indicator circle (scale & color modulated by pulse_lfo signal)
+ comp.add(
+ Layer.shape("circle", {
+ id: "pulse-dot",
+ x: 100,
+ y: 520,
+ width: 14,
+ height: 14,
+ borderRadius: 7,
+ fillColor: "#38bdf8",
+ }).animate(
+ new LayerAnimation()
+ .signal("scale", "pulse_lfo", { multiplier: 0.35, offset: 0.85 })
+ .colorSignal("fillColor", "pulse_lfo", {
+ colorA: "#38bdf8",
+ colorB: "#ec4899",
+ })
+ .fadeIn(0, 18),
+ ),
+ );
+
+ // 3. Accent pill badge (scale modulated by signal, organic rotation wiggle)
+ comp.add(
+ Layer.shape("rect", {
+ id: "vfx-badge",
+ x: 126,
+ y: 514,
+ width: 185,
+ height: 26,
+ borderRadius: 6,
+ fillColor: "#38bdf8",
+ }).animate(
+ new LayerAnimation()
+ .signal("scale", "pulse_lfo", { multiplier: 0.08, offset: 1.0 })
+ .wiggle("rotation", { frequency: 1.2, amplitude: 3.0, seed: 101 })
+ .slideInY(536, 514, 6, 22)
+ .fadeIn(6, 18),
+ ),
+ );
+
+ // 4. Badge label
+ comp.add(
+ Layer.text("AUDIO SIGNAL EXTRACTOR", {
+ id: "badge-label",
+ x: 136,
+ y: 520,
+ fontSize: 11,
+ fill: "#0f172a",
+ fontFamily: "Inter",
+ fontWeight: "bold",
+ }).animate(new LayerAnimation().slideInY(542, 520, 6, 22).fadeIn(6, 18)),
+ );
+
+ // 5. Signal reactive level track (meter background)
+ comp.add(
+ Layer.shape("rect", {
+ id: "meter-bg",
+ x: 330,
+ y: 524,
+ width: 220,
+ height: 6,
+ borderRadius: 3,
+ fillColor: "rgba(255, 255, 255, 0.12)",
+ }).animate(new LayerAnimation().fadeIn(10, 24)),
+ );
+
+ // 6. Active signal meter bar (width & color dynamically modulated by speech energy_signal)
+ comp.add(
+ Layer.shape("rect", {
+ id: "meter-active",
+ x: 330,
+ y: 524,
+ width: 60,
+ height: 6,
+ borderRadius: 3,
+ fillColor: "#38bdf8",
+ }).animate(
+ new LayerAnimation()
+ .signal("width", "energy_signal", { multiplier: 190, offset: 25 })
+ .colorSignal("fillColor", "energy_signal", {
+ colorA: "#38bdf8",
+ colorB: "#f43f5e",
+ })
+ .fadeIn(10, 24),
+ ),
+ );
+
+ // 7. Signal meter label
+ comp.add(
+ Layer.text("SIGNAL MODULATION", {
+ id: "meter-label",
+ x: 566,
+ y: 522,
+ fontSize: 11,
+ fill: "#94a3b8",
+ fontFamily: "Inter",
+ fontWeight: "bold",
+ }).animate(new LayerAnimation().fadeIn(12, 26)),
+ );
+
+ // 8. Main title (entrance slide-up & fade-in)
+ comp.add(
+ Layer.text("Framefields Render Engine", {
+ id: "main-title",
+ x: 100,
+ y: 554,
+ fontSize: 32,
+ fill: "#ffffff",
+ fontFamily: "Inter",
+ }).animate(new LayerAnimation().slideInY(580, 554, 8, 26).fadeIn(8, 22)),
+ );
+
+ // 9. Subtitle (entrance slide-up & fade-in)
+ comp.add(
+ Layer.text(
+ "node-audio-signal-extractor • WebGPU Audio DSP • Reactive Speech Blur",
+ {
+ id: "subtitle",
+ x: 100,
+ y: 605,
+ fontSize: 15,
+ fill: "#94a3b8",
+ fontFamily: "Inter",
+ },
+ ).animate(new LayerAnimation().slideInY(625, 605, 14, 32).fadeIn(14, 28)),
+ );
+
+ console.log(
+ "3. Dynamically computing composition duration & configuring reactive effects...",
+ );
+ const durationMs = await comp.computeDuration();
+ console.log(
+ ` Dynamically computed duration: ${durationMs}ms (${(durationMs / 1000).toFixed(2)}s)`,
+ );
+
+ console.log(
+ "4. Instantiating reactive effect classes with game-engine per-frame lifecycle hooks...",
+ );
+
+ // 1. First-class ColorBalance effect
+ const colorBalance = new ColorBalance({
+ shadows: { cyanRed: 12, magentaGreen: -4, yellowBlue: -12 },
+ midtones: { cyanRed: 4, magentaGreen: 6, yellowBlue: 2 },
+ highlights: { cyanRed: -4, magentaGreen: 0, yellowBlue: 8 },
+ preserveLuminosity: true,
+ });
+
+ // 2. Procedural breathing LFO for subtle vignette modulation
+ const vignetteBreathing = Signal.builder({
+ type: "sine",
+ frequency: 0.8,
+ amplitude: 8.0,
+ offset: 50.0,
+ });
+
+ // 3. First-class Vignette effect bound to reactive signals with per-frame mutation
+ const vignette = new Vignette({
+ strength: vignetteBreathing,
+ radius: 0.85,
+ softness: 0.55,
+ });
+
+ // Per-frame game engine hook: dynamically adjust radius over frames
+ vignette.onRequestFrame(({ frame }) => {
+ if (frame < 50) {
+ vignette.radius = 0.75 + frame * 0.002;
+ }
+ });
+
+ // 4. Reactive Speech Blur: Screen blurs dynamically when the man is talking!
+ // When quiet/silent: 0 blur (crystal clear).
+ // When speech detected: blur smoothly scales with voice energy (up to 20px).
+ const speechBlurStrength = computed(() => {
+ const energy = audio.energy.value;
+ if (energy < 0.15) return 0;
+ return Math.min(22, Math.max(0, (energy - 0.15) * 32 + 2));
+ });
+
+ const blur = new Blur({
+ blurType: "Gaussian",
+ strength: speechBlurStrength,
+ });
+
+ blur.onRequestFrame(({ frame }) => {
+ audio.seekFrame(frame);
+ });
+
+ const compVM = await comp.toVirtualMediaAsync();
+ const styledMedia = new Media(compVM)
+ .apply(colorBalance)
+ .apply(vignette)
+ .apply(blur);
+
+ const finalVM = styledMedia.toVirtualMedia();
+ const renderer = new HeadlessMediaRenderer();
+
+ console.log("5. Rendering high-quality preview frame at frame 48 (2.0s)...");
+ const t0 = Date.now();
+ const pngBuffer = await renderer.renderImage(finalVM, 48, 24);
+ await fs.writeFile(outputPng, pngBuffer);
+ console.log(
+ ` Saved preview PNG (${(pngBuffer.length / 1024).toFixed(1)} KB) in ${Date.now() - t0}ms`,
+ );
+
+ console.log(
+ `6. Rendering and encoding full-duration MP4 video (${(durationMs / 1000).toFixed(2)}s, ${Math.round((durationMs / 1000) * 24)} frames @ 24fps)...`,
+ );
+ const t1 = Date.now();
+ const { filePath: tempVideoPath, cleanup } = await renderer.renderVideo(
+ finalVM,
+ {
+ codec: "h264",
+ quality: "medium",
+ },
+ );
+
+ await fs.copyFile(tempVideoPath, outputMp4);
+ await cleanup();
+ const stat = await fs.stat(outputMp4);
+ console.log(
+ ` Saved full-length MP4 video (${(stat.size / 1024 / 1024).toFixed(2)} MB) in ${Date.now() - t1}ms`,
+ );
+
+ console.log("=== Render Completed Successfully ===");
}
-main().catch(err => {
- console.error("Pipeline execution failed:", err);
- process.exit(1);
+main().catch((err) => {
+ console.error("Pipeline execution failed:", err);
+ process.exit(1);
});
diff --git a/scripts/render-text-2d-3d.mjs b/scripts/render-text-2d-3d.mjs
index b0372909..882a58ea 100644
--- a/scripts/render-text-2d-3d.mjs
+++ b/scripts/render-text-2d-3d.mjs
@@ -1,3 +1,6 @@
+import fs from "node:fs/promises";
+import path from "node:path";
+import { Canvas, loadImage } from "skia-canvas";
import {
CameraAnimation,
Composition,
@@ -6,16 +9,14 @@ import {
Layer3D,
LayerAnimation,
} from "../packages/framefields/dist/index.mjs";
-import { loadImage, Canvas } from "skia-canvas";
-import fs from "node:fs/promises";
-import path from "node:path";
async function main() {
console.log("=== WebGPU 2D & 3D Typography Render Engine ===");
- const artifactDir =
- "/Users/okanaslankan/.gemini/antigravity-ide/brain/b888d40e-44c5-4686-bebd-a131288bd97e";
- const outputDir = "/Users/okanaslankan/framefields/examples/output/text_2d_3d";
+ const outputDir =
+ process.env.OUTPUT_DIR ||
+ path.resolve(process.cwd(), "examples/output/text_2d_3d");
+ const artifactDir = process.env.ARTIFACT_DIR || outputDir;
await fs.mkdir(artifactDir, { recursive: true });
await fs.mkdir(outputDir, { recursive: true });
@@ -250,13 +251,16 @@ async function main() {
fill: "#ffffff",
letterSpacing: 0, // Tests pair kerning: AV, VA, AT, AR
}),
- Layer.text("Zero-error pair kerning verified against Skia reference rasters (Δ = 0px).", {
- id: "card-sub-2d",
- fontSize: 13,
- fontWeight: 400,
- fill: "#94a3b8",
- lineHeight: 1.4,
- }),
+ Layer.text(
+ "Zero-error pair kerning verified against Skia reference rasters (Δ = 0px).",
+ {
+ id: "card-sub-2d",
+ fontSize: 13,
+ fontWeight: 400,
+ fill: "#94a3b8",
+ lineHeight: 1.4,
+ },
+ ),
],
}),
);
@@ -278,13 +282,16 @@ async function main() {
borderWidth: 1,
borderColor: "rgba(30, 41, 59, 0.7)",
children: [
- Layer.text("WEBGPU PIPELINE: 0 CHURN / 0 PIPELINE REBUILDS / 120 FPS ZERO-CPU-RE-RASTER", {
- id: "footer-left",
- fontSize: 12,
- fontWeight: "bold",
- fill: "#10b981",
- letterSpacing: 2,
- }),
+ Layer.text(
+ "WEBGPU PIPELINE: 0 CHURN / 0 PIPELINE REBUILDS / 120 FPS ZERO-CPU-RE-RASTER",
+ {
+ id: "footer-left",
+ fontSize: 12,
+ fontWeight: "bold",
+ fill: "#10b981",
+ letterSpacing: 2,
+ },
+ ),
Layer.text("FRAMEFIELDS CORE ENGINE", {
id: "footer-right",
fontSize: 12,
@@ -343,7 +350,11 @@ async function main() {
const text3DData = ctx.getImageData(width / 2 - 200, 420, 400, 100).data;
let has3DTextPixels = false;
for (let i = 0; i < text3DData.length; i += 4) {
- if (text3DData[i] > 150 || text3DData[i + 1] > 120 || text3DData[i + 2] > 180) {
+ if (
+ text3DData[i] > 150 ||
+ text3DData[i + 1] > 120 ||
+ text3DData[i + 2] > 180
+ ) {
has3DTextPixels = true;
break;
}
@@ -353,7 +364,9 @@ async function main() {
console.log(`3D Extruded Text Rasterization Verified: ${has3DTextPixels}`);
if (!has2DTextPixels || !has3DTextPixels) {
- throw new Error("Validation failed: text pixels not detected in expected regions.");
+ throw new Error(
+ "Validation failed: text pixels not detected in expected regions.",
+ );
}
console.log("✅ Successfully rendered 2D & 3D text!");