Skip to content

Code-first video & image, rendered natively on WebGPU.

upt readme

okanaslbb68a8c88686
diff --git a/README.md b/README.md
index 9ebc8b11..7737b714 100644
--- a/README.md
+++ b/README.md
@@ -1,171 +1,246 @@
+<div align="center">
+
# Gitframes
-> **High-Performance WebGPU Rendering Engine & SDK for Programmatic Video, Motion Graphics, Audio DSP, and Neural Conditioning.**
+**High-Performance WebGPU Rendering Engine & SDK for Programmatic Video, Motion Graphics, Audio DSP, and Neural Vision.**
+
+[![npm](https://img.shields.io/badge/npm-gitframes-cb3837?logo=npm&logoColor=white)](https://www.npmjs.com/package/gitframes)
+[![license](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENCE)
+[![node](https://img.shields.io/badge/node-%E2%89%A522-3c873a?logo=node.js&logoColor=white)](https://nodejs.org)
+[![engine](https://img.shields.io/badge/engine-WebGPU%20%C2%B7%20WGSL-6366f1)](https://www.w3.org/TR/webgpu/)
+[![gpu](https://img.shields.io/badge/GPU-Dawn%20%C2%B7%20Metal%20%C2%B7%20Vulkan-8b5cf6)](#monorepo-architecture)
+[![vision](https://img.shields.io/badge/vision-YOLO11%20ONNX-ff5a1f)](#6-yolo11-neural-vision--tracking)
-Gitframes is a code-first video composition engine built directly on native WebGPU and modern JavaScript Signals. It brings Photoshop-grade 2D image processing, After Effects-grade kinetic typography, unified 3D spatial camera motion, procedural audio DSP, and real-time neural vision pipelines into a single, deterministic rendering pipeline.
+**Code-first video as pure software engineering** — no headless browser, no DOM reflow, no screenshot pipeline.
+Renders directly on GPU hardware via Dawn / WebGPU / Metal / Vulkan in Node.js and modern WebGPU browsers.
-Unlike browser-dependent frameworks that run inside headless Chromium, Gitframes executes directly on GPU hardware via Dawn / WebGPU / Vulkan / Metal in Node.js and in modern WebGPU-enabled browsers—delivering 60–120 FPS rendering speeds with a tiny resource footprint.
+</div>
+> [!NOTE]
> **Using an AI coding agent?** Install the gitframes skills in **Claude Code** with `/plugin marketplace add gatewai-dev/gitframes` → `/plugin install gitframes@gitframes-plugins`, or in **Codex** with `codex plugin marketplace add gatewai-dev/gitframes`. See [Agent Skills & Plugins](#agent-skills--plugins).
---
## Table of Contents
-- [Core Value Proposition](#core-value-proposition)
-- [Architectural Comparison: Gitframes vs. Remotion vs. Hyperframes](#architectural-comparison-gitframes-vs-remotion-vs-hyperframes)
- - [Detailed Comparison Matrix](#detailed-comparison-matrix)
- - [Why Gitframes? Deep Architectural Analysis](#why-gitframes-deep-architectural-analysis)
+- [Why Gitframes](#why-gitframes)
+- [Architectural Comparison](#architectural-comparison-gitframes-vs-remotion-vs-hyperframes)
- [Key Features & Capabilities](#key-features--capabilities)
- [1. Slug GPU Vector Typography & AE Parity](#1-slug-gpu-vector-typography--ae-parity)
- - [2. Photoshop-Grade WebGPU 2D VFX (50+ Shaders)](#2-photoshop-grade-webgpu-2d-vfx-50-shaders)
+ - [2. Photoshop-Grade WebGPU 2D VFX](#2-photoshop-grade-webgpu-2d-vfx-50-shaders)
- [3. Unified 3D Scene Graph, Camera & Mesh Shading](#3-unified-3d-scene-graph-camera--mesh-shading)
- [4. Audio Layers, Procedural SFX & Reactive Signals](#4-audio-layers-procedural-sfx--reactive-signals)
- [5. WebGPU ChartGPU Bridge](#5-webgpu-chartgpu-bridge)
- - [6. Neural Tensor & MediaPipe AI Conditioning](#6-neural-tensor--mediapipe-ai-conditioning)
+ - [6. YOLO11 Neural Vision & Tracking](#6-yolo11-neural-vision--tracking)
- [7. Headless Conformance & FrameGrid Testing](#7-headless-conformance--framegrid-testing)
- [Monorepo Architecture](#monorepo-architecture)
- [Quickstart Guide](#quickstart-guide)
- - [Installation](#installation)
- [1. Basic Composition & Kinetic Auto-Layout](#1-basic-composition--kinetic-auto-layout)
- [2. Unified 3D Scene with Camera & 3D Model](#2-unified-3d-scene-with-camera--3d-model)
- [3. Audio Soundtrack, Procedural SFX & Reactive Signals](#3-audio-soundtrack-procedural-sfx--reactive-signals)
- [4. Chained WebGPU Post-Processing VFX](#4-chained-webgpu-post-processing-vfx)
- - [5. Headless Video & FrameGrid Rendering](#5-headless-video--framegrid-rendering)
+ - [5. YOLO11 Vision: Pin, Matte & Reframe](#5-yolo11-vision-pin-matte--reframe)
+ - [6. Headless Video & FrameGrid Rendering](#6-headless-video--framegrid-rendering)
- [Engineering Doctrines & Best Practices](#engineering-doctrines--best-practices)
- [Agent Skills & Plugins](#agent-skills--plugins)
- - [Claude Code](#claude-code)
- - [Codex](#codex)
- - [Any other agent (skills only)](#any-other-agent-skills-only)
- [Reference Showcase Examples](#reference-showcase-examples)
- [Development & Building](#development--building)
+- [License](#license)
---
-## Core Value Proposition
+## Why Gitframes
-Modern automated video generation workflows are often constrained by the architectures of general-purpose web browsers: heavy process overhead, non-deterministic DOM layout reflows, and slow screenshot capture bottlenecks. Gitframes treats **video composition as pure software engineering**:
+Modern automated video generation is usually constrained by the architectures of general-purpose web browsers: process overhead, non-deterministic DOM layout reflows, and slow screenshot capture. Gitframes treats **video composition as software engineering**:
-1. **Zero Headless Browser Overhead**: Eliminates Puppeteer, Chromium IPC serialization, and `page.screenshot()` transfers. Gitframes connects directly to native GPU devices via Dawn (`webgpu`) in Node.js and `@napi-rs/webcodecs` hardware encoding.
-2. **Deterministic Frame-Accurate Clock**: Compositions run against absolute frame clocks, discrete sample points, and frame-accurate audio BeatGrids. No floating timers, drift, or dropped frames.
-3. **Analytic Resolution-Independent Typography**: Powered by the Slug algorithm in WebGPU WGSL shaders. Glyph contours are mathematically evaluated on the GPU per-pixel without texture atlases, memory bloat, or scaling artifacts.
-4. **Photoshop-Grade Tonal & Spatial VFX**: 50+ modular GPU shader operations including Curves, Levels, Selective Color, 3D LUTs, Halftone Screens, Film Grain, Unsharp Mask, Mesh Warping, and Screen-Space Relighting.
-5. **Unified 3D & 2D Depth Compositing**: Seamlessly nest 2D flex/box layout trees inside 3D homography planes, multiplane camera rigs, 3D meshes (OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF), PBR glass refraction, and SSAO contact shadows.
-6. **Built-in Procedural Audio DSP**: Renders multi-track soundtracks, deterministic procedural transition effects (whoosh, impact, riser), and drives visual keyframes reactively.
-7. **Cloud-Native & CI/CD Ready**: Extremely low RAM usage (~200–400 MB per worker vs. 2–4 GB for Chromium), making it ideal for high-throughput serverless rendering clusters (AWS G4/G5, Modal, RunPod, Kubernetes).
+| | Pillar | What it means |
+|---|---|---|
+| 🚀 | **Zero Headless-Browser Overhead** | No Puppeteer, no Chromium IPC, no `page.screenshot()`. Gitframes talks straight to native GPU devices via Dawn/WebGPU and hardware-encodes with `@napi-rs/webcodecs`. |
+| 🎯 | **Deterministic Frame-Accurate Clock** | Absolute frame clocks, discrete sample points, and frame-accurate audio BeatGrids. No floating timers, no drift, no dropped frames. |
+| 🔠 | **Analytic, Resolution-Independent Type** | The Slug algorithm evaluates glyph contours per-pixel in WGSL — no texture atlases, no scaling artifacts, razor-sharp from 10 px to 10,000 px. |
+| 🎨 | **Photoshop-Grade Tonal & Spatial VFX** | 50+ modular GPU shaders: Curves, Levels, Selective Color, 3D LUTs, Halftone, Film Grain, Unsharp Mask, Mesh Warp, and Screen-Space Relighting. |
+| 🧊 | **Unified 3D & 2D Depth Compositing** | Nest 2D flex/box trees inside 3D homography planes, multiplane rigs, and meshes (OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF), with PBR glass and SSAO. |
+| 🔊 | **Built-in Procedural Audio DSP** | Multi-track soundtracks, deterministic procedural transition SFX (whoosh, impact, riser), and reactive signals that drive visuals from audio. |
+| 👁️ | **On-Device Neural Vision** | Native YOLO11 ONNX inference for detection, instance segmentation, pose, OBB, classification, and open-vocabulary prompting — feeding reactive signals without a round trip to disk. |
+| ☁️ | **Cloud-Native & CI/CD Ready** | ~200–400 MB RAM per worker (vs. 2–4 GB for Chromium), ideal for serverless GPU render clusters (AWS G4/G5, Modal, RunPod, Kubernetes). |
---
## Architectural Comparison: Gitframes vs. Remotion vs. Hyperframes
-Developers generating video programmatically commonly consider **Remotion** (React/Chromium) or **Hyperframes** (Canvas2D/SVG web animation). The table below details how Gitframes compares across fundamental engineering dimensions.
+Developers generating video programmatically commonly weigh **Remotion** (React/Chromium) or **Hyperframes** (Canvas2D/SVG web animation). The matrix below compares the fundamental engineering dimensions.
### Detailed Comparison Matrix
| Capability / Dimension | **Gitframes** | **Remotion** | **Hyperframes** |
|---|---|---|---|
-| **Underlying Engine** | **Native WebGPU** (WGSL Compute & Render Pipelines via Dawn / Metal / Vulkan) | **Chromium / Puppeteer** (React DOM, HTML/CSS layout engine) | **Canvas2D / WebGL / SVG** (Browser or Node Skia runtime) |
-| **Rendering Architecture** | Direct hardware GPU framebuffer rendering & hardware video encoding ([`@napi-rs/webcodecs`](file:///Users/okanaslankan/gitframes/packages/renderer/package.json#L49)) | Spawns headless Chrome instances; captures frames via CDP / `page.screenshot()` | Software or hardware 2D canvas context execution |
-| **Rendering Speed & FPS** | **60–120+ FPS** (Real-time to faster-than-real-time GPU execution) | **5–20 FPS** (Bottlenecked by DOM reflow, IPC serialization, and rasterization) | **20–40 FPS** (Bottlenecked by CPU canvas draw commands or JS execution) |
-| **Memory Footprint** | **~200–400 MB** per render process (zero browser overhead) | **1.5–4.0 GB+** per render worker (Full Chromium instance + V8 DOM heap) | **~500 MB–1 GB** (Depends on Skia/Canvas node bindings) |
-| **Typography Engine** | **Slug GPU** (Analytic cubic/quadratic Bezier evaluation in WGSL; infinite zoom; After Effects parity selectors) | **Browser DOM Text** (CSS fonts; rasterized to bitmap; blurry under 3D transforms) | **Canvas2D / Path Text** (CPU rasterized glyph curves or standard text rendering) |
-| **2D VFX & Post-Processing** | **50+ WebGPU Shaders** (Curves, Levels, Selective Color, 3D LUT, Film Grain, Halftone, Liquify, PBR Glass, Relighting) | Limited to **CSS Filters** (`filter: blur(...)`) or custom WebGL canvas wrappers | Basic Canvas2D composite operations and simple 2D filters |
-| **3D Graphics & Spatial Depth** | **Native 3D Scene Graph** (LookAt/Turntable camera, multiplane depth, mesh skinning for OBJ/FBX/glTF, SSAO, PCSS shadows, DoF) | **None built-in** (Requires embedding Three.js/Fiber inside canvas inside React DOM) | Minimal 2.5D canvas layers; no unified 3D mesh pipeline |
-| **Motion Blur & Physics** | **Physical Shutter Motion Blur** ($180^\circ$ shutter velocity buffers in MRT) + closed-form spring kinematics | CSS transitions or JS interpolations; synthetic CSS blur hacks | Frame interpolation or manual multi-pass rendering |
-| **Audio Engine & DSP** | **Native WebGPU Audio DSP & Procedural SFX** (Audio layers, multi-track mixing, procedural SFX triggers, reactive signals) | `<Audio>` component for static audio playback; basic volume curves | Basic static audio playback |
-| **Live Charting & Telemetry** | **ChartGPU Bridge** (Hardware-accelerated line, bar, OHLC, area charts rendered directly to GPU textures) | HTML/SVG chart libraries (Recharts, Chart.js) rendered in DOM | Custom canvas draw operations |
-| **AI & Computer Vision** | **WebGPU Tensor Pipelines & MediaPipe** (Canny, Depth-to-Normals, Optical Flow, 52-blendshape facial tracking) | External pre-rendered assets; no native GPU tensor conditioning | External pre-rendered assets |
-| **Headless Verification** | **FrameGrid contact sheets**, single-frame snapshots, and Skia MSE mathematical pixel invariant assertions | Visual snapshot testing via Playwright/Puppeteer | Manual frame inspection or canvas pixel diffing |
-| **Docker / Cloud Portability** | **Compact Docker Container** (~500 MB Alpine/Debian slim with native GPU/Vulkan drivers) | **Heavy Docker Container** (~2–3 GB with Chromium, fonts, and X11/Mesa dependencies) | Moderate container size |
-
----
+| **Underlying Engine** | **Native WebGPU** (WGSL compute & render pipelines via Dawn / Metal / Vulkan) | **Chromium / Puppeteer** (React DOM, HTML/CSS layout) | **Canvas2D / WebGL / SVG** (browser or Node Skia) |
+| **Rendering Architecture** | Direct hardware framebuffer rendering & hardware video encoding (`@napi-rs/webcodecs`) | Spawns headless Chrome; captures frames via CDP / `page.screenshot()` | Software or hardware 2D canvas context |
+| **Throughput** | **60–120+ FPS** (real-time to faster-than-real-time GPU execution) | **5–20 FPS** (DOM reflow, IPC, rasterization) | **20–40 FPS** (CPU draw commands / JS) |
+| **Memory Footprint** | **~200–400 MB** per render (zero browser) | **1.5–4.0 GB+** per worker (Chromium + V8 DOM heap) | **~500 MB–1 GB** (Skia/Canvas bindings) |
+| **Typography Engine** | **Slug GPU** — analytic Bézier evaluation in WGSL, infinite zoom, After Effects selectors | Browser DOM text (CSS fonts, rasterized, blurry under 3D transforms) | Canvas2D / path text (CPU-rasterized glyphs) |
+| **2D VFX & Post-Processing** | **50+ WebGPU shaders** (Curves, Levels, Selective Color, 3D LUT, Film Grain, Halftone, Liquify, PBR Glass, Relight) | CSS Filters or custom WebGL canvas wrappers | Basic Canvas2D composites and 2D filters |
+| **3D Graphics & Depth** | **Native 3D scene graph** — LookAt/Turntable camera, multiplane, skinning (OBJ/FBX/glTF), SSAO, PCSS, DoF | None built-in (embed Three.js/Fiber inside React DOM) | Minimal 2.5D layers; no unified mesh pipeline |
+| **Motion Blur & Physics** | Physical 180° shutter velocity buffers in MRT + closed-form spring kinematics | CSS transitions / JS interpolation; synthetic blur hacks | Frame interpolation or manual multipass |
+| **Audio Engine & DSP** | Native audio DSP & procedural SFX (multi-track mixing, beat grids, reactive signals) | `<Audio>` playback; basic volume curves | Basic static audio playback |
+| **Live Charting & Telemetry** | **ChartGPU bridge** — GPU-rendered line, bar, OHLC, area charts into GPU textures | DOM chart libraries (Recharts, Chart.js) | Custom canvas draw operations |
+| **AI & Computer Vision** | **YOLO11 ONNX** — detect, instance segmentation, COCO-17 pose, OBB, classify, and open-vocabulary YOLO-World; WebGPU tensor conditioning (Canny, depth-to-normals, optical flow, deflicker) | External pre-rendered assets; no native GPU tensor conditioning | External pre-rendered assets |
+| **Headless Verification** | **FrameGrid contact sheets**, single-frame snapshots, Skia MSE pixel-invariant assertions | Playwright/Puppeteer visual snapshots | Manual frame inspection / canvas diffing |
+| **Docker / Cloud Portability** | **Compact** (~500 MB slim image with native GPU/Vulkan drivers) | **Heavy** (~2–3 GB with Chromium, fonts, X11/Mesa) | Moderate container size |
### Why Gitframes? Deep Architectural Analysis
-#### 1. Why Not Remotion for High-Performance Pipelines?
-Remotion deserves credit for pioneering the "video in React" paradigm. For developers building basic marketing templates using existing web UI components, Remotion provides an accessible bridge. However, because Remotion is coupled to **headless Chromium**:
-- **Scalability Ceiling**: Chromium is an interactive document browser, not a real-time compositor. Every frame requires DOM style recalculation, layout reflow, paint tree traversal, and cross-process image buffer copying over Chrome DevTools Protocol (CDP).
-- **GPU Inefficiencies**: WebGL inside headless Chromium operates within sandbox constraints and cannot leverage modern compute shaders, storage buffers, or native WebGPU multi-render targets (MRT) efficiently.
-- **Server Infrastructure Cost**: Scaling Remotion in production requires high-memory CPU instances to prevent Chromium OOM crashes, driving cloud rendering bills significantly higher.
+#### 1. Why not Remotion for high-performance pipelines?
+Remotion pioneered the "video in React" paradigm and is a great fit for marketing templates built from existing web UI. But because it is coupled to **headless Chromium**:
+- **Scalability ceiling** — Chromium is an interactive document browser, not a real-time compositor. Every frame pays DOM style recalculation, layout reflow, paint traversal, and cross-process buffer copies over CDP.
+- **GPU inefficiency** — WebGL inside headless Chromium is sandboxed and cannot use compute shaders, storage buffers, or native WebGPU multi-render targets (MRT) efficiently.
+- **Infrastructure cost** — Scaling in production needs high-memory CPU instances to survive Chromium OOM crashes.
-**Gitframes eliminates the browser entirely.** By executing directly on the GPU via native WebGPU, Gitframes operates with the architecture of a AAA game engine or digital audio workstation (DAW), yielding massive throughput improvements.
+**Gitframes eliminates the browser entirely.** Running directly on the GPU via native WebGPU, it behaves like a game engine or DAW and delivers order-of-magnitude throughput gains.
-#### 2. Why Not Hyperframes or Traditional Canvas2D?
-Hyperframes and standard 2D canvas engines improve on DOM overhead by drawing directly to canvas surfaces. However:
-- **CPU Bottlenecks**: Canvas2D API calls (`ctx.arc`, `ctx.bezierCurveTo`) are largely evaluated on the CPU. Under complex scenes with hundreds of animated typography layers or vector paths, CPU rasterization quickly stalls.
-- **Shallow Post-Processing**: Canvas2D lacks multi-pass fragment shader pipelines. Professional tonal grading (3D LUT color cubes, selective color isolation, bilateral SSAO, optical depth-of-field bokeh) is virtually impossible without native shaders.
-- **Audio Disconnect**: Canvas frameworks treat audio as an external soundtrack. Gitframes unifies audio synthesis and visual animation through frame-accurate reactive signals and stem extraction.
+#### 2. Why not Hyperframes or Canvas2D?
+Canvas engines avoid DOM overhead but:
+- **CPU-bound rasterization** — `ctx.arc` / `ctx.bezierCurveTo` execute largely on the CPU; hundreds of animated type layers stall quickly.
+- **Shallow post-processing** — no multi-pass fragment pipelines, so pro grading (3D LUTs, selective color, bilateral SSAO, bokeh DoF) is effectively impossible.
+- **Disconnected audio** — Canvas frameworks treat audio as an external soundtrack; Gitframes unifies synthesis and animation through frame-accurate signals.
---
## Key Features & Capabilities
### 1. Slug GPU Vector Typography & AE Parity
-Traditional text rendering relies on CPU font rasterization or low-resolution signed distance field (SDF) glyph atlases that become soft under 3D camera sweeps. Gitframes integrates the **Slug algorithm** ([`SlugPipeline`](file:///Users/okanaslankan/gitframes/packages/webgpu-renderers/src/slug/slug-pipeline.ts)):
-- **Analytic GPU Evaluation**: WGSL fragment shaders evaluate exact cubic and quadratic Bezier curves per-pixel. Glyphs remain razor-sharp at $10\text{px}$ or $10,000\text{px}$ with zero CPU re-rasterization.
-- **After Effects-Parity Text Animators**: Full parity with After Effects range selectors (`square`, `ramp_up`, `ramp_down`, `triangle`, `smooth`), non-linear `easeHigh`/`easeLow` curves, and seeded PRNG character shuffling ([`TextAnimator`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/index.ts#L250)).
-- **Human Typing Cadence**: Dynamic typewriter simulation with weighted punctuation delays (commas $3\times$, sentence ends $5.5\times$, newlines $7\times$) and trailing scramble character resolution ([`TypewriterAnimator`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/index.ts#L275)).
-- **3D Volumetric Text Formations**: Map text onto 3D cylindrical drums, logarithmic vortex spirals, and double-helix DNA ribbons with surface-normal banking ([`evaluateVolumetricFormation`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/index.ts#L198)).
-- **Dynamic Leading & Skew**: Area-preserving unimodular shear matrices and accordion line leading anchored to baseline, center, or top edges.
+Traditional text relies on CPU rasterization or low-res SDF atlases that soften under 3D camera sweeps. Gitframes integrates the **Slug algorithm** ([`SlugPipeline`](packages/webgpu-renderers/src/slug/slug-pipeline.ts)):
+- **Analytic GPU evaluation** — WGSL fragment shaders solve exact cubic/quadratic Béziers per-pixel. Glyphs stay sharp at 10 px or 10,000 px with zero CPU re-rasterization.
+- **After Effects–parity animators** — range selectors (`square`, `ramp_up`, `ramp_down`, `triangle`, `smooth`), `easeHigh`/`easeLow` curves, and seeded PRNG character shuffling ([`TextAnimator`](packages/gitframes/src/index.ts)).
+- **Human typing cadence** — weighted punctuation delays (commas 3×, sentence ends 5.5×, newlines 7×) and trailing scramble resolution ([`TypewriterAnimator`](packages/gitframes/src/index.ts)).
+- **3D volumetric formations** — map text onto cylindrical drums, logarithmic vortex spirals, and double-helix ribbons with surface-normal banking ([`evaluateVolumetricFormation`](packages/gitframes/src/index.ts)).
+- **Dynamic leading & skew** — area-preserving unimodular shear and accordion line-leading anchored to baseline, center, or top.
### 2. Photoshop-Grade WebGPU 2D VFX (50+ Shaders)
-Gitframes houses a comprehensive suite of professional image and video processing shader nodes in [`nodes/`](file:///Users/okanaslankan/gitframes/nodes) and [`packages/webgpu-renderers`](file:///Users/okanaslankan/gitframes/packages/webgpu-renderers):
-- **Tonal Grading**: Curves (RGB, R, G, B spline interpolation), Levels (black/white point, gamma, output levels), Shadows/Highlights, Selective Color (CMYK color gamut isolation), and 3D Cube LUT grading ([`ApplyLUT`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/index.ts#L392)).
-- **Stylization & Grain**: Film Grain (Gaussian film emulsion with spatial seed variation), Halftone Screen (monochrome, RGB, or CMYK with adjustable dot shapes and angles), Gradient Map, and High Pass filtering.
-- **Optics & Lens**: Bilateral Gaussian Blur, Unsharp Mask, Vignette, Refraction Caustics, and PBR Glassmorphism with chromatic dispersion ([`PBRGlass`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/index.ts#L434)).
-- **Distortion & Warping**: Displacement Maps, Liquify, Mesh Warping, and Corner Pin homography.
+A comprehensive suite of professional image/video shader nodes in [`nodes/`](nodes) and [`packages/webgpu-renderers`](packages/webgpu-renderers):
+- **Tonal grading** — Curves (RGB/R/G/B spline), Levels (black/white point, gamma, output), Shadows/Highlights, Selective Color (CMYK gamut isolation), 3D Cube LUT ([`ApplyLUT`](packages/gitframes/src/effects)).
+- **Stylization & grain** — Film Grain (Gaussian emulsion with spatial seed variation), Halftone (mono/RGB/CMYK, adjustable dot shape & angle), Gradient Map, High Pass.
+- **Optics & lens** — Bilateral Gaussian Blur, Unsharp Mask, Vignette, Refraction Caustics, PBR Glassmorphism with chromatic dispersion ([`PBRGlass`](packages/gitframes/src/effects)).
+- **Distortion & warping** — Displacement Maps, Liquify, Mesh Warp, Corner Pin homography.
### 3. Unified 3D Scene Graph, Camera & Mesh Shading
-- **Calibrated 3D Camera Rig**: LookAt and Turntable cameras ([`Camera3D`](file:///Users/okanaslankan/gitframes/packages/webgpu-renderers/src/math3d/camera3d.ts#L80)) calibrated so that $z=0$ matches 2D canvas pixel coordinates $1:1$.
-- **3D Layout Primitives**: Native containers for [`Layer3D.cube`](file:///Users/okanaslankan/gitframes/packages/gitframes/src/shapes3d.ts#L204), `Layer3D.carousel`, `Layer3D.prism`, `Layer3D.plane`, and `Layer3D.grid` with unified depth buffer testing.
-- **Zero-Dependency 3D Model Parsers**: Built-in parsers for **OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, and OFF** files.
-- **Skeletal Animation & Shading**: 128-bone Linear Blend Skinning (LBS), multi-light Blinn-Phong and PBR shading, directional contact soft shadows (PCSS / Poisson PCF), Screen-Space Ambient Occlusion (SSAO), and optical Depth of Field (DoF).
-- **Physical Motion Blur**: $180^\circ$ standard physical camera shutter motion blur powered by per-vertex velocity vectors packed into `rg16float` MRT buffers ([`MotionBlurPipeline`](file:///Users/okanaslankan/gitframes/packages/webgpu-renderers/src/index.ts)).
+- **Calibrated camera rig** — LookAt and Turntable cameras ([`Camera3D`](packages/webgpu-renderers/src/math3d/camera3d.ts)) calibrated so `z = 0` matches 2D canvas pixel coordinates 1:1.
+- **3D layout primitives** — [`Layer3D.cube`](packages/gitframes/src/shapes3d.ts), `carousel`, `prism`, `plane`, `grid` with unified depth-buffer testing.
+- **Zero-dependency model parsers** — OBJ, FBX, glTF/GLB, STL, PLY, VOX, 3DS, OFF.
+- **Skeletal animation & shading** — 128-bone Linear Blend Skinning, Blinn-Phong & PBR multi-light shading, PCSS/Poisson contact shadows, SSAO, and optical DoF.
+- **Physical motion blur** — 180° shutter motion blur with per-vertex velocity vectors packed into `rg16float` MRT buffers.
### 4. Audio Layers, Procedural SFX & Reactive Signals
-Gitframes provides deterministic audio handling and transition sound design:
-- **Soundtrack Audio Layers**: Load and mix soundtrack audio directly via `Layer.audio` with frame-exact lifecycle control.
-- **Procedural Transition SFX**: Deterministic CPU-synthesized whooshes, impacts, risers, downshifters, and glitches placed precisely on the bar/beat grid (`renderSfx`, `mixSfxInto`, `softLimit`).
-- **Multi-Track Mixing**: Combine and master audio tracks headlessly with `mixAudioTracks` and `encodeStereoWav`.
-- **Reactive Signals**: Drive layer transformations, scales, borders, or shader uniforms directly from procedural tempo signals (`Signal.builder`) or audio analysis.
+- **Soundtrack layers** — `.audio` media nodes with frame-exact lifecycle control.
+- **Procedural SFX** — deterministic CPU-synthesized whooshes, impacts, risers, downshifters, and glitches placed on the bar/beat grid (`renderSfx`, `mixSfxInto`, `softLimit`).
+- **Multi-track mixing** — master tracks headlessly with `mixAudioTracks` and `encodeStereoWav`.
+- **Reactive signals** — drive transforms, scale, borders, or shader uniforms from tempo signals (`Signal.builder`) or audio analysis.
### 5. WebGPU ChartGPU Bridge
-Through native integration with [`ChartGPU`](file:///Users/okanaslankan/gitframes/packages/webgpu-renderers/src/chartgpu/chartgpu-bridge.ts):
-- Render real-time financial, scientific, and metric charts (line, bar, candlestick/OHLC, area, scatter) directly onto offscreen GPU textures.
-- Composite live charts directly into 3D floating perspective cards, flex layouts, or HUD overlays without canvas DOM elements.
+Through native [`ChartGPU`](packages/webgpu-renderers/src/chartgpu/chartgpu-bridge.ts) integration:
+- Render real-time financial, scientific, and metric charts (line, bar, OHLC/candlestick, area, scatter) directly onto offscreen GPU textures.
+- Composite live charts into 3D perspective cards, flex layouts, or HUD overlays — no canvas DOM elements.
-### 6. Neural Tensor & MediaPipe AI Conditioning
-Native WebGPU compute pipelines in [`@gitframes/tensor-webgpu`](file:///Users/okanaslankan/gitframes/packages/tensor-webgpu) and [`@gitframes/mediapipe`](file:///Users/okanaslankan/gitframes/packages/mediapipe):
-- **WebGPU Tensor Conditioning**: Real-time Canny edge detection, Monocular Depth-to-Normals, Optical Flow, and Temporal Deflicker pipelines for ControlNet and video generation pipelines.
-- **Zero-Copy Vision Tracking**: MediaPipe Face Mesh and Pose tracking. Extracts 52 facial blendshapes directly into reactive signals to drive 3D avatars, digital humans, or pinned UI elements in GPU memory without disk I/O.
+### 6. YOLO11 Neural Vision & Tracking
+The MediaPipe engine has been **replaced by a first-class YOLO11 pipeline** ([`@gitframes/yolo`](packages/yolo)). It runs ONNX models via `onnxruntime-node` (CPU EP) or `onnxruntime-web` (WebGPU), and wires every result into the same reactive signal surface the rest of Gitframes consumes.
+
+> [!TIP]
+> **Lazy by construction.** `YoloVisionRunner.create()` and `new YoloNode(...)` perform **zero I/O** — no downloads, no sessions, no file probes. A model is fetched the first time a task actually runs. Call `await vision.ready()` to warm up ahead of time.
+
+#### Six task heads from one engine
+
+| Task | Default model | Classes | Output |
+|---|---|---|---|
+| **Detect** | `yolo11{n,s,m,l,x}` | COCO 80 | Axis-aligned boxes + scores |
+| **Segment** | `yolo11{n,s,m}-seg` | COCO 80 | Per-instance masks (frame-aligned alpha) |
+| **Pose** | `yolo11{n,s,m,l}-pose` | COCO-17 | 17 keypoints + visibility per person |
+| **OBB** | `yolo11{n,s,m,l}-obb` | DOTA-15 | Rotated boxes + 4 corners + angle |
+| **Classify** | `yolo11{n,s}-cls` | ImageNet-1k | Top-1 / Top-5 |
+| **World** | `yolov8{s,m}-worldv2` | Open vocabulary | Prompt-driven detection |
+
+- **Variants `n/s/m/l/x`** and **imgsz 320–1280** (640 default; OBB 1024; classify 224). Tune `confidence`, `iouThreshold`, and a COCO `classes` filter per composition.
+- **Custom models** — point `customModel` at a local `path` or `url` with a `task` and `classes` array to run your own ONNX export.
+- **Configurable asset origin** — override `baseUrl` per run or via `GITFRAMES_YOLO_BASE_URL`; models are cached atomically (temp + rename) and evicted if corrupt.
+
+#### Hardware-accelerated vision (WebGPU T2)
+- **GPU letterbox resize** — WGSL compute letterboxing straight into the inference tensor ([`YoloGpuPipeline`](packages/yolo/src/gpu/yolo-gpu-pipeline.ts)).
+- **GPU mask upscaling** — prototype-coefficient mask assembly with threshold + feather in compute ([`mask-upscale.wgsl`](packages/yolo/src/gpu/mask-upscale.wgsl.ts)).
+- **OpenPose-style skeleton textures** — rasterize COCO-17 keypoints into a VRAM conditioning texture ([`PoseSkeletonRenderer`](packages/yolo/src/gpu/pose-skeleton-renderer.ts)).
+- **GPU segmentation texture pool** — reusable `r8unorm`/`rgba8unorm` silhouette textures ([`SegmentationTexturePool`](packages/yolo/src/gpu/segmentation-texture-pool.ts)).
+- **Optical-flow forward warping** — advect detections and masks between inference calls for up to ~4× effective throughput ([`optical-flow-warper`](packages/yolo/src/gpu/optical-flow-warper.ts)).
+
+#### Temporal tracking & analysis
+- **Multi-object tracker** ([`TemporalObjectTracker`](packages/yolo/src/tracking/temporal-object-tracker.ts)) assigns stable `trackId`s via IoU association, with configurable `minHits`, `positionSmoothing`, and velocity-based **coasting** for up to `maxMissedFrames` (default 15) so a transient miss holds the track instead of flashing.
+- **Pose↔track matching** ([`pose-track-matcher`](packages/yolo/src/tracking/pose-track-matcher.ts)) binds keypoints to the right track by id, then by spatial IoU fallback.
+- **One-shot sequence analysis** — `comp.analyzeVisionSequence(src, { tasks, categories })` decodes frames through the mediabunny pipeline, tracks them, and returns a **zod-serializable** report (per-track frame ranges, mean speed, sampled center paths, per-class presence/confidence, mean mask coverage, model download bytes/timing) ([`analyzeSequence`](packages/yolo/src/analysis/analyze-sequence.ts)).
+
+#### Reactive vision signals
+Every tracked entity is exposed as reactive `ProgrammaticSignal`s that animate layers and shader uniforms:
+
+| Group | Highlights |
+|---|---|
+| `objects` | `get(trackId)`, `byCategory(cat, rank)`, `primary`, `count`, `hasCategory`, `detectedCategories` |
+| `objects.*.bounds` | `x/y/width/height`, `screenX/screenY/screenWidth/screenHeight`, `aspectRatio`, `area`, `angle` |
+| `objects.*.anchors` | 9 anchors (corners, edges, center) ready for pinning |
+| `objects.*.kinematics` | `vx`, `vy`, `speed`, `acceleration`, `headingRad/Deg` |
+| `objects.*.pose` | All 17 COCO keypoints, plus `hasPose`, `wristSpeed`, `handRaised`, `bodyTiltAngle` |
+| `masks` | `get(trackId)`, `subject`, `count`; per-mask `area`, `coverage`, `solidity`, `bboxFill` |
+| `segmentation` | `subject`, `humanSilhouette`, `instanceMasks`, GPU `stencilTexture` |
+| `classes` | Per-class `count`, `maxConfidence`, `present`, `primary`, plus a detection `histogram` |
+| `classification` | `top1`, `top1Confidence`, `top5`, `top5At(frame)` |
+| Tensors | `poseLandmarksTensor [17,3]`, `objectsTensor [16,8]`, `masksTensor [16,2]`, `histogramTensor [80]` |
+
+#### Spatial pinning
+Project normalized landmarks to screen space with a configurable camera FOV, then bind any node to a track or landmark ([`SpatialLandmarkTransformer`](packages/yolo/src/spatial/camera-space-transformer.ts), [`spatial-pin`](packages/yolo/src/spatial/spatial-pin.ts)):
+- `pinToObject(track, { anchor, offsetX/Y/Z, matchWidth, matchHeight, smoothFrames, hideWhenLost })`
+- `pinToLandmark(coord, { offsetX/Y/Z })`
+
+#### High-level composition helpers
+- **Subject Sandwich** — `comp.addSubjectSandwich({ source, behind, feather, fit })` cuts the foreground subject out and places typography/graphics behind them.
+- **Smart Reframing** — `comp.addSmartFraming({ source, targetAspect, damping, leadHeadroom })` auto-crops 16:9 → 9:16 while tracking the focal subject.
+- **Subject Outline** — `comp.addSubjectOutline({ source, color, width, blur })` strokes the segmented boundary as an audio-reactive contour glow.
+- **Tracked Region Blur** — `layer.blurRegion(track, { strength })` blurs faces, plates, or any detected class.
+- **Node modes** — `passthrough`, `mask`, `matte`, `crop`, `skeleton`, `boxes`, `tracking`, `obb`, plus optional `keyBackground`/`backgroundKeyThreshold` to grow a person mask into connected foreground (e.g. a dress).
+
+#### Agent-first DX
+- **Runtime config is zod-validated** and available from a **zod-only entry** (`@gitframes/yolo/schemas`) so the hot path stays zod-free.
+- **`vision.summary(frame)`** returns a deterministic, serializable snapshot (objects, classes, masks) safe to call inside a frame hook.
+- **Browser entry** — `@gitframes/yolo/web` re-exports the engine plus `createWebGPUProvider()` / `hasWebGPU()`; `onnxruntime-web` is an optional lazy peer.
+
+#### Migration from MediaPipe
+The `MediaPipe` effect class and `comp.withMediaPipe(...)` remain as **deprecated, source-compatible aliases** that normalize legacy options (`enablePoseLandmarks`, `enableSegmentation`, `mode: "skeleton"`, `objectScoreThreshold`, …) onto the YOLO engine. Prefer [`Yolo`](packages/gitframes/src/effects/classes.ts) and `comp.withYolo(...)`. Face blendshape signals were removed with the old engine; `driveBlendshapes` is accepted as a no-op.
### 7. Headless Conformance & FrameGrid Testing
-- **Pixel-Sampling Invariant Assertions**: Test compositions in Vitest headlessly using `skia-canvas` to verify shader mathematics, font coverage, and Mean Squared Error (MSE) temporal frame deltas.
-- **FrameGrid Visual Contact Sheets**: Call `comp.renderFrameGrid(...)` to output contact sheets of sequential frames across a timeline, enabling instant visual review of easing curves, kinetic typography, and transitions.
+- **Pixel-sampling invariant assertions** — test compositions in Vitest with `skia-canvas` to verify shader math, font coverage, and Mean Squared Error (MSE) temporal deltas.
+- **FrameGrid contact sheets** — `comp.renderFrameGrid(...)` outputs sequential-frame contact sheets for instant review of easing, kinetic type, and transitions.
---
## Monorepo Architecture
-Gitframes is structured as a modular monorepo managed with `pnpm` and `turbo`:
+Managed with `pnpm` workspaces and `turbo`:
```
gitframes/
├── packages/
-│ ├── gitframes/ # Primary unified SDK (Composition, Layer, LayerAnimation, Signal)
-│ ├── core/ # Core AST definitions, Effect base class, VirtualMediaData
+│ ├── gitframes/ # Unified SDK (Composition, Layer, LayerAnimation, Signal, effects)
+│ ├── core/ # Core AST, Effect base class, VirtualMediaData, vision types
│ ├── compositions/ # Layout engine, Flex/Box AST compiler, timeline evaluator
-│ ├── webgpu-renderers/ # WGSL shaders, Slug GPU text engine, 3D renderer, camera, lights, materials
-│ ├── tensor-webgpu/ # WebGPU compute pipelines (Canny, Depth-to-Normals, Flow, Deflicker)
-│ ├── mediapipe/ # MediaPipe vision runner, 52 blendshape signals, landmark pinning
-│ ├── renderer/ # Headless Node.js WebGPU renderer via Dawn, WebCodecs, and skia-canvas
-│ └── server-utils/ # Shared server infrastructure, file storage, and asset caches
-├── nodes/ # 58+ specialized domain nodes (VFX, audio processors, layout generators)
+│ ├── webgpu-renderers/ # WGSL shaders, Slug text engine, 3D renderer, camera, lights, materials
+│ ├── tensor-webgpu/ # WebGPU compute pipelines (Canny, depth-to-normals, flow, deflicker, landmarks)
+│ ├── yolo/ # YOLO11 ONNX engine: tasks, decode, tracking, GPU accel, signals
+│ ├── renderer/ # Headless Node.js WebGPU renderer via Dawn, WebCodecs, skia-canvas
+│ ├── renderers/ # Higher-level render orchestration
+│ ├── media/ # Media decoding / encoding adapters
+│ ├── node-sdk/ # Node renderer contracts and result schemas
+│ ├── server-utils/ # Server infrastructure, storage, asset caches
+│ └── client-utils/ # Shared browser utilities
+├── nodes/ # 58+ specialized domain nodes (VFX, audio, layout, node-yolo)
├── apps/
│ └── renderer-service/ # Production HTTP / gRPC rendering microservice container
-├── examples/ # 19 production-grade reference compositions and films
-├── recipes/ # Architectural composition guides and best practices
-└── specs/ # Engineering specifications and mathematical audits
+├── examples/ # Reference compositions and films
+├── skills/ # Portable agent skills (compose, effects, render)
+└── scripts/ # Build, release, and plugin validation tooling
```
---
@@ -178,7 +253,7 @@ gitframes/
pnpm add gitframes
```
-> **Note**: Node.js $\ge 22$ is required. Gitframes uses native GPU acceleration via Dawn / WebGPU or Vulkan.
+> **Requirements:** Node.js ≥ 22. Gitframes uses native GPU acceleration via Dawn / WebGPU or Vulkan.
---
@@ -197,13 +272,13 @@ const comp = new Composition({
fonts: ["assets/fonts/Inter.ttf", "assets/fonts/SpaceGrotesk.ttf"],
});
-// 2. Define physical snap overshoot animations
+// 2. Define physical snap-overshoot animations
const cardEntrance = LayerAnimation.create()
.fadeIn(0, 20, "power2.out")
.fromTo("y", 60, 0, { start: 0, end: 35, ease: "back.out(1.5)" })
.fromTo("scale", 0.92, 1.0, { start: 0, end: 35, ease: "back.out(1.2)" });
-// 3. Assemble responsive flex layout card
+// 3. Assemble a responsive flex-layout card
const heroCard = Layer.box({
width: 720,
height: 380,
@@ -251,17 +326,19 @@ import { Composition, Layer, Layer3D, CameraAnimation, Light } from "gitframes";
const comp = new Composition({ width: 1920, height: 1080, fps: 60, durationFrames: 300 });
-// 1. Declare LookAt 3D Camera with continuous orbit
+// 1. LookAt 3D camera with a continuous orbit
const cameraAnim = CameraAnimation.create()
.orbit({ fromAzimuth: -30, toAzimuth: 30, elevation: 15, radius: 1200, start: 0, end: 300 });
-comp.add(Layer.camera({ x: 960, y: 540, z: -1000, targetX: 960, targetY: 540, targetZ: 0 }).animate(cameraAnim));
+comp.add(
+ Layer.camera({ x: 960, y: 540, z: -1000, targetX: 960, targetY: 540, targetZ: 0 }).animate(cameraAnim)
+);
-// 2. Add Studio Lighting
+// 2. Studio lighting
comp.add(Light.ambient("#ffffff", 0.4));
comp.add(Light.directional({ color: "#e0e7ff", intensity: 1.2, x: 500, y: -800, z: -600 }));
-// 3. Add 3D Model with skeletal animation
+// 3. 3D model with skeletal animation
comp.add(
Layer.glb("assets/models/character.glb", {
x: 960,
@@ -273,7 +350,7 @@ comp.add(
})
);
-// 4. Add 3D Prism Layout Carousel
+// 4. 3D prism layout carousel
comp.add(
Layer3D.carousel({
radius: 400,
@@ -296,10 +373,10 @@ import { Composition, Layer, LayerAnimation, Signal, renderSfx, mixSfxInto, soft
const comp = new Composition({ width: 1920, height: 1080, fps: 60 });
const totalFrames = 240;
-// 1. Add soundtrack audio
+// 1. Soundtrack layer
comp.addAudio(Layer.audio("assets/score.mp3", { volume: 0.9, durationFrames: totalFrames }));
-// 2. Synthesize frame-accurate procedural SFX hits
+// 2. Frame-accurate procedural SFX on the beat grid
const bed: [Float32Array, Float32Array] = [
new Float32Array(Math.ceil((totalFrames / 60) * 48000)),
new Float32Array(Math.ceil((totalFrames / 60) * 48000)),
@@ -310,20 +387,20 @@ mixSfxInto(bed, [
]);
softLimit(bed);
-// 3. Generate reactive signals synced to tempo (e.g. 120 BPM = 2 Hz)
+// 3. Tempo signal (120 BPM = 2 Hz)
const beatPulse = Signal.builder({ type: "sawtooth", frequency: 2, amplitude: 0.08, offset: 1.0 });
-// 4. Bind reactive signals to visual properties
+// 4. Bind it to visuals
const reactiveCard = Layer.box({ width: 400, height: 250, background: "#1c202e", borderRadius: 20 })
.animate(
LayerAnimation.create()
.signal("scale", beatPulse, { multiplier: 1.0, offset: 0.0 })
.fromTo("opacity", 0, 1, { start: 0, end: 15, ease: "power2.out" }),
);
+
comp.add(reactiveCard);
```
-
---
### 4. Chained WebGPU Post-Processing VFX
@@ -333,7 +410,7 @@ import { Composition, FilmGrain, Vignette, ColorBalance } from "gitframes";
const comp = new Composition({ width: 1920, height: 1080, fps: 60 });
-// Apply whole-composition cinematic color grading and film emulsion
+// Whole-composition cinematic grade + film emulsion
comp.apply(new Vignette({ strength: 0.28, radius: 0.85 }));
comp.apply(new FilmGrain({ strength: 0.06, size: 1.5, animated: true }));
comp.apply(new ColorBalance({ shadows: [0, 2, 6], highlights: [4, 1, -2] }));
@@ -341,17 +418,83 @@ comp.apply(new ColorBalance({ shadows: [0, 2, 6], highlights: [4, 1, -2] }));
---
-### 5. Headless Video & FrameGrid Rendering
+### 5. YOLO11 Vision: Pin, Matte & Reframe
+
+```typescript
+import { Composition, Layer, Vignette } from "gitframes";
+
+const comp = new Composition({ width: 1920, height: 1080, fps: 30 });
+
+// Attach YOLO11 to the whole composition. Models download lazily on first use.
+const vision = comp.withYolo({
+ enableDetection: true,
+ enableSegmentation: true,
+ enablePose: true,
+ variant: "s",
+ confidence: 0.35,
+});
+
+// Pin a caption to the primary tracked subject (smoothing + auto-hide when lost)
+comp.add(
+ Layer.text("SUBJECT 01", { fontSize: 40, fill: "#f8fafc" }).pinToObject(
+ vision.objects.primary,
+ { anchor: "topCenter", offsetY: -48, smoothFrames: 5, hideWhenLost: true },
+ ),
+);
+
+// Drive a shader uniform from a reactive signal — here, subject mask coverage
+comp.add(
+ Layer.box({ width: 1920, height: 1080, background: "#000000" }).withEffect(
+ new Vignette({ strength: vision.segmentation.subject.coverage, radius: 0.9 }),
+ ),
+);
+
+// Or use the one-liners for the common editorial moves:
+// comp.addSubjectSandwich({ source: "assets/dancer.mp4", behind: [headline], feather: 4 });
+// comp.addSmartFraming({ source: "assets/action.mp4", targetAspect: 9 / 16, damping: 0.15 });
+// comp.addSubjectOutline({ source: "assets/character.mp4", color: "#FF5A1F", width: 6, blur: 12 });
+
+// Inspect a source before authoring: one-shot, ffmpeg-free, zod-serializable report
+const report = await comp.analyzeVisionSequence("assets/street.mp4", {
+ tasks: ["detect", "pose"],
+ categories: ["person"],
+});
+console.log(report.tracks.map((t) => `${t.category}#${t.trackId} ${t.frames.join("–")}`));
+```
+
+**Standalone runner (no composition):**
+
+```typescript
+import { YoloVisionRunner } from "@gitframes/yolo";
+
+const runner = YoloVisionRunner.create({ variant: "n", confidence: 0.25 }); // zero I/O
+const boxes = await runner.detect({ data: rgba, width: 1920, height: 1080 });
+runner.close();
+```
+
+**In the browser (WebGPU EP):**
+
+```typescript
+import { YoloVisionRunner, createWebGPUProvider, hasWebGPU } from "@gitframes/yolo/web";
+
+if (hasWebGPU()) {
+ const runner = YoloVisionRunner.create({ provider: createWebGPUProvider() });
+}
+```
+
+---
+
+### 6. Headless Video & FrameGrid Rendering
```typescript
import { buildMyComposition } from "./my-composition.js";
const comp = await buildMyComposition();
-// 1. Render single frame to PNG buffer for visual inspection
+// 1. Single frame to a PNG buffer for visual inspection
const frameBuffer = await comp.renderFrame({ frame: 45 });
-// 2. Render contact sheet grid of 12 sequential frames
+// 2. Contact-sheet grid of 12 sequential frames
const gridBuffer = await comp.renderFrameGrid({
startFrame: 0,
endFrame: 120,
@@ -360,7 +503,7 @@ const gridBuffer = await comp.renderFrameGrid({
showLabels: true,
});
-// 3. Render final hardware-encoded MP4 video with mixed audio
+// 3. Final hardware-encoded MP4 with mixed audio
const { filePath } = await comp.renderVideo({
outputPath: "output/final-product-film.mp4",
quality: "high",
@@ -372,32 +515,30 @@ console.log(`Video rendered successfully to: ${filePath}`);
---
-### Key Rules to Follow
+## Engineering Doctrines & Best Practices
-1. **Design Tokens & Theme Contracts**: Always define a centralized `THEME` object for colors, typography, border radii, and spacing scales. Never hardcode magic hex values or ad-hoc margins.
-2. **WebGPU Premultiplied Alpha Invariant**: Fragment shaders outputting premultiplied alpha (`color * opacity * alpha`) must use `srcFactor: "one"` in their blend state (`color: { srcFactor: "one", dstFactor: "one-minus-src-alpha", operation: "add" }`). Never use `srcFactor: "src-alpha"` for premultiplied outputs, as squaring alpha causes fade-ins to darken into murky gray.
-3. **Carrier Match Cuts**: Carry visual elements (a badge, card, cursor, or container) across scene boundaries with continuous velocity and position to eliminate jarring transition cuts.
-4. **Physical Easing Vocabulary**: Use `back.out(1.4–1.7)` for snap overshoot entrances, `spring` or `expo.out` for decelerating motion, and `power2.in` for exits. Reserve `linear` solely for infinite spinners or time counters.
-5. **Headless Invariant Verification**: Verify true shader transformations, font glyph rasterization coverage, and temporal frame deltas (MSE) headlessly via pixel sampling with `skia-canvas` in Vitest before deploying.
+1. **Design tokens & theme contracts** — define a centralized `THEME` for colors, type, radii, and spacing. Never hardcode magic hex values or ad-hoc margins.
+2. **WebGPU premultiplied-alpha invariant** — fragment shaders outputting premultiplied alpha (`color * opacity * alpha`) must use `srcFactor: "one"` in their blend state (`{ srcFactor: "one", dstFactor: "one-minus-src-alpha", operation: "add" }`). Never use `srcFactor: "src-alpha"` for premultiplied output — squaring alpha darkens fades into murky gray.
+3. **Carrier match cuts** — carry a visual element (badge, card, cursor, container) across scene boundaries with continuous velocity and position to avoid jarring cuts.
+4. **Physical easing vocabulary** — `back.out(1.4–1.7)` for snap-overshoot entrances, `spring` / `expo.out` for decelerating motion, `power2.in` for exits. Reserve `linear` for infinite spinners and time counters.
+5. **Headless invariant verification** — verify shader transforms, glyph coverage, and temporal MSE deltas with `skia-canvas` pixel sampling in Vitest before shipping.
---
## Agent Skills & Plugins
-Gitframes ships agent skills that teach Claude, Codex and other coding agents how to write, render and check gitframes compositions. The repository is a plugin marketplace (`gitframes-plugins`) with a single plugin (`gitframes`), and that plugin contains only skills: no MCP servers, hooks or commands.
+Gitframes ships agent skills that teach Claude, Codex, and other coding agents how to write, render, and check compositions. The repository is a plugin marketplace (`gitframes-plugins`) with a single plugin (`gitframes`) that contains **only skills** — no MCP servers, hooks, or commands.
| Skill | Use it for |
| --- | --- |
| `gitframes-compose` | Compositions, layer trees, layout, animation and easing, beat grids, film structure |
-| `gitframes-effects` | Effect classes, the unified section architecture, premultiplied-alpha invariants |
+| `gitframes-effects` | Effect classes, the unified section architecture, premultiplied-alpha invariants, YOLO vision conditioning |
| `gitframes-render` | Headless rendering, FrameGrid inspection, pixel probes, MP4 delivery checks |
-Once installed, the skills load on their own when a task matches them (e.g. *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*). You don't need to invoke them.
+Once installed, skills load automatically when a task matches (e.g. *"add a film-grain pass to this scene"* or *"render a frame grid of intro.ts"*).
### Claude Code
-Inside a Claude Code session:
-
```text
/plugin marketplace add gatewai-dev/gitframes
/plugin install gitframes@gitframes-plugins
@@ -410,9 +551,9 @@ claude plugin marketplace add gatewai-dev/gitframes
claude plugin install gitframes@gitframes-plugins
```
-Add `--scope project` to the install command to record the plugin in the project's `.claude/settings.json` so your whole team gets it. To pull new skill versions later, run `/plugin marketplace update gitframes-plugins`.
+Add `--scope project` to record the plugin in `.claude/settings.json` for the whole team. Update later with `/plugin marketplace update gitframes-plugins`.
-**Enable it for everyone working in your repo.** Commit this to `.claude/settings.json`; Claude Code prompts teammates to install it when they trust the folder:
+**Enable it for everyone in your repo.** Commit this to `.claude/settings.json`; Claude Code prompts teammates to install it when they trust the folder:
```json
{
@@ -427,17 +568,15 @@ Add `--scope project` to the install command to record the plugin in the project
}
```
-**Claude desktop app (Code tab):** open the plugins settings, add the marketplace `gatewai-dev/gitframes`, then install **Gitframes**.
+**Claude desktop app (Code tab):** open plugin settings, add the marketplace `gatewai-dev/gitframes`, then install **Gitframes**.
### Codex
-Add the marketplace:
-
```bash
codex plugin marketplace add gatewai-dev/gitframes
```
-Then install **gitframes** from `/plugins` in the Codex TUI, or turn it on in `~/.codex/config.toml` (or in a project's `.codex/config.toml`):
+Then install **gitframes** from `/plugins` in the TUI, or enable it in `~/.codex/config.toml` (or a project's `.codex/config.toml`):
```toml
[plugins."gitframes@gitframes-plugins"]
@@ -446,53 +585,58 @@ enabled = true
### Any other agent (skills only)
-With the [`skills`](https://skills.sh) CLI, which supports Claude Code, Codex, Cursor, Copilot and others:
+With the [`skills`](https://skills.sh) CLI, which supports Claude Code, Codex, Cursor, Copilot, and others:
```bash
npx skills add gatewai-dev/gitframes
```
-Or copy the folders by hand: put `skills/<name>/` into `.claude/skills/`, `.agents/skills/` or `~/.agents/skills/`. VS Code / Copilot / Cursor / Kiro can also load the portable root [`plugin.json`](plugin.json) through their plugin UI.
+Or copy the folders by hand: put `skills/<name>/` into `.claude/skills/`, `.agents/skills/`, or `~/.agents/skills/`. VS Code / Copilot / Cursor / Kiro can load the portable root [`plugin.json`](plugin.json) through their plugin UI.
### Maintaining the plugin
-Three manifests describe the same plugin: [`plugin.json`](plugin.json) (portable [Agent Plugins 1.0](https://agent-plugins.org)), [`.claude-plugin/plugin.json`](.claude-plugin/plugin.json) (Claude Code) and [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json) (Codex). The marketplace catalogs are [`.claude-plugin/marketplace.json`](.claude-plugin/marketplace.json) and [`.agents/plugins/marketplace.json`](.agents/plugins/marketplace.json). The portable field set is closed, so client-specific fields go in that client's manifest, not in `plugin.json`. Bump `version` in all three manifests together, because clients use it to decide when to update.
+Three manifests describe the same plugin: [`plugin.json`](plugin.json) (portable [Agent Plugins 1.0](https://agent-plugins.org)), [`.claude-plugin/plugin.json`](.claude-plugin/plugin.json), and [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json). The marketplace catalogs are [`.claude-plugin/marketplace.json`](.claude-plugin/marketplace.json) and [`.agents/plugins/marketplace.json`](.agents/plugins/marketplace.json). The portable field set is closed — client-specific fields go in that client's manifest, not in `plugin.json`. Bump `version` in all three together, since clients use it to decide when to update.
-When you work inside this repository, Codex and Claude pick up the skills automatically through the symlinks in `.agents/skills/` and `.claude/skills/`. Skills are written only under `skills/`, so never copy them anywhere else. `pnpm run check:plugins` validates the manifests, skill frontmatter, marketplace catalogs, symlinks and the generated effects catalog. `pnpm run sync:effects-catalog` regenerates `gitframes-effects`'s catalog after any change to the `Effect` classes.
+Inside this repository, Codex and Claude pick up skills through the symlinks in `.agents/skills/` and `.claude/skills/`. Skills live only under `skills/`; never copy them elsewhere. `pnpm run check:plugins` validates manifests, skill frontmatter, marketplace catalogs, symlinks, and the generated effects catalog. `pnpm run sync:effects-catalog` regenerates the `gitframes-effects` catalog after any `Effect` class change.
---
-## Development & Building
+## Reference Showcase Examples
+
+The [`examples/`](examples) directory holds production-grade reference compositions:
+
+| Example | What it demonstrates |
+| --- | --- |
+| [`19_gitframes_film`](examples/19_gitframes_film) | The 30-second master brand film — full pipeline, audio, VFX, 3D |
+| [`21_full_circle`](examples/21_full_circle) | Multi-scene narrative composition |
+| [`22_gitframes_launch`](examples/22_gitframes_launch) | Launch/product-motion composition |
-Gitframes uses `pnpm` (version 10+) and `turbo` for monorepo orchestration.
+---
+
+## Development & Building
-### Build All Packages
+Gitframes uses `pnpm` (10+) and `turbo` for orchestration.
```bash
+# Install
pnpm install
-pnpm build
-```
-### Run Conformance Tests
+# Build all packages
+pnpm build
-```bash
+# Run conformance tests
pnpm test
-```
-
-### Run an Example Renderer
-```bash
# Render a specific showcase example
-pnpm --filter @gitframes/example-04-blending render
+pnpm --filter @gitframes/example-21-full-circle render
-# Render the 30-second master brand film
-cd examples/19_gitframes_film
-pnpm render
+# Render the master brand film
+cd examples/19_gitframes_film && pnpm render
```
### Docker Container for Production Rendering
-Gitframes provides an optimized Docker build ([`Dockerfile.renderer`](file:///Users/okanaslankan/gitframes/Dockerfile.renderer)) for deploying the renderer service in cloud GPU clusters:
+An optimized [`Dockerfile.renderer`](Dockerfile.renderer) deploys the renderer service into cloud GPU clusters:
```bash
docker build -t gitframes-renderer -f Dockerfile.renderer .
@@ -502,4 +646,4 @@ docker build -t gitframes-renderer -f Dockerfile.renderer .
## License
-Gitframes is open-source software licensed under [Apache-2.0]
+Gitframes is open-source software licensed under [Apache-2.0](LICENCE). Downloaded YOLO model weights follow their upstream Ultralytics licensing.