Code-first video & image, rendered natively on WebGPU.
Merge remote-tracking branch 'origin/main' into fix/gltf-compatibility
okanasl0f2a545fac6a
diff --cc nodes/node-compositor/SKILL.mdindex 810d97d1,e36910ce..45502a05--- a/nodes/node-compositor/SKILL.md+++ b/nodes/node-compositor/SKILL.md@@@ -210,7 -210,11 +210,11 @@@ Each track represents animatable proper- **Audio-Reactive 3D Models:** a `kind: "model3d"` node loading an OBJ/FBX/GLTF/STL/PLY mesh with `audioDeform: { audioTrackId, mode: "normal_extrusion" | "radial_pulse" | "harmonic_wave" | "twist" | "ripple", amplitudeMultiplier, damping }`, deformed directly in VRAM via WebGPU compute shaders driven by audio frequency spectra and transient energies. `twist` turns each slice about Y by its height (harder on the bass); `ripple` runs concentric waves across XY, struck by the drums. `audioTrackId` may be an audio layer's id or an audio file path.- **Transparent meshes:** glTF `BLEND` materials and meshes with reduced layer, material, tint, or vertex alpha draw after opaque items, back to front by the camera-space center of each mesh’s transformed bounds, without writing depth. This is object-level sorting: intersecting meshes, self-overlap, skinned/deformed bounds, translucent flat layers, glass, and transparent shadows have separate limitations.- **Mesh wireframe:** `wireframe: true` on a `model3d` draws triangle edges as one-pixel lines, including triangulation and back edges; it also applies to skinned and audio-deformed meshes and their shadows.-- **glTF scenes:** a glTF/GLB `model3d` keeps its node hierarchy, every skin and its animation clips (translation, rotation, scale; linear, step and cubic-spline). Skinned meshes are posed per frame from the clip (`animationName`, `animationTime`/`animationProgress`, `loop`); base-colour textures (with mipmaps), alpha `MASK`/`BLEND` and VRM MToon materials (`material: "toon"`, picked automatically when `material` is unset) render as authored. `center`/`normalizeSize` apply above the scene graph, so skinning stays intact.+- **glTF scenes:** a glTF/GLB `model3d` keeps its node hierarchy, every skin and its animation clips (translation, rotation, scale and morph weights; linear, step and cubic-spline). Draco and meshopt compressed geometry and all seven core primitive modes are supported; per-map UV sets and texture transforms are independent. Basis KTX2 (ETC1S/UASTC) textures transcode offline to RGBA8 with their mip levels. Unsupported required extensions fail clearly. Morph targets run on the GPU before skinning and audio deformation. Skinned meshes are posed per frame from the clip (`animationName`, `animationTime`/`animationProgress`, `loop`); base-colour textures (with mipmaps), alpha `MASK`/`BLEND` and VRM MToon materials (`material: "toon"`, picked automatically when `material` is unset) render as authored. `center`/`normalizeSize` apply above the scene graph, so skinning stays intact.+ - **Procedural Shards (`Layer3D.shard`):** Procedural polyhedron primitive generated by Sutherland-Hodgman convex half-space clipping against Fibonacci lattice normals with PCG jitter. Faceted flat normals, automatic planar UVs, Rapier 3D convex hull collision.+ - **Instanced 3D Meshes (`instances3d`):** `kind: "model3d"` with `instances3d` renders hundreds or thousands of mesh instances in 1 GPU draw call (`drawIndexed(indexCount, instanceCount)`). Supports single or multi-mesh variants with round-robin or variant indexing, dynamic per-instance transforms, and per-instance emissive / color modulation. In `framefields`, created with `Layer3D.instances({ count, mesh, instanceAt, bodies, emissiveBySpeed })`.+ - **GPU 3D Swarms (`swarm3d`):** `kind: "model3d"` with `swarm3d` runs continuous 3D particle simulation in a WebGPU compute shader with 3D curl noise, 3D vortex fields (`swirl`, `pull`, `axial`), ground plane collisions with Coulomb sliding friction and rotational roll torque, and kinetic heat accumulation/cooling, packing directly into GPU instance buffers.+ - **Planar Reflective Floor (`floor` in `environment`):** `kind: "light"` with `lightType: "environment"` and `floor: { y, color, reflectivity, roughness, fadeDistance, opacity }` renders a mirrored-camera reflection pass, subterranean fragment clipping, 2-pass anisotropic roughness blur ($\sigma_y \approx 3.5 \sigma_x$), Schlick Fresnel grazing reflections, and analytical depth unprojection.## Don't Forget- If text rendering required, you must include a font in spec (CLI TOOL only). No emoji font ships with framefields: to draw emoji, provide a color emoji font yourself with `FontManager.registerEmojiFont(path)` (a CBDT/CBLC bitmap font such as Noto Color Emoji).diff --cc nodes/node-compositor/src/renderers/webgpu-renderer.tsindex 922c23a4,a5deac3d..208fc313--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts@@@ -67,7 -67,9 +67,10 @@@ importtype ShapeGeometryConfig,SlugFontCache,type SlugGlyphBatch,+ type Swarm3DFrameInput,+ type Swarm3DState,+ type Swarm3DStaticConfig,+ sampleMorphWeights,spansText,srtLoader,Vector3Math,diff --cc packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.tsindex b07b2a62,8d3af72c..a823e4a3--- a/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts+++ b/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts@@@ -49,10 -49,12 +49,14 @@@ export interface MeshUniformData extendwireframe?: boolean;/** Blend without writing depth; the caller draws transparent meshes back to front. */transparent?: boolean;+ /** Extra per-map UV vertex attributes are present. */+ hasMapUvs?: boolean;isSkinned?: boolean;skinningMatrices?: Float32Array; // 128 * 16 floats+ /** Hardware-instanced draw path. */+ isInstanced?: boolean;+ /** Storage buffer containing `array<InstanceData>` (64 bytes per instance). */+ instanceBuffer?: GPUBuffer;material?: "lit" | "unlit" | "toon";/** Screen-space track matte: the mesh shows where this texture's alpha does. */matte?: GPUTexture;@@@ -418,7 -401,7 +456,10 @@@ export class Mesh3DPipelinethis.modelData[36] = data.opacity ?? 1.0;this.modelData[37] = data.twoSided ? 1.0 : 0.0;this.modelData[38] = data.isSkinned ? 1.0 : 0.0;- this.modelData[39] = data.hasMapUvs ? 1 : 0;- this.modelData[39] = data.isInstanced || data.instanceBuffer ? 1.0 : 0.0;++ // Bit 0 selects instance data; bit 1 selects the extra map UVs.++ this.modelData[39] =++ (data.isInstanced || data.instanceBuffer ? 1 : 0) +++ (data.hasMapUvs ? 2 : 0);const shininess =data.roughness !== undefined@@@ -494,11 -483,10 +541,13 @@@data: MeshUniformData,useMrt = false,useBloom = false,+ instanceCount = 1,+ firstInstance = 0,): void {- const modelBindGroup = this.packModel(data);+ const modelBindGroup = this.packModel({+ ...data,+ hasMapUvs: Boolean(meshBuffers.mapUvBuffer),+ });const white = this.defaultWhiteTexture;const maps = data.maps;const textureBindGroup = this.device.createBindGroup({@@@ -578,10 -560,10 +637,12 @@@lightCameraBindGroup: GPUBindGroup,meshBuffers: GPUMeshBuffers,data: MeshUniformData,+ instanceCount = 1,+ firstInstance = 0,): void {- const wireframe = data.wireframe === true;+ const wireframe =+ data.wireframe === true &&+ (!meshBuffers.topology || meshBuffers.topology === "triangle-list");if (wireframe && !meshBuffers.wireIndexBuffer) {throw new Error("Wireframe requires mesh buffers with edge indices");}diff --cc packages/webgpu-renderers/src/renderer3d/renderer3d.tsindex 3282d6db,dd8bbd21..81bd0753--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts@@@ -45,7 -45,7 +46,8 @@@ import { dofBokehWgsl } from "../shaderimport type { SlugGlyphBatch } from "../slug/slug-pipeline.js";import { AudioMeshDeformPipeline } from "./audio-mesh-deform-pipeline.js";import { type BloomOptions, BloomPipeline } from "./bloom-pipeline.js";+ import { FloorReflectionPipeline } from "./floor-reflection-pipeline.js";+import { decodeKtx2, isKtx2 } from "./ktx2.js";import {type GPUMeshBuffers,Mesh3DPipeline,@@@ -237,7 -243,8 +248,9 @@@ export class Renderer3Dpublic slug3dPipeline: Slug3DPipeline;public mesh3dPipeline: Mesh3DPipeline;public audioMeshDeformPipeline: AudioMeshDeformPipeline;+ private morphPipeline: MorphPipeline;+ public swarm3dPipeline: Swarm3DPipeline;+ public floorReflectionPipeline: FloorReflectionPipeline;private dofPipeline: GPURenderPipeline;private dofBindGroupLayout: GPUBindGroupLayout;private cameraPool: BufferPool;@@@ -307,7 -320,15 +326,16 @@@sampleCount,);this.audioMeshDeformPipeline = new AudioMeshDeformPipeline(device);+ this.morphPipeline = new MorphPipeline(device);+ this.swarm3dPipeline = new Swarm3DPipeline(device);+ this.floorReflectionPipeline = new FloorReflectionPipeline(+ device,+ format,+ sampleCount,+ this.quad3dPipeline.cameraLayout,+ this.quad3dPipeline.lightsLayout,+ "depth24plus",+ );this.cameraPool = new BufferPool(320,@@@ -1682,9 -1869,15 +1977,16 @@@this.quad3dPipeline.destroy();this.slug3dPipeline.destroy();this.mesh3dPipeline.destroy();+ this.morphPipeline.destroy();+ this.floorReflectionPipeline.destroy();+ this.reflectionColorTexture?.destroy();+ this.reflectionMsaaColorTexture?.destroy();+ this.reflectionDepthTexture?.destroy();+ this.reflectionTempTexture?.destroy();+ this.reflectionBlurredTexture?.destroy();this.cameraPool.destroy();this.lightsPool.destroy();+ this.instancePool.destroy();this.cameraUniformBindGroupCache.destroy();}}diff --cc packages/webgpu-renderers/src/shaders/mesh3d.tsindex 99c740ce,f36a3b53..7b02d92a--- a/packages/webgpu-renderers/src/shaders/mesh3d.ts+++ b/packages/webgpu-renderers/src/shaders/mesh3d.ts@@@ -17,7 -17,7 +17,7 @@@ struct ModelUniformsmodelMatrix : mat4x4<f32>,normalMatrix : mat4x4<f32>,colorTint : vec4<f32>,- params : vec4<f32>, // x = opacity, y = twoSided (1.0 = yes), z = isSkinned (1.0 = yes), w = hasMapUvs- params : vec4<f32>, // x = opacity, y = twoSided (1.0 = yes), z = isSkinned (1.0 = yes), w = isInstanced (1.0 = yes)++ params : vec4<f32>, // x = opacity, y = twoSided, z = isSkinned, w = flags (bit 0 instanced, bit 1 map UVs)materialParams : vec4<f32>, // x = shininess, y = specularIntensity, z = ambientIntensity, w = materialMode (0=unlit, 1=lit, 2=toon)shade : vec4<f32>, // rgb = toon shadow-side color multiplier, a = alpha cutoff (0 = none)pbr : vec4<f32>, // x = roughness (<0 = default), y = metallic@@@ -61,14 -65,19 +69,23 @@@ struct VertexOutput@location(2) worldNormal : vec3<f32>,@location(3) linearDepth : f32,@location(4) color : vec4<f32>,- @location(5) emissive : vec4<f32>,+ @location(5) normalUv : vec2<f32>,+ @location(6) mrUv : vec2<f32>,+ @location(7) aoUv : vec2<f32>,+ @location(8) emissiveUv : vec2<f32>,++ @location(9) emissive : vec4<f32>,};+ fn rotateVecByQuat(v: vec3<f32>, q: vec4<f32>) -> vec3<f32> {+ let t = 2.0 * cross(q.xyz, v);+ return v + q.w * t + cross(q.xyz, t);+ }+@vertex- fn vs_main(in: VertexInput) -> VertexOutput {+ fn vs_main(+ in: VertexInput,+ @builtin(instance_index) instanceIndex: u32,+ ) -> VertexOutput {var out: VertexOutput;var localPos = in.position;@@@ -101,6 -110,23 +118,23 @@@}}+ // Hardware instancing: transform local vertex by instance transform+ // Static read prevents Tint from dropping the storage buffer binding+ let dummyInst = instances[0].posScale.x;+ var instColor = vec4<f32>(1.0);+ var instEmissive = vec4<f32>(0.0);+- if (model.params.w > 0.5) {++ if ((u32(model.params.w) & 1u) != 0u) {+ let inst = instances[instanceIndex];+ let scaledPos = localPos * inst.posScale.w;+ localPos = rotateVecByQuat(scaledPos, inst.quat) + inst.posScale.xyz;+ localNorm = rotateVecByQuat(localNorm, inst.quat);+ instColor = inst.color;+ instEmissive = inst.emissive;+ } else {+ localPos.x += dummyInst * 0.0;+ }+let worldPos4 = model.modelMatrix * vec4<f32>(localPos, 1.0);out.worldPos = worldPos4.xyz;@@@ -109,11 -135,8 +143,13 @@@out.clipPos = cam.viewProjMatrix * worldPos4;out.uv = in.uv;- out.normalUv = select(in.uv, in.normalUv, model.params.w > 0.5);- out.mrUv = select(in.uv, in.mrUv, model.params.w > 0.5);- out.aoUv = select(in.uv, in.aoUv, model.params.w > 0.5);- out.emissiveUv = select(in.uv, in.emissiveUv, model.params.w > 0.5);- out.color = in.color;++ let hasMapUvs = (u32(model.params.w) & 2u) != 0u;++ out.normalUv = select(in.uv, in.normalUv, hasMapUvs);++ out.mrUv = select(in.uv, in.mrUv, hasMapUvs);++ out.aoUv = select(in.uv, in.aoUv, hasMapUvs);++ out.emissiveUv = select(in.uv, in.emissiveUv, hasMapUvs);+ out.color = in.color * instColor;+ out.emissive = instEmissive;out.linearDepth = dot(worldPos4.xyz - cam.cameraPos.xyz, cam.cameraDir.xyz);return out;diff --cc plugins/framefields/skills/framefields-compose/SKILL.mdindex 2403724b,f88c5a62..c6eec60f--- a/plugins/framefields/skills/framefields-compose/SKILL.md+++ b/plugins/framefields/skills/framefields-compose/SKILL.md@@@ -443,8 -446,9 +446,9 @@@ Layer3D.cylinder({ ..., color: "#111",- **Bloom only takes light well past white** (emissive ≳ 3, hot speculars), so ordinary bright surfaces don't glow. Raise `emissiveIntensity` to make something glow.- **Shadows:** one `castShadow` directional or spot light per scene. Meshes cast by default (`castShadow: false` opts out); flat layers (planes, cards) cast only with `castShadow: true`. Everything receives.- **Fog** fades to `fog.color` exactly, so set it to the composition background for a seamless horizon.-- **Models:** glTF materials bring base color, normal, metallic-roughness, occlusion and emissive maps; OBJ (`map_Kd`, `norm` in the `.mtl`) and FBX bring diffuse and normal maps, looked up next to the model. Textures must be PNG, JPEG, WebP, GIF, TIFF or AVIF: TGA, PSD, DDS and KTX2 are skipped with a warning (convert them). Draco / meshopt-compressed glTF files fail to load: re-export without mesh compression. Vertex colors (VOX, PLY, OFF, glTF `COLOR_0`) multiply the base color.+- **Models:** glTF materials bring base color, normal, metallic-roughness, occlusion and emissive maps; OBJ (`map_Kd`, `norm` in the `.mtl`) and FBX bring diffuse and normal maps, looked up next to the model. Textures can be PNG, JPEG, WebP, GIF, TIFF, AVIF or Basis KTX2 (ETC1S/UASTC, transcoded offline to RGBA8); TGA, PSD and DDS are skipped with a warning. Draco and meshopt-compressed glTF geometry is decoded by the asynchronous compositor loader; individual maps retain their own UV sets and texture transforms. glTF morph weights animate on the GPU before skinning. Unknown required glTF extensions fail clearly. Vertex colors (VOX, PLY, OFF, glTF `COLOR_0`) multiply the base color.- **Model orientation:** the world is y-down, so a y-up model (most glTF, OBJ and FBX files) needs `rotateX: 180` to stand upright.+ - **Planar reflective floor:** `floor: { y, color, reflectivity, roughness, fadeDistance, opacity }` renders an exact mirrored-camera reflection pass with subterranean fragment clipping, 2-pass anisotropic roughness blur ($\sigma_y \approx 3.5 \sigma_x$ along the reflection axis), Schlick Fresnel grazing angle boost, linear distance fade, and HDR radiance bloom output.- `emissive` / `emissiveIntensity` / `exposure` animate like any prop (`.keys("emissiveIntensity", …)`).- **Transparent 3D meshes:** glTF `BLEND` materials and meshes with reduced opacity draw after opaque scene items, sorted back to front by each mesh's bounds center without writing depth. This is an object-level approximation: intersecting or self-overlapping meshes cannot always sort correctly; translucent flat layers and glass use separate rendering paths.- **Glass:** `material: "glass" | "acrylic" | "frosted"` on any Layer3D primitive, extruded text (`Layer3D.extrudedText`) or model refracts whatever is behind it on screen — the video, 2D layers, particles — with a rainbow fringe (`dispersion`, default 0.04), bend strength `thickness` (px, default 80) and `ior` (1.5). `color` tints the glass (deeper toward the edges); `frosted` blurs by `roughness`. Glass needs no environment: it draws its own studio reflections, and against a dark grade it reads best — skip `Lighting.*` rigs for glass-only scenes (their environment washes glass out). Glass shows only 2D content and opaque 3D drawn before it, not other glass.