Code-first video & image, rendered natively on WebGPU.
feat(3d): cube cameras with equirectangular, fisheye and relativistic lenses
okanasla6fedc691340
diff --git a/.changeset/cube-cameras.md b/.changeset/cube-cameras.mdnew file mode 100644index 00000000..27c547a1--- /dev/null+++ b/.changeset/cube-cameras.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++Cube cameras: `lens.projection` `"equirectangular"` (360° panoramas) and `"fisheye"` (up to 360°), drawn as six mipmapped cube faces and remapped. `relativity: { speed, direction, doppler, beaming, exposure }` films a scene as seen near the speed of light: aberration, spectral Doppler shift and beaming, animatable with `relativitySpeed`. Lens options are now typed as authored, so fields with defaults may be left out.diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.mdindex e6302c9c..cd2488f8 100644--- a/nodes/node-compositor/SKILL.md+++ b/nodes/node-compositor/SKILL.md@@ -213,6 +213,7 @@ Each track represents animatable property modifications:- **Several 3D scenes:** a `box`/`flex`/`block` with a numeric `width` and `height` that holds a `camera` and the layers it films renders as its own scene (one holding only a camera stays a shot in a camera cut list): a nested composition with only its own camera, lights and environment, on its own clock (children start from the box's frame 0), placed in its parent like a media layer. Transitions, opacity and mattes on the box then blend finished pictures. Without that, every 3D layer shares one world and the active camera (the latest to start).- **3D scene capacity:** direct lights and skin joints use dynamically sized GPU storage; directional and spot lights can each cast shadows, up to the device texture-array layer limit.- **Art-directed 3D lighting:** `material: "toon"` on meshes, flat 3D cards and native 3D text uses the shared direct lights, colors, point/spot attenuation, cast shadows, environment fog and tone mapping. Optional `toon: { shadowColor, lightColor, threshold, softness, diffuse }` controls the shadow/lit albedo multipliers, terminator cosine/width and Lambert modelling (defaults: model shade color or `#80808c`, white, 0.075, 0.25, 0). Lit/physical materials retain their own shading.+- **Cube cameras:** a camera whose `lens.projection` is `"equirectangular"` or `"fisheye"`, or that has `relativity`, draws the scene as six 90° faces into a mipmapped cube (`cubeFaceCameras`, layer order +X −X +Y −Y +Z −Z; the right-handed view and left-handed cube sampling differ by a z reflection, `CUBE_FLIP`) and `Renderer3D.remapLens` turns it into the output picture. Each face gets its own floor reflection and outline pass (ink width rescaled to the output's density at the view centre); shadows are drawn once; bloom reads the remapped radiance. `relativity` aberrates every ray (`cos θ = (cos θ' − β)/(1 − β cos θ')`) and applies a spectral Doppler/beaming matrix tabulated in ln D. Depth of field throws; glass gets no 2D backdrop.- **Geometry outlines:** an environment light's optional `outlines: true | { color, width, referenceHeight, creaseAngle, depthThreshold, opacity }` runs a depth/normal ink pass before camera depth blur and bloom (defaults: `#16131c`, 1.5 px at height 1080, 45°, 0.01, 1). It reuses the linear-depth attachment's spare channels for octahedral surface normals and outline participation. Inverse-depth curvature avoids false contours on perspective planes; normal discontinuities find creases without inking texture/shadow/triangulation seams. `outline: false` on a layer/model opts out while retaining depth occlusion; `creaseAngle: 180` disables creases. Ink fades into the environment fog. 3D `anchorX/Y` pivots remain fixed under scale/rotation and leave identity placement unchanged.With MSAA enabled, retain original scene color and geometry samples and shade ink per sample before resolving color. A single-sample ink pass after the scene resolve loses subpixel coverage between outlined surfaces. Lens blur uses a separate nearest-sample geometry resolve. The scene's geometry is still drawn once.- **Transparent meshes:** glTF `BLEND` materials and meshes with reduced layer, material, tint, or vertex alpha draw after opaque items, back to front by the camera-space center of each mesh’s transformed bounds, without writing depth. This is object-level sorting: intersecting meshes, self-overlap, skinned/deformed bounds, translucent flat layers, glass, and transparent shadows have separate limitations.diff --git a/nodes/node-compositor/src/renderers/cube-camera.ts b/nodes/node-compositor/src/renderers/cube-camera.tsnew file mode 100644index 00000000..4b4aa6e6--- /dev/null+++ b/nodes/node-compositor/src/renderers/cube-camera.ts@@ -0,0 +1,90 @@+import {+ type Camera3DPose,+ type CubeLensOptions,+ cameraBasis,+ defaultCubeFaceSize,+ type LensProjection,+ type LensRemapOptions,+ lensCenterDensity,+ type RelativisticObserver,+ type Vec3,+ Vector3Math,+} from "@framefields/webgpu-renderers";++/** Largest face a cube camera picks on its own; `lens.cubeSize` may go higher. */+const DEFAULT_MAX_FACE = 4096;++/** A camera drawn as a cube map and remapped through its lens. */+export interface CubeCamera {+ faceSize: number;+ remap: LensRemapOptions;+ /** Output pixels per radian at the centre of the view. */+ centerDensity: number;+}++interface AuthoredCamera {+ lens?: {+ projection?: LensProjection;+ fov?: number;+ zoom?: number;+ cubeSize?: number;+ };+ relativity?: {+ speed?: number;+ direction?: Vec3;+ doppler?: boolean;+ beaming?: boolean;+ exposure?: number | "auto";+ };+}++/**+ * The cube camera an authored camera asks for, or null for an ordinary+ * perspective view. A camera with `relativity` is always a cube camera, even+ * at speed 0, so a shot that accelerates from rest never switches pipelines.+ */+export function resolveCubeCamera(+ authored: AuthoredCamera | undefined,+ camera: Camera3DPose,+ speed: number,+ width: number,+ height: number,+ maxTextureSize: number,+): CubeCamera | null {+ const projection = authored?.lens?.projection ?? "perspective";+ const relativity = authored?.relativity;+ if (projection === "perspective" && !relativity) return null;+ const lens: CubeLensOptions = {+ projection,+ fov: camera.fov,+ zoom: camera.zoom,+ };+ const requested = authored?.lens?.cubeSize;+ const faceSize = Math.min(+ maxTextureSize,+ requested ??+ defaultCubeFaceSize(+ lens,+ width,+ height,+ Math.min(DEFAULT_MAX_FACE, maxTextureSize),+ ),+ );+ const basis = cameraBasis(camera);+ const observer: RelativisticObserver | null = relativity+ ? {+ speed,+ direction: relativity.direction+ ? Vector3Math.normalize(relativity.direction)+ : basis.forward,+ doppler: relativity.doppler ?? true,+ beaming: relativity.beaming ?? true,+ exposure: relativity.exposure ?? "auto",+ }+ : null;+ return {+ faceSize,+ remap: { ...lens, basis, observer },+ centerDensity: lensCenterDensity(lens, width, height),+ };+}diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.tsindex 37d3a0a1..f29d95c2 100644--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts@@ -41,6 +41,7 @@ import {Camera3D,type Camera3DPose,type Color,+ cubeFaceCameras,type DrawMesh3DOpts,type DrawQuad3DOpts,type Environment3DOptions,@@ -82,6 +83,7 @@ import {import { compileTimeline, seekTimeline } from "../shared/compiler.js";import type { CompositorOperation } from "../shared/config.js";import { shouldPaintContainerBackground } from "./container-bg.js";+import { type CubeCamera, resolveCubeCamera } from "./cube-camera.js";import { compatibleMeshDraws } from "./mesh-instancing.js";import {buildLayerMatrix,@@ -1240,6 +1242,8 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {/** Where a relief keeps its look from: the default camera at the scene's field of view. */let reliefEye: [number, number, number] = [0, 0, 0];let dofEnabled = false;+ /** Set when the camera draws a cube map and remaps it through its lens. */+ let cubeCamera: CubeCamera | null = null;const sceneLights: Light3DOptions[] = [];let sceneEnvironment: Environment3DOptions | null = null;let sceneBloom: BloomOptions | null = null;@@ -1344,6 +1348,20 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {camStub?.maxBlurRadius ?? camOp?.dof?.maxBlurRadius ?? 24;}+ cubeCamera = resolveCubeCamera(+ camOp,+ camera,+ camStub?.relativitySpeed ?? camOp?.relativity?.speed ?? 0,+ nativeWidth,+ nativeHeight,+ ctx.device.limits.maxTextureDimension2D,+ );+ if (cubeCamera && dofEnabled) {+ throw new Error(+ `Camera '${camOp?.id}' is a cube camera (${camOp?.relativity ? "relativity" : camOp?.lens?.projection}) with depth of field: depth of field needs a single perspective view.`,+ );+ }+// Collect Active Lightsfor (const lvm of lightVMs) {const lop = lvm.operation as any;@@ -1680,147 +1698,235 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {...glass,];- // Planar floor reflection pass (if environment.floor is active)+ // One view of the scene: its floor reflection, then every item.const floorSpec = sceneEnvironment?.floor;- let blurredReflectionTex: GPUTexture | null = null;- if (- floorSpec &&- typeof floorSpec.y === "number" &&- camera.eye[1] < floorSpec.y- ) {- const reflectedCamera: Camera3DPose = {- ...camera,- eye: [camera.eye[0], 2 * floorSpec.y - camera.eye[1], camera.eye[2]],- target: [- camera.target[0],- 2 * floorSpec.y - camera.target[1],- camera.target[2],- ],- up: [camera.up[0], camera.up[1], camera.up[2]],- };+ const drawView = (+ view: Camera3DPose,+ width: number,+ height: number,+ colorView: GPUTextureView,+ geometryView: GPUTextureView | undefined,+ hdrTargetView: GPUTextureView | undefined,+ /** The 2D picture glass refracts; a cube face has none. */+ backdrop: GPUTexture | undefined,+ ): GPUBuffer => {+ let blurredReflectionTex: GPUTexture | null = null;+ if (+ floorSpec &&+ typeof floorSpec.y === "number" &&+ view.eye[1] < floorSpec.y+ ) {+ const reflectedCamera: Camera3DPose = {+ ...view,+ eye: [view.eye[0], 2 * floorSpec.y - view.eye[1], view.eye[2]],+ target: [+ view.target[0],+ 2 * floorSpec.y - view.target[1],+ view.target[2],+ ],+ up: [view.up[0], view.up[1], view.up[2]],+ };+ const {+ pass: reflPass,+ cameraBindGroup: reflCamBind,+ lightsBindGroup: reflLightsBind,+ reflectionTexture: reflTex,+ } = r3d.beginReflectionPass(+ encoder,+ width,+ height,+ reflectedCamera,+ sceneLights,+ { environment: sceneEnvironment, shadow: null },+ );+ for (const item of ordered) {+ if (item.kind === "quad") {+ r3d.drawTextureQuad(+ reflPass,+ reflCamBind,+ reflLightsBind,+ item.texture,+ item.opts,+ false,+ );+ } else if (item.kind === "slug") {+ r3d.drawSlugText(+ reflPass,+ reflCamBind,+ reflLightsBind,+ item.batches,+ item.opts,+ item.stackBase,+ false,+ );+ } else if (item.kind === "mesh") {+ r3d.drawMesh3D(+ reflPass,+ reflCamBind,+ reflLightsBind,+ item.meshBuffers ?? item.mesh,+ item.texture,+ item.opts,+ false,+ );+ }+ }+ reflPass.end();+ blurredReflectionTex = r3d.blurReflection(+ encoder,+ reflTex,+ floorSpec.roughness ?? 0.35,+ );+ }+const {- pass: reflPass,- cameraBindGroup: reflCamBind,- lightsBindGroup: reflLightsBind,- reflectionTexture: reflTex,- } = r3d.beginReflectionPass(+ pass: pass3d,+ cameraBindGroup,+ cameraBuffer,+ lightsBindGroup,+ } = r3d.beginPass(encoder,- nativeWidth,- nativeHeight,- reflectedCamera,+ colorView,+ width,+ height,+ view,+ "clear",+ { r: 0, g: 0, b: 0, a: 0 },+ geometryView,sceneLights,- { environment: sceneEnvironment, shadow: null },+ {+ environment: sceneEnvironment,+ shadow: shadow.length ? shadow : null,+ },+ Boolean(sceneBloom),+ hdrTargetView,);++ if (floorSpec && blurredReflectionTex) {+ r3d.drawFloor(+ pass3d,+ cameraBindGroup,+ lightsBindGroup,+ blurredReflectionTex,+ floorSpec,+ geometryEnabled,+ Boolean(sceneBloom),+ );+ }for (const item of ordered) {if (item.kind === "quad") {r3d.drawTextureQuad(- reflPass,- reflCamBind,- reflLightsBind,+ pass3d,+ cameraBindGroup,+ lightsBindGroup,item.texture,item.opts,- false,+ geometryEnabled,);} else if (item.kind === "slug") {r3d.drawSlugText(- reflPass,- reflCamBind,- reflLightsBind,+ pass3d,+ cameraBindGroup,+ lightsBindGroup,item.batches,item.opts,item.stackBase,- false,+ geometryEnabled,);} else if (item.kind === "mesh") {r3d.drawMesh3D(- reflPass,- reflCamBind,- reflLightsBind,+ pass3d,+ cameraBindGroup,+ lightsBindGroup,item.meshBuffers ?? item.mesh,item.texture,- item.opts,- false,+ isGlass(item) ? { ...item.opts, backdrop } : item.opts,+ geometryEnabled,);}}- reflPass.end();- blurredReflectionTex = r3d.blurReflection(- encoder,- reflTex,- floorSpec.roughness ?? 0.35,- );- }- const {- pass: pass3d,- cameraBindGroup,- cameraBuffer,- lightsBindGroup,- } = r3d.beginPass(- encoder,- scene3dView,- nativeWidth,- nativeHeight,- camera,- "clear",- { r: 0, g: 0, b: 0, a: 0 },- linearDepthView,- sceneLights,- {- environment: sceneEnvironment,- shadow: shadow.length ? shadow : null,- },- Boolean(sceneBloom),- );+ pass3d.end();+ return cameraBuffer;+ };- if (floorSpec && blurredReflectionTex) {- r3d.drawFloor(- pass3d,- cameraBindGroup,- lightsBindGroup,- blurredReflectionTex,- floorSpec,- geometryEnabled,- Boolean(sceneBloom),+ let final3dTex = scene3dTex;+ if (cubeCamera) {+ // Six 90° faces around the eye, then the lens's picture of them.+ const size = cubeCamera.faceSize;+ const cube = r3d.getOrCreateCubeTexture(size, false);+ const hdrCube = sceneBloom+ ? r3d.getOrCreateCubeTexture(size, true)+ : null;+ // Ink as wide on screen as without a cube: a face has size / 2+ // pixels per radian at its centre, the output centerDensity.+ const faceOutlines = outlines && {+ ...outlines,+ referenceHeight:+ ((outlines.referenceHeight ?? 1080) *+ 2 *+ cubeCamera.centerDensity) /+ nativeHeight,+ };+ const faceTex = faceOutlines+ ? ctx.renderer.getTemporaryTexture(size, size, [+ curTex,+ targetTexture,+ scene3dTex,+ ...curExcludes,+ ...itemTextures,+ ])+ : null;+ const faceGeometry = faceOutlines+ ? r3d.getOrCreateLinearDepthTexture(size, size)+ : null;+ cubeFaceCameras(camera.eye, camera.near, camera.far).forEach(+ (face, i) => {+ const faceView = r3d.cubeFaceView(cube, i);+ drawView(+ face,+ size,+ size,+ faceTex ? faceTex.createView() : faceView,+ faceGeometry?.createView(),+ hdrCube ? r3d.cubeFaceView(hdrCube, i) : undefined,+ undefined,+ );+ if (faceTex && faceGeometry && faceOutlines) {+ r3d.applyOutlines(+ encoder,+ faceTex,+ faceGeometry,+ faceView,+ face,+ faceOutlines,+ sceneEnvironment?.fog,+ );+ }+ },+ );+ r3d.remapLens(+ encoder,+ cube,+ hdrCube,+ scene3dView,+ nativeWidth,+ nativeHeight,+ cubeCamera.remap,);}- for (const item of ordered) {- if (item.kind === "quad") {- r3d.drawTextureQuad(- pass3d,- cameraBindGroup,- lightsBindGroup,- item.texture,- item.opts,- geometryEnabled,- );- } else if (item.kind === "slug") {- r3d.drawSlugText(- pass3d,- cameraBindGroup,- lightsBindGroup,- item.batches,- item.opts,- item.stackBase,- geometryEnabled,- );- } else if (item.kind === "mesh") {- r3d.drawMesh3D(- pass3d,- cameraBindGroup,- lightsBindGroup,- item.meshBuffers ?? item.mesh,- item.texture,- isGlass(item) ? { ...item.opts, backdrop: curTex } : item.opts,- geometryEnabled,+ const cameraBuffer = cubeCamera+ ? null+ : drawView(+ camera,+ nativeWidth,+ nativeHeight,+ scene3dView,+ linearDepthView,+ undefined,+ curTex,);- }- }-- pass3d.end();-- let final3dTex = scene3dTex;- if (outlines && linearDepthTex) {+ if (!cubeCamera && outlines && linearDepthTex) {if (dofEnabled) r3d.resolveGeometry(encoder, linearDepthTex);const outlinedTex = ctx.renderer.getTemporaryTexture(nativeWidth,@@ -1838,7 +1944,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {);final3dTex = outlinedTex;}- if (dofEnabled && linearDepthTex) {+ if (dofEnabled && linearDepthTex && cameraBuffer) {const dofResultTex = ctx.renderer.getTemporaryTexture(nativeWidth,nativeHeight,diff --git a/nodes/node-compositor/src/shared/compiler.ts b/nodes/node-compositor/src/shared/compiler.tsindex 98031ec1..fadf3849 100644--- a/nodes/node-compositor/src/shared/compiler.ts+++ b/nodes/node-compositor/src/shared/compiler.ts@@ -78,6 +78,7 @@ export interface LayerStub {maxBlurRadius?: number;shakeTranslation?: number;shakeRotation?: number;+ relativitySpeed?: number;orbitRadius?: number;orbitAzimuth?: number;orbitElevation?: number;@@ -718,6 +719,7 @@ function hashNodes(maxBlurRadius: lop.maxBlurRadius,shakeTranslation: lop.shakeTranslation,shakeRotation: lop.shakeRotation,+ relativitySpeed: lop.relativitySpeed,orbitRadius: lop.orbitRadius,orbitAzimuth: lop.orbitAzimuth,orbitElevation: lop.orbitElevation,@@ -906,6 +908,7 @@ export function compileTimeline(shakeRotation:lop.shakeRotation ??(lop.shake?.enabled ? (lop.shake?.rotationAmplitude ?? 0) : 0),+ relativitySpeed: lop.relativitySpeed ?? lop.relativity?.speed ?? 0,orbitRadius: lop.orbitRadius,orbitAzimuth: lop.orbitAzimuth,orbitElevation: lop.orbitElevation,diff --git a/packages/compositions/src/program/lint.ts b/packages/compositions/src/program/lint.tsindex 957b4040..2d14fbc0 100644--- a/packages/compositions/src/program/lint.ts+++ b/packages/compositions/src/program/lint.ts@@ -99,6 +99,39 @@ function solidColor(value: unknown): string | null {return c && c[3] > 0.99 ? value : null;}+type Push = (+ severity: RenderDiagnostic["severity"],+ code: string,+ message: string,+ nodeId?: string,+) => void;++/** Cube cameras (panoramic lenses, relativity) and what they cannot combine with. */+function lintCamera(node: AnyNode, id: string, push: Push): void {+ const lens = node.lens as { projection?: string; fov?: number } | undefined;+ const projection = lens?.projection ?? "perspective";+ if (projection === "perspective" && (lens?.fov ?? 0) > 160)+ push(+ "error",+ "camera_fov_range",+ `'${id}' has a ${lens?.fov}° perspective lens: perspective goes up to 160°. Use projection "fisheye" for wider views.`,+ id,+ );+ const relativity = node.relativity as { speed?: number } | undefined;+ const cube =+ projection !== "perspective" ||+ (relativity?.speed ?? 0) > 0 ||+ animates(node, ["relativitySpeed"]);+ const dof = node.dof as { enabled?: boolean } | undefined;+ if (cube && (dof?.enabled || animates(node, ["focusDistance"])))+ push(+ "error",+ "camera_cube_dof",+ `'${id}' is a cube camera (${relativity ? "relativity" : projection}) with depth of field: depth of field needs a single perspective view.`,+ id,+ );+}+export function lintComposition(config: Pick<CompositorProgramConfig, "width" | "height" | "layout"> & {backgroundColor?: string;@@ -178,7 +211,10 @@ export function lintComposition(}return;}- if (node.kind === "camera") return;+ if (node.kind === "camera") {+ lintCamera(node, id, push);+ return;+ }if (node.kind === "media") {const src = node.inputHandleId;diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.tsindex 6f061b55..fb57a2ac 100644--- a/packages/compositions/src/program/schema.ts+++ b/packages/compositions/src/program/schema.ts@@ -165,6 +165,7 @@ export const AnimatablePropSchema = z.enum(["maxBlurRadius","shakeTranslation","shakeRotation",+ "relativitySpeed","orbitRadius","orbitAzimuth","orbitElevation",@@ -298,6 +299,7 @@ export type CameraAnimatableProp =| "maxBlurRadius"| "shakeTranslation"| "shakeRotation"+ | "relativitySpeed"| "orbitRadius"| "orbitAzimuth"| "orbitElevation";@@ -381,6 +383,7 @@ export const EXTENDED_3D_ANIMATABLE_PROPS = ["maxBlurRadius","shakeTranslation","shakeRotation",+ "relativitySpeed","orbitRadius","orbitAzimuth","orbitElevation",@@ -1250,18 +1253,42 @@ export interface ShapeNode extends CompositionNodeBase {trimOffset?: number;}+/**+ * The picture a camera draws. `perspective` is an ordinary lens; the others+ * are cube cameras: the scene is drawn as six 90° faces around the eye and+ * remapped, so they can show what no single frustum holds.+ */+export const CameraProjectionSchema = z.enum([+ "perspective",+ "equirectangular",+ "fisheye",+]);+export type CameraProjection = z.infer<typeof CameraProjectionSchema>;+export const CameraLensSchema = z.object({focalLength: z.number().min(5).max(500).default(50),- fov: z.number().min(5).max(160).optional(),+ /**+ * Degrees. Perspective: vertical field of view, up to 160. Fisheye: the+ * full angle across the frame's height (equidistant), up to 360.+ * Equirectangular always shows the whole sphere and ignores it.+ */+ fov: z.number().min(5).max(360).optional(),sensorWidth: z.number().default(36),near: z.number().min(0.1).default(1),far: z.number().min(10).default(50000),zoom: z.number().min(0.01).default(1.0),+ projection: CameraProjectionSchema.default("perspective"),+ /**+ * Cube face size in pixels for cube cameras. Default: matches the+ * output's sharpness at the centre of the view, up to 4096.+ */+ cubeSize: z.number().int().min(64).max(8192).optional(),}).strict();-export type CameraLens = z.infer<typeof CameraLensSchema>;+/** A lens as authored: fields with schema defaults may be left out. */+export type CameraLens = z.input<typeof CameraLensSchema>;export const CameraDoFSchema = z.object({@@ -1289,6 +1316,36 @@ export const CameraShakeSchema = zexport type CameraShake = z.infer<typeof CameraShakeSchema>;+/**+ * The camera moves through the scene near the speed of light. What it sees+ * is physics, not grading: rays crowd towards the direction of motion+ * (aberration), colours shift blue ahead and red behind (Doppler) and ahead+ * brightens (beaming). Always drawn by a cube camera.+ */+export const CameraRelativitySchema = z+ .object({+ /** β = v / c. Animate it with the `relativitySpeed` prop. */+ speed: z.number().min(0).max(0.9999).default(0),+ /** World-space direction of motion. Default: where the camera looks. */+ direction: z.tuple([z.number(), z.number(), z.number()]).optional(),+ /** Shift colours along the spectrum. Default true. */+ doppler: z.boolean().default(true),+ /** Brighten ahead, dim behind. Default true. */+ beaming: z.boolean().default(true),+ /**+ * Exposure for the shifted light: "auto" (default) meters the centre of+ * the view, so what the camera looks at keeps its brightness at rest;+ * a number is fixed stops (0 = the true, often blinding, brightness).+ */+ exposure: z+ .union([z.literal("auto"), z.number().min(-64).max(16)])+ .default("auto"),+ })+ .strict();++/** Relativity as authored: fields with schema defaults may be left out. */+export type CameraRelativity = z.input<typeof CameraRelativitySchema>;+export interface CameraNode {id: string;kind: "camera";@@ -1305,6 +1362,7 @@ export interface CameraNode {lens?: CameraLens;dof?: CameraDoF;shake?: CameraShake;+ relativity?: CameraRelativity;startFrame?: number;durationFrames?: number;animation?: LayerAnimationSpec;@@ -1327,6 +1385,7 @@ export const CameraNodeSchema: z.ZodType<CameraNode> = zlens: CameraLensSchema.optional(),dof: CameraDoFSchema.optional(),shake: CameraShakeSchema.optional(),+ relativity: CameraRelativitySchema.optional(),startFrame: z.number().int().min(0).default(0),durationFrames: z.number().int().min(1).optional(),animation: LayerAnimationSchema.default({ tracks: [] }),diff --git a/packages/framefields/src/cube-camera-render.test.ts b/packages/framefields/src/cube-camera-render.test.tsnew file mode 100644index 00000000..dbabd00f--- /dev/null+++ b/packages/framefields/src/cube-camera-render.test.ts@@ -0,0 +1,236 @@+import { describe, expect, it } from "vitest";+import {+ CameraAnimation,+ type CameraNode,+ Composition,+ Layer,+ Layer3D,+} from "./index.js";+import { computeMse, decodePngPixels, getPixel } from "./test-utils/pixels.js";++const EYE = { x: 400, y: 300, z: 0 } as const;+const DISTANCE = 800;++/** Unlit markers in the six axis directions around the eye. */+const MARKERS = [+ { id: "front", dir: [0, 0, 1], color: "#ff0000" },+ { id: "back", dir: [0, 0, -1], color: "#00ffff" },+ { id: "right", dir: [1, 0, 0], color: "#00ff00" },+ { id: "left", dir: [-1, 0, 0], color: "#0000ff" },+ { id: "up", dir: [0, -1, 0], color: "#ffff00" },+ { id: "down", dir: [0, 1, 0], color: "#ff00ff" },+] as const;++function scene(+ width: number,+ height: number,+ camera: Partial<CameraNode>,+ markers: readonly (typeof MARKERS)[number][] = MARKERS,+) {+ const comp = new Composition({+ width,+ height,+ fps: 30,+ durationMs: 1000,+ backgroundColor: "#000000",+ });+ comp.add([+ Layer.camera({+ x: EYE.x,+ y: EYE.y,+ z: EYE.z,+ targetX: EYE.x,+ targetY: EYE.y,+ targetZ: EYE.z + 1000,+ ...camera,+ }),+ Layer.ambientLight("#ffffff", 1),+ ...markers.map((m) =>+ Layer3D.sphere({+ id: m.id,+ radius: 150,+ x: EYE.x + m.dir[0] * DISTANCE,+ y: EYE.y + m.dir[1] * DISTANCE,+ z: EYE.z + m.dir[2] * DISTANCE,+ color: m.color,+ material: "unlit",+ }),+ ),+ ]);+ return comp;+}++const near = (+ p: { r: number; g: number; b: number },+ hex: string,+ tolerance = 40,+) => {+ const v = Number.parseInt(hex.slice(1), 16);+ return (+ Math.abs(p.r - ((v >> 16) & 255)) <= tolerance &&+ Math.abs(p.g - ((v >> 8) & 255)) <= tolerance &&+ Math.abs(p.b - (v & 255)) <= tolerance+ );+};++/** Mean x of the pixels close to `hex`. */+function centroidX(+ img: Awaited<ReturnType<typeof decodePngPixels>>,+ hex: string,+): number {+ let sum = 0;+ let count = 0;+ for (let y = 0; y < img.height; y += 2) {+ for (let x = 0; x < img.width; x += 2) {+ if (near(getPixel(img, x, y), hex)) {+ sum += x;+ count++;+ }+ }+ }+ return count ? sum / count : Number.NaN;+}++describe("cube cameras", () => {+ it("draws the whole sphere around the eye as an equirectangular panorama", async () => {+ const img = await decodePngPixels(+ await scene(1024, 512, {+ lens: { projection: "equirectangular" },+ }).renderFrame({ frame: 0 }),+ );+ // Forward in the middle, right and left a quarter turn either side,+ // behind at both edges, up at the top and down at the bottom.+ expect(near(getPixel(img, 512, 256), "#ff0000")).toBe(true);+ expect(near(getPixel(img, 768, 256), "#00ff00")).toBe(true);+ expect(near(getPixel(img, 256, 256), "#0000ff")).toBe(true);+ expect(near(getPixel(img, 3, 256), "#00ffff")).toBe(true);+ expect(near(getPixel(img, 1020, 256), "#00ffff")).toBe(true);+ expect(near(getPixel(img, 512, 6), "#ffff00")).toBe(true);+ expect(near(getPixel(img, 512, 505), "#ff00ff")).toBe(true);+ // Between markers: empty sky.+ expect(near(getPixel(img, 640, 256), "#000000", 10)).toBe(true);+ });++ it("matches the ordinary camera at rest", async () => {+ const plain = scene(640, 360, { lens: { fov: 60 } }, [MARKERS[0]]);+ const cube = scene(+ 640,+ 360,+ { lens: { fov: 60 }, relativity: { speed: 0 } },+ [MARKERS[0]],+ );+ const a = await decodePngPixels(await plain.renderFrame({ frame: 0 }));+ const b = await decodePngPixels(await cube.renderFrame({ frame: 0 }));+ expect(near(getPixel(b, 320, 180), "#ff0000")).toBe(true);+ // Resampling softens the sphere's edge by a pixel at most.+ expect(computeMse(a.pixels, b.pixels)).toBeLessThan(40);+ });++ it("aberrates the view towards the direction of motion", async () => {+ // A marker 70° to the right: at rest, outside a 100° view even counting+ // its 10.8° angular radius.+ const angle = (70 * Math.PI) / 180;+ const side = {+ id: "side",+ dir: [Math.sin(angle), 0, Math.cos(angle)],+ color: "#00ff00",+ } as unknown as (typeof MARKERS)[number];+ const render = (speed: number) =>+ scene(+ 600,+ 600,+ {+ lens: { fov: 100 },+ relativity: { speed, doppler: false, beaming: false },+ },+ [side],+ ).renderFrame({ frame: 0 });+ const rest = await decodePngPixels(await render(0));+ expect(Number.isNaN(centroidX(rest, "#00ff00"))).toBe(true);+ // At β = 0.8 it is seen at cos θ' = (cos θ + β) / (1 + β cos θ) ≈ 26.3°.+ const beta = 0.8;+ const seen = Math.acos(+ (Math.cos(angle) + beta) / (1 + beta * Math.cos(angle)),+ );+ const fast = await decodePngPixels(await render(beta));+ const expected =+ 300 + (300 * Math.tan(seen)) / Math.tan((50 * Math.PI) / 180);+ expect(centroidX(fast, "#00ff00")).toBeCloseTo(expected, -1);+ });++ it("shifts light blue ahead and red behind", async () => {+ const grey = [+ { id: "ahead", dir: [0, 0, 1], color: "#808080" },+ { id: "behind", dir: [0, 0, -1], color: "#808080" },+ ] as unknown as (typeof MARKERS)[number][];+ const render = (relativity: CameraNode["relativity"]) =>+ scene(+ 1024,+ 512,+ { lens: { projection: "equirectangular" }, relativity },+ grey,+ ).renderFrame({ frame: 0 });+ const sum = (p: { r: number; g: number; b: number }) => p.r + p.g + p.b;+ const hue = await decodePngPixels(+ await render({ speed: 0.3, beaming: false, exposure: 0 }),+ );+ const ahead = getPixel(hue, 512, 256);+ const behind = getPixel(hue, 2, 256);+ expect(ahead.b).toBeGreaterThan(ahead.r + 20);+ expect(behind.r).toBeGreaterThan(behind.b + 20);+ // True brightness: beaming brightens ahead and dims behind.+ const beamed = await decodePngPixels(+ await render({ speed: 0.3, exposure: 0 }),+ );+ expect(sum(getPixel(beamed, 512, 256))).toBeGreaterThan(sum(ahead));+ expect(sum(getPixel(beamed, 2, 256))).toBeLessThan(sum(behind));+ // Auto exposure meters the centre: ahead keeps its brightness at rest.+ const metered = await decodePngPixels(await render({ speed: 0.3 }));+ const luma = (p: { r: number; g: number; b: number }) =>+ 0.2126 * p.r + 0.7152 * p.g + 0.0722 * p.b;+ expect(luma(getPixel(metered, 512, 256))).toBeCloseTo(128, -1.3);+ });++ it("animates the speed with relativitySpeed", async () => {+ const angle = (70 * Math.PI) / 180;+ const side = {+ id: "side",+ dir: [Math.sin(angle), 0, Math.cos(angle)],+ color: "#00ff00",+ } as unknown as (typeof MARKERS)[number];+ const comp = scene(+ 600,+ 600,+ {+ lens: { fov: 100 },+ relativity: { doppler: false, beaming: false },+ animation: CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.8, {+ start: 0,+ end: 10,+ ease: "none",+ }),+ },+ [side],+ );+ const rest = await decodePngPixels(await comp.renderFrame({ frame: 0 }));+ const fast = await decodePngPixels(await comp.renderFrame({ frame: 10 }));+ expect(Number.isNaN(centroidX(rest, "#00ff00"))).toBe(true);+ expect(centroidX(fast, "#00ff00")).toBeGreaterThan(300);+ });++ it("sees behind the camera with a 360° fisheye", async () => {+ const img = await decodePngPixels(+ await scene(600, 600, {+ lens: { projection: "fisheye", fov: 360 },+ }).renderFrame({ frame: 0 }),+ );+ expect(near(getPixel(img, 300, 300), "#ff0000")).toBe(true);+ // Behind the eye is the rim of the circle; outside it, nothing.+ expect(near(getPixel(img, 300, 4), "#00ffff")).toBe(true);+ expect(near(getPixel(img, 300, 596), "#00ffff")).toBe(true);+ expect(getPixel(img, 10, 10).a).toBe(255);+ expect(near(getPixel(img, 10, 10), "#000000", 10)).toBe(true);+ // Right, at 90°, is halfway out.+ expect(near(getPixel(img, 450, 300), "#00ff00")).toBe(true);+ });+});diff --git a/packages/webgpu-renderers/src/math3d/cube-lens.test.ts b/packages/webgpu-renderers/src/math3d/cube-lens.test.tsnew file mode 100644index 00000000..5d5e3746--- /dev/null+++ b/packages/webgpu-renderers/src/math3d/cube-lens.test.ts@@ -0,0 +1,96 @@+import { describe, expect, it } from "vitest";+import { Camera3D } from "./camera3d.js";+import {+ cameraBasis,+ cubeFaceCameras,+ cubeTexel,+ defaultCubeFaceSize,+} from "./cube-lens.js";+import type { Vec3 } from "./vector3.js";++/** Clip-space position of world point `p` (column-major matrix). */+function project(m: number[], p: Vec3): [number, number, number, number] {+ const out: [number, number, number, number] = [0, 0, 0, 0];+ for (let r = 0; r < 4; r++) {+ out[r] = m[r] * p[0] + m[4 + r] * p[1] + m[8 + r] * p[2] + m[12 + r];+ }+ return out;+}++describe("cube-map cameras", () => {+ const eye: Vec3 = [120, -40, 300];+ const faces = cubeFaceCameras(eye, 1, 10000);++ it("draws each face where WebGPU samples it", () => {+ let checked = 0;+ for (let i = 0; i < 6; i++) {+ const viewProj = Camera3D.computeViewProjectionMatrix(faces[i]);+ for (const x of [-0.9, -0.3, 0, 0.45, 0.8]) {+ for (const y of [-0.85, -0.2, 0, 0.5, 0.95]) {+ // A world ray through this face's NDC (x, y).+ const { right, up, forward } = cameraBasis(faces[i]);+ const dir: Vec3 = [0, 1, 2].map(+ (k) => forward[k] + x * right[k] + y * up[k],+ ) as Vec3;+ const point: Vec3 = [0, 1, 2].map(+ (k) => eye[k] + dir[k] * 500,+ ) as Vec3;+ const clip = project(viewProj, point);+ const ndcX = clip[0] / clip[3];+ const ndcY = clip[1] / clip[3];+ expect(ndcX).toBeCloseTo(x, 5);+ expect(ndcY).toBeCloseTo(y, 5);+ const texel = cubeTexel(dir);+ expect(texel.face).toBe(i);+ expect(texel.u).toBeCloseTo((ndcX + 1) / 2, 5);+ expect(texel.v).toBeCloseTo((1 - ndcY) / 2, 5);+ checked++;+ }+ }+ }+ expect(checked).toBe(150);+ });++ it("covers every direction with square 90° faces", () => {+ for (const face of faces) {+ expect(face.fov).toBe(90);+ expect(face.aspect).toBe(1);+ }+ const forwards = faces.map((f) => cameraBasis(f).forward);+ for (let a = 0; a < 6; a++) {+ for (let b = a + 1; b < 6; b++) {+ const dot = forwards[a].reduce((s, v, k) => s + v * forwards[b][k], 0);+ expect([0, -1]).toContain(Math.round(dot));+ }+ }+ });++ it("sizes faces to the lens's density at the centre", () => {+ const max = 4096;+ // 1080p height, 90° vertical: 540 px per unit tangent → 1080-wide faces.+ expect(+ defaultCubeFaceSize(+ { projection: "perspective", fov: 90, zoom: 1 },+ 1920,+ 1080,+ max,+ ),+ ).toBe(2048);+ expect(+ defaultCubeFaceSize(+ { projection: "equirectangular", fov: 90, zoom: 1 },+ 2048,+ 1024,+ max,+ ),+ ).toBe(1024);+ expect(+ defaultCubeFaceSize(+ { projection: "perspective", fov: 10, zoom: 1 },+ 1920,+ 1080,+ max,+ ),+ ).toBe(max);+ });+});diff --git a/packages/webgpu-renderers/src/math3d/cube-lens.ts b/packages/webgpu-renderers/src/math3d/cube-lens.tsnew file mode 100644index 00000000..dae327e1--- /dev/null+++ b/packages/webgpu-renderers/src/math3d/cube-lens.ts@@ -0,0 +1,130 @@+/**+ * Cube-map cameras: a scene is drawn as six 90° faces around the eye, then+ * remapped into any lens, including ones no single perspective frustum can+ * hold (a full panorama, a fisheye past 180°, a relativistic observer).+ *+ * The engine's view matrix (`lookAt`) is right-handed while cube sampling is+ * left-handed, so every lookup direction goes through `CUBE_FLIP`, a+ * reflection of z. With it, the face drawn by `cubeFaceCameras()[i]` is exactly+ * what WebGPU returns for layer `i` of a cube texture.+ */++import type { Camera3DPose } from "./camera3d.js";+import { Matrix4Math } from "./matrix4.js";+import { type Vec3, Vector3Math } from "./vector3.js";++/** The picture a cube camera's lens draws. */+export type LensProjection = "perspective" | "equirectangular" | "fisheye";++/** The six faces in WebGPU layer order (+X, −X, +Y, −Y, +Z, −Z), in cube space. */+const CUBE_FACES: readonly { major: Vec3; s: Vec3; t: Vec3 }[] = [+ { major: [1, 0, 0], s: [0, 0, -1], t: [0, -1, 0] },+ { major: [-1, 0, 0], s: [0, 0, 1], t: [0, -1, 0] },+ { major: [0, 1, 0], s: [1, 0, 0], t: [0, 0, 1] },+ { major: [0, -1, 0], s: [1, 0, 0], t: [0, 0, -1] },+ { major: [0, 0, 1], s: [1, 0, 0], t: [0, -1, 0] },+ { major: [0, 0, -1], s: [-1, 0, 0], t: [0, -1, 0] },+];++/** World direction ↔ cube-space direction (a reflection, its own inverse). */+export const CUBE_FLIP = (v: Vec3): Vec3 => [v[0], v[1], -v[2]];++/**+ * The six cameras that draw a cube map around `eye`: 90° square frusta in+ * layer order. Their faces meet edge to edge, so the cube covers every+ * direction once.+ */+export function cubeFaceCameras(+ eye: Vec3,+ near: number,+ far: number,+): Camera3DPose[] {+ return CUBE_FACES.map(({ major, t }) => ({+ eye: [...eye] as Vec3,+ target: Vector3Math.add(eye, CUBE_FLIP(major)),+ up: CUBE_FLIP([-t[0], -t[1], -t[2]]),+ fov: 90,+ aspect: 1,+ near,+ far,+ zoom: 1,+ }));+}++/**+ * Where WebGPU samples a cube for world direction `dir`: the layer and its+ * (u, v), v down. The selection rule of the WebGPU (Vulkan) spec.+ */+export function cubeTexel(dir: Vec3): { face: number; u: number; v: number } {+ const c = CUBE_FLIP(dir);+ const ax = Math.abs(c[0]);+ const ay = Math.abs(c[1]);+ const az = Math.abs(c[2]);+ const axis = ax >= ay && ax >= az ? 0 : ay >= az ? 1 : 2;+ const face = axis * 2 + (c[axis] >= 0 ? 0 : 1);+ const { s, t } = CUBE_FACES[face];+ const ma = Math.abs(c[axis]);+ const sc = Vector3Math.dot(s, c);+ const tc = Vector3Math.dot(t, c);+ return { face, u: (sc / ma + 1) / 2, v: (tc / ma + 1) / 2 };+}++/** A camera's orthonormal frame: screen right, screen up and forward. */+export function cameraBasis(camera: Camera3DPose): {+ right: Vec3;+ up: Vec3;+ forward: Vec3;+} {+ const view = Matrix4Math.lookAt(camera.eye, camera.target, camera.up);+ return {+ right: [view[0], view[4], view[8]],+ up: [view[1], view[5], view[9]],+ forward: [view[2], view[6], view[10]],+ };+}++/** What a lens needs to turn an output pixel into a ray. */+export interface CubeLensOptions {+ projection: LensProjection;+ /**+ * Perspective: vertical field of view. Fisheye: the full angle across the+ * frame's height (equidistant, up to 360°). Ignored by equirectangular,+ * which always shows the whole sphere.+ */+ fov: number;+ /** Optical zoom (perspective and fisheye). */+ zoom: number;+}++/** Output pixels per radian at the centre of the view. */+export function lensCenterDensity(+ lens: CubeLensOptions,+ width: number,+ height: number,+): number {+ const half = (lens.fov * Math.PI) / 360;+ switch (lens.projection) {+ case "perspective":+ return ((height / 2) * lens.zoom) / Math.tan(half);+ case "fisheye":+ return ((height / 2) * lens.zoom) / half;+ case "equirectangular":+ return width / (2 * Math.PI);+ }+}++/**+ * Face size that matches the lens's pixel density at the centre of the view:+ * a face holds `size / 2` pixels per radian at its centre. Rounded up to a+ * power of two (so a moving lens reuses its targets) and clamped.+ */+export function defaultCubeFaceSize(+ lens: CubeLensOptions,+ width: number,+ height: number,+ maxSize: number,+): number {+ const wanted = 2 * lensCenterDensity(lens, width, height);+ const pow2 = 2 ** Math.ceil(Math.log2(Math.max(1, wanted)));+ return Math.min(maxSize, Math.max(64, pow2));+}diff --git a/packages/webgpu-renderers/src/math3d/index.ts b/packages/webgpu-renderers/src/math3d/index.tsindex ac4aafbc..d9157005 100644--- a/packages/webgpu-renderers/src/math3d/index.ts+++ b/packages/webgpu-renderers/src/math3d/index.ts@@ -1,6 +1,8 @@export * from "./camera3d.js";+export * from "./cube-lens.js";export * from "./environment3d.js";export * from "./light3d.js";export * from "./matrix4.js";export * from "./path3d.js";+export * from "./relativistic-doppler.js";export * from "./vector3.js";diff --git a/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.ts b/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.tsnew file mode 100644index 00000000..89df9b9d--- /dev/null+++ b/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.ts@@ -0,0 +1,64 @@+import { describe, expect, it } from "vitest";+import {+ aberrate,+ DOPPLER_LUT_SIZE,+ dopplerColorMatrix,+ dopplerLut,+ lorentzFactor,+} from "./relativistic-doppler.js";++const apply = (m: number[][], c: number[]) =>+ m.map((row) => row[0] * c[0] + row[1] * c[1] + row[2] * c[2]);++describe("relativistic observer", () => {+ it("aberrates rays towards the direction of motion", () => {+ // A ray seen at cos θ' = β left the scene at 90°.+ expect(aberrate(0.8, 0.8).sourceCos).toBeCloseTo(0, 10);+ // Straight ahead and behind are fixed points.+ expect(aberrate(1, 0.9).sourceCos).toBeCloseTo(1, 10);+ expect(aberrate(-1, 0.9).sourceCos).toBeCloseTo(-1, 10);+ // D ahead is √((1 + β) / (1 − β)).+ expect(aberrate(1, 0.8).doppler).toBeCloseTo(3, 10);+ expect(aberrate(-1, 0.8).doppler).toBeCloseTo(1 / 3, 10);+ expect(lorentzFactor(0.99)).toBeCloseTo(7.0888, 3);+ });++ it("leaves colour untouched at rest", () => {+ for (const beaming of [true, false]) {+ const m = dopplerColorMatrix(1, beaming);+ for (let r = 0; r < 3; r++) {+ for (let c = 0; c < 3; c++) {+ expect(m[r][c]).toBeCloseTo(r === c ? 1 : 0, 6);+ }+ }+ }+ });++ it("shifts ahead to blue and brighter, behind to red and darker", () => {+ const grey = [0.5, 0.5, 0.5];+ const ahead = apply(dopplerColorMatrix(1.3), grey);+ expect(ahead[2]).toBeGreaterThan(ahead[0]);+ expect(ahead[0] + ahead[1] + ahead[2]).toBeGreaterThan(1.5);+ const behind = apply(dopplerColorMatrix(1 / 1.3), grey);+ expect(behind[0]).toBeGreaterThan(behind[2]);+ expect(behind[0] + behind[1] + behind[2]).toBeLessThan(1.5);+ // Without beaming only the hue moves; brightness stays near rest.+ const shiftOnly = apply(dopplerColorMatrix(1.3, false), grey);+ expect(shiftOnly[0] + shiftOnly[1] + shiftOnly[2]).toBeLessThan(+ ahead[0] + ahead[1] + ahead[2],+ );+ });++ it("tabulates the matrix in ln D, identity in the middle", () => {+ const lut = dopplerLut(true);+ expect(lut).toHaveLength(DOPPLER_LUT_SIZE * 12);+ expect(lut.every(Number.isFinite)).toBe(true);+ // An even table has no entry at exactly D = 1; its two middle entries+ // straddle it, so their average is nearly identity.+ const mid = DOPPLER_LUT_SIZE / 2;+ const diag = (n: number, r: number) => lut[n * 12 + r * 5];+ for (let r = 0; r < 3; r++) {+ expect((diag(mid - 1, r) + diag(mid, r)) / 2).toBeCloseTo(1, 1);+ }+ });+});diff --git a/packages/webgpu-renderers/src/math3d/relativistic-doppler.ts b/packages/webgpu-renderers/src/math3d/relativistic-doppler.tsnew file mode 100644index 00000000..1812bb6d--- /dev/null+++ b/packages/webgpu-renderers/src/math3d/relativistic-doppler.ts@@ -0,0 +1,208 @@+/**+ * Relativistic observer: what a camera moving at a fraction `β` of the speed+ * of light sees of a scene at rest.+ *+ * Aberration: a ray arriving at angle θ' from the direction of motion left+ * the scene at θ, with cos θ = (cos θ' − β) / (1 − β cos θ'). The Doppler+ * factor D = γ (1 + β cos θ') scales every frequency, and spectral radiance+ * transforms as L'(λ') = D⁵ L(D λ') (the "searchlight": ahead is brighter).+ *+ * A pixel only stores a colour, so its spectrum is modelled: the colour is a+ * reflectance (three smooth bands that sum to one, held flat past the visible+ * range, so red surfaces stay red into the infrared) lit by daylight (a+ * 5778 K black body). Shifting that spectrum and integrating it against the+ * CIE 1931 observer gives the colour seen. `dopplerColorMatrix(D)` is the+ * resulting linear map on linear sRGB, exactly the identity at D = 1.+ */++/** Wavelength range integrated by the observer, nm. */+const LAMBDA_MIN = 360;+const LAMBDA_MAX = 830;+const LAMBDA_STEP = 1;+/** Daylight: the Sun's effective temperature, K. */+const DAYLIGHT_KELVIN = 5778;+/** Second radiation constant hc/k in nm·K. */+const C2 = 1.4387769e7;++/** Piecewise Gaussian used by the CIE fit. */+function lobe(l: number, mu: number, below: number, above: number): number {+ const t = (l - mu) / (l < mu ? below : above);+ return Math.exp(-0.5 * t * t);+}++/**+ * CIE 1931 2° colour matching functions, multi-lobe fit of Wyman, Sloan and+ * Shirley (JCGT 2013).+ */+function cieXyz(l: number): [number, number, number] {+ return [+ 1.056 * lobe(l, 599.8, 37.9, 31.0) ++ 0.362 * lobe(l, 442.0, 16.0, 26.7) -+ 0.065 * lobe(l, 501.1, 20.4, 26.2),+ 0.821 * lobe(l, 568.8, 46.9, 40.5) + 0.286 * lobe(l, 530.9, 16.3, 31.1),+ 1.217 * lobe(l, 437.0, 11.8, 36.0) + 0.681 * lobe(l, 459.0, 26.0, 13.8),+ ];+}++/** XYZ → linear sRGB (D65). */+const XYZ_TO_RGB = [+ [3.2406, -1.5372, -0.4986],+ [-0.9689, 1.8758, 0.0415],+ [0.0557, -0.204, 1.057],+] as const;++function smoothstep(a: number, b: number, x: number): number {+ const t = Math.min(1, Math.max(0, (x - a) / (b - a)));+ return t * t * (3 - 2 * t);+}++/** The blue, green and red reflectance bands at wavelength `l` (sum to 1). */+function bands(l: number): [number, number, number] {+ const blueToGreen = smoothstep(440, 510, l);+ const greenToRed = smoothstep(570, 630, l);+ return [1 - blueToGreen, blueToGreen * (1 - greenToRed), greenToRed];+}++/** Black-body spectral radiance shape at wavelength `l` nm (unnormalised). */+function planck(l: number, kelvin: number): number {+ const x = C2 / (l * kelvin);+ // Far in the ultraviolet the exponential overflows: there is no light.+ if (x > 700) return 0;+ return 1 / (l ** 5 * Math.expm1(x));+}++/**+ * K[k][i]: channel k of linear sRGB seen from band i at Doppler factor `d`,+ * with the D⁵ searchlight factor when `beaming`.+ */+function bandResponse(d: number, beaming: boolean): number[][] {+ const k = [+ [0, 0, 0],+ [0, 0, 0],+ [0, 0, 0],+ ];+ const scale = beaming ? d ** 5 : 1;+ for (let l = LAMBDA_MIN; l <= LAMBDA_MAX; l += LAMBDA_STEP) {+ const xyz = cieXyz(l);+ const source = l * d;+ const b = bands(source);+ const e = planck(source, DAYLIGHT_KELVIN) * scale;+ for (let c = 0; c < 3; c++) {+ const bar =+ XYZ_TO_RGB[c][0] * xyz[0] ++ XYZ_TO_RGB[c][1] * xyz[1] ++ XYZ_TO_RGB[c][2] * xyz[2];+ for (let i = 0; i < 3; i++) k[c][i] += bar * b[i] * e;+ }+ }+ return k;+}++function invert3(m: number[][]): number[][] {+ const [[a, b, c], [d, e, f], [g, h, i]] = m;+ const A = e * i - f * h;+ const B = -(d * i - f * g);+ const C = d * h - e * g;+ const det = a * A + b * B + c * C;+ if (Math.abs(det) < 1e-12) throw new Error("Singular colour basis");+ const s = 1 / det;+ return [+ [A * s, -(b * i - c * h) * s, (b * f - c * e) * s],+ [B * s, (a * i - c * g) * s, -(a * f - c * d) * s],+ [C * s, -(a * h - b * g) * s, (a * e - b * d) * s],+ ];+}++function multiply3(a: number[][], b: number[][]): number[][] {+ return a.map((row) =>+ [0, 1, 2].map(+ (j) => row[0] * b[0][j] + row[1] * b[1][j] + row[2] * b[2][j],+ ),+ );+}++let restFrame: { white: number[]; toBands: number[][] } | null = null;++/** The rest-frame response, white-balanced so white reflectance is white. */+function rest(): { white: number[]; toBands: number[][] } {+ if (restFrame) return restFrame;+ const k = bandResponse(1, false);+ const white = k.map((row) => row[0] + row[1] + row[2]);+ const balanced = k.map((row, c) => row.map((v) => v / white[c]));+ restFrame = { white, toBands: invert3(balanced) };+ return restFrame;+}++/**+ * The linear-sRGB matrix (row-major 3×3) that turns a colour at rest into the+ * colour seen at Doppler factor `d` (> 1 towards the direction of motion).+ */+export function dopplerColorMatrix(d: number, beaming = true): number[][] {+ const { white, toBands } = rest();+ const k = bandResponse(d, beaming).map((row, c) =>+ row.map((v) => v / white[c]),+ );+ return multiply3(k, toBands);+}++/** Largest Doppler factor the lookup table covers (β ≈ 0.99995). */+export const DOPPLER_LUT_MAX = 200;+/** Entries in the lookup table, evenly spaced in ln D. */+export const DOPPLER_LUT_SIZE = 256;++const luts = new Map<boolean, Float32Array>();++/**+ * `dopplerColorMatrix` sampled at DOPPLER_LUT_SIZE factors evenly spaced in+ * ln D over [1/DOPPLER_LUT_MAX, DOPPLER_LUT_MAX]: three vec4 rows per entry.+ */+export function dopplerLut(beaming: boolean): Float32Array {+ const cached = luts.get(beaming);+ if (cached) return cached;+ const data = new Float32Array(DOPPLER_LUT_SIZE * 12);+ const span = Math.log(DOPPLER_LUT_MAX);+ for (let n = 0; n < DOPPLER_LUT_SIZE; n++) {+ const lnD = -span + (2 * span * n) / (DOPPLER_LUT_SIZE - 1);+ const m = dopplerColorMatrix(Math.exp(lnD), beaming);+ for (let r = 0; r < 3; r++) data.set(m[r], n * 12 + r * 4);+ }+ luts.set(beaming, data);+ return data;+}++/** Rec. 709 luminance of a linear-sRGB colour. */+const luminance = (c: number[]) =>+ 0.2126 * c[0] + 0.7152 * c[1] + 0.0722 * c[2];++/**+ * How much brighter a neutral surface looks at Doppler factor `d`: the gain+ * an exposure meter would compensate for.+ */+export function dopplerLuminanceGain(+ d: number,+ doppler: boolean,+ beaming: boolean,+): number {+ if (!doppler) return beaming ? d ** 4 : 1;+ const m = dopplerColorMatrix(d, beaming);+ return luminance(m.map((row) => row[0] + row[1] + row[2]));+}++/** Lorentz factor of speed β. */+export function lorentzFactor(beta: number): number {+ return 1 / Math.sqrt(1 - beta * beta);+}++/**+ * Where a ray seen at angle `observedCos` (cosine to the direction of motion)+ * left the scene at rest, and its Doppler factor.+ */+export function aberrate(+ observedCos: number,+ beta: number,+): { sourceCos: number; doppler: number } {+ return {+ sourceCos: (observedCos - beta) / (1 - beta * observedCos),+ doppler: lorentzFactor(beta) * (1 + beta * observedCos),+ };+}diff --git a/packages/webgpu-renderers/src/renderer3d/index.ts b/packages/webgpu-renderers/src/renderer3d/index.tsindex 041d021c..b7fafa41 100644--- a/packages/webgpu-renderers/src/renderer3d/index.ts+++ b/packages/webgpu-renderers/src/renderer3d/index.ts@@ -2,6 +2,7 @@ export * from "./audio-mesh-deform-pipeline.js";export * from "./bloom-pipeline.js";export * from "./floor-reflection-pipeline.js";export * from "./glass-pipeline.js";+export * from "./lens-remap-pipeline.js";export * from "./matte3d.js";export * from "./mesh3d-pipeline.js";export * from "./motion-blur-pipeline.js";diff --git a/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.tsnew file mode 100644index 00000000..dba74336--- /dev/null+++ b/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.ts@@ -0,0 +1,586 @@+/**+ * Lens remap: turns a cube map drawn around the eye (`cubeFaceCameras`) into+ * the picture of a lens. Every output pixel becomes a ray (perspective,+ * equirectangular or equidistant fisheye), optionally seen by an observer+ * moving near the speed of light: the ray is aberrated back to the direction+ * it left the scene, and its colour Doppler-shifted (`relativistic-doppler`).+ *+ * The cube is mipmapped first, so a lens or a fast observer that squeezes+ * much of the scene into few pixels filters it instead of aliasing.+ */++import type { LensProjection } from "../math3d/cube-lens.js";+import {+ DOPPLER_LUT_MAX,+ DOPPLER_LUT_SIZE,+ dopplerLuminanceGain,+ dopplerLut,+ lorentzFactor,+} from "../math3d/relativistic-doppler.js";+import type { Vec3 } from "../math3d/vector3.js";+import { BufferPool } from "../renderer2d/buffer-pool.js";++/** An observer moving at `speed` (a fraction of c) through a scene at rest. */+export interface RelativisticObserver {+ /** β = v / c, 0 ≤ β < 1. */+ speed: number;+ /** Unit direction of motion in world space. */+ direction: Vec3;+ /** Shift colours along the spectrum (blue ahead, red behind). */+ doppler: boolean;+ /** Brighten ahead and dim behind (relativistic beaming). */+ beaming: boolean;+ /**+ * Exposure for the shifted light: stops, or "auto" to meter the centre of+ * the view, so what the camera looks at keeps its brightness at rest.+ */+ exposure: number | "auto";+}++export interface LensRemapOptions {+ projection: LensProjection;+ /** Degrees; see `CubeLensOptions.fov`. */+ fov: number;+ zoom: number;+ /** The camera's frame: screen right, screen up, forward (world space). */+ basis: { right: Vec3; up: Vec3; forward: Vec3 };+ observer?: RelativisticObserver | null;+}++const PROJECTION_INDEX: Record<LensProjection, number> = {+ perspective: 0,+ equirectangular: 1,+ fisheye: 2,+};++/** How the observer changes colour: none, spectral table, or brightness only. */+const ColorMode = { None: 0, Spectral: 1, Beaming: 2 } as const;++const mipWgsl = /* wgsl */ `+@group(0) @binding(0) var samp : sampler;+@group(0) @binding(1) var src : texture_2d<f32>;++struct VsOut {+ @builtin(position) pos : vec4<f32>,+ @location(0) uv : vec2<f32>,+};++@vertex+fn vs(@builtin(vertex_index) i : u32) -> VsOut {+ let p = vec2<f32>(f32((i << 1u) & 2u), f32(i & 2u));+ var o : VsOut;+ o.pos = vec4<f32>(p * 2.0 - 1.0, 0.0, 1.0);+ o.uv = vec2<f32>(p.x, 1.0 - p.y);+ return o;+}++// One bilinear tap between four texels of the level above: a 2x2 box.+@fragment+fn fs(in : VsOut) -> @location(0) vec4<f32> {+ return textureSampleLevel(src, samp, in.uv, 0.0);+}+`;++const remapWgsl = /* wgsl */ `+struct Lens {+ right : vec4<f32>, // xyz screen right, w projection (0 persp, 1 equirect, 2 fisheye)+ up : vec4<f32>, // xyz screen up, w scale (persp: tan(fov/2)/zoom, fisheye: fov/2/zoom)+ forward : vec4<f32>, // xyz forward, w aspect (width / height)+ motion : vec4<f32>, // xyz direction of motion, w β+ params : vec4<f32>, // x color mode, y exposure gain, z γ, w ln(max D)+};++@group(0) @binding(0) var<uniform> lens : Lens;+@group(0) @binding(1) var samp : sampler;+@group(0) @binding(2) var colorCube : texture_cube<f32>;+@group(0) @binding(3) var hdrCube : texture_cube<f32>;+@group(0) @binding(4) var<storage, read> dopplerLut : array<vec4<f32>>;++const PI = 3.14159265358979;+const LUT_SIZE = ${DOPPLER_LUT_SIZE}.0;++struct VsOut {+ @builtin(position) pos : vec4<f32>,+ @location(0) ndc : vec2<f32>,+};++@vertex+fn vs(@builtin(vertex_index) i : u32) -> VsOut {+ let p = vec2<f32>(f32((i << 1u) & 2u), f32(i & 2u));+ var o : VsOut;+ o.pos = vec4<f32>(p * 2.0 - 1.0, 0.0, 1.0);+ o.ndc = p * 2.0 - 1.0;+ return o;+}++struct Ray {+ dir : vec3<f32>,+ inside : f32,+};++// The ray an output pixel sees, in world space.+fn lensRay(ndc : vec2<f32>) -> Ray {+ let r = lens.right.xyz;+ let u = lens.up.xyz;+ let f = lens.forward.xyz;+ let scale = lens.up.w;+ let aspect = lens.forward.w;+ let projection = lens.right.w;+ var out : Ray;+ out.inside = 1.0;+ if (projection < 0.5) {+ out.dir = normalize(f + ndc.x * aspect * scale * r + ndc.y * scale * u);+ } else if (projection < 1.5) {+ let lon = ndc.x * PI;+ let lat = ndc.y * 0.5 * PI;+ out.dir = cos(lat) * (sin(lon) * r + cos(lon) * f) + sin(lat) * u;+ } else {+ let q = vec2<f32>(ndc.x * aspect, ndc.y);+ let rho = length(q);+ let theta = rho * scale;+ let side = select(vec2<f32>(0.0), q / rho, rho > 1e-6);+ out.dir = cos(theta) * f + sin(theta) * (side.x * r + side.y * u);+ out.inside = select(0.0, 1.0, theta <= PI);+ }+ return out;+}++struct Seen {+ dir : vec3<f32>,+ doppler : f32,+};++// Aberration: where a ray seen by the moving observer left the scene.+fn aberrate(observed : vec3<f32>) -> Seen {+ let beta = lens.motion.w;+ let n = lens.motion.xyz;+ let c = clamp(dot(observed, n), -1.0, 1.0);+ let cs = (c - beta) / (1.0 - beta * c);+ let side = observed - c * n;+ let len = length(side);+ // Exactly ahead or behind, any perpendicular will do: sin θ is 0.+ let fallback = normalize(cross(n, select(vec3<f32>(1.0, 0.0, 0.0), vec3<f32>(0.0, 1.0, 0.0), abs(n.x) > 0.9)));+ let perp = select(fallback, side / len, len > 1e-6);+ var out : Seen;+ out.dir = cs * n + sqrt(max(0.0, 1.0 - cs * cs)) * perp;+ out.doppler = lens.params.z * (1.0 + beta * c);+ return out;+}++// Linear-sRGB matrix for Doppler factor d, interpolated in ln d.+fn dopplerMatrix(d : f32) -> mat3x3<f32> {+ let span = lens.params.w;+ let t = clamp((log(d) + span) / (2.0 * span), 0.0, 1.0) * (LUT_SIZE - 1.0);+ let i0 = u32(floor(t));+ let i1 = min(i0 + 1u, u32(LUT_SIZE) - 1u);+ let w = t - f32(i0);+ let r0 = mix(dopplerLut[i0 * 3u], dopplerLut[i1 * 3u], w).xyz;+ let r1 = mix(dopplerLut[i0 * 3u + 1u], dopplerLut[i1 * 3u + 1u], w).xyz;+ let r2 = mix(dopplerLut[i0 * 3u + 2u], dopplerLut[i1 * 3u + 2u], w).xyz;+ // Rows in the table; WGSL matrices take columns.+ return transpose(mat3x3<f32>(r0, r1, r2));+}++// The observer's light from scene light \`c\` (linear rgb) at Doppler factor d.+fn shiftLight(c : vec3<f32>, d : f32) -> vec3<f32> {+ let mode = lens.params.x;+ var out = c;+ if (mode > 1.5) {+ out = c * pow(d, 4.0);+ } else if (mode > 0.5) {+ out = dopplerMatrix(d) * c;+ }+ return max(out * lens.params.y, vec3<f32>(0.0));+}++fn srgbToLinear(c : vec3<f32>) -> vec3<f32> {+ let lo = c / 12.92;+ let hi = pow((max(c, vec3<f32>(0.0)) + 0.055) / 1.055, vec3<f32>(2.4));+ return select(hi, lo, c <= vec3<f32>(0.04045));+}++fn linearToSrgb(c : vec3<f32>) -> vec3<f32> {+ let x = clamp(c, vec3<f32>(0.0), vec3<f32>(1.0));+ let lo = x * 12.92;+ let hi = 1.055 * pow(x, vec3<f32>(1.0 / 2.4)) - 0.055;+ return select(hi, lo, x <= vec3<f32>(0.0031308));+}++struct Sampled {+ color : vec4<f32>,+ hdr : vec4<f32>,+};++fn sampleLens(ndc : vec2<f32>) -> Sampled {+ let ray = lensRay(ndc);+ var seen : Seen;+ seen.dir = ray.dir;+ seen.doppler = 1.0;+ if (lens.motion.w > 0.0) {+ seen = aberrate(ray.dir);+ }+ // The engine's frames are right-handed, cube lookups left-handed.+ let lookup = vec3<f32>(seen.dir.xy, -seen.dir.z);+ var color = textureSample(colorCube, samp, lookup);+ var hdr = textureSample(hdrCube, samp, lookup);+ if (lens.params.x > 0.5) {+ // Premultiplied display colour: shift its light, then re-encode.+ let a = color.a;+ let lin = srgbToLinear(color.rgb / max(a, 1e-4));+ color = vec4<f32>(linearToSrgb(shiftLight(lin, seen.doppler)) * a, a);+ // Radiance is linear and premultiplied: the shift applies directly.+ hdr = vec4<f32>(shiftLight(hdr.rgb, seen.doppler), hdr.a);+ }+ var out : Sampled;+ out.color = color * ray.inside;+ out.hdr = hdr * ray.inside;+ return out;+}++@fragment+fn fs_color(in : VsOut) -> @location(0) vec4<f32> {+ return sampleLens(in.ndc).color;+}++struct HdrOut {+ @location(0) color : vec4<f32>,+ @location(1) hdr : vec4<f32>,+};++@fragment+fn fs_color_hdr(in : VsOut) -> HdrOut {+ let s = sampleLens(in.ndc);+ var out : HdrOut;+ out.color = s.color;+ out.hdr = s.hdr;+ return out;+}+`;++const HDR_FORMAT: GPUTextureFormat = "rgba16float";++export class LensRemapPipeline {+ private device: GPUDevice;+ private colorFormat: GPUTextureFormat;+ private mipModule: GPUShaderModule;+ private mipLayout: GPUBindGroupLayout;+ private mipPipes = new Map<GPUTextureFormat, GPURenderPipeline>();+ private remapLayout: GPUBindGroupLayout;+ private remapModule: GPUShaderModule;+ private remapPipe: GPURenderPipeline | null = null;+ private remapHdrPipe: GPURenderPipeline | null = null;+ private mipSampler: GPUSampler;+ private cubeSampler: GPUSampler;+ private placeholderCube: GPUTexture;+ private luts = new Map<boolean, GPUBuffer>();+ private uniformPool = new BufferPool(+ 80,+ GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,+ );+ private uniformData = new Float32Array(20);++ constructor(device: GPUDevice, colorFormat: GPUTextureFormat) {+ this.device = device;+ this.colorFormat = colorFormat;+ this.mipModule = device.createShaderModule({+ label: "lens-mip.wgsl",+ code: mipWgsl,+ });+ this.remapModule = device.createShaderModule({+ label: "lens-remap.wgsl",+ code: remapWgsl,+ });+ this.mipLayout = device.createBindGroupLayout({+ label: "LensMipLayout",+ entries: [+ {+ binding: 0,+ visibility: GPUShaderStage.FRAGMENT,+ sampler: { type: "filtering" },+ },+ {+ binding: 1,+ visibility: GPUShaderStage.FRAGMENT,+ texture: { sampleType: "float" },+ },+ ],+ });+ this.remapLayout = device.createBindGroupLayout({+ label: "LensRemapLayout",+ entries: [+ {+ binding: 0,+ visibility: GPUShaderStage.FRAGMENT,+ buffer: { type: "uniform" },+ },+ {+ binding: 1,+ visibility: GPUShaderStage.FRAGMENT,+ sampler: { type: "filtering" },+ },+ {+ binding: 2,+ visibility: GPUShaderStage.FRAGMENT,+ texture: { sampleType: "float", viewDimension: "cube" },+ },+ {+ binding: 3,+ visibility: GPUShaderStage.FRAGMENT,+ texture: { sampleType: "float", viewDimension: "cube" },+ },+ {+ binding: 4,+ visibility: GPUShaderStage.FRAGMENT,+ buffer: { type: "read-only-storage" },+ },+ ],+ });+ this.mipSampler = device.createSampler({+ label: "LensMipSampler",+ magFilter: "linear",+ minFilter: "linear",+ });+ this.cubeSampler = device.createSampler({+ label: "LensCubeSampler",+ magFilter: "linear",+ minFilter: "linear",+ mipmapFilter: "linear",+ maxAnisotropy: 16,+ });+ this.placeholderCube = device.createTexture({+ label: "LensPlaceholderCube",+ size: { width: 1, height: 1, depthOrArrayLayers: 6 },+ format: HDR_FORMAT,+ usage: GPUTextureUsage.TEXTURE_BINDING,+ });+ }++ /** Mip levels a cube face of `size` gets. */+ static mipLevels(size: number): number {+ return Math.floor(Math.log2(Math.max(1, size))) + 1;+ }++ /** Fills every mip level of every layer of `texture` from level 0. */+ generateMips(encoder: GPUCommandEncoder, texture: GPUTexture): void {+ const pipe = this.mipPipeline(texture.format);+ for (let layer = 0; layer < texture.depthOrArrayLayers; layer++) {+ for (let level = 1; level < texture.mipLevelCount; level++) {+ const view = (mip: number) =>+ texture.createView({+ dimension: "2d",+ baseArrayLayer: layer,+ arrayLayerCount: 1,+ baseMipLevel: mip,+ mipLevelCount: 1,+ });+ const pass = encoder.beginRenderPass({+ label: "LensMip",+ colorAttachments: [+ { view: view(level), loadOp: "clear", storeOp: "store" },+ ],+ });+ pass.setPipeline(pipe);+ pass.setBindGroup(+ 0,+ this.device.createBindGroup({+ layout: this.mipLayout,+ entries: [+ { binding: 0, resource: this.mipSampler },+ { binding: 1, resource: view(level - 1) },+ ],+ }),+ );+ pass.draw(3);+ pass.end();+ }+ }+ }++ /**+ * Draws the lens's picture of `color` (and of `hdr`, the scene radiance,+ * into `destHdr` when both are given). Generates the cubes' mips first.+ */+ remap(+ encoder: GPUCommandEncoder,+ color: GPUTexture,+ hdr: GPUTexture | null,+ dest: GPUTextureView,+ destHdr: GPUTextureView | null,+ width: number,+ height: number,+ options: LensRemapOptions,+ ): void {+ this.generateMips(encoder, color);+ if (hdr) this.generateMips(encoder, hdr);+ const withHdr = Boolean(hdr && destHdr);+ const pass = encoder.beginRenderPass({+ label: "LensRemap",+ colorAttachments: [+ { view: dest, loadOp: "clear", storeOp: "store" },+ ...(withHdr && destHdr+ ? [+ {+ view: destHdr,+ loadOp: "clear" as const,+ storeOp: "store" as const,+ },+ ]+ : []),+ ],+ });+ pass.setPipeline(this.remapPipeline(withHdr));+ pass.setBindGroup(+ 0,+ this.device.createBindGroup({+ layout: this.remapLayout,+ entries: [+ {+ binding: 0,+ resource: {+ buffer: this.uniformPool.getBuffer(+ this.device,+ this.packUniforms(options, width, height),+ ),+ },+ },+ { binding: 1, resource: this.cubeSampler },+ { binding: 2, resource: color.createView({ dimension: "cube" }) },+ {+ binding: 3,+ resource: (hdr ?? this.placeholderCube).createView({+ dimension: "cube",+ }),+ },+ {+ binding: 4,+ resource: { buffer: this.lut(options.observer?.beaming ?? true) },+ },+ ],+ }),+ );+ pass.draw(3);+ pass.end();+ }++ private packUniforms(+ options: LensRemapOptions,+ width: number,+ height: number,+ ): Float32Array {+ const u = this.uniformData;+ const { right, up, forward } = options.basis;+ const half = (options.fov * Math.PI) / 360;+ const scale =+ options.projection === "fisheye"+ ? half / options.zoom+ : Math.tan(half) / options.zoom;+ u.set([...right, PROJECTION_INDEX[options.projection]], 0);+ u.set([...up, scale], 4);+ u.set([...forward, width / Math.max(1, height)], 8);+ const observer = options.observer;+ const beta = observer ? Math.min(Math.max(observer.speed, 0), 0.99999) : 0;+ const direction = observer?.direction ?? forward;+ u.set([...direction, beta], 12);+ u.set(+ [+ observer && beta > 0 ? colorMode(observer) : ColorMode.None,+ observer ? exposureGain(observer, beta, forward) : 1,+ lorentzFactor(beta),+ Math.log(DOPPLER_LUT_MAX),+ ],+ 16,+ );+ return u;+ }++ private lut(beaming: boolean): GPUBuffer {+ let buffer = this.luts.get(beaming);+ if (!buffer) {+ const data = dopplerLut(beaming);+ buffer = this.device.createBuffer({+ label: `DopplerLut${beaming ? "Beaming" : ""}`,+ size: data.byteLength,+ usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST,+ });+ this.device.queue.writeBuffer(buffer, 0, data);+ this.luts.set(beaming, buffer);+ }+ return buffer;+ }++ private mipPipeline(format: GPUTextureFormat): GPURenderPipeline {+ let pipe = this.mipPipes.get(format);+ if (!pipe) {+ pipe = this.device.createRenderPipeline({+ label: `LensMip:${format}`,+ layout: this.device.createPipelineLayout({+ bindGroupLayouts: [this.mipLayout],+ }),+ vertex: { module: this.mipModule, entryPoint: "vs" },+ fragment: {+ module: this.mipModule,+ entryPoint: "fs",+ targets: [{ format }],+ },+ primitive: { topology: "triangle-list" },+ });+ this.mipPipes.set(format, pipe);+ }+ return pipe;+ }++ private remapPipeline(withHdr: boolean): GPURenderPipeline {+ const cached = withHdr ? this.remapHdrPipe : this.remapPipe;+ if (cached) return cached;+ const pipe = this.device.createRenderPipeline({+ label: withHdr ? "LensRemapHdr" : "LensRemap",+ layout: this.device.createPipelineLayout({+ bindGroupLayouts: [this.remapLayout],+ }),+ vertex: { module: this.remapModule, entryPoint: "vs" },+ fragment: {+ module: this.remapModule,+ entryPoint: withHdr ? "fs_color_hdr" : "fs_color",+ targets: withHdr+ ? [{ format: this.colorFormat }, { format: HDR_FORMAT }]+ : [{ format: this.colorFormat }],+ },+ primitive: { topology: "triangle-list" },+ });+ if (withHdr) this.remapHdrPipe = pipe;+ else this.remapPipe = pipe;+ return pipe;+ }++ /** Call once per frame: recycles the per-pass uniforms. */+ resetPools(): void {+ this.uniformPool.reset();+ }++ destroy(): void {+ this.uniformPool.destroy();+ this.placeholderCube.destroy();+ for (const buffer of this.luts.values()) buffer.destroy();+ this.luts.clear();+ }+}++/** The gain on the shifted light: metered at the centre of the view, or fixed stops. */+function exposureGain(+ observer: RelativisticObserver,+ beta: number,+ forward: Vec3,+): number {+ if (observer.exposure !== "auto") return 2 ** observer.exposure;+ const c =+ forward[0] * observer.direction[0] ++ forward[1] * observer.direction[1] ++ forward[2] * observer.direction[2];+ const d = lorentzFactor(beta) * (1 + beta * c);+ return 1 / dopplerLuminanceGain(d, observer.doppler, observer.beaming);+}++function colorMode(observer: RelativisticObserver): number {+ if (observer.doppler) return ColorMode.Spectral;+ return observer.beaming ? ColorMode.Beaming : ColorMode.None;+}diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.tsindex 3f28422f..cfb3484d 100644--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts@@ -54,6 +54,10 @@ import { FloorReflectionPipeline } from "./floor-reflection-pipeline.js";import { GeometryOutlinePipeline } from "./geometry-outline-pipeline.js";import { GeometryResolvePipeline } from "./geometry-resolve-pipeline.js";import { decodeKtx2, isKtx2 } from "./ktx2.js";+import {+ type LensRemapOptions,+ LensRemapPipeline,+} from "./lens-remap-pipeline.js";import {type GPUMeshBuffers,Mesh3DPipeline,@@ -298,7 +302,8 @@ export class Renderer3D {>();/** Parsed models by source and load options: a model file is read once, not every frame. */private modelCache = new Map<string, Model3DData>();- private lastCameraPose: Camera3DPose | null = null;+ /** What the cached camera buffer was packed from (pose, lens, surface). */+ private lastCameraKey: string | null = null;private shadowTexture: GPUTexture | null = null;private shadowTextures = new Map<string, GPUTexture>();private placeholderShadowTexture: GPUTexture | null = null;@@ -308,6 +313,7 @@ export class Renderer3D {private outlinePipeline: GeometryOutlinePipeline | null = null;private geometryResolvePipeline: GeometryResolvePipeline | null = null;private lastCameraBuffer: GPUBuffer | null = null;+ private lensPipeline: LensRemapPipeline | null = null;/*** Samples per pixel in the 3D pass. Mesh silhouettes (extruded text,@@ -438,16 +444,23 @@ export class Renderer3D {label: string,width: number,height: number,- desc: { format: GPUTextureFormat; usage: number; sampleCount?: number },+ desc: {+ format: GPUTextureFormat;+ usage: number;+ sampleCount?: number;+ layers?: number;+ mipLevelCount?: number;+ },): GPUTexture {- const key = `${label}:${width}x${height}:${desc.sampleCount ?? 1}:${desc.usage}`;+ const { layers = 1, ...texture2d } = desc;+ const key = `${label}:${width}x${height}x${layers}:${desc.sampleCount ?? 1}:${desc.mipLevelCount ?? 1}:${desc.format}:${desc.usage}`;this.attachmentsUsed.add(key);let texture = this.attachments.get(key);if (!texture) {texture = this.device.createTexture({label,- size: { width, height },- ...desc,+ size: { width, height, depthOrArrayLayers: layers },+ ...texture2d,});this.attachments.set(key, texture);}@@ -463,11 +476,12 @@ export class Renderer3D {this.attachmentsUsed.clear();this.outlinePipeline?.reset();this.bloomPipeline?.resetPools();+ this.lensPipeline?.resetPools();this.cameraPool.reset();// The cached camera buffer goes back to the pool and may be handed// out again (e.g. to a shadow pass's light camera): never reuse it// across a reset.- this.lastCameraPose = null;+ this.lastCameraKey = null;this.lastCameraBuffer = null;this.lightsPool.reset();this.directLightsPool.reset();@@ -515,6 +529,67 @@ export class Renderer3D {return this.hdrTexture;}+ /**+ * A cube camera's target: six `size`² faces with a full mip chain, drawn+ * into face by face and sampled by `remapLens`. `hdr` holds the scene+ * radiance (the bloom source) instead of display colour.+ */+ getOrCreateCubeTexture(size: number, hdr: boolean): GPUTexture {+ return this.attachment(+ hdr ? "Renderer3DCubeHdr" : "Renderer3DCube",+ size,+ size,+ {+ format: hdr ? "rgba16float" : this.format,+ usage:+ GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,+ layers: 6,+ mipLevelCount: LensRemapPipeline.mipLevels(size),+ },+ );+ }++ /** Face `face` (WebGPU layer order) of a cube target, as a render attachment. */+ cubeFaceView(cube: GPUTexture, face: number): GPUTextureView {+ return cube.createView({+ dimension: "2d",+ baseArrayLayer: face,+ arrayLayerCount: 1,+ baseMipLevel: 0,+ mipLevelCount: 1,+ });+ }++ /**+ * Draws a cube camera's picture into `dest` (`width` × `height`) through+ * its lens. With an `hdr` cube the radiance is remapped too and becomes+ * `hdrSource()`, so bloom glows where the lens put the light.+ */+ remapLens(+ encoder: GPUCommandEncoder,+ color: GPUTexture,+ hdr: GPUTexture | null,+ dest: GPUTextureView,+ width: number,+ height: number,+ options: LensRemapOptions,+ ): void {+ this.lensPipeline ??= new LensRemapPipeline(this.device, this.format);+ const destHdr = hdr+ ? this.getOrCreateHdrTexture(width, height).createView()+ : null;+ this.lensPipeline.remap(+ encoder,+ color,+ hdr,+ dest,+ destHdr,+ width,+ height,+ options,+ );+ }+private getOrCreateHdrTexture(width: number, height: number): GPUTexture {this.hdrTexture = this.attachment("Renderer3DHdrRadiance", width, height, {format: "rgba16float",@@ -622,23 +697,25 @@ export class Renderer3D {surfaceHeight: number,time = 0,): GPUBuffer {- // Skip recomputation if camera pose hasn't changed- if (- this.lastCameraPose &&- this.lastCameraBuffer &&- this.lastCameraPose.eye[0] === camera.eye[0] &&- this.lastCameraPose.eye[1] === camera.eye[1] &&- this.lastCameraPose.eye[2] === camera.eye[2] &&- this.lastCameraPose.target[0] === camera.target[0] &&- this.lastCameraPose.target[1] === camera.target[1] &&- this.lastCameraPose.target[2] === camera.target[2] &&- this.lastCameraPose.up[0] === camera.up[0] &&- this.lastCameraPose.up[1] === camera.up[1] &&- this.lastCameraPose.up[2] === camera.up[2] &&- this.lastCameraPose.fov === camera.fov &&- this.lastCameraPose.near === camera.near &&- this.lastCameraPose.far === camera.far- ) {+ // Every packed field: a face of a cube camera differs from the main+ // view in aspect and surface size alone.+ const key = [+ ...camera.eye,+ ...camera.target,+ ...camera.up,+ camera.fov,+ camera.aspect,+ camera.near,+ camera.far,+ camera.zoom,+ camera.focusDistance,+ camera.fStop,+ camera.maxBlurRadius,+ surfaceWidth,+ surfaceHeight,+ time,+ ].join(",");+ if (this.lastCameraBuffer && this.lastCameraKey === key) {return this.lastCameraBuffer;}@@ -687,7 +764,7 @@ export class Renderer3D {this.device,this.cameraUniformData,);- this.lastCameraPose = camera;+ this.lastCameraKey = key;this.lastCameraBuffer = buffer;return buffer;}@@ -954,6 +1031,8 @@ export class Renderer3D {world?: Scene3DWorld,/** Also write unclamped scene radiance (`hdrTexture()`), the bloom source. */writeHdr = false,+ /** Where the radiance resolves instead of `hdrSource()` (a cube face). */+ hdrTargetView?: GPUTextureView,): {pass: GPURenderPassEncoder;cameraBindGroup: GPUBindGroup;@@ -1056,7 +1135,9 @@ export class Renderer3D {this.passWritesHdr = writeHdr;if (writeHdr) {- this.hdrTexture = this.getOrCreateHdrTexture(surfaceWidth, surfaceHeight);+ const hdrView =+ hdrTargetView ??+ this.getOrCreateHdrTexture(surfaceWidth, surfaceHeight).createView();if (msaa) {this.msaaHdrTexture = this.getOrCreateMsaaTexture(this.msaaHdrTexture,@@ -1068,7 +1149,7 @@ export class Renderer3D {}colorAttachments.push(attach(- this.hdrTexture.createView(),+ hdrView,msaa ? this.msaaHdrTexture : null,{ r: 0, g: 0, b: 0, a: 0 },"clear",@@ -2101,6 +2182,7 @@ export class Renderer3D {for (const texture of this.attachments.values()) texture.destroy();this.attachments.clear();this.bloomPipeline?.destroy();+ this.lensPipeline?.destroy();this.outlinePipeline?.destroy();this.depthTexture?.destroy();for (const texture of this.shadowTextures.values()) texture.destroy();diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.mdindex 000d3558..33574062 100644--- a/plugins/framefields/skills/framefields-compose/SKILL.md+++ b/plugins/framefields/skills/framefields-compose/SKILL.md@@ -244,6 +244,20 @@ const camAnim = CameraAnimation.camera()});```+**Cube cameras.** `lens.projection` picks the picture: `"perspective"` (default), `"equirectangular"` (the whole sphere around the eye, 360 × 180, for VR/360 video; `fov` is ignored), or `"fisheye"` (equidistant; `fov` is the full angle across the frame's height, up to 360°, so it can see behind the camera). Anything but perspective draws the scene as six 90° cube faces around the eye and remaps them, with mipmaps, so it costs about six renders. `lens.cubeSize` overrides the face size (default: matches the output's sharpness at the centre of the view, up to 4096).++**Relativity.** `relativity: { speed, direction?, doppler?, beaming?, exposure? }` films the scene as seen by a camera moving at `speed` × c (0 to 0.9999) along `direction` (world space, default where the camera looks). It is physics, not grading: rays crowd towards the direction of motion (aberration; objects at the side swing forward and look turned, not squashed), colours shift blue ahead and red behind (spectral Doppler, each colour modelled as a reflectance lit by daylight) and ahead brightens with beaming. `exposure: "auto"` (default) meters the centre of the view so what the camera looks at keeps its brightness; a number is fixed stops, where `0` is the true, often blinding, light. Animate the speed with the `relativitySpeed` prop, e.g. `CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.99, { start: 0, end: 90, ease: "expo.in" })`. A camera with `relativity` is always a cube camera, even at speed 0, and at rest matches the ordinary camera. Works with every projection.++```typescript+Layer.camera({+ x: 0, y: -170, z: 0, targetX: 0, targetY: -170, targetZ: 1000,+ lens: { fov: 90 },+ relativity: { speed: 0.9 },+});+```++Cube cameras draw outlines, shadows, floor reflections and bloom, but not depth of field (`comp.lint()` reports `camera_cube_dof`), and glass refracts only the 3D scene, not the 2D layers below.+### 4. Procedural & Audio-Reactive DriversProperties can be bound to Perlin wiggles, harmonic springs, or audio-extracted signals:```typescript