Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(3d): cube cameras with equirectangular, fisheye and relativistic lenses

okanasla6fedc691340
diff --git a/.changeset/cube-cameras.md b/.changeset/cube-cameras.md
new file mode 100644
index 00000000..27c547a1
--- /dev/null
+++ b/.changeset/cube-cameras.md
@@ -0,0 +1,5 @@
+---
+"framefields": minor
+---
+
+Cube cameras: `lens.projection` `"equirectangular"` (360° panoramas) and `"fisheye"` (up to 360°), drawn as six mipmapped cube faces and remapped. `relativity: { speed, direction, doppler, beaming, exposure }` films a scene as seen near the speed of light: aberration, spectral Doppler shift and beaming, animatable with `relativitySpeed`. Lens options are now typed as authored, so fields with defaults may be left out.
diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.md
index e6302c9c..cd2488f8 100644
--- a/nodes/node-compositor/SKILL.md
+++ b/nodes/node-compositor/SKILL.md
@@ -213,6 +213,7 @@ Each track represents animatable property modifications:
- **Several 3D scenes:** a `box`/`flex`/`block` with a numeric `width` and `height` that holds a `camera` and the layers it films renders as its own scene (one holding only a camera stays a shot in a camera cut list): a nested composition with only its own camera, lights and environment, on its own clock (children start from the box's frame 0), placed in its parent like a media layer. Transitions, opacity and mattes on the box then blend finished pictures. Without that, every 3D layer shares one world and the active camera (the latest to start).
- **3D scene capacity:** direct lights and skin joints use dynamically sized GPU storage; directional and spot lights can each cast shadows, up to the device texture-array layer limit.
- **Art-directed 3D lighting:** `material: "toon"` on meshes, flat 3D cards and native 3D text uses the shared direct lights, colors, point/spot attenuation, cast shadows, environment fog and tone mapping. Optional `toon: { shadowColor, lightColor, threshold, softness, diffuse }` controls the shadow/lit albedo multipliers, terminator cosine/width and Lambert modelling (defaults: model shade color or `#80808c`, white, 0.075, 0.25, 0). Lit/physical materials retain their own shading.
+- **Cube cameras:** a camera whose `lens.projection` is `"equirectangular"` or `"fisheye"`, or that has `relativity`, draws the scene as six 90° faces into a mipmapped cube (`cubeFaceCameras`, layer order +X −X +Y −Y +Z −Z; the right-handed view and left-handed cube sampling differ by a z reflection, `CUBE_FLIP`) and `Renderer3D.remapLens` turns it into the output picture. Each face gets its own floor reflection and outline pass (ink width rescaled to the output's density at the view centre); shadows are drawn once; bloom reads the remapped radiance. `relativity` aberrates every ray (`cos θ = (cos θ' − β)/(1 − β cos θ')`) and applies a spectral Doppler/beaming matrix tabulated in ln D. Depth of field throws; glass gets no 2D backdrop.
- **Geometry outlines:** an environment light's optional `outlines: true | { color, width, referenceHeight, creaseAngle, depthThreshold, opacity }` runs a depth/normal ink pass before camera depth blur and bloom (defaults: `#16131c`, 1.5 px at height 1080, 45°, 0.01, 1). It reuses the linear-depth attachment's spare channels for octahedral surface normals and outline participation. Inverse-depth curvature avoids false contours on perspective planes; normal discontinuities find creases without inking texture/shadow/triangulation seams. `outline: false` on a layer/model opts out while retaining depth occlusion; `creaseAngle: 180` disables creases. Ink fades into the environment fog. 3D `anchorX/Y` pivots remain fixed under scale/rotation and leave identity placement unchanged.
With MSAA enabled, retain original scene color and geometry samples and shade ink per sample before resolving color. A single-sample ink pass after the scene resolve loses subpixel coverage between outlined surfaces. Lens blur uses a separate nearest-sample geometry resolve. The scene's geometry is still drawn once.
- **Transparent meshes:** glTF `BLEND` materials and meshes with reduced layer, material, tint, or vertex alpha draw after opaque items, back to front by the camera-space center of each mesh’s transformed bounds, without writing depth. This is object-level sorting: intersecting meshes, self-overlap, skinned/deformed bounds, translucent flat layers, glass, and transparent shadows have separate limitations.
diff --git a/nodes/node-compositor/src/renderers/cube-camera.ts b/nodes/node-compositor/src/renderers/cube-camera.ts
new file mode 100644
index 00000000..4b4aa6e6
--- /dev/null
+++ b/nodes/node-compositor/src/renderers/cube-camera.ts
@@ -0,0 +1,90 @@
+import {
+ type Camera3DPose,
+ type CubeLensOptions,
+ cameraBasis,
+ defaultCubeFaceSize,
+ type LensProjection,
+ type LensRemapOptions,
+ lensCenterDensity,
+ type RelativisticObserver,
+ type Vec3,
+ Vector3Math,
+} from "@framefields/webgpu-renderers";
+
+/** Largest face a cube camera picks on its own; `lens.cubeSize` may go higher. */
+const DEFAULT_MAX_FACE = 4096;
+
+/** A camera drawn as a cube map and remapped through its lens. */
+export interface CubeCamera {
+ faceSize: number;
+ remap: LensRemapOptions;
+ /** Output pixels per radian at the centre of the view. */
+ centerDensity: number;
+}
+
+interface AuthoredCamera {
+ lens?: {
+ projection?: LensProjection;
+ fov?: number;
+ zoom?: number;
+ cubeSize?: number;
+ };
+ relativity?: {
+ speed?: number;
+ direction?: Vec3;
+ doppler?: boolean;
+ beaming?: boolean;
+ exposure?: number | "auto";
+ };
+}
+
+/**
+ * The cube camera an authored camera asks for, or null for an ordinary
+ * perspective view. A camera with `relativity` is always a cube camera, even
+ * at speed 0, so a shot that accelerates from rest never switches pipelines.
+ */
+export function resolveCubeCamera(
+ authored: AuthoredCamera | undefined,
+ camera: Camera3DPose,
+ speed: number,
+ width: number,
+ height: number,
+ maxTextureSize: number,
+): CubeCamera | null {
+ const projection = authored?.lens?.projection ?? "perspective";
+ const relativity = authored?.relativity;
+ if (projection === "perspective" && !relativity) return null;
+ const lens: CubeLensOptions = {
+ projection,
+ fov: camera.fov,
+ zoom: camera.zoom,
+ };
+ const requested = authored?.lens?.cubeSize;
+ const faceSize = Math.min(
+ maxTextureSize,
+ requested ??
+ defaultCubeFaceSize(
+ lens,
+ width,
+ height,
+ Math.min(DEFAULT_MAX_FACE, maxTextureSize),
+ ),
+ );
+ const basis = cameraBasis(camera);
+ const observer: RelativisticObserver | null = relativity
+ ? {
+ speed,
+ direction: relativity.direction
+ ? Vector3Math.normalize(relativity.direction)
+ : basis.forward,
+ doppler: relativity.doppler ?? true,
+ beaming: relativity.beaming ?? true,
+ exposure: relativity.exposure ?? "auto",
+ }
+ : null;
+ return {
+ faceSize,
+ remap: { ...lens, basis, observer },
+ centerDensity: lensCenterDensity(lens, width, height),
+ };
+}
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index 37d3a0a1..f29d95c2 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -41,6 +41,7 @@ import {
Camera3D,
type Camera3DPose,
type Color,
+ cubeFaceCameras,
type DrawMesh3DOpts,
type DrawQuad3DOpts,
type Environment3DOptions,
@@ -82,6 +83,7 @@ import {
import { compileTimeline, seekTimeline } from "../shared/compiler.js";
import type { CompositorOperation } from "../shared/config.js";
import { shouldPaintContainerBackground } from "./container-bg.js";
+import { type CubeCamera, resolveCubeCamera } from "./cube-camera.js";
import { compatibleMeshDraws } from "./mesh-instancing.js";
import {
buildLayerMatrix,
@@ -1240,6 +1242,8 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
/** Where a relief keeps its look from: the default camera at the scene's field of view. */
let reliefEye: [number, number, number] = [0, 0, 0];
let dofEnabled = false;
+ /** Set when the camera draws a cube map and remaps it through its lens. */
+ let cubeCamera: CubeCamera | null = null;
const sceneLights: Light3DOptions[] = [];
let sceneEnvironment: Environment3DOptions | null = null;
let sceneBloom: BloomOptions | null = null;
@@ -1344,6 +1348,20 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
camStub?.maxBlurRadius ?? camOp?.dof?.maxBlurRadius ?? 24;
}
+ cubeCamera = resolveCubeCamera(
+ camOp,
+ camera,
+ camStub?.relativitySpeed ?? camOp?.relativity?.speed ?? 0,
+ nativeWidth,
+ nativeHeight,
+ ctx.device.limits.maxTextureDimension2D,
+ );
+ if (cubeCamera && dofEnabled) {
+ throw new Error(
+ `Camera '${camOp?.id}' is a cube camera (${camOp?.relativity ? "relativity" : camOp?.lens?.projection}) with depth of field: depth of field needs a single perspective view.`,
+ );
+ }
+
// Collect Active Lights
for (const lvm of lightVMs) {
const lop = lvm.operation as any;
@@ -1680,147 +1698,235 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
...glass,
];
- // Planar floor reflection pass (if environment.floor is active)
+ // One view of the scene: its floor reflection, then every item.
const floorSpec = sceneEnvironment?.floor;
- let blurredReflectionTex: GPUTexture | null = null;
- if (
- floorSpec &&
- typeof floorSpec.y === "number" &&
- camera.eye[1] < floorSpec.y
- ) {
- const reflectedCamera: Camera3DPose = {
- ...camera,
- eye: [camera.eye[0], 2 * floorSpec.y - camera.eye[1], camera.eye[2]],
- target: [
- camera.target[0],
- 2 * floorSpec.y - camera.target[1],
- camera.target[2],
- ],
- up: [camera.up[0], camera.up[1], camera.up[2]],
- };
+ const drawView = (
+ view: Camera3DPose,
+ width: number,
+ height: number,
+ colorView: GPUTextureView,
+ geometryView: GPUTextureView | undefined,
+ hdrTargetView: GPUTextureView | undefined,
+ /** The 2D picture glass refracts; a cube face has none. */
+ backdrop: GPUTexture | undefined,
+ ): GPUBuffer => {
+ let blurredReflectionTex: GPUTexture | null = null;
+ if (
+ floorSpec &&
+ typeof floorSpec.y === "number" &&
+ view.eye[1] < floorSpec.y
+ ) {
+ const reflectedCamera: Camera3DPose = {
+ ...view,
+ eye: [view.eye[0], 2 * floorSpec.y - view.eye[1], view.eye[2]],
+ target: [
+ view.target[0],
+ 2 * floorSpec.y - view.target[1],
+ view.target[2],
+ ],
+ up: [view.up[0], view.up[1], view.up[2]],
+ };
+ const {
+ pass: reflPass,
+ cameraBindGroup: reflCamBind,
+ lightsBindGroup: reflLightsBind,
+ reflectionTexture: reflTex,
+ } = r3d.beginReflectionPass(
+ encoder,
+ width,
+ height,
+ reflectedCamera,
+ sceneLights,
+ { environment: sceneEnvironment, shadow: null },
+ );
+ for (const item of ordered) {
+ if (item.kind === "quad") {
+ r3d.drawTextureQuad(
+ reflPass,
+ reflCamBind,
+ reflLightsBind,
+ item.texture,
+ item.opts,
+ false,
+ );
+ } else if (item.kind === "slug") {
+ r3d.drawSlugText(
+ reflPass,
+ reflCamBind,
+ reflLightsBind,
+ item.batches,
+ item.opts,
+ item.stackBase,
+ false,
+ );
+ } else if (item.kind === "mesh") {
+ r3d.drawMesh3D(
+ reflPass,
+ reflCamBind,
+ reflLightsBind,
+ item.meshBuffers ?? item.mesh,
+ item.texture,
+ item.opts,
+ false,
+ );
+ }
+ }
+ reflPass.end();
+ blurredReflectionTex = r3d.blurReflection(
+ encoder,
+ reflTex,
+ floorSpec.roughness ?? 0.35,
+ );
+ }
+
const {
- pass: reflPass,
- cameraBindGroup: reflCamBind,
- lightsBindGroup: reflLightsBind,
- reflectionTexture: reflTex,
- } = r3d.beginReflectionPass(
+ pass: pass3d,
+ cameraBindGroup,
+ cameraBuffer,
+ lightsBindGroup,
+ } = r3d.beginPass(
encoder,
- nativeWidth,
- nativeHeight,
- reflectedCamera,
+ colorView,
+ width,
+ height,
+ view,
+ "clear",
+ { r: 0, g: 0, b: 0, a: 0 },
+ geometryView,
sceneLights,
- { environment: sceneEnvironment, shadow: null },
+ {
+ environment: sceneEnvironment,
+ shadow: shadow.length ? shadow : null,
+ },
+ Boolean(sceneBloom),
+ hdrTargetView,
);
+
+ if (floorSpec && blurredReflectionTex) {
+ r3d.drawFloor(
+ pass3d,
+ cameraBindGroup,
+ lightsBindGroup,
+ blurredReflectionTex,
+ floorSpec,
+ geometryEnabled,
+ Boolean(sceneBloom),
+ );
+ }
for (const item of ordered) {
if (item.kind === "quad") {
r3d.drawTextureQuad(
- reflPass,
- reflCamBind,
- reflLightsBind,
+ pass3d,
+ cameraBindGroup,
+ lightsBindGroup,
item.texture,
item.opts,
- false,
+ geometryEnabled,
);
} else if (item.kind === "slug") {
r3d.drawSlugText(
- reflPass,
- reflCamBind,
- reflLightsBind,
+ pass3d,
+ cameraBindGroup,
+ lightsBindGroup,
item.batches,
item.opts,
item.stackBase,
- false,
+ geometryEnabled,
);
} else if (item.kind === "mesh") {
r3d.drawMesh3D(
- reflPass,
- reflCamBind,
- reflLightsBind,
+ pass3d,
+ cameraBindGroup,
+ lightsBindGroup,
item.meshBuffers ?? item.mesh,
item.texture,
- item.opts,
- false,
+ isGlass(item) ? { ...item.opts, backdrop } : item.opts,
+ geometryEnabled,
);
}
}
- reflPass.end();
- blurredReflectionTex = r3d.blurReflection(
- encoder,
- reflTex,
- floorSpec.roughness ?? 0.35,
- );
- }
- const {
- pass: pass3d,
- cameraBindGroup,
- cameraBuffer,
- lightsBindGroup,
- } = r3d.beginPass(
- encoder,
- scene3dView,
- nativeWidth,
- nativeHeight,
- camera,
- "clear",
- { r: 0, g: 0, b: 0, a: 0 },
- linearDepthView,
- sceneLights,
- {
- environment: sceneEnvironment,
- shadow: shadow.length ? shadow : null,
- },
- Boolean(sceneBloom),
- );
+ pass3d.end();
+ return cameraBuffer;
+ };
- if (floorSpec && blurredReflectionTex) {
- r3d.drawFloor(
- pass3d,
- cameraBindGroup,
- lightsBindGroup,
- blurredReflectionTex,
- floorSpec,
- geometryEnabled,
- Boolean(sceneBloom),
+ let final3dTex = scene3dTex;
+ if (cubeCamera) {
+ // Six 90° faces around the eye, then the lens's picture of them.
+ const size = cubeCamera.faceSize;
+ const cube = r3d.getOrCreateCubeTexture(size, false);
+ const hdrCube = sceneBloom
+ ? r3d.getOrCreateCubeTexture(size, true)
+ : null;
+ // Ink as wide on screen as without a cube: a face has size / 2
+ // pixels per radian at its centre, the output centerDensity.
+ const faceOutlines = outlines && {
+ ...outlines,
+ referenceHeight:
+ ((outlines.referenceHeight ?? 1080) *
+ 2 *
+ cubeCamera.centerDensity) /
+ nativeHeight,
+ };
+ const faceTex = faceOutlines
+ ? ctx.renderer.getTemporaryTexture(size, size, [
+ curTex,
+ targetTexture,
+ scene3dTex,
+ ...curExcludes,
+ ...itemTextures,
+ ])
+ : null;
+ const faceGeometry = faceOutlines
+ ? r3d.getOrCreateLinearDepthTexture(size, size)
+ : null;
+ cubeFaceCameras(camera.eye, camera.near, camera.far).forEach(
+ (face, i) => {
+ const faceView = r3d.cubeFaceView(cube, i);
+ drawView(
+ face,
+ size,
+ size,
+ faceTex ? faceTex.createView() : faceView,
+ faceGeometry?.createView(),
+ hdrCube ? r3d.cubeFaceView(hdrCube, i) : undefined,
+ undefined,
+ );
+ if (faceTex && faceGeometry && faceOutlines) {
+ r3d.applyOutlines(
+ encoder,
+ faceTex,
+ faceGeometry,
+ faceView,
+ face,
+ faceOutlines,
+ sceneEnvironment?.fog,
+ );
+ }
+ },
+ );
+ r3d.remapLens(
+ encoder,
+ cube,
+ hdrCube,
+ scene3dView,
+ nativeWidth,
+ nativeHeight,
+ cubeCamera.remap,
);
}
- for (const item of ordered) {
- if (item.kind === "quad") {
- r3d.drawTextureQuad(
- pass3d,
- cameraBindGroup,
- lightsBindGroup,
- item.texture,
- item.opts,
- geometryEnabled,
- );
- } else if (item.kind === "slug") {
- r3d.drawSlugText(
- pass3d,
- cameraBindGroup,
- lightsBindGroup,
- item.batches,
- item.opts,
- item.stackBase,
- geometryEnabled,
- );
- } else if (item.kind === "mesh") {
- r3d.drawMesh3D(
- pass3d,
- cameraBindGroup,
- lightsBindGroup,
- item.meshBuffers ?? item.mesh,
- item.texture,
- isGlass(item) ? { ...item.opts, backdrop: curTex } : item.opts,
- geometryEnabled,
+ const cameraBuffer = cubeCamera
+ ? null
+ : drawView(
+ camera,
+ nativeWidth,
+ nativeHeight,
+ scene3dView,
+ linearDepthView,
+ undefined,
+ curTex,
);
- }
- }
-
- pass3d.end();
-
- let final3dTex = scene3dTex;
- if (outlines && linearDepthTex) {
+ if (!cubeCamera && outlines && linearDepthTex) {
if (dofEnabled) r3d.resolveGeometry(encoder, linearDepthTex);
const outlinedTex = ctx.renderer.getTemporaryTexture(
nativeWidth,
@@ -1838,7 +1944,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
);
final3dTex = outlinedTex;
}
- if (dofEnabled && linearDepthTex) {
+ if (dofEnabled && linearDepthTex && cameraBuffer) {
const dofResultTex = ctx.renderer.getTemporaryTexture(
nativeWidth,
nativeHeight,
diff --git a/nodes/node-compositor/src/shared/compiler.ts b/nodes/node-compositor/src/shared/compiler.ts
index 98031ec1..fadf3849 100644
--- a/nodes/node-compositor/src/shared/compiler.ts
+++ b/nodes/node-compositor/src/shared/compiler.ts
@@ -78,6 +78,7 @@ export interface LayerStub {
maxBlurRadius?: number;
shakeTranslation?: number;
shakeRotation?: number;
+ relativitySpeed?: number;
orbitRadius?: number;
orbitAzimuth?: number;
orbitElevation?: number;
@@ -718,6 +719,7 @@ function hashNodes(
maxBlurRadius: lop.maxBlurRadius,
shakeTranslation: lop.shakeTranslation,
shakeRotation: lop.shakeRotation,
+ relativitySpeed: lop.relativitySpeed,
orbitRadius: lop.orbitRadius,
orbitAzimuth: lop.orbitAzimuth,
orbitElevation: lop.orbitElevation,
@@ -906,6 +908,7 @@ export function compileTimeline(
shakeRotation:
lop.shakeRotation ??
(lop.shake?.enabled ? (lop.shake?.rotationAmplitude ?? 0) : 0),
+ relativitySpeed: lop.relativitySpeed ?? lop.relativity?.speed ?? 0,
orbitRadius: lop.orbitRadius,
orbitAzimuth: lop.orbitAzimuth,
orbitElevation: lop.orbitElevation,
diff --git a/packages/compositions/src/program/lint.ts b/packages/compositions/src/program/lint.ts
index 957b4040..2d14fbc0 100644
--- a/packages/compositions/src/program/lint.ts
+++ b/packages/compositions/src/program/lint.ts
@@ -99,6 +99,39 @@ function solidColor(value: unknown): string | null {
return c && c[3] > 0.99 ? value : null;
}
+type Push = (
+ severity: RenderDiagnostic["severity"],
+ code: string,
+ message: string,
+ nodeId?: string,
+) => void;
+
+/** Cube cameras (panoramic lenses, relativity) and what they cannot combine with. */
+function lintCamera(node: AnyNode, id: string, push: Push): void {
+ const lens = node.lens as { projection?: string; fov?: number } | undefined;
+ const projection = lens?.projection ?? "perspective";
+ if (projection === "perspective" && (lens?.fov ?? 0) > 160)
+ push(
+ "error",
+ "camera_fov_range",
+ `'${id}' has a ${lens?.fov}° perspective lens: perspective goes up to 160°. Use projection "fisheye" for wider views.`,
+ id,
+ );
+ const relativity = node.relativity as { speed?: number } | undefined;
+ const cube =
+ projection !== "perspective" ||
+ (relativity?.speed ?? 0) > 0 ||
+ animates(node, ["relativitySpeed"]);
+ const dof = node.dof as { enabled?: boolean } | undefined;
+ if (cube && (dof?.enabled || animates(node, ["focusDistance"])))
+ push(
+ "error",
+ "camera_cube_dof",
+ `'${id}' is a cube camera (${relativity ? "relativity" : projection}) with depth of field: depth of field needs a single perspective view.`,
+ id,
+ );
+}
+
export function lintComposition(
config: Pick<CompositorProgramConfig, "width" | "height" | "layout"> & {
backgroundColor?: string;
@@ -178,7 +211,10 @@ export function lintComposition(
}
return;
}
- if (node.kind === "camera") return;
+ if (node.kind === "camera") {
+ lintCamera(node, id, push);
+ return;
+ }
if (node.kind === "media") {
const src = node.inputHandleId;
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 6f061b55..fb57a2ac 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -165,6 +165,7 @@ export const AnimatablePropSchema = z.enum([
"maxBlurRadius",
"shakeTranslation",
"shakeRotation",
+ "relativitySpeed",
"orbitRadius",
"orbitAzimuth",
"orbitElevation",
@@ -298,6 +299,7 @@ export type CameraAnimatableProp =
| "maxBlurRadius"
| "shakeTranslation"
| "shakeRotation"
+ | "relativitySpeed"
| "orbitRadius"
| "orbitAzimuth"
| "orbitElevation";
@@ -381,6 +383,7 @@ export const EXTENDED_3D_ANIMATABLE_PROPS = [
"maxBlurRadius",
"shakeTranslation",
"shakeRotation",
+ "relativitySpeed",
"orbitRadius",
"orbitAzimuth",
"orbitElevation",
@@ -1250,18 +1253,42 @@ export interface ShapeNode extends CompositionNodeBase {
trimOffset?: number;
}
+/**
+ * The picture a camera draws. `perspective` is an ordinary lens; the others
+ * are cube cameras: the scene is drawn as six 90° faces around the eye and
+ * remapped, so they can show what no single frustum holds.
+ */
+export const CameraProjectionSchema = z.enum([
+ "perspective",
+ "equirectangular",
+ "fisheye",
+]);
+export type CameraProjection = z.infer<typeof CameraProjectionSchema>;
+
export const CameraLensSchema = z
.object({
focalLength: z.number().min(5).max(500).default(50),
- fov: z.number().min(5).max(160).optional(),
+ /**
+ * Degrees. Perspective: vertical field of view, up to 160. Fisheye: the
+ * full angle across the frame's height (equidistant), up to 360.
+ * Equirectangular always shows the whole sphere and ignores it.
+ */
+ fov: z.number().min(5).max(360).optional(),
sensorWidth: z.number().default(36),
near: z.number().min(0.1).default(1),
far: z.number().min(10).default(50000),
zoom: z.number().min(0.01).default(1.0),
+ projection: CameraProjectionSchema.default("perspective"),
+ /**
+ * Cube face size in pixels for cube cameras. Default: matches the
+ * output's sharpness at the centre of the view, up to 4096.
+ */
+ cubeSize: z.number().int().min(64).max(8192).optional(),
})
.strict();
-export type CameraLens = z.infer<typeof CameraLensSchema>;
+/** A lens as authored: fields with schema defaults may be left out. */
+export type CameraLens = z.input<typeof CameraLensSchema>;
export const CameraDoFSchema = z
.object({
@@ -1289,6 +1316,36 @@ export const CameraShakeSchema = z
export type CameraShake = z.infer<typeof CameraShakeSchema>;
+/**
+ * The camera moves through the scene near the speed of light. What it sees
+ * is physics, not grading: rays crowd towards the direction of motion
+ * (aberration), colours shift blue ahead and red behind (Doppler) and ahead
+ * brightens (beaming). Always drawn by a cube camera.
+ */
+export const CameraRelativitySchema = z
+ .object({
+ /** β = v / c. Animate it with the `relativitySpeed` prop. */
+ speed: z.number().min(0).max(0.9999).default(0),
+ /** World-space direction of motion. Default: where the camera looks. */
+ direction: z.tuple([z.number(), z.number(), z.number()]).optional(),
+ /** Shift colours along the spectrum. Default true. */
+ doppler: z.boolean().default(true),
+ /** Brighten ahead, dim behind. Default true. */
+ beaming: z.boolean().default(true),
+ /**
+ * Exposure for the shifted light: "auto" (default) meters the centre of
+ * the view, so what the camera looks at keeps its brightness at rest;
+ * a number is fixed stops (0 = the true, often blinding, brightness).
+ */
+ exposure: z
+ .union([z.literal("auto"), z.number().min(-64).max(16)])
+ .default("auto"),
+ })
+ .strict();
+
+/** Relativity as authored: fields with schema defaults may be left out. */
+export type CameraRelativity = z.input<typeof CameraRelativitySchema>;
+
export interface CameraNode {
id: string;
kind: "camera";
@@ -1305,6 +1362,7 @@ export interface CameraNode {
lens?: CameraLens;
dof?: CameraDoF;
shake?: CameraShake;
+ relativity?: CameraRelativity;
startFrame?: number;
durationFrames?: number;
animation?: LayerAnimationSpec;
@@ -1327,6 +1385,7 @@ export const CameraNodeSchema: z.ZodType<CameraNode> = z
lens: CameraLensSchema.optional(),
dof: CameraDoFSchema.optional(),
shake: CameraShakeSchema.optional(),
+ relativity: CameraRelativitySchema.optional(),
startFrame: z.number().int().min(0).default(0),
durationFrames: z.number().int().min(1).optional(),
animation: LayerAnimationSchema.default({ tracks: [] }),
diff --git a/packages/framefields/src/cube-camera-render.test.ts b/packages/framefields/src/cube-camera-render.test.ts
new file mode 100644
index 00000000..dbabd00f
--- /dev/null
+++ b/packages/framefields/src/cube-camera-render.test.ts
@@ -0,0 +1,236 @@
+import { describe, expect, it } from "vitest";
+import {
+ CameraAnimation,
+ type CameraNode,
+ Composition,
+ Layer,
+ Layer3D,
+} from "./index.js";
+import { computeMse, decodePngPixels, getPixel } from "./test-utils/pixels.js";
+
+const EYE = { x: 400, y: 300, z: 0 } as const;
+const DISTANCE = 800;
+
+/** Unlit markers in the six axis directions around the eye. */
+const MARKERS = [
+ { id: "front", dir: [0, 0, 1], color: "#ff0000" },
+ { id: "back", dir: [0, 0, -1], color: "#00ffff" },
+ { id: "right", dir: [1, 0, 0], color: "#00ff00" },
+ { id: "left", dir: [-1, 0, 0], color: "#0000ff" },
+ { id: "up", dir: [0, -1, 0], color: "#ffff00" },
+ { id: "down", dir: [0, 1, 0], color: "#ff00ff" },
+] as const;
+
+function scene(
+ width: number,
+ height: number,
+ camera: Partial<CameraNode>,
+ markers: readonly (typeof MARKERS)[number][] = MARKERS,
+) {
+ const comp = new Composition({
+ width,
+ height,
+ fps: 30,
+ durationMs: 1000,
+ backgroundColor: "#000000",
+ });
+ comp.add([
+ Layer.camera({
+ x: EYE.x,
+ y: EYE.y,
+ z: EYE.z,
+ targetX: EYE.x,
+ targetY: EYE.y,
+ targetZ: EYE.z + 1000,
+ ...camera,
+ }),
+ Layer.ambientLight("#ffffff", 1),
+ ...markers.map((m) =>
+ Layer3D.sphere({
+ id: m.id,
+ radius: 150,
+ x: EYE.x + m.dir[0] * DISTANCE,
+ y: EYE.y + m.dir[1] * DISTANCE,
+ z: EYE.z + m.dir[2] * DISTANCE,
+ color: m.color,
+ material: "unlit",
+ }),
+ ),
+ ]);
+ return comp;
+}
+
+const near = (
+ p: { r: number; g: number; b: number },
+ hex: string,
+ tolerance = 40,
+) => {
+ const v = Number.parseInt(hex.slice(1), 16);
+ return (
+ Math.abs(p.r - ((v >> 16) & 255)) <= tolerance &&
+ Math.abs(p.g - ((v >> 8) & 255)) <= tolerance &&
+ Math.abs(p.b - (v & 255)) <= tolerance
+ );
+};
+
+/** Mean x of the pixels close to `hex`. */
+function centroidX(
+ img: Awaited<ReturnType<typeof decodePngPixels>>,
+ hex: string,
+): number {
+ let sum = 0;
+ let count = 0;
+ for (let y = 0; y < img.height; y += 2) {
+ for (let x = 0; x < img.width; x += 2) {
+ if (near(getPixel(img, x, y), hex)) {
+ sum += x;
+ count++;
+ }
+ }
+ }
+ return count ? sum / count : Number.NaN;
+}
+
+describe("cube cameras", () => {
+ it("draws the whole sphere around the eye as an equirectangular panorama", async () => {
+ const img = await decodePngPixels(
+ await scene(1024, 512, {
+ lens: { projection: "equirectangular" },
+ }).renderFrame({ frame: 0 }),
+ );
+ // Forward in the middle, right and left a quarter turn either side,
+ // behind at both edges, up at the top and down at the bottom.
+ expect(near(getPixel(img, 512, 256), "#ff0000")).toBe(true);
+ expect(near(getPixel(img, 768, 256), "#00ff00")).toBe(true);
+ expect(near(getPixel(img, 256, 256), "#0000ff")).toBe(true);
+ expect(near(getPixel(img, 3, 256), "#00ffff")).toBe(true);
+ expect(near(getPixel(img, 1020, 256), "#00ffff")).toBe(true);
+ expect(near(getPixel(img, 512, 6), "#ffff00")).toBe(true);
+ expect(near(getPixel(img, 512, 505), "#ff00ff")).toBe(true);
+ // Between markers: empty sky.
+ expect(near(getPixel(img, 640, 256), "#000000", 10)).toBe(true);
+ });
+
+ it("matches the ordinary camera at rest", async () => {
+ const plain = scene(640, 360, { lens: { fov: 60 } }, [MARKERS[0]]);
+ const cube = scene(
+ 640,
+ 360,
+ { lens: { fov: 60 }, relativity: { speed: 0 } },
+ [MARKERS[0]],
+ );
+ const a = await decodePngPixels(await plain.renderFrame({ frame: 0 }));
+ const b = await decodePngPixels(await cube.renderFrame({ frame: 0 }));
+ expect(near(getPixel(b, 320, 180), "#ff0000")).toBe(true);
+ // Resampling softens the sphere's edge by a pixel at most.
+ expect(computeMse(a.pixels, b.pixels)).toBeLessThan(40);
+ });
+
+ it("aberrates the view towards the direction of motion", async () => {
+ // A marker 70° to the right: at rest, outside a 100° view even counting
+ // its 10.8° angular radius.
+ const angle = (70 * Math.PI) / 180;
+ const side = {
+ id: "side",
+ dir: [Math.sin(angle), 0, Math.cos(angle)],
+ color: "#00ff00",
+ } as unknown as (typeof MARKERS)[number];
+ const render = (speed: number) =>
+ scene(
+ 600,
+ 600,
+ {
+ lens: { fov: 100 },
+ relativity: { speed, doppler: false, beaming: false },
+ },
+ [side],
+ ).renderFrame({ frame: 0 });
+ const rest = await decodePngPixels(await render(0));
+ expect(Number.isNaN(centroidX(rest, "#00ff00"))).toBe(true);
+ // At β = 0.8 it is seen at cos θ' = (cos θ + β) / (1 + β cos θ) ≈ 26.3°.
+ const beta = 0.8;
+ const seen = Math.acos(
+ (Math.cos(angle) + beta) / (1 + beta * Math.cos(angle)),
+ );
+ const fast = await decodePngPixels(await render(beta));
+ const expected =
+ 300 + (300 * Math.tan(seen)) / Math.tan((50 * Math.PI) / 180);
+ expect(centroidX(fast, "#00ff00")).toBeCloseTo(expected, -1);
+ });
+
+ it("shifts light blue ahead and red behind", async () => {
+ const grey = [
+ { id: "ahead", dir: [0, 0, 1], color: "#808080" },
+ { id: "behind", dir: [0, 0, -1], color: "#808080" },
+ ] as unknown as (typeof MARKERS)[number][];
+ const render = (relativity: CameraNode["relativity"]) =>
+ scene(
+ 1024,
+ 512,
+ { lens: { projection: "equirectangular" }, relativity },
+ grey,
+ ).renderFrame({ frame: 0 });
+ const sum = (p: { r: number; g: number; b: number }) => p.r + p.g + p.b;
+ const hue = await decodePngPixels(
+ await render({ speed: 0.3, beaming: false, exposure: 0 }),
+ );
+ const ahead = getPixel(hue, 512, 256);
+ const behind = getPixel(hue, 2, 256);
+ expect(ahead.b).toBeGreaterThan(ahead.r + 20);
+ expect(behind.r).toBeGreaterThan(behind.b + 20);
+ // True brightness: beaming brightens ahead and dims behind.
+ const beamed = await decodePngPixels(
+ await render({ speed: 0.3, exposure: 0 }),
+ );
+ expect(sum(getPixel(beamed, 512, 256))).toBeGreaterThan(sum(ahead));
+ expect(sum(getPixel(beamed, 2, 256))).toBeLessThan(sum(behind));
+ // Auto exposure meters the centre: ahead keeps its brightness at rest.
+ const metered = await decodePngPixels(await render({ speed: 0.3 }));
+ const luma = (p: { r: number; g: number; b: number }) =>
+ 0.2126 * p.r + 0.7152 * p.g + 0.0722 * p.b;
+ expect(luma(getPixel(metered, 512, 256))).toBeCloseTo(128, -1.3);
+ });
+
+ it("animates the speed with relativitySpeed", async () => {
+ const angle = (70 * Math.PI) / 180;
+ const side = {
+ id: "side",
+ dir: [Math.sin(angle), 0, Math.cos(angle)],
+ color: "#00ff00",
+ } as unknown as (typeof MARKERS)[number];
+ const comp = scene(
+ 600,
+ 600,
+ {
+ lens: { fov: 100 },
+ relativity: { doppler: false, beaming: false },
+ animation: CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.8, {
+ start: 0,
+ end: 10,
+ ease: "none",
+ }),
+ },
+ [side],
+ );
+ const rest = await decodePngPixels(await comp.renderFrame({ frame: 0 }));
+ const fast = await decodePngPixels(await comp.renderFrame({ frame: 10 }));
+ expect(Number.isNaN(centroidX(rest, "#00ff00"))).toBe(true);
+ expect(centroidX(fast, "#00ff00")).toBeGreaterThan(300);
+ });
+
+ it("sees behind the camera with a 360° fisheye", async () => {
+ const img = await decodePngPixels(
+ await scene(600, 600, {
+ lens: { projection: "fisheye", fov: 360 },
+ }).renderFrame({ frame: 0 }),
+ );
+ expect(near(getPixel(img, 300, 300), "#ff0000")).toBe(true);
+ // Behind the eye is the rim of the circle; outside it, nothing.
+ expect(near(getPixel(img, 300, 4), "#00ffff")).toBe(true);
+ expect(near(getPixel(img, 300, 596), "#00ffff")).toBe(true);
+ expect(getPixel(img, 10, 10).a).toBe(255);
+ expect(near(getPixel(img, 10, 10), "#000000", 10)).toBe(true);
+ // Right, at 90°, is halfway out.
+ expect(near(getPixel(img, 450, 300), "#00ff00")).toBe(true);
+ });
+});
diff --git a/packages/webgpu-renderers/src/math3d/cube-lens.test.ts b/packages/webgpu-renderers/src/math3d/cube-lens.test.ts
new file mode 100644
index 00000000..5d5e3746
--- /dev/null
+++ b/packages/webgpu-renderers/src/math3d/cube-lens.test.ts
@@ -0,0 +1,96 @@
+import { describe, expect, it } from "vitest";
+import { Camera3D } from "./camera3d.js";
+import {
+ cameraBasis,
+ cubeFaceCameras,
+ cubeTexel,
+ defaultCubeFaceSize,
+} from "./cube-lens.js";
+import type { Vec3 } from "./vector3.js";
+
+/** Clip-space position of world point `p` (column-major matrix). */
+function project(m: number[], p: Vec3): [number, number, number, number] {
+ const out: [number, number, number, number] = [0, 0, 0, 0];
+ for (let r = 0; r < 4; r++) {
+ out[r] = m[r] * p[0] + m[4 + r] * p[1] + m[8 + r] * p[2] + m[12 + r];
+ }
+ return out;
+}
+
+describe("cube-map cameras", () => {
+ const eye: Vec3 = [120, -40, 300];
+ const faces = cubeFaceCameras(eye, 1, 10000);
+
+ it("draws each face where WebGPU samples it", () => {
+ let checked = 0;
+ for (let i = 0; i < 6; i++) {
+ const viewProj = Camera3D.computeViewProjectionMatrix(faces[i]);
+ for (const x of [-0.9, -0.3, 0, 0.45, 0.8]) {
+ for (const y of [-0.85, -0.2, 0, 0.5, 0.95]) {
+ // A world ray through this face's NDC (x, y).
+ const { right, up, forward } = cameraBasis(faces[i]);
+ const dir: Vec3 = [0, 1, 2].map(
+ (k) => forward[k] + x * right[k] + y * up[k],
+ ) as Vec3;
+ const point: Vec3 = [0, 1, 2].map(
+ (k) => eye[k] + dir[k] * 500,
+ ) as Vec3;
+ const clip = project(viewProj, point);
+ const ndcX = clip[0] / clip[3];
+ const ndcY = clip[1] / clip[3];
+ expect(ndcX).toBeCloseTo(x, 5);
+ expect(ndcY).toBeCloseTo(y, 5);
+ const texel = cubeTexel(dir);
+ expect(texel.face).toBe(i);
+ expect(texel.u).toBeCloseTo((ndcX + 1) / 2, 5);
+ expect(texel.v).toBeCloseTo((1 - ndcY) / 2, 5);
+ checked++;
+ }
+ }
+ }
+ expect(checked).toBe(150);
+ });
+
+ it("covers every direction with square 90° faces", () => {
+ for (const face of faces) {
+ expect(face.fov).toBe(90);
+ expect(face.aspect).toBe(1);
+ }
+ const forwards = faces.map((f) => cameraBasis(f).forward);
+ for (let a = 0; a < 6; a++) {
+ for (let b = a + 1; b < 6; b++) {
+ const dot = forwards[a].reduce((s, v, k) => s + v * forwards[b][k], 0);
+ expect([0, -1]).toContain(Math.round(dot));
+ }
+ }
+ });
+
+ it("sizes faces to the lens's density at the centre", () => {
+ const max = 4096;
+ // 1080p height, 90° vertical: 540 px per unit tangent → 1080-wide faces.
+ expect(
+ defaultCubeFaceSize(
+ { projection: "perspective", fov: 90, zoom: 1 },
+ 1920,
+ 1080,
+ max,
+ ),
+ ).toBe(2048);
+ expect(
+ defaultCubeFaceSize(
+ { projection: "equirectangular", fov: 90, zoom: 1 },
+ 2048,
+ 1024,
+ max,
+ ),
+ ).toBe(1024);
+ expect(
+ defaultCubeFaceSize(
+ { projection: "perspective", fov: 10, zoom: 1 },
+ 1920,
+ 1080,
+ max,
+ ),
+ ).toBe(max);
+ });
+});
diff --git a/packages/webgpu-renderers/src/math3d/cube-lens.ts b/packages/webgpu-renderers/src/math3d/cube-lens.ts
new file mode 100644
index 00000000..dae327e1
--- /dev/null
+++ b/packages/webgpu-renderers/src/math3d/cube-lens.ts
@@ -0,0 +1,130 @@
+/**
+ * Cube-map cameras: a scene is drawn as six 90° faces around the eye, then
+ * remapped into any lens, including ones no single perspective frustum can
+ * hold (a full panorama, a fisheye past 180°, a relativistic observer).
+ *
+ * The engine's view matrix (`lookAt`) is right-handed while cube sampling is
+ * left-handed, so every lookup direction goes through `CUBE_FLIP`, a
+ * reflection of z. With it, the face drawn by `cubeFaceCameras()[i]` is exactly
+ * what WebGPU returns for layer `i` of a cube texture.
+ */
+
+import type { Camera3DPose } from "./camera3d.js";
+import { Matrix4Math } from "./matrix4.js";
+import { type Vec3, Vector3Math } from "./vector3.js";
+
+/** The picture a cube camera's lens draws. */
+export type LensProjection = "perspective" | "equirectangular" | "fisheye";
+
+/** The six faces in WebGPU layer order (+X, −X, +Y, −Y, +Z, −Z), in cube space. */
+const CUBE_FACES: readonly { major: Vec3; s: Vec3; t: Vec3 }[] = [
+ { major: [1, 0, 0], s: [0, 0, -1], t: [0, -1, 0] },
+ { major: [-1, 0, 0], s: [0, 0, 1], t: [0, -1, 0] },
+ { major: [0, 1, 0], s: [1, 0, 0], t: [0, 0, 1] },
+ { major: [0, -1, 0], s: [1, 0, 0], t: [0, 0, -1] },
+ { major: [0, 0, 1], s: [1, 0, 0], t: [0, -1, 0] },
+ { major: [0, 0, -1], s: [-1, 0, 0], t: [0, -1, 0] },
+];
+
+/** World direction ↔ cube-space direction (a reflection, its own inverse). */
+export const CUBE_FLIP = (v: Vec3): Vec3 => [v[0], v[1], -v[2]];
+
+/**
+ * The six cameras that draw a cube map around `eye`: 90° square frusta in
+ * layer order. Their faces meet edge to edge, so the cube covers every
+ * direction once.
+ */
+export function cubeFaceCameras(
+ eye: Vec3,
+ near: number,
+ far: number,
+): Camera3DPose[] {
+ return CUBE_FACES.map(({ major, t }) => ({
+ eye: [...eye] as Vec3,
+ target: Vector3Math.add(eye, CUBE_FLIP(major)),
+ up: CUBE_FLIP([-t[0], -t[1], -t[2]]),
+ fov: 90,
+ aspect: 1,
+ near,
+ far,
+ zoom: 1,
+ }));
+}
+
+/**
+ * Where WebGPU samples a cube for world direction `dir`: the layer and its
+ * (u, v), v down. The selection rule of the WebGPU (Vulkan) spec.
+ */
+export function cubeTexel(dir: Vec3): { face: number; u: number; v: number } {
+ const c = CUBE_FLIP(dir);
+ const ax = Math.abs(c[0]);
+ const ay = Math.abs(c[1]);
+ const az = Math.abs(c[2]);
+ const axis = ax >= ay && ax >= az ? 0 : ay >= az ? 1 : 2;
+ const face = axis * 2 + (c[axis] >= 0 ? 0 : 1);
+ const { s, t } = CUBE_FACES[face];
+ const ma = Math.abs(c[axis]);
+ const sc = Vector3Math.dot(s, c);
+ const tc = Vector3Math.dot(t, c);
+ return { face, u: (sc / ma + 1) / 2, v: (tc / ma + 1) / 2 };
+}
+
+/** A camera's orthonormal frame: screen right, screen up and forward. */
+export function cameraBasis(camera: Camera3DPose): {
+ right: Vec3;
+ up: Vec3;
+ forward: Vec3;
+} {
+ const view = Matrix4Math.lookAt(camera.eye, camera.target, camera.up);
+ return {
+ right: [view[0], view[4], view[8]],
+ up: [view[1], view[5], view[9]],
+ forward: [view[2], view[6], view[10]],
+ };
+}
+
+/** What a lens needs to turn an output pixel into a ray. */
+export interface CubeLensOptions {
+ projection: LensProjection;
+ /**
+ * Perspective: vertical field of view. Fisheye: the full angle across the
+ * frame's height (equidistant, up to 360°). Ignored by equirectangular,
+ * which always shows the whole sphere.
+ */
+ fov: number;
+ /** Optical zoom (perspective and fisheye). */
+ zoom: number;
+}
+
+/** Output pixels per radian at the centre of the view. */
+export function lensCenterDensity(
+ lens: CubeLensOptions,
+ width: number,
+ height: number,
+): number {
+ const half = (lens.fov * Math.PI) / 360;
+ switch (lens.projection) {
+ case "perspective":
+ return ((height / 2) * lens.zoom) / Math.tan(half);
+ case "fisheye":
+ return ((height / 2) * lens.zoom) / half;
+ case "equirectangular":
+ return width / (2 * Math.PI);
+ }
+}
+
+/**
+ * Face size that matches the lens's pixel density at the centre of the view:
+ * a face holds `size / 2` pixels per radian at its centre. Rounded up to a
+ * power of two (so a moving lens reuses its targets) and clamped.
+ */
+export function defaultCubeFaceSize(
+ lens: CubeLensOptions,
+ width: number,
+ height: number,
+ maxSize: number,
+): number {
+ const wanted = 2 * lensCenterDensity(lens, width, height);
+ const pow2 = 2 ** Math.ceil(Math.log2(Math.max(1, wanted)));
+ return Math.min(maxSize, Math.max(64, pow2));
+}
diff --git a/packages/webgpu-renderers/src/math3d/index.ts b/packages/webgpu-renderers/src/math3d/index.ts
index ac4aafbc..d9157005 100644
--- a/packages/webgpu-renderers/src/math3d/index.ts
+++ b/packages/webgpu-renderers/src/math3d/index.ts
@@ -1,6 +1,8 @@
export * from "./camera3d.js";
+export * from "./cube-lens.js";
export * from "./environment3d.js";
export * from "./light3d.js";
export * from "./matrix4.js";
export * from "./path3d.js";
+export * from "./relativistic-doppler.js";
export * from "./vector3.js";
diff --git a/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.ts b/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.ts
new file mode 100644
index 00000000..89df9b9d
--- /dev/null
+++ b/packages/webgpu-renderers/src/math3d/relativistic-doppler.test.ts
@@ -0,0 +1,64 @@
+import { describe, expect, it } from "vitest";
+import {
+ aberrate,
+ DOPPLER_LUT_SIZE,
+ dopplerColorMatrix,
+ dopplerLut,
+ lorentzFactor,
+} from "./relativistic-doppler.js";
+
+const apply = (m: number[][], c: number[]) =>
+ m.map((row) => row[0] * c[0] + row[1] * c[1] + row[2] * c[2]);
+
+describe("relativistic observer", () => {
+ it("aberrates rays towards the direction of motion", () => {
+ // A ray seen at cos θ' = β left the scene at 90°.
+ expect(aberrate(0.8, 0.8).sourceCos).toBeCloseTo(0, 10);
+ // Straight ahead and behind are fixed points.
+ expect(aberrate(1, 0.9).sourceCos).toBeCloseTo(1, 10);
+ expect(aberrate(-1, 0.9).sourceCos).toBeCloseTo(-1, 10);
+ // D ahead is √((1 + β) / (1 − β)).
+ expect(aberrate(1, 0.8).doppler).toBeCloseTo(3, 10);
+ expect(aberrate(-1, 0.8).doppler).toBeCloseTo(1 / 3, 10);
+ expect(lorentzFactor(0.99)).toBeCloseTo(7.0888, 3);
+ });
+
+ it("leaves colour untouched at rest", () => {
+ for (const beaming of [true, false]) {
+ const m = dopplerColorMatrix(1, beaming);
+ for (let r = 0; r < 3; r++) {
+ for (let c = 0; c < 3; c++) {
+ expect(m[r][c]).toBeCloseTo(r === c ? 1 : 0, 6);
+ }
+ }
+ }
+ });
+
+ it("shifts ahead to blue and brighter, behind to red and darker", () => {
+ const grey = [0.5, 0.5, 0.5];
+ const ahead = apply(dopplerColorMatrix(1.3), grey);
+ expect(ahead[2]).toBeGreaterThan(ahead[0]);
+ expect(ahead[0] + ahead[1] + ahead[2]).toBeGreaterThan(1.5);
+ const behind = apply(dopplerColorMatrix(1 / 1.3), grey);
+ expect(behind[0]).toBeGreaterThan(behind[2]);
+ expect(behind[0] + behind[1] + behind[2]).toBeLessThan(1.5);
+ // Without beaming only the hue moves; brightness stays near rest.
+ const shiftOnly = apply(dopplerColorMatrix(1.3, false), grey);
+ expect(shiftOnly[0] + shiftOnly[1] + shiftOnly[2]).toBeLessThan(
+ ahead[0] + ahead[1] + ahead[2],
+ );
+ });
+
+ it("tabulates the matrix in ln D, identity in the middle", () => {
+ const lut = dopplerLut(true);
+ expect(lut).toHaveLength(DOPPLER_LUT_SIZE * 12);
+ expect(lut.every(Number.isFinite)).toBe(true);
+ // An even table has no entry at exactly D = 1; its two middle entries
+ // straddle it, so their average is nearly identity.
+ const mid = DOPPLER_LUT_SIZE / 2;
+ const diag = (n: number, r: number) => lut[n * 12 + r * 5];
+ for (let r = 0; r < 3; r++) {
+ expect((diag(mid - 1, r) + diag(mid, r)) / 2).toBeCloseTo(1, 1);
+ }
+ });
+});
diff --git a/packages/webgpu-renderers/src/math3d/relativistic-doppler.ts b/packages/webgpu-renderers/src/math3d/relativistic-doppler.ts
new file mode 100644
index 00000000..1812bb6d
--- /dev/null
+++ b/packages/webgpu-renderers/src/math3d/relativistic-doppler.ts
@@ -0,0 +1,208 @@
+/**
+ * Relativistic observer: what a camera moving at a fraction `β` of the speed
+ * of light sees of a scene at rest.
+ *
+ * Aberration: a ray arriving at angle θ' from the direction of motion left
+ * the scene at θ, with cos θ = (cos θ' − β) / (1 − β cos θ'). The Doppler
+ * factor D = γ (1 + β cos θ') scales every frequency, and spectral radiance
+ * transforms as L'(λ') = D⁵ L(D λ') (the "searchlight": ahead is brighter).
+ *
+ * A pixel only stores a colour, so its spectrum is modelled: the colour is a
+ * reflectance (three smooth bands that sum to one, held flat past the visible
+ * range, so red surfaces stay red into the infrared) lit by daylight (a
+ * 5778 K black body). Shifting that spectrum and integrating it against the
+ * CIE 1931 observer gives the colour seen. `dopplerColorMatrix(D)` is the
+ * resulting linear map on linear sRGB, exactly the identity at D = 1.
+ */
+
+/** Wavelength range integrated by the observer, nm. */
+const LAMBDA_MIN = 360;
+const LAMBDA_MAX = 830;
+const LAMBDA_STEP = 1;
+/** Daylight: the Sun's effective temperature, K. */
+const DAYLIGHT_KELVIN = 5778;
+/** Second radiation constant hc/k in nm·K. */
+const C2 = 1.4387769e7;
+
+/** Piecewise Gaussian used by the CIE fit. */
+function lobe(l: number, mu: number, below: number, above: number): number {
+ const t = (l - mu) / (l < mu ? below : above);
+ return Math.exp(-0.5 * t * t);
+}
+
+/**
+ * CIE 1931 2° colour matching functions, multi-lobe fit of Wyman, Sloan and
+ * Shirley (JCGT 2013).
+ */
+function cieXyz(l: number): [number, number, number] {
+ return [
+ 1.056 * lobe(l, 599.8, 37.9, 31.0) +
+ 0.362 * lobe(l, 442.0, 16.0, 26.7) -
+ 0.065 * lobe(l, 501.1, 20.4, 26.2),
+ 0.821 * lobe(l, 568.8, 46.9, 40.5) + 0.286 * lobe(l, 530.9, 16.3, 31.1),
+ 1.217 * lobe(l, 437.0, 11.8, 36.0) + 0.681 * lobe(l, 459.0, 26.0, 13.8),
+ ];
+}
+
+/** XYZ → linear sRGB (D65). */
+const XYZ_TO_RGB = [
+ [3.2406, -1.5372, -0.4986],
+ [-0.9689, 1.8758, 0.0415],
+ [0.0557, -0.204, 1.057],
+] as const;
+
+function smoothstep(a: number, b: number, x: number): number {
+ const t = Math.min(1, Math.max(0, (x - a) / (b - a)));
+ return t * t * (3 - 2 * t);
+}
+
+/** The blue, green and red reflectance bands at wavelength `l` (sum to 1). */
+function bands(l: number): [number, number, number] {
+ const blueToGreen = smoothstep(440, 510, l);
+ const greenToRed = smoothstep(570, 630, l);
+ return [1 - blueToGreen, blueToGreen * (1 - greenToRed), greenToRed];
+}
+
+/** Black-body spectral radiance shape at wavelength `l` nm (unnormalised). */
+function planck(l: number, kelvin: number): number {
+ const x = C2 / (l * kelvin);
+ // Far in the ultraviolet the exponential overflows: there is no light.
+ if (x > 700) return 0;
+ return 1 / (l ** 5 * Math.expm1(x));
+}
+
+/**
+ * K[k][i]: channel k of linear sRGB seen from band i at Doppler factor `d`,
+ * with the D⁵ searchlight factor when `beaming`.
+ */
+function bandResponse(d: number, beaming: boolean): number[][] {
+ const k = [
+ [0, 0, 0],
+ [0, 0, 0],
+ [0, 0, 0],
+ ];
+ const scale = beaming ? d ** 5 : 1;
+ for (let l = LAMBDA_MIN; l <= LAMBDA_MAX; l += LAMBDA_STEP) {
+ const xyz = cieXyz(l);
+ const source = l * d;
+ const b = bands(source);
+ const e = planck(source, DAYLIGHT_KELVIN) * scale;
+ for (let c = 0; c < 3; c++) {
+ const bar =
+ XYZ_TO_RGB[c][0] * xyz[0] +
+ XYZ_TO_RGB[c][1] * xyz[1] +
+ XYZ_TO_RGB[c][2] * xyz[2];
+ for (let i = 0; i < 3; i++) k[c][i] += bar * b[i] * e;
+ }
+ }
+ return k;
+}
+
+function invert3(m: number[][]): number[][] {
+ const [[a, b, c], [d, e, f], [g, h, i]] = m;
+ const A = e * i - f * h;
+ const B = -(d * i - f * g);
+ const C = d * h - e * g;
+ const det = a * A + b * B + c * C;
+ if (Math.abs(det) < 1e-12) throw new Error("Singular colour basis");
+ const s = 1 / det;
+ return [
+ [A * s, -(b * i - c * h) * s, (b * f - c * e) * s],
+ [B * s, (a * i - c * g) * s, -(a * f - c * d) * s],
+ [C * s, -(a * h - b * g) * s, (a * e - b * d) * s],
+ ];
+}
+
+function multiply3(a: number[][], b: number[][]): number[][] {
+ return a.map((row) =>
+ [0, 1, 2].map(
+ (j) => row[0] * b[0][j] + row[1] * b[1][j] + row[2] * b[2][j],
+ ),
+ );
+}
+
+let restFrame: { white: number[]; toBands: number[][] } | null = null;
+
+/** The rest-frame response, white-balanced so white reflectance is white. */
+function rest(): { white: number[]; toBands: number[][] } {
+ if (restFrame) return restFrame;
+ const k = bandResponse(1, false);
+ const white = k.map((row) => row[0] + row[1] + row[2]);
+ const balanced = k.map((row, c) => row.map((v) => v / white[c]));
+ restFrame = { white, toBands: invert3(balanced) };
+ return restFrame;
+}
+
+/**
+ * The linear-sRGB matrix (row-major 3×3) that turns a colour at rest into the
+ * colour seen at Doppler factor `d` (> 1 towards the direction of motion).
+ */
+export function dopplerColorMatrix(d: number, beaming = true): number[][] {
+ const { white, toBands } = rest();
+ const k = bandResponse(d, beaming).map((row, c) =>
+ row.map((v) => v / white[c]),
+ );
+ return multiply3(k, toBands);
+}
+
+/** Largest Doppler factor the lookup table covers (β ≈ 0.99995). */
+export const DOPPLER_LUT_MAX = 200;
+/** Entries in the lookup table, evenly spaced in ln D. */
+export const DOPPLER_LUT_SIZE = 256;
+
+const luts = new Map<boolean, Float32Array>();
+
+/**
+ * `dopplerColorMatrix` sampled at DOPPLER_LUT_SIZE factors evenly spaced in
+ * ln D over [1/DOPPLER_LUT_MAX, DOPPLER_LUT_MAX]: three vec4 rows per entry.
+ */
+export function dopplerLut(beaming: boolean): Float32Array {
+ const cached = luts.get(beaming);
+ if (cached) return cached;
+ const data = new Float32Array(DOPPLER_LUT_SIZE * 12);
+ const span = Math.log(DOPPLER_LUT_MAX);
+ for (let n = 0; n < DOPPLER_LUT_SIZE; n++) {
+ const lnD = -span + (2 * span * n) / (DOPPLER_LUT_SIZE - 1);
+ const m = dopplerColorMatrix(Math.exp(lnD), beaming);
+ for (let r = 0; r < 3; r++) data.set(m[r], n * 12 + r * 4);
+ }
+ luts.set(beaming, data);
+ return data;
+}
+
+/** Rec. 709 luminance of a linear-sRGB colour. */
+const luminance = (c: number[]) =>
+ 0.2126 * c[0] + 0.7152 * c[1] + 0.0722 * c[2];
+
+/**
+ * How much brighter a neutral surface looks at Doppler factor `d`: the gain
+ * an exposure meter would compensate for.
+ */
+export function dopplerLuminanceGain(
+ d: number,
+ doppler: boolean,
+ beaming: boolean,
+): number {
+ if (!doppler) return beaming ? d ** 4 : 1;
+ const m = dopplerColorMatrix(d, beaming);
+ return luminance(m.map((row) => row[0] + row[1] + row[2]));
+}
+
+/** Lorentz factor of speed β. */
+export function lorentzFactor(beta: number): number {
+ return 1 / Math.sqrt(1 - beta * beta);
+}
+
+/**
+ * Where a ray seen at angle `observedCos` (cosine to the direction of motion)
+ * left the scene at rest, and its Doppler factor.
+ */
+export function aberrate(
+ observedCos: number,
+ beta: number,
+): { sourceCos: number; doppler: number } {
+ return {
+ sourceCos: (observedCos - beta) / (1 - beta * observedCos),
+ doppler: lorentzFactor(beta) * (1 + beta * observedCos),
+ };
+}
diff --git a/packages/webgpu-renderers/src/renderer3d/index.ts b/packages/webgpu-renderers/src/renderer3d/index.ts
index 041d021c..b7fafa41 100644
--- a/packages/webgpu-renderers/src/renderer3d/index.ts
+++ b/packages/webgpu-renderers/src/renderer3d/index.ts
@@ -2,6 +2,7 @@ export * from "./audio-mesh-deform-pipeline.js";
export * from "./bloom-pipeline.js";
export * from "./floor-reflection-pipeline.js";
export * from "./glass-pipeline.js";
+export * from "./lens-remap-pipeline.js";
export * from "./matte3d.js";
export * from "./mesh3d-pipeline.js";
export * from "./motion-blur-pipeline.js";
diff --git a/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.ts
new file mode 100644
index 00000000..dba74336
--- /dev/null
+++ b/packages/webgpu-renderers/src/renderer3d/lens-remap-pipeline.ts
@@ -0,0 +1,586 @@
+/**
+ * Lens remap: turns a cube map drawn around the eye (`cubeFaceCameras`) into
+ * the picture of a lens. Every output pixel becomes a ray (perspective,
+ * equirectangular or equidistant fisheye), optionally seen by an observer
+ * moving near the speed of light: the ray is aberrated back to the direction
+ * it left the scene, and its colour Doppler-shifted (`relativistic-doppler`).
+ *
+ * The cube is mipmapped first, so a lens or a fast observer that squeezes
+ * much of the scene into few pixels filters it instead of aliasing.
+ */
+
+import type { LensProjection } from "../math3d/cube-lens.js";
+import {
+ DOPPLER_LUT_MAX,
+ DOPPLER_LUT_SIZE,
+ dopplerLuminanceGain,
+ dopplerLut,
+ lorentzFactor,
+} from "../math3d/relativistic-doppler.js";
+import type { Vec3 } from "../math3d/vector3.js";
+import { BufferPool } from "../renderer2d/buffer-pool.js";
+
+/** An observer moving at `speed` (a fraction of c) through a scene at rest. */
+export interface RelativisticObserver {
+ /** β = v / c, 0 ≤ β < 1. */
+ speed: number;
+ /** Unit direction of motion in world space. */
+ direction: Vec3;
+ /** Shift colours along the spectrum (blue ahead, red behind). */
+ doppler: boolean;
+ /** Brighten ahead and dim behind (relativistic beaming). */
+ beaming: boolean;
+ /**
+ * Exposure for the shifted light: stops, or "auto" to meter the centre of
+ * the view, so what the camera looks at keeps its brightness at rest.
+ */
+ exposure: number | "auto";
+}
+
+export interface LensRemapOptions {
+ projection: LensProjection;
+ /** Degrees; see `CubeLensOptions.fov`. */
+ fov: number;
+ zoom: number;
+ /** The camera's frame: screen right, screen up, forward (world space). */
+ basis: { right: Vec3; up: Vec3; forward: Vec3 };
+ observer?: RelativisticObserver | null;
+}
+
+const PROJECTION_INDEX: Record<LensProjection, number> = {
+ perspective: 0,
+ equirectangular: 1,
+ fisheye: 2,
+};
+
+/** How the observer changes colour: none, spectral table, or brightness only. */
+const ColorMode = { None: 0, Spectral: 1, Beaming: 2 } as const;
+
+const mipWgsl = /* wgsl */ `
+@group(0) @binding(0) var samp : sampler;
+@group(0) @binding(1) var src : texture_2d<f32>;
+
+struct VsOut {
+ @builtin(position) pos : vec4<f32>,
+ @location(0) uv : vec2<f32>,
+};
+
+@vertex
+fn vs(@builtin(vertex_index) i : u32) -> VsOut {
+ let p = vec2<f32>(f32((i << 1u) & 2u), f32(i & 2u));
+ var o : VsOut;
+ o.pos = vec4<f32>(p * 2.0 - 1.0, 0.0, 1.0);
+ o.uv = vec2<f32>(p.x, 1.0 - p.y);
+ return o;
+}
+
+// One bilinear tap between four texels of the level above: a 2x2 box.
+@fragment
+fn fs(in : VsOut) -> @location(0) vec4<f32> {
+ return textureSampleLevel(src, samp, in.uv, 0.0);
+}
+`;
+
+const remapWgsl = /* wgsl */ `
+struct Lens {
+ right : vec4<f32>, // xyz screen right, w projection (0 persp, 1 equirect, 2 fisheye)
+ up : vec4<f32>, // xyz screen up, w scale (persp: tan(fov/2)/zoom, fisheye: fov/2/zoom)
+ forward : vec4<f32>, // xyz forward, w aspect (width / height)
+ motion : vec4<f32>, // xyz direction of motion, w β
+ params : vec4<f32>, // x color mode, y exposure gain, z γ, w ln(max D)
+};
+
+@group(0) @binding(0) var<uniform> lens : Lens;
+@group(0) @binding(1) var samp : sampler;
+@group(0) @binding(2) var colorCube : texture_cube<f32>;
+@group(0) @binding(3) var hdrCube : texture_cube<f32>;
+@group(0) @binding(4) var<storage, read> dopplerLut : array<vec4<f32>>;
+
+const PI = 3.14159265358979;
+const LUT_SIZE = ${DOPPLER_LUT_SIZE}.0;
+
+struct VsOut {
+ @builtin(position) pos : vec4<f32>,
+ @location(0) ndc : vec2<f32>,
+};
+
+@vertex
+fn vs(@builtin(vertex_index) i : u32) -> VsOut {
+ let p = vec2<f32>(f32((i << 1u) & 2u), f32(i & 2u));
+ var o : VsOut;
+ o.pos = vec4<f32>(p * 2.0 - 1.0, 0.0, 1.0);
+ o.ndc = p * 2.0 - 1.0;
+ return o;
+}
+
+struct Ray {
+ dir : vec3<f32>,
+ inside : f32,
+};
+
+// The ray an output pixel sees, in world space.
+fn lensRay(ndc : vec2<f32>) -> Ray {
+ let r = lens.right.xyz;
+ let u = lens.up.xyz;
+ let f = lens.forward.xyz;
+ let scale = lens.up.w;
+ let aspect = lens.forward.w;
+ let projection = lens.right.w;
+ var out : Ray;
+ out.inside = 1.0;
+ if (projection < 0.5) {
+ out.dir = normalize(f + ndc.x * aspect * scale * r + ndc.y * scale * u);
+ } else if (projection < 1.5) {
+ let lon = ndc.x * PI;
+ let lat = ndc.y * 0.5 * PI;
+ out.dir = cos(lat) * (sin(lon) * r + cos(lon) * f) + sin(lat) * u;
+ } else {
+ let q = vec2<f32>(ndc.x * aspect, ndc.y);
+ let rho = length(q);
+ let theta = rho * scale;
+ let side = select(vec2<f32>(0.0), q / rho, rho > 1e-6);
+ out.dir = cos(theta) * f + sin(theta) * (side.x * r + side.y * u);
+ out.inside = select(0.0, 1.0, theta <= PI);
+ }
+ return out;
+}
+
+struct Seen {
+ dir : vec3<f32>,
+ doppler : f32,
+};
+
+// Aberration: where a ray seen by the moving observer left the scene.
+fn aberrate(observed : vec3<f32>) -> Seen {
+ let beta = lens.motion.w;
+ let n = lens.motion.xyz;
+ let c = clamp(dot(observed, n), -1.0, 1.0);
+ let cs = (c - beta) / (1.0 - beta * c);
+ let side = observed - c * n;
+ let len = length(side);
+ // Exactly ahead or behind, any perpendicular will do: sin θ is 0.
+ let fallback = normalize(cross(n, select(vec3<f32>(1.0, 0.0, 0.0), vec3<f32>(0.0, 1.0, 0.0), abs(n.x) > 0.9)));
+ let perp = select(fallback, side / len, len > 1e-6);
+ var out : Seen;
+ out.dir = cs * n + sqrt(max(0.0, 1.0 - cs * cs)) * perp;
+ out.doppler = lens.params.z * (1.0 + beta * c);
+ return out;
+}
+
+// Linear-sRGB matrix for Doppler factor d, interpolated in ln d.
+fn dopplerMatrix(d : f32) -> mat3x3<f32> {
+ let span = lens.params.w;
+ let t = clamp((log(d) + span) / (2.0 * span), 0.0, 1.0) * (LUT_SIZE - 1.0);
+ let i0 = u32(floor(t));
+ let i1 = min(i0 + 1u, u32(LUT_SIZE) - 1u);
+ let w = t - f32(i0);
+ let r0 = mix(dopplerLut[i0 * 3u], dopplerLut[i1 * 3u], w).xyz;
+ let r1 = mix(dopplerLut[i0 * 3u + 1u], dopplerLut[i1 * 3u + 1u], w).xyz;
+ let r2 = mix(dopplerLut[i0 * 3u + 2u], dopplerLut[i1 * 3u + 2u], w).xyz;
+ // Rows in the table; WGSL matrices take columns.
+ return transpose(mat3x3<f32>(r0, r1, r2));
+}
+
+// The observer's light from scene light \`c\` (linear rgb) at Doppler factor d.
+fn shiftLight(c : vec3<f32>, d : f32) -> vec3<f32> {
+ let mode = lens.params.x;
+ var out = c;
+ if (mode > 1.5) {
+ out = c * pow(d, 4.0);
+ } else if (mode > 0.5) {
+ out = dopplerMatrix(d) * c;
+ }
+ return max(out * lens.params.y, vec3<f32>(0.0));
+}
+
+fn srgbToLinear(c : vec3<f32>) -> vec3<f32> {
+ let lo = c / 12.92;
+ let hi = pow((max(c, vec3<f32>(0.0)) + 0.055) / 1.055, vec3<f32>(2.4));
+ return select(hi, lo, c <= vec3<f32>(0.04045));
+}
+
+fn linearToSrgb(c : vec3<f32>) -> vec3<f32> {
+ let x = clamp(c, vec3<f32>(0.0), vec3<f32>(1.0));
+ let lo = x * 12.92;
+ let hi = 1.055 * pow(x, vec3<f32>(1.0 / 2.4)) - 0.055;
+ return select(hi, lo, x <= vec3<f32>(0.0031308));
+}
+
+struct Sampled {
+ color : vec4<f32>,
+ hdr : vec4<f32>,
+};
+
+fn sampleLens(ndc : vec2<f32>) -> Sampled {
+ let ray = lensRay(ndc);
+ var seen : Seen;
+ seen.dir = ray.dir;
+ seen.doppler = 1.0;
+ if (lens.motion.w > 0.0) {
+ seen = aberrate(ray.dir);
+ }
+ // The engine's frames are right-handed, cube lookups left-handed.
+ let lookup = vec3<f32>(seen.dir.xy, -seen.dir.z);
+ var color = textureSample(colorCube, samp, lookup);
+ var hdr = textureSample(hdrCube, samp, lookup);
+ if (lens.params.x > 0.5) {
+ // Premultiplied display colour: shift its light, then re-encode.
+ let a = color.a;
+ let lin = srgbToLinear(color.rgb / max(a, 1e-4));
+ color = vec4<f32>(linearToSrgb(shiftLight(lin, seen.doppler)) * a, a);
+ // Radiance is linear and premultiplied: the shift applies directly.
+ hdr = vec4<f32>(shiftLight(hdr.rgb, seen.doppler), hdr.a);
+ }
+ var out : Sampled;
+ out.color = color * ray.inside;
+ out.hdr = hdr * ray.inside;
+ return out;
+}
+
+@fragment
+fn fs_color(in : VsOut) -> @location(0) vec4<f32> {
+ return sampleLens(in.ndc).color;
+}
+
+struct HdrOut {
+ @location(0) color : vec4<f32>,
+ @location(1) hdr : vec4<f32>,
+};
+
+@fragment
+fn fs_color_hdr(in : VsOut) -> HdrOut {
+ let s = sampleLens(in.ndc);
+ var out : HdrOut;
+ out.color = s.color;
+ out.hdr = s.hdr;
+ return out;
+}
+`;
+
+const HDR_FORMAT: GPUTextureFormat = "rgba16float";
+
+export class LensRemapPipeline {
+ private device: GPUDevice;
+ private colorFormat: GPUTextureFormat;
+ private mipModule: GPUShaderModule;
+ private mipLayout: GPUBindGroupLayout;
+ private mipPipes = new Map<GPUTextureFormat, GPURenderPipeline>();
+ private remapLayout: GPUBindGroupLayout;
+ private remapModule: GPUShaderModule;
+ private remapPipe: GPURenderPipeline | null = null;
+ private remapHdrPipe: GPURenderPipeline | null = null;
+ private mipSampler: GPUSampler;
+ private cubeSampler: GPUSampler;
+ private placeholderCube: GPUTexture;
+ private luts = new Map<boolean, GPUBuffer>();
+ private uniformPool = new BufferPool(
+ 80,
+ GPUBufferUsage.UNIFORM | GPUBufferUsage.COPY_DST,
+ );
+ private uniformData = new Float32Array(20);
+
+ constructor(device: GPUDevice, colorFormat: GPUTextureFormat) {
+ this.device = device;
+ this.colorFormat = colorFormat;
+ this.mipModule = device.createShaderModule({
+ label: "lens-mip.wgsl",
+ code: mipWgsl,
+ });
+ this.remapModule = device.createShaderModule({
+ label: "lens-remap.wgsl",
+ code: remapWgsl,
+ });
+ this.mipLayout = device.createBindGroupLayout({
+ label: "LensMipLayout",
+ entries: [
+ {
+ binding: 0,
+ visibility: GPUShaderStage.FRAGMENT,
+ sampler: { type: "filtering" },
+ },
+ {
+ binding: 1,
+ visibility: GPUShaderStage.FRAGMENT,
+ texture: { sampleType: "float" },
+ },
+ ],
+ });
+ this.remapLayout = device.createBindGroupLayout({
+ label: "LensRemapLayout",
+ entries: [
+ {
+ binding: 0,
+ visibility: GPUShaderStage.FRAGMENT,
+ buffer: { type: "uniform" },
+ },
+ {
+ binding: 1,
+ visibility: GPUShaderStage.FRAGMENT,
+ sampler: { type: "filtering" },
+ },
+ {
+ binding: 2,
+ visibility: GPUShaderStage.FRAGMENT,
+ texture: { sampleType: "float", viewDimension: "cube" },
+ },
+ {
+ binding: 3,
+ visibility: GPUShaderStage.FRAGMENT,
+ texture: { sampleType: "float", viewDimension: "cube" },
+ },
+ {
+ binding: 4,
+ visibility: GPUShaderStage.FRAGMENT,
+ buffer: { type: "read-only-storage" },
+ },
+ ],
+ });
+ this.mipSampler = device.createSampler({
+ label: "LensMipSampler",
+ magFilter: "linear",
+ minFilter: "linear",
+ });
+ this.cubeSampler = device.createSampler({
+ label: "LensCubeSampler",
+ magFilter: "linear",
+ minFilter: "linear",
+ mipmapFilter: "linear",
+ maxAnisotropy: 16,
+ });
+ this.placeholderCube = device.createTexture({
+ label: "LensPlaceholderCube",
+ size: { width: 1, height: 1, depthOrArrayLayers: 6 },
+ format: HDR_FORMAT,
+ usage: GPUTextureUsage.TEXTURE_BINDING,
+ });
+ }
+
+ /** Mip levels a cube face of `size` gets. */
+ static mipLevels(size: number): number {
+ return Math.floor(Math.log2(Math.max(1, size))) + 1;
+ }
+
+ /** Fills every mip level of every layer of `texture` from level 0. */
+ generateMips(encoder: GPUCommandEncoder, texture: GPUTexture): void {
+ const pipe = this.mipPipeline(texture.format);
+ for (let layer = 0; layer < texture.depthOrArrayLayers; layer++) {
+ for (let level = 1; level < texture.mipLevelCount; level++) {
+ const view = (mip: number) =>
+ texture.createView({
+ dimension: "2d",
+ baseArrayLayer: layer,
+ arrayLayerCount: 1,
+ baseMipLevel: mip,
+ mipLevelCount: 1,
+ });
+ const pass = encoder.beginRenderPass({
+ label: "LensMip",
+ colorAttachments: [
+ { view: view(level), loadOp: "clear", storeOp: "store" },
+ ],
+ });
+ pass.setPipeline(pipe);
+ pass.setBindGroup(
+ 0,
+ this.device.createBindGroup({
+ layout: this.mipLayout,
+ entries: [
+ { binding: 0, resource: this.mipSampler },
+ { binding: 1, resource: view(level - 1) },
+ ],
+ }),
+ );
+ pass.draw(3);
+ pass.end();
+ }
+ }
+ }
+
+ /**
+ * Draws the lens's picture of `color` (and of `hdr`, the scene radiance,
+ * into `destHdr` when both are given). Generates the cubes' mips first.
+ */
+ remap(
+ encoder: GPUCommandEncoder,
+ color: GPUTexture,
+ hdr: GPUTexture | null,
+ dest: GPUTextureView,
+ destHdr: GPUTextureView | null,
+ width: number,
+ height: number,
+ options: LensRemapOptions,
+ ): void {
+ this.generateMips(encoder, color);
+ if (hdr) this.generateMips(encoder, hdr);
+ const withHdr = Boolean(hdr && destHdr);
+ const pass = encoder.beginRenderPass({
+ label: "LensRemap",
+ colorAttachments: [
+ { view: dest, loadOp: "clear", storeOp: "store" },
+ ...(withHdr && destHdr
+ ? [
+ {
+ view: destHdr,
+ loadOp: "clear" as const,
+ storeOp: "store" as const,
+ },
+ ]
+ : []),
+ ],
+ });
+ pass.setPipeline(this.remapPipeline(withHdr));
+ pass.setBindGroup(
+ 0,
+ this.device.createBindGroup({
+ layout: this.remapLayout,
+ entries: [
+ {
+ binding: 0,
+ resource: {
+ buffer: this.uniformPool.getBuffer(
+ this.device,
+ this.packUniforms(options, width, height),
+ ),
+ },
+ },
+ { binding: 1, resource: this.cubeSampler },
+ { binding: 2, resource: color.createView({ dimension: "cube" }) },
+ {
+ binding: 3,
+ resource: (hdr ?? this.placeholderCube).createView({
+ dimension: "cube",
+ }),
+ },
+ {
+ binding: 4,
+ resource: { buffer: this.lut(options.observer?.beaming ?? true) },
+ },
+ ],
+ }),
+ );
+ pass.draw(3);
+ pass.end();
+ }
+
+ private packUniforms(
+ options: LensRemapOptions,
+ width: number,
+ height: number,
+ ): Float32Array {
+ const u = this.uniformData;
+ const { right, up, forward } = options.basis;
+ const half = (options.fov * Math.PI) / 360;
+ const scale =
+ options.projection === "fisheye"
+ ? half / options.zoom
+ : Math.tan(half) / options.zoom;
+ u.set([...right, PROJECTION_INDEX[options.projection]], 0);
+ u.set([...up, scale], 4);
+ u.set([...forward, width / Math.max(1, height)], 8);
+ const observer = options.observer;
+ const beta = observer ? Math.min(Math.max(observer.speed, 0), 0.99999) : 0;
+ const direction = observer?.direction ?? forward;
+ u.set([...direction, beta], 12);
+ u.set(
+ [
+ observer && beta > 0 ? colorMode(observer) : ColorMode.None,
+ observer ? exposureGain(observer, beta, forward) : 1,
+ lorentzFactor(beta),
+ Math.log(DOPPLER_LUT_MAX),
+ ],
+ 16,
+ );
+ return u;
+ }
+
+ private lut(beaming: boolean): GPUBuffer {
+ let buffer = this.luts.get(beaming);
+ if (!buffer) {
+ const data = dopplerLut(beaming);
+ buffer = this.device.createBuffer({
+ label: `DopplerLut${beaming ? "Beaming" : ""}`,
+ size: data.byteLength,
+ usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST,
+ });
+ this.device.queue.writeBuffer(buffer, 0, data);
+ this.luts.set(beaming, buffer);
+ }
+ return buffer;
+ }
+
+ private mipPipeline(format: GPUTextureFormat): GPURenderPipeline {
+ let pipe = this.mipPipes.get(format);
+ if (!pipe) {
+ pipe = this.device.createRenderPipeline({
+ label: `LensMip:${format}`,
+ layout: this.device.createPipelineLayout({
+ bindGroupLayouts: [this.mipLayout],
+ }),
+ vertex: { module: this.mipModule, entryPoint: "vs" },
+ fragment: {
+ module: this.mipModule,
+ entryPoint: "fs",
+ targets: [{ format }],
+ },
+ primitive: { topology: "triangle-list" },
+ });
+ this.mipPipes.set(format, pipe);
+ }
+ return pipe;
+ }
+
+ private remapPipeline(withHdr: boolean): GPURenderPipeline {
+ const cached = withHdr ? this.remapHdrPipe : this.remapPipe;
+ if (cached) return cached;
+ const pipe = this.device.createRenderPipeline({
+ label: withHdr ? "LensRemapHdr" : "LensRemap",
+ layout: this.device.createPipelineLayout({
+ bindGroupLayouts: [this.remapLayout],
+ }),
+ vertex: { module: this.remapModule, entryPoint: "vs" },
+ fragment: {
+ module: this.remapModule,
+ entryPoint: withHdr ? "fs_color_hdr" : "fs_color",
+ targets: withHdr
+ ? [{ format: this.colorFormat }, { format: HDR_FORMAT }]
+ : [{ format: this.colorFormat }],
+ },
+ primitive: { topology: "triangle-list" },
+ });
+ if (withHdr) this.remapHdrPipe = pipe;
+ else this.remapPipe = pipe;
+ return pipe;
+ }
+
+ /** Call once per frame: recycles the per-pass uniforms. */
+ resetPools(): void {
+ this.uniformPool.reset();
+ }
+
+ destroy(): void {
+ this.uniformPool.destroy();
+ this.placeholderCube.destroy();
+ for (const buffer of this.luts.values()) buffer.destroy();
+ this.luts.clear();
+ }
+}
+
+/** The gain on the shifted light: metered at the centre of the view, or fixed stops. */
+function exposureGain(
+ observer: RelativisticObserver,
+ beta: number,
+ forward: Vec3,
+): number {
+ if (observer.exposure !== "auto") return 2 ** observer.exposure;
+ const c =
+ forward[0] * observer.direction[0] +
+ forward[1] * observer.direction[1] +
+ forward[2] * observer.direction[2];
+ const d = lorentzFactor(beta) * (1 + beta * c);
+ return 1 / dopplerLuminanceGain(d, observer.doppler, observer.beaming);
+}
+
+function colorMode(observer: RelativisticObserver): number {
+ if (observer.doppler) return ColorMode.Spectral;
+ return observer.beaming ? ColorMode.Beaming : ColorMode.None;
+}
diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
index 3f28422f..cfb3484d 100644
--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
@@ -54,6 +54,10 @@ import { FloorReflectionPipeline } from "./floor-reflection-pipeline.js";
import { GeometryOutlinePipeline } from "./geometry-outline-pipeline.js";
import { GeometryResolvePipeline } from "./geometry-resolve-pipeline.js";
import { decodeKtx2, isKtx2 } from "./ktx2.js";
+import {
+ type LensRemapOptions,
+ LensRemapPipeline,
+} from "./lens-remap-pipeline.js";
import {
type GPUMeshBuffers,
Mesh3DPipeline,
@@ -298,7 +302,8 @@ export class Renderer3D {
>();
/** Parsed models by source and load options: a model file is read once, not every frame. */
private modelCache = new Map<string, Model3DData>();
- private lastCameraPose: Camera3DPose | null = null;
+ /** What the cached camera buffer was packed from (pose, lens, surface). */
+ private lastCameraKey: string | null = null;
private shadowTexture: GPUTexture | null = null;
private shadowTextures = new Map<string, GPUTexture>();
private placeholderShadowTexture: GPUTexture | null = null;
@@ -308,6 +313,7 @@ export class Renderer3D {
private outlinePipeline: GeometryOutlinePipeline | null = null;
private geometryResolvePipeline: GeometryResolvePipeline | null = null;
private lastCameraBuffer: GPUBuffer | null = null;
+ private lensPipeline: LensRemapPipeline | null = null;
/**
* Samples per pixel in the 3D pass. Mesh silhouettes (extruded text,
@@ -438,16 +444,23 @@ export class Renderer3D {
label: string,
width: number,
height: number,
- desc: { format: GPUTextureFormat; usage: number; sampleCount?: number },
+ desc: {
+ format: GPUTextureFormat;
+ usage: number;
+ sampleCount?: number;
+ layers?: number;
+ mipLevelCount?: number;
+ },
): GPUTexture {
- const key = `${label}:${width}x${height}:${desc.sampleCount ?? 1}:${desc.usage}`;
+ const { layers = 1, ...texture2d } = desc;
+ const key = `${label}:${width}x${height}x${layers}:${desc.sampleCount ?? 1}:${desc.mipLevelCount ?? 1}:${desc.format}:${desc.usage}`;
this.attachmentsUsed.add(key);
let texture = this.attachments.get(key);
if (!texture) {
texture = this.device.createTexture({
label,
- size: { width, height },
- ...desc,
+ size: { width, height, depthOrArrayLayers: layers },
+ ...texture2d,
});
this.attachments.set(key, texture);
}
@@ -463,11 +476,12 @@ export class Renderer3D {
this.attachmentsUsed.clear();
this.outlinePipeline?.reset();
this.bloomPipeline?.resetPools();
+ this.lensPipeline?.resetPools();
this.cameraPool.reset();
// The cached camera buffer goes back to the pool and may be handed
// out again (e.g. to a shadow pass's light camera): never reuse it
// across a reset.
- this.lastCameraPose = null;
+ this.lastCameraKey = null;
this.lastCameraBuffer = null;
this.lightsPool.reset();
this.directLightsPool.reset();
@@ -515,6 +529,67 @@ export class Renderer3D {
return this.hdrTexture;
}
+ /**
+ * A cube camera's target: six `size`² faces with a full mip chain, drawn
+ * into face by face and sampled by `remapLens`. `hdr` holds the scene
+ * radiance (the bloom source) instead of display colour.
+ */
+ getOrCreateCubeTexture(size: number, hdr: boolean): GPUTexture {
+ return this.attachment(
+ hdr ? "Renderer3DCubeHdr" : "Renderer3DCube",
+ size,
+ size,
+ {
+ format: hdr ? "rgba16float" : this.format,
+ usage:
+ GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
+ layers: 6,
+ mipLevelCount: LensRemapPipeline.mipLevels(size),
+ },
+ );
+ }
+
+ /** Face `face` (WebGPU layer order) of a cube target, as a render attachment. */
+ cubeFaceView(cube: GPUTexture, face: number): GPUTextureView {
+ return cube.createView({
+ dimension: "2d",
+ baseArrayLayer: face,
+ arrayLayerCount: 1,
+ baseMipLevel: 0,
+ mipLevelCount: 1,
+ });
+ }
+
+ /**
+ * Draws a cube camera's picture into `dest` (`width` × `height`) through
+ * its lens. With an `hdr` cube the radiance is remapped too and becomes
+ * `hdrSource()`, so bloom glows where the lens put the light.
+ */
+ remapLens(
+ encoder: GPUCommandEncoder,
+ color: GPUTexture,
+ hdr: GPUTexture | null,
+ dest: GPUTextureView,
+ width: number,
+ height: number,
+ options: LensRemapOptions,
+ ): void {
+ this.lensPipeline ??= new LensRemapPipeline(this.device, this.format);
+ const destHdr = hdr
+ ? this.getOrCreateHdrTexture(width, height).createView()
+ : null;
+ this.lensPipeline.remap(
+ encoder,
+ color,
+ hdr,
+ dest,
+ destHdr,
+ width,
+ height,
+ options,
+ );
+ }
+
private getOrCreateHdrTexture(width: number, height: number): GPUTexture {
this.hdrTexture = this.attachment("Renderer3DHdrRadiance", width, height, {
format: "rgba16float",
@@ -622,23 +697,25 @@ export class Renderer3D {
surfaceHeight: number,
time = 0,
): GPUBuffer {
- // Skip recomputation if camera pose hasn't changed
- if (
- this.lastCameraPose &&
- this.lastCameraBuffer &&
- this.lastCameraPose.eye[0] === camera.eye[0] &&
- this.lastCameraPose.eye[1] === camera.eye[1] &&
- this.lastCameraPose.eye[2] === camera.eye[2] &&
- this.lastCameraPose.target[0] === camera.target[0] &&
- this.lastCameraPose.target[1] === camera.target[1] &&
- this.lastCameraPose.target[2] === camera.target[2] &&
- this.lastCameraPose.up[0] === camera.up[0] &&
- this.lastCameraPose.up[1] === camera.up[1] &&
- this.lastCameraPose.up[2] === camera.up[2] &&
- this.lastCameraPose.fov === camera.fov &&
- this.lastCameraPose.near === camera.near &&
- this.lastCameraPose.far === camera.far
- ) {
+ // Every packed field: a face of a cube camera differs from the main
+ // view in aspect and surface size alone.
+ const key = [
+ ...camera.eye,
+ ...camera.target,
+ ...camera.up,
+ camera.fov,
+ camera.aspect,
+ camera.near,
+ camera.far,
+ camera.zoom,
+ camera.focusDistance,
+ camera.fStop,
+ camera.maxBlurRadius,
+ surfaceWidth,
+ surfaceHeight,
+ time,
+ ].join(",");
+ if (this.lastCameraBuffer && this.lastCameraKey === key) {
return this.lastCameraBuffer;
}
@@ -687,7 +764,7 @@ export class Renderer3D {
this.device,
this.cameraUniformData,
);
- this.lastCameraPose = camera;
+ this.lastCameraKey = key;
this.lastCameraBuffer = buffer;
return buffer;
}
@@ -954,6 +1031,8 @@ export class Renderer3D {
world?: Scene3DWorld,
/** Also write unclamped scene radiance (`hdrTexture()`), the bloom source. */
writeHdr = false,
+ /** Where the radiance resolves instead of `hdrSource()` (a cube face). */
+ hdrTargetView?: GPUTextureView,
): {
pass: GPURenderPassEncoder;
cameraBindGroup: GPUBindGroup;
@@ -1056,7 +1135,9 @@ export class Renderer3D {
this.passWritesHdr = writeHdr;
if (writeHdr) {
- this.hdrTexture = this.getOrCreateHdrTexture(surfaceWidth, surfaceHeight);
+ const hdrView =
+ hdrTargetView ??
+ this.getOrCreateHdrTexture(surfaceWidth, surfaceHeight).createView();
if (msaa) {
this.msaaHdrTexture = this.getOrCreateMsaaTexture(
this.msaaHdrTexture,
@@ -1068,7 +1149,7 @@ export class Renderer3D {
}
colorAttachments.push(
attach(
- this.hdrTexture.createView(),
+ hdrView,
msaa ? this.msaaHdrTexture : null,
{ r: 0, g: 0, b: 0, a: 0 },
"clear",
@@ -2101,6 +2182,7 @@ export class Renderer3D {
for (const texture of this.attachments.values()) texture.destroy();
this.attachments.clear();
this.bloomPipeline?.destroy();
+ this.lensPipeline?.destroy();
this.outlinePipeline?.destroy();
this.depthTexture?.destroy();
for (const texture of this.shadowTextures.values()) texture.destroy();
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index 000d3558..33574062 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -244,6 +244,20 @@ const camAnim = CameraAnimation.camera()
});
```
+**Cube cameras.** `lens.projection` picks the picture: `"perspective"` (default), `"equirectangular"` (the whole sphere around the eye, 360 × 180, for VR/360 video; `fov` is ignored), or `"fisheye"` (equidistant; `fov` is the full angle across the frame's height, up to 360°, so it can see behind the camera). Anything but perspective draws the scene as six 90° cube faces around the eye and remaps them, with mipmaps, so it costs about six renders. `lens.cubeSize` overrides the face size (default: matches the output's sharpness at the centre of the view, up to 4096).
+
+**Relativity.** `relativity: { speed, direction?, doppler?, beaming?, exposure? }` films the scene as seen by a camera moving at `speed` × c (0 to 0.9999) along `direction` (world space, default where the camera looks). It is physics, not grading: rays crowd towards the direction of motion (aberration; objects at the side swing forward and look turned, not squashed), colours shift blue ahead and red behind (spectral Doppler, each colour modelled as a reflectance lit by daylight) and ahead brightens with beaming. `exposure: "auto"` (default) meters the centre of the view so what the camera looks at keeps its brightness; a number is fixed stops, where `0` is the true, often blinding, light. Animate the speed with the `relativitySpeed` prop, e.g. `CameraAnimation.camera().fromTo("relativitySpeed", 0, 0.99, { start: 0, end: 90, ease: "expo.in" })`. A camera with `relativity` is always a cube camera, even at speed 0, and at rest matches the ordinary camera. Works with every projection.
+
+```typescript
+Layer.camera({
+ x: 0, y: -170, z: 0, targetX: 0, targetY: -170, targetZ: 1000,
+ lens: { fov: 90 },
+ relativity: { speed: 0.9 },
+});
+```
+
+Cube cameras draw outlines, shadows, floor reflections and bloom, but not depth of field (`comp.lint()` reports `camera_cube_dof`), and glass refracts only the 3D scene, not the 2D layers below.
+
### 4. Procedural & Audio-Reactive Drivers
Properties can be bound to Perlin wiggles, harmonic springs, or audio-extracted signals:
```typescript