Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Render output: transparent PNG, 16-bit PNG and EXR, render region, 60 fps (issue #611)

Nguyen Xuan An2a00a7b39662
diff --git a/changes/611-render-output.md b/changes/611-render-output.md
new file mode 100644
index 00000000..f3862cd4
--- /dev/null
+++ b/changes/611-render-output.md
@@ -0,0 +1 @@
+- Render jobs gain four optional output settings, all off by default so a plain job writes the same bytes: `transparent: true` (PNG with alpha, for `image` and `stills`), `format: "png16"` or `"exr"` (16-bit PNG and OpenEXR from a half-float frame, passes included: depth in metres as float32 `Z`, object IDs as float32 `id`), `region: { x, y, width, height }` (a crop of the whole frame, for `image`, `stills` and `video`) and `fps` up to 60. The encoders are small hand-written Node code with no new dependency. `docs/agent-render-jobs.md` has the details.
diff --git a/changes/611-render-output.size.md b/changes/611-render-output.size.md
new file mode 100644
index 00000000..b288b24d
--- /dev/null
+++ b/changes/611-render-output.size.md
@@ -0,0 +1,6 @@
+- entry:render-passes: the pass target can be half float (173 bytes gzip), which 16-bit PNG and EXR passes read back instead of 8-bit bytes.
+- scene:agent-scene: the preview page carries the transparent flag, the half-float output target and its readback (about 400 bytes gzip), the compositing output of issue 611.
+- scene:agent-scene+lazy: the same 400 bytes, nothing else lazy changed.
+- goal:lit-box: 10 bytes gzip for one `frame.outputFramebuffer ?? null` in the final pass, the only core change that lets a render job draw the finished frame into a float target; there is no addon seam for it.
+- goal:gltf-box: the same 11 bytes in the core renderer.
+- goal:agent-scene: the same 411 bytes, a visible capability (alpha, 16-bit and EXR render output) for agent jobs.
diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.md
index 58b5142e..2bce116e 100644
--- a/docs/agent-render-jobs.md
+++ b/docs/agent-render-jobs.md
@@ -125,6 +125,23 @@ Videos hold the lit image only.
A seek to any frame gives the same pixels: a job samples each frame from its time and keeps no state between frames.
`npm run smoke:render-passes` proves it through MCP: it renders eight stills with every pass, then renders frame 3 again as an image in a different order of seeks and compares the bytes of the lit image and all five passes.
+## Output options: transparent, 16-bit PNG, EXR, region and 60 fps
+
+Four optional settings, each off by default. A job without them writes the same bytes as before.
+
+| Setting | Meaning |
+| --- | --- |
+| `transparent: true` | `image` and `stills`: the lit image keeps alpha (a PNG of colour type 6). The sky, the environment background and the automatic ground are not drawn, so give the scene its own ground or a `shadowCatcher` node (see `renderFlags` in `scene_capabilities`). Volume fog is refused. Passes ignore it. |
+| `format` | `image` and `stills`: `png` (8-bit, default), `png16` or `exr`. The lit image is drawn into a half-float target and read back, so the file holds the same graded display colour as the 8-bit PNG (within one 8-bit step) with 16-bit steps. |
+| `region: { x, y, width, height }` | Pixels from the top left, each side at least 16. The frame is rendered whole and the crop is kept, so the look is the same as the full frame. Works for `image`, `stills` and `video` (a video region has even sides). Not with `renderer`. |
+| `fps` | Up to 60 for video, within the 900 frame limit. |
+
+`png16` writes `render.png` (RGB, or RGBA with `transparent`; alpha is straight) and one PNG per pass. A depth pass is a 16-bit gray PNG holding the raw step 0 to 65535 over 100 m, an `objectId` pass is a 16-bit gray PNG of the ID (a scene with more than 65535 objects fails: use `exr`), and `albedo`, `normal` and `emission` are 16-bit RGB with the same values as the 8-bit pass.
+`exr` writes `render.exr` (half-float R, G, B and A when transparent; alpha stays premultiplied) and `render-<pass>.exr`. `depth` is one float32 channel `Z` in metres (the far limit, 100, is background), `objectId` one float32 channel `id` (0 is background), and `albedo`, `normal` and `emission` are half-float RGB. Files use ZIP compression and open in Blender, Nuke and OpenEXR tools.
+Passes in these formats are drawn into half-float targets, so albedo, normal and emission carry half-float precision, not the 8 bits of the default passes.
+The path tracer writes an 8-bit PNG of the whole frame: `transparent`, `format` and `region` are refused with `renderer`. H.264 has no alpha, so `transparent` and `format` are refused for video. 16-bit and EXR output need `EXT_color_buffer_float`; without it the job fails with a message that says to use `png`.
+A `png16` or `exr` frame is up to four times the 8-bit frame limit (48 MiB). A region with the browser H.264 encoder (`encoder: "nvenc"`) uses screenshot frames instead.
+
## Path tracer
An `image`, `stills` or `video` job can ask for the offline [path tracer](https://github.com/codepawl/67.js/blob/main/docs/path-tracer.md) with `settings.renderer`:
@@ -278,11 +295,11 @@ This conservative repair step prevents two recovery processes from deleting a ne
| Retained records | 32 |
| Total store bytes | 512 MiB |
| Combined output bytes per job | 128 MiB |
-| PNG artifact bytes | 12 MiB each (a 1920x1080 frame) |
+| PNG artifact bytes | 12 MiB each (a 1920x1080 frame); 48 MiB each for `png16` and `exr` |
| Snapshot JSON | 1 MiB |
| Asset bytes | 64 MiB |
| Dimensions | 64 to 1920 pixels each, at most 1920x1080 pixels in all (1080x1920 fits) |
-| Video frame rate | 1 to 30 fps |
+| Video frame rate | 1 to 60 fps |
| Duration | 0.1 to 60 seconds |
| Maximum video frames | 900 |
| Still frames | 8 |
diff --git a/docs/learn/agent-workflows.md b/docs/learn/agent-workflows.md
index 5aeb63d2..0e515e39 100644
--- a/docs/learn/agent-workflows.md
+++ b/docs/learn/agent-workflows.md
@@ -395,6 +395,8 @@ The result call returns the path of the file in the jobs folder. The agent reads
}
```
+For compositing, add `"transparent": true` (PNG with alpha), `"format": "png16"` or `"exr"`, or a `"region": {"x": 0, "y": 0, "width": 640, "height": 360}` crop to an image or stills job; a video can run at up to 60 fps. Each is off unless set.
+
While a job runs, previews and edits pause. Inspection and recipe export still work. After the job, call `scene_render_release` to delete files that you no longer need. [Render jobs](/agent-render-jobs.md) covers sequences, video, render passes and the path tracer.
## Skills
diff --git a/scripts/agent-preview-browser.js b/scripts/agent-preview-browser.js
index 1fae93e1..222edeff 100644
--- a/scripts/agent-preview-browser.js
+++ b/scripts/agent-preview-browser.js
@@ -24,6 +24,7 @@ import { hasPatternTextures } from './agent-preview-textures.js';
import { TEXTURE_SEQUENCE_LIMITS } from '../src/texture-sequence-limits.js';
import { registerCatalogEnvironment, registerCatalogLook } from '../src/look-catalog.js';
import { parseRadianceHdr } from '../src/hdri.js';
+import { createFloatOutput, readFloatPixels } from '../src/float-output.js';
import { parseCube } from '../src/view-transform.js';
import { weightTracksFromGltfAnimation } from '../src/export-skeleton.js';
import { clipFromGltfAnimation } from '../src/skeleton-pose.js';
@@ -83,6 +84,16 @@ function candidateCamera(record, operations) {
return createAgentScene({ format: '67js.scene', schemaVersion: 1, geometries: [], materials: [], nodes: [], camera }).camera;
}
+// The half-float target a floatOutput frame draws into; rebuilt when the renderer or the canvas size changes.
+function floatOutputFor(record) {
+ const cached = record.floatOutput;
+ if (cached && cached.renderer === record.renderer && cached.output.width === canvas.width && cached.output.height === canvas.height) return cached.output;
+ cached?.output.dispose();
+ const output = createFloatOutput(record.renderer.context, canvas.width, canvas.height);
+ record.floatOutput = { renderer: record.renderer, output };
+ return output;
+}
+
function renderInfo(renderer) {
const context = renderer.context;
const debug = context.getExtension('WEBGL_debug_renderer_info');
@@ -347,7 +358,7 @@ function cascadesFor(scene) {
return count > 0 ? agentSceneModule('cascades').createCascadedShadows({ count }) : null;
}
-function initializeRenderer(scene, { memoryBudgetMiB, backend }) {
+function initializeRenderer(scene, { memoryBudgetMiB, backend, transparent = false }) {
probeSnippets(scene);
const placement = agentGroundPlacement(scene.parts, scene.environment);
const renderer = installDepthPrecision(installRenderPasses(installCameraEffects(new Renderer67(canvas, scene.parts, {
@@ -363,6 +374,7 @@ function initializeRenderer(scene, { memoryBudgetMiB, backend }) {
pixelRatio: 1,
samples: backend === 'cpu' ? 1 : 4,
memoryBudgetMiB,
+ transparent,
}))));
// Outlines, toon light and polylines draw only for a frame that carries environment.lines; user passes run last, on the finished frame.
// Mesh blend installs before line art, so its stage runs first and the lines draw over the blended colour; it blends only the
@@ -388,15 +400,15 @@ function previousCameraFor(record, motionTime) {
return previousCamera;
}
-function createPreview(scene, { sessionId, memoryBudgetMiB, backend }, asset, placedAssets, sequenceFrames) {
+function createPreview(scene, { sessionId, memoryBudgetMiB, backend, transparent = false }, asset, placedAssets, sequenceFrames) {
if (active) {
throw new Error('Dispose the current scene before creating another.');
}
validatePreviewCamera(scene.camera);
let renderer;
try {
- renderer = initializeRenderer(scene, { memoryBudgetMiB, backend });
- const record = { scene, renderer, asset, placedAssets, sessionId, options: { memoryBudgetMiB, backend },
+ renderer = initializeRenderer(scene, { memoryBudgetMiB, backend, transparent });
+ const record = { scene, renderer, asset, placedAssets, sessionId, options: { memoryBudgetMiB, backend, transparent },
checkpoints: new Map(), branches: createBranchStore(), rendererTopology: scene.topologyRevision, rendererTextured: sceneIsTextured(scene), sequenceFrames };
draw(record);
const result = status(record);
@@ -740,7 +752,7 @@ window.__agentPreview = {
},
// `pass` draws one render pass instead of the lit image. Object ID frames also list the part behind each ID.
// `camera` overrides camera fields and `depthOfField` the blur and focus distance (a timeline shot); `brightness` scales exposure for a dip through black.
- frame({ sessionId, animation, motionTime, rendererTime = 0, pass, passDepthRange, camera: cameraOverrides, depthOfField: depthOfFieldOverrides, brightness = 1 }) {
+ frame({ sessionId, animation, motionTime, rendererTime = 0, pass, passDepthRange, passFloat = false, floatOutput = false, camera: cameraOverrides, depthOfField: depthOfFieldOverrides, brightness = 1 }) {
const record = requireSession(sessionId);
if (!Number.isFinite(brightness) || brightness < 0 || brightness > 1) throw new RangeError('Invalid frame brightness.');
if (!Number.isFinite(rendererTime) || rendererTime < 0 || rendererTime > 3600) throw new RangeError('Invalid render time.');
@@ -757,11 +769,21 @@ window.__agentPreview = {
const shotEffects = depthOfFieldOverrides
? { ...record.scene.effects, depthOfField: { ...record.scene.effects.depthOfField, ...depthOfFieldOverrides } }
: undefined;
- draw(record, rendererTime, shotCamera, { pass, passDepthRange, previousCamera,
+ const outputFramebuffer = floatOutput && !pass ? floatOutputFor(record).framebuffer : undefined;
+ draw(record, rendererTime, shotCamera, { pass, passDepthRange, passFloat, previousCamera, outputFramebuffer,
...(brightness === 1 ? {} : { exposure }), ...(shotEffects ? { effects: shotEffects } : {}) });
const info = renderInfo(record.renderer);
return pass === 'objectId' ? { ...info, objectNames: record.renderer.passSupport.objectNames() } : info;
},
+ // Reads the last frame as RGBA float32, rows bottom to top, cropped to `region` (pixels from the top left): the lit image
+ // when it was drawn with floatOutput, a pass when it was drawn with passFloat. Base64 of the float bytes.
+ readFloat({ sessionId, source, region }) {
+ const record = requireSession(sessionId);
+ const context = record.renderer.context;
+ const framebuffer = source === 'pass' ? record.renderer.passSupport.framebuffer : floatOutputFor(record).framebuffer;
+ const pixels = readFloatPixels(context, framebuffer, canvas.width, canvas.height, region);
+ return { width: region?.width ?? canvas.width, height: region?.height ?? canvas.height, data: bytesToBase64(new Uint8Array(pixels.buffer)) };
+ },
// Offline path trace of the scene at one instant. The tracer loads here, on first use, and not with the preview.
async pathTrace({ sessionId, animation, motionTime, options }) {
const record = requireSession(sessionId);
diff --git a/scripts/agent-render-jobs.mjs b/scripts/agent-render-jobs.mjs
index d8e65c8c..53b62827 100644
--- a/scripts/agent-render-jobs.mjs
+++ b/scripts/agent-render-jobs.mjs
@@ -3,7 +3,7 @@ import { mkdir, open, readFile, writeFile, rename, unlink, rmdir, readdir, lstat
import { randomUUID, createHash } from 'node:crypto';
import { isAbsolute, join, resolve } from 'node:path';
-import { RENDER_PASS_NAMES, VIDEO_ENCODERS, AUDIO_FILE_PATTERN, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS } from '../src/agent-scene-capabilities.js';
+import { RENDER_PASS_NAMES, RENDER_OUTPUT_FORMATS, MAXIMUM_JOB_FPS, VIDEO_ENCODERS, AUDIO_FILE_PATTERN, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS } from '../src/agent-scene-capabilities.js';
import { recipeAssetNeeds } from '../src/agent-scene-assets.js';
import { normalizeFilmTimeline, filmTimelineDuration } from '../src/film-timeline.js';
import { OIDN_DEFAULT_SAMPLES, PATH_TRACER_DENOISERS, PATH_TRACER_JOB_LIMITS } from '../src/path-tracer-capabilities.js';
@@ -33,6 +33,9 @@ export function jobDeadlineMs({ frameCount, width, height, renderer }, maximumMs
const traceDeadline = renderer?.type === 'pathTracer' ? Math.ceil(renderer.maxSeconds * 1000 * frameCount) + PATH_TRACE_OVERHEAD_MS : 0;
return Math.min(maximumMs, Math.max(frameDeadline, traceDeadline));
}
+const MINIMUM_REGION_SIDE = 16;
+/** A 16-bit PNG or an EXR of a busy 1920x1080 frame is bigger than the 8-bit PNG the frame limit was sized for. */
+export const HIGH_PRECISION_FRAME_FACTOR = 4;
const MAXIMUM_JSON_BYTES = 1024 ** 2;
const MAXIMUM_ASSET_BYTES = AGENT_ASSET_LIMITS.maxBytes;
/** Sibling folder of the job directories where the runtime keeps GLB exports; the store never reads or counts it. */
@@ -82,13 +85,13 @@ function jsonCopy(value, maximum = MAXIMUM_JSON_BYTES) {
}
function settingsFrom(input, clips, recipe) {
- objectKeys(input, ['kind', 'width', 'height', 'fps', 'duration', 'animation', 'motion', 'rendererTime', 'passes', 'renderer', 'timeline', 'audio', 'encoder'], 'settings');
+ objectKeys(input, ['kind', 'width', 'height', 'fps', 'duration', 'animation', 'motion', 'rendererTime', 'passes', 'renderer', 'timeline', 'audio', 'encoder', 'transparent', 'format', 'region'], 'settings');
if (!['image', 'stills', 'video'].includes(input.kind)) throw new TypeError('kind must be image, stills or video.');
const width = boundedNumber(input.width, 64, MAXIMUM_JOB_SIDE, 'width', true);
const height = boundedNumber(input.height, 64, MAXIMUM_JOB_SIDE, 'height', true);
if (width * height > MAXIMUM_JOB_PIXELS) throw new RangeError('width x height must not exceed 1920x1080 pixels.');
if (input.kind === 'video' && (width % 2 || height % 2)) throw new RangeError('Video dimensions must be even.');
- const fps = boundedNumber(input.fps ?? 24, 1, 30, 'fps', true);
+ const fps = boundedNumber(input.fps ?? 24, 1, MAXIMUM_JOB_FPS, 'fps', true);
const timelineStart = timelineStartFrom(input, recipe);
const defaultDuration = timelineStart === undefined || input.kind === 'image' ? 1 : filmTimelineDuration(recipe.timeline) - timelineStart;
const duration = boundedNumber(input.duration ?? defaultDuration, 0.1, MAXIMUM_VIDEO_SECONDS, 'duration');
@@ -97,6 +100,7 @@ function settingsFrom(input, clips, recipe) {
const result = { kind: input.kind, width, height, fps, duration, frameCount };
if (input.passes !== undefined) result.passes = passesFrom(input.passes, input.kind);
if (input.renderer !== undefined) result.renderer = rendererFrom(input.renderer, input.passes);
+ Object.assign(result, outputOptionsFrom(input, result));
const lastSampleTime = input.kind === 'image' ? 0 : input.kind === 'stills' ? duration : (frameCount - 1) / fps;
if (timelineStart !== undefined) {
if (input.motion !== undefined || input.animation !== undefined) throw new TypeError('A timeline owns the motion clock; leave motion and animation out.');
@@ -146,6 +150,38 @@ function settingsFrom(input, clips, recipe) {
return result;
}
+/** The optional transparent, format and region settings, kept only when they differ from the defaults so old records read the same. */
+function outputOptionsFrom(input, { kind, width, height, renderer }) {
+ const options = {};
+ const tracerNote = 'The path tracer writes an opaque 8-bit PNG of the whole frame.';
+ if (input.transparent !== undefined) {
+ if (typeof input.transparent !== 'boolean') throw new TypeError('transparent must be boolean.');
+ if (input.transparent && kind === 'video') throw new RangeError('H.264 video has no alpha: transparent needs kind image or stills.');
+ if (input.transparent && renderer) throw new RangeError(`transparent cannot combine with renderer. ${tracerNote}`);
+ if (input.transparent) options.transparent = true;
+ }
+ if (input.format !== undefined) {
+ if (!RENDER_OUTPUT_FORMATS.includes(input.format)) throw new TypeError(`format must be one of ${RENDER_OUTPUT_FORMATS.join(', ')}.`);
+ if (input.format !== 'png' && kind === 'video') throw new RangeError('A video is H.264: format needs kind image or stills.');
+ if (input.format !== 'png' && renderer) throw new RangeError(`format cannot combine with renderer. ${tracerNote}`);
+ if (input.format !== 'png') options.format = input.format;
+ }
+ if (input.region !== undefined) {
+ objectKeys(input.region, ['x', 'y', 'width', 'height'], 'region');
+ if (renderer) throw new RangeError(`region cannot combine with renderer. ${tracerNote}`);
+ const region = {
+ x: boundedNumber(input.region.x, 0, width - 1, 'region x', true),
+ y: boundedNumber(input.region.y, 0, height - 1, 'region y', true),
+ width: boundedNumber(input.region.width, MINIMUM_REGION_SIDE, width, 'region width', true),
+ height: boundedNumber(input.region.height, MINIMUM_REGION_SIDE, height, 'region height', true),
+ };
+ if (region.x + region.width > width || region.y + region.height > height) throw new RangeError('region must lie inside the frame.');
+ if (kind === 'video' && (region.width % 2 || region.height % 2)) throw new RangeError('A video region must have even width and height.');
+ options.region = region;
+ }
+ return options;
+}
+
const PATH_TRACER_BACKENDS = ['cpu', 'gpu', 'native'];
const FRACTIONAL_RENDERER_FIELDS = new Set(['maxSeconds', 'noiseThreshold']);
@@ -195,15 +231,26 @@ function passesFrom(passes, kind) {
}
/** Every artifact filename of a job, lit image first and then each requested pass, frame by frame. */
-export function artifactNames(kind, passes = []) {
+export function artifactNames(kind, passes = [], format = 'png') {
if (kind === 'video') return ['render.mp4'];
+ const extension = format === 'exr' ? 'exr' : 'png';
const stems = kind === 'image' ? ['render'] : Array.from({ length: 8 }, (_, index) => `frame-${index}`);
- return stems.flatMap(stem => [`${stem}.png`, ...passes.map(pass => `${stem}-${pass}.png`)]);
+ return stems.flatMap(stem => [`${stem}.${extension}`, ...passes.map(pass => `${stem}-${pass}.${extension}`)]);
+}
+
+function artifactMimeType(settings) {
+ if (settings.kind === 'video') return 'video/mp4';
+ return settings.format === 'exr' ? 'image/x-exr' : 'image/png';
+}
+
+function artifactByteLimit(settings, limits) {
+ if (settings.kind === 'video') return limits.maxOutputBytes;
+ return settings.format === undefined ? limits.maxFrameBytes : limits.maxFrameBytes * HIGH_PRECISION_FRAME_FACTOR;
}
function allowedFile(name) {
const passNames = RENDER_PASS_NAMES.join('|');
- const artifact = new RegExp(`^(?:render\\.(?:png|mp4)|(?:render|frame-[0-7])(?:-(?:${passNames}))?\\.png)(?:\\.partial)?$`);
+ const artifact = new RegExp(`^(?:render\\.(?:png|mp4|exr)|(?:render|frame-[0-7])(?:-(?:${passNames}))?\\.(?:png|exr))(?:\\.partial)?$`);
return INTERNAL_FILES.has(name) || artifact.test(name);
}
@@ -415,15 +462,15 @@ export async function createAgentRenderJobs({ root, run, limits: inputLimits = {
boundedNumber(record.progress.maximumPendingFrames, 0, 1, 'persisted pending frames', true);
if (record.metrics !== undefined) record.metrics = jsonCopy(record.metrics, 65536);
if (record.reason !== undefined && (typeof record.reason !== 'string' || record.reason.length > 2048)) throw new Error('Invalid persisted job reason.');
- const namesAllowed = artifactNames(settings.kind, settings.passes);
+ const namesAllowed = artifactNames(settings.kind, settings.passes, settings.format);
if (!Array.isArray(record.artifacts) || record.artifacts.length > namesAllowed.length) throw new Error('Invalid persisted artifacts.');
const seen = new Set();
for (const artifact of record.artifacts) {
objectKeys(artifact, ['filename', 'mimeType', 'bytes', 'sha256'], 'artifact');
if (!namesAllowed.includes(artifact.filename) || seen.has(artifact.filename)) throw new Error('Invalid persisted artifact name.');
seen.add(artifact.filename);
- const actual = await hashFile(join(directory, artifact.filename), settings.kind === 'video' ? limits.maxOutputBytes : limits.maxFrameBytes);
- if (artifact.mimeType !== (settings.kind === 'video' ? 'video/mp4' : 'image/png') || artifact.bytes !== actual.bytes || artifact.sha256 !== actual.sha256) throw new Error('Persisted artifact integrity failed.');
+ const actual = await hashFile(join(directory, artifact.filename), artifactByteLimit(settings, limits));
+ if (artifact.mimeType !== artifactMimeType(settings) || artifact.bytes !== actual.bytes || artifact.sha256 !== actual.sha256) throw new Error('Persisted artifact integrity failed.');
}
if (record.state === 'completed' && record.artifacts.length !== namesAllowed.length) throw new Error('Completed job is missing its artifacts.');
const job = { record, persistence: Promise.resolve(), settled: Promise.resolve() };
@@ -479,7 +526,7 @@ export async function createAgentRenderJobs({ root, run, limits: inputLimits = {
record.state = 'finalizing';
await persist(job);
objectKeys(completed, ['artifacts', 'metrics'], 'runner result');
- const expected = artifactNames(record.settings.kind, record.settings.passes);
+ const expected = artifactNames(record.settings.kind, record.settings.passes, record.settings.format);
if (!Array.isArray(completed.artifacts) || completed.artifacts.length !== expected.length) throw new Error('Runner did not return the required fixed artifacts.');
const directory = await directoryFor(record.jobId);
const metrics = jsonCopy(completed.metrics ?? {}, 65536);
@@ -488,9 +535,9 @@ export async function createAgentRenderJobs({ root, run, limits: inputLimits = {
let outputBytes = 0;
for (const output of completed.artifacts) {
objectKeys(output, ['temporary', 'filename', 'mimeType'], 'runner artifact');
- if (!expected.includes(output.filename) || seen.has(output.filename) || output.temporary !== `${output.filename}.partial` || output.mimeType !== (record.settings.kind === 'video' ? 'video/mp4' : 'image/png')) throw new Error('Runner artifact must use the fixed filename and MIME type.');
+ if (!expected.includes(output.filename) || seen.has(output.filename) || output.temporary !== `${output.filename}.partial` || output.mimeType !== artifactMimeType(record.settings)) throw new Error('Runner artifact must use the fixed filename and MIME type.');
seen.add(output.filename);
- const artifact = { filename: output.filename, mimeType: output.mimeType, ...await hashFile(join(directory, output.temporary), record.settings.kind === 'video' ? limits.maxOutputBytes : limits.maxFrameBytes) };
+ const artifact = { filename: output.filename, mimeType: output.mimeType, ...await hashFile(join(directory, output.temporary), artifactByteLimit(record.settings, limits)) };
outputBytes += artifact.bytes;
if (outputBytes > limits.maxOutputBytes) throw new RangeError('Job exceeds its combined output byte budget.');
artifacts.push(artifact);
diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjs
index 189caede..cf636f19 100644
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@ -22,7 +22,8 @@ import { recipeAssetNeeds } from '../src/agent-scene-assets.js';
import { sequenceFrameIds } from '../src/agent-texture-sequence.js';
import { catalogLookReferences, BUNDLED_ENVIRONMENT_NAMES, BUNDLED_CUBE_LOOK_NAMES } from '../src/look-catalog.js';
import { TEXTURE_SEQUENCE_CAPABILITIES } from '../src/texture-sequence-limits.js';
-import { createAgentRenderJobs, EXPORT_DIRECTORY_NAME } from './agent-render-jobs.mjs';
+import { createAgentRenderJobs, EXPORT_DIRECTORY_NAME, HIGH_PRECISION_FRAME_FACTOR } from './agent-render-jobs.mjs';
+import { encodeRenderOutput } from './render-output-encoders.mjs';
import { createFrameEncoder } from './frame-encoder.mjs';
import { DEFAULT_PAGE_ENCODER_QUANTIZER, createFrameReceiver, startVideoEncoderInPage } from './gpu-frame-reader.mjs';
import { audioInputArguments, createAudioLibrary, measureStreamSeconds } from './job-audio.mjs';
@@ -426,7 +427,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
jobPage.setDefaultTimeout(15000);
await jobPage.goto(origin);
await jobPage.waitForFunction(() => Boolean(window.__agentPreview));
- const input = { sessionId: jobId, recipe: snapshot.recipe, memoryBudgetMiB: memory, backend,
+ const input = { sessionId: jobId, recipe: snapshot.recipe, memoryBudgetMiB: memory, backend, ...(settings.transparent ? { transparent: true } : {}),
...(snapshot.assetBytes ? { data: snapshot.assetBytes.toString('base64'), ...(await motionSources([snapshot.recipe.motion])) } : await placedAssetData(snapshot.recipe)) };
const created = await jobPage.evaluate(({ method, input }) => window.__agentPreview[method](input),
{ method: snapshot.assetBytes ? 'createAsset' : 'create', input });
@@ -435,6 +436,25 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
if (width < 64 || height < 64) throw new Error('Render budget reduced the job below its minimum output size.');
await jobPage.setViewportSize({ width, height });
const session = await jobPage.context().newCDPSession(jobPage);
+ const region = settings.region;
+ // A memory budget can shrink the frame below the size the region was chosen for.
+ if (region && (region.x + region.width > width || region.y + region.height > height)) {
+ throw new Error(`The region lies outside the rendered frame, which the memory budget reduced to ${width}x${height}.`);
+ }
+ const outputWidth = region?.width ?? width;
+ const outputHeight = region?.height ?? height;
+ const format = settings.format ?? 'png';
+ const highPrecision = format !== 'png';
+ const artifactExtension = format === 'exr' ? 'exr' : 'png';
+ const artifactMimeType = format === 'exr' ? 'image/x-exr' : 'image/png';
+ // A transparent canvas shows the page behind it in a screenshot, so the page and the browser's default are made clear.
+ if (settings.transparent) {
+ await session.send('Emulation.setDefaultBackgroundColorOverride', { color: { r: 0, g: 0, b: 0, a: 0 } });
+ await jobPage.evaluate(() => {
+ document.documentElement.style.background = 'transparent';
+ document.body.style.background = 'transparent';
+ });
+ }
const artifacts = [];
const timeline = settings.timeline ? snapshot.recipe.timeline : undefined;
const audio = settings.audio ? await prepareAudio(settings) : undefined;
@@ -445,13 +465,13 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
activeFrameReceiver = frameReceiver;
const frameSocketUrl = `ws://127.0.0.1:${httpServer.address().port}${frameReceiver.path}`;
// A path traced frame is a PNG made outside the page's drawing buffer, so the browser encoder has nothing to encode.
- if (settings.encoder === 'nvenc' && !settings.renderer) {
+ if (settings.encoder === 'nvenc' && !settings.renderer && !region) {
const pageEncoder = await startVideoEncoderInPage(jobPage, frameSocketUrl,
{ width, height, fps: settings.fps, quantizer: DEFAULT_PAGE_ENCODER_QUANTIZER });
pageEncodes = pageEncoder.supported;
}
encoder = createFrameEncoder({ executable: ffmpeg, output: join(directory, 'render.mp4.partial'),
- fps: settings.fps, format: pageEncodes ? 'h264' : settings.renderer ? 'png' : 'jpeg', width, height, signal, maximumFrameBytes: MAXIMUM_JOB_FRAME_BYTES, backend: settings.encoder === 'nvenc' ? 'gpu' : 'cpu',
+ fps: settings.fps, format: pageEncodes ? 'h264' : settings.renderer ? 'png' : 'jpeg', width: outputWidth, height: outputHeight, signal, maximumFrameBytes: MAXIMUM_JOB_FRAME_BYTES, backend: settings.encoder === 'nvenc' ? 'gpu' : 'cpu',
audioArguments: settings.audio ? audioInputArguments(audio.path, settings.audio.offset) : [],
audioCodec: settings.audio?.track === 'pcm' ? 'pcm' : 'aac' });
if (pageEncodes) frameReceiver.onChunk(chunk => encoder.writeChunk(chunk));
@@ -465,7 +485,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
// A timeline job reads one clock for the shot, the motion and the renderer, so a seek matches the frame in a video.
const timelineTime = timeline ? Math.min(filmTimelineDuration(timeline), settings.timeline.start + elapsed) : undefined;
const shotSample = timeline ? sampleFilmTimeline(timeline, snapshot.recipe, timelineTime) : undefined;
- const frameInput = { sessionId: jobId,
+ const frameInput = { sessionId: jobId, ...(highPrecision ? { floatOutput: true } : {}),
rendererTime: (settings.rendererTime?.start ?? 0) + (timelineTime ?? elapsed) * (settings.rendererTime?.rate ?? 1),
...(settings.motion ? { motionTime: settings.motion.start + elapsed } : {}),
...(shotSample?.motionTime == null ? {} : { motionTime: shotSample.motionTime }),
@@ -548,6 +568,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
const captureStarted = performance.now();
const captured = await session.send('Page.captureScreenshot', { fromSurface: true,
format: settings.kind === 'video' ? 'jpeg' : 'png',
+ ...(region ? { clip: { ...region, scale: 1 } } : {}),
...(settings.kind === 'video' ? { quality: 95 } : { optimizeForSpeed: true }) });
const captureBytes = Buffer.from(captured.data, 'base64');
metrics.captureSeconds += (performance.now() - captureStarted) / 1000;
@@ -555,22 +576,36 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
signal.throwIfAborted();
return captureBytes;
}
+ // 16-bit PNG and EXR come from a float readback of the same frame (or of its pass), encoded here.
+ async function captureHighPrecisionBytes(pass) {
+ const captureStarted = performance.now();
+ const read = await jobPage.evaluate(readInput => window.__agentPreview.readFloat(readInput),
+ { sessionId: jobId, source: pass ? 'pass' : 'frame', ...(region ? { region } : {}) });
+ const readBytes = Buffer.from(read.data, 'base64');
+ const pixels = new Float32Array(readBytes.buffer.slice(readBytes.byteOffset, readBytes.byteOffset + readBytes.length));
+ const encodedBytes = encodeRenderOutput({ format, pixels, width: read.width, height: read.height, pass,
+ hasAlpha: settings.transparent === true, depthRange: RENDER_PASS_DEPTH_RANGE });
+ metrics.captureSeconds += (performance.now() - captureStarted) / 1000;
+ if (encodedBytes.length > MAXIMUM_JOB_FRAME_BYTES * HIGH_PRECISION_FRAME_FACTOR) throw new Error('Rendered frame exceeded the high-precision size limit.');
+ signal.throwIfAborted();
+ return encodedBytes;
+ }
async function writeArtifact(filename, artifactBytes) {
await writeFile(join(directory, `${filename}.partial`), artifactBytes, { flag: 'wx' });
- artifacts.push({ temporary: `${filename}.partial`, filename, mimeType: 'image/png' });
+ artifacts.push({ temporary: `${filename}.partial`, filename, mimeType: artifactMimeType });
}
- const bytes = await captureFrameBytes();
+ const bytes = highPrecision ? await captureHighPrecisionBytes(undefined) : await captureFrameBytes();
if (encoder) {
if (bytes) await encoder.writeFrame(bytes);
}
else {
const stem = settings.kind === 'image' ? 'render' : `frame-${frame}`;
- await writeArtifact(`${stem}.png`, bytes);
+ await writeArtifact(`${stem}.${artifactExtension}`, bytes);
// Passes follow the lit image of the same instant, so each seek yields every file for that frame.
for (const pass of settings.passes ?? []) {
const passInfo = await jobPage.evaluate(input => window.__agentPreview.frame(input),
- { ...frameInput, pass, passDepthRange: RENDER_PASS_DEPTH_RANGE });
- await writeArtifact(`${stem}-${pass}.png`, await captureFrameBytes());
+ { ...frameInput, floatOutput: false, pass, passDepthRange: RENDER_PASS_DEPTH_RANGE, ...(highPrecision ? { passFloat: true } : {}) });
+ await writeArtifact(`${stem}-${pass}.${artifactExtension}`, highPrecision ? await captureHighPrecisionBytes(pass) : await captureFrameBytes());
if (pass === 'depth') metrics.depthRangeMetres = RENDER_PASS_DEPTH_RANGE;
if (pass === 'objectId') {
metrics.objectNames = passInfo.objectNames.slice(0, MAXIMUM_OBJECT_NAMES);
diff --git a/scripts/mcp-server.mjs b/scripts/mcp-server.mjs
index aaea8696..7bb75aff 100644
--- a/scripts/mcp-server.mjs
+++ b/scripts/mcp-server.mjs
@@ -9,7 +9,7 @@ import packageMetadata from '../package.json' with { type: 'json' };
import { MODIFIER_OPERATIONS } from '../src/mesh-operations.js';
import { VIEW_TRANSFORM_NAMES, LOOK_NAMES } from '../src/view-transform-names.js';
import { BUNDLED_ENVIRONMENT_NAMES, BUNDLED_CUBE_LOOK_NAMES, CATALOG_FILE_ID } from '../src/look-catalog.js';
-import { MATERIAL_PATTERN_TYPES, MATERIAL_MAPPINGS, LIGHTING_SETUP_NAMES, RENDER_PASS_NAMES, VIDEO_ENCODERS, MAXIMUM_JOB_SIDE, MAXIMUM_MEMORY_MIB } from '../src/agent-scene-capabilities.js';
+import { MATERIAL_PATTERN_TYPES, MATERIAL_MAPPINGS, LIGHTING_SETUP_NAMES, RENDER_PASS_NAMES, RENDER_OUTPUT_FORMATS, MAXIMUM_JOB_FPS, VIDEO_ENCODERS, MAXIMUM_JOB_SIDE, MAXIMUM_MEMORY_MIB } from '../src/agent-scene-capabilities.js';
import { MATERIAL_LAYER_SCALAR_RANGES, MATERIAL_LAYER_COLOR_RANGES } from '../src/material-layers.js';
import { SUBSURFACE_RANGES } from '../src/subsurface-limits.js';
import { MAX_POINT_LIGHTS, MAX_SPOT_LIGHTS, MAX_AREA_LIGHTS } from '../src/light-limits.js';
@@ -695,7 +695,7 @@ const renderSettings = z.strictObject({
kind: z.enum(['image', 'stills', 'video']),
width: z.number().int().min(64).max(MAXIMUM_JOB_SIDE),
height: z.number().int().min(64).max(MAXIMUM_JOB_SIDE),
- fps: z.number().int().min(1).max(30).optional(),
+ fps: z.number().int().min(1).max(MAXIMUM_JOB_FPS).optional(),
duration: z.number().min(0.1).max(60).optional(),
animation: z.strictObject({ clip: z.number().int().min(0), start: z.number().min(0).max(3600).optional() }).optional(),
motion: z.strictObject({ start: z.number().min(0).max(3600).optional() }).optional(),
@@ -705,6 +705,9 @@ const renderSettings = z.strictObject({
audio: z.strictObject({ file: z.string().max(68), offset: z.number().min(0).max(3600).optional(), track: z.enum(['aac', 'pcm']).optional() }).optional().describe('Video only: a file name in the --audio folder (.wav, .mp3, .ogg, .flac) mixed in from offset seconds.'),
encoder: z.enum(VIDEO_ENCODERS).optional().describe('Video only: cpu (default) or nvenc.'),
passes: z.array(z.enum(RENDER_PASS_NAMES)).min(1).max(RENDER_PASS_NAMES.length).optional().describe('image and stills only: also write raw PNGs per frame for these passes; opaque geometry only.'),
+ transparent: z.boolean().optional().describe('image and stills only: lit image with alpha. No sky, background or auto ground; no volume fog.'),
+ format: z.enum(RENDER_OUTPUT_FORMATS).optional().describe('image and stills only: png (8-bit, default), png16 or exr (half float; depth in metres).'),
+ region: z.strictObject({ x: z.number().int().min(0), y: z.number().int().min(0), width: z.number().int().min(16), height: z.number().int().min(16) }).optional().describe('Pixels from the top left to keep; the frame is rendered whole. Not with renderer.'),
}).describe('Immutable snapshot render: one PNG, eight evenly spaced PNGs, or CPU H.264 MP4. Video dimensions must be even. Defaults: 24 fps, one second. Use motion.start for authored recipe tracks, animation for an imported clip, or both to layer them. Omitted motion holds time zero. Nonlooping samples must fit the authored duration.');
const readOnly = { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false };
const edit = { readOnlyHint: false, destructiveHint: true, idempotentHint: false, openWorldHint: false };
diff --git a/scripts/render-output-encoders.mjs b/scripts/render-output-encoders.mjs
new file mode 100644
index 00000000..1bce1264
--- /dev/null
+++ b/scripts/render-output-encoders.mjs
@@ -0,0 +1,257 @@
+import { deflateSync } from 'node:zlib';
+
+import { PNG_SIGNATURE, pngChunk } from '../src/export-png.js';
+
+// Node-side writers for the high-precision files of a render job: 16-bit PNG and OpenEXR. Both are small hand-written
+// encoders (no dependency) and both take the float pixels the page reads back (RGBA float32, rows bottom to top).
+
+const EXR_MAGIC = [0x76, 0x2f, 0x31, 0x01];
+const EXR_VERSION = [2, 0, 0, 0];
+const EXR_ZIP_COMPRESSION = 3;
+const EXR_ZIP_ROWS_PER_BLOCK = 16;
+const EXR_HALF = 1;
+const EXR_FLOAT = 2;
+const HALF_MAXIMUM = 65504;
+const PNG_COLOR_TYPES = Object.freeze({ 1: 0, 3: 2, 4: 6 });
+const PNG_SUB_FILTER = 1;
+const MAXIMUM_PNG16_OBJECT_ID = 65535;
+
+/** The 16 bit value of a colour channel in 0 to 1. */
+function toSixteenBits(value) {
+ return Math.round(Math.min(1, Math.max(0, value)) * 65535);
+}
+
+/**
+ * A 16-bit PNG from interleaved samples (1 gray, 3 RGB or 4 RGBA channels), top row first, compressed with the Sub filter.
+ * `samples` holds integers 0 to 65535.
+ */
+export function encodePng16(samples, width, height, channels) {
+ if (!Object.hasOwn(PNG_COLOR_TYPES, channels)) throw new RangeError('A 16-bit PNG has 1, 3 or 4 channels.');
+ if (samples.length !== width * height * channels) throw new RangeError('The sample count does not match the image size.');
+ const bytesPerPixel = channels * 2;
+ const rowBytes = width * bytesPerPixel;
+ const filtered = Buffer.alloc((rowBytes + 1) * height);
+ const rowSamples = width * channels;
+ const row = Buffer.alloc(rowBytes);
+ for (let rowIndex = 0; rowIndex < height; rowIndex++) {
+ for (let index = 0; index < rowSamples; index++) row.writeUInt16BE(samples[rowIndex * rowSamples + index], index * 2);
+ const target = rowIndex * (rowBytes + 1);
+ filtered[target] = PNG_SUB_FILTER;
+ for (let byteIndex = 0; byteIndex < rowBytes; byteIndex++) {
+ const left = byteIndex >= bytesPerPixel ? row[byteIndex - bytesPerPixel] : 0;
+ filtered[target + 1 + byteIndex] = (row[byteIndex] - left) & 0xff;
+ }
+ }
+ const header = Buffer.alloc(13);
+ header.writeUInt32BE(width, 0);
+ header.writeUInt32BE(height, 4);
+ header[8] = 16;
+ header[9] = PNG_COLOR_TYPES[channels];
+ return Buffer.concat([Buffer.from(PNG_SIGNATURE), pngChunk('IHDR', header), pngChunk('IDAT', deflateSync(filtered, { level: 6 })), pngChunk('IEND', new Uint8Array(0))]);
+}
+
+const float32Scratch = new Float32Array(1);
+const uint32Scratch = new Uint32Array(float32Scratch.buffer);
+
+/** IEEE half float bits of a number, rounded to nearest and clamped to the finite range. */
+export function toHalfBits(value) {
+ const clamped = Number.isNaN(value) ? 0 : Math.min(HALF_MAXIMUM, Math.max(-HALF_MAXIMUM, value));
+ float32Scratch[0] = clamped;
+ const bits = uint32Scratch[0];
+ const sign = (bits >>> 16) & 0x8000;
+ const exponent = ((bits >>> 23) & 0xff) - 127 + 15;
+ const mantissa = bits & 0x7fffff;
+ if (exponent <= 0) {
+ if (exponent < -10) return sign;
+ const shifted = (mantissa | 0x800000) >> (1 - exponent);
+ return sign | ((shifted + 0x1000) >> 13);
+ }
+ return sign | (((exponent << 10) | (mantissa >> 13)) + ((mantissa >> 12) & 1));
+}
+
+function exrAttribute(name, type, value) {
+ return Buffer.concat([Buffer.from(`${name}\0${type}\0`), uint32(value.length), value]);
+}
+
+function uint32(value) {
+ const buffer = Buffer.alloc(4);
+ buffer.writeInt32LE(value);
+ return buffer;
+}
+
+function box(minimumX, minimumY, maximumX, maximumY) {
+ const buffer = Buffer.alloc(16);
+ [minimumX, minimumY, maximumX, maximumY].forEach((value, index) => buffer.writeInt32LE(value, index * 4));
+ return buffer;
+}
+
+function channelList(channels) {
+ const entries = channels.map(({ name, pixelType }) => {
+ const entry = Buffer.alloc(name.length + 1 + 16);
+ entry.write(name, 0, 'latin1');
+ entry.writeInt32LE(pixelType, name.length + 1);
+ entry.writeInt32LE(1, name.length + 9);
+ entry.writeInt32LE(1, name.length + 13);
+ return entry;
+ });
+ return Buffer.concat([...entries, Buffer.alloc(1)]);
+}
+
+/** The OpenEXR ZIP step: split even and odd bytes, delta code, deflate; a block that does not shrink is stored as it is. */
+function zipBlock(raw) {
+ const reordered = Buffer.alloc(raw.length);
+ const half = (raw.length + 1) >> 1;
+ for (let index = 0; index < raw.length; index++) {
+ const target = index % 2 === 0 ? index / 2 : half + (index - 1) / 2;
+ reordered[target] = raw[index];
+ }
+ let previous = reordered[0];
+ for (let index = 1; index < reordered.length; index++) {
+ const current = reordered[index];
+ reordered[index] = (current - previous + 128 + 256) & 0xff;
+ previous = current;
+ }
+ const compressed = deflateSync(reordered);
+ return compressed.length < raw.length ? compressed : raw;
+}
+
+/**
+ * An OpenEXR file (scanline, ZIP compression, one part). `channels` is [{ name, type: 'half' | 'float', values }] where
+ * `values` is a Float32Array of width x height numbers, top row first. Channels are written in name order, as the format requires.
+ */
+export function encodeExr(width, height, channels) {
+ const ordered = [...channels].sort((left, right) => (left.name < right.name ? -1 : 1))
+ .map(channel => ({ ...channel, pixelType: channel.type === 'float' ? EXR_FLOAT : EXR_HALF }));
+ const bytesPerPixel = ordered.reduce((total, channel) => total + (channel.pixelType === EXR_FLOAT ? 4 : 2), 0);
+ const header = Buffer.concat([
+ Buffer.from([...EXR_MAGIC, ...EXR_VERSION]),
+ exrAttribute('channels', 'chlist', channelList(ordered)),
+ exrAttribute('compression', 'compression', Buffer.from([EXR_ZIP_COMPRESSION])),
+ exrAttribute('dataWindow', 'box2i', box(0, 0, width - 1, height - 1)),
+ exrAttribute('displayWindow', 'box2i', box(0, 0, width - 1, height - 1)),
+ exrAttribute('lineOrder', 'lineOrder', Buffer.from([0])),
+ exrAttribute('pixelAspectRatio', 'float', Buffer.from(new Float32Array([1]).buffer)),
+ exrAttribute('screenWindowCenter', 'v2f', Buffer.from(new Float32Array([0, 0]).buffer)),
+ exrAttribute('screenWindowWidth', 'float', Buffer.from(new Float32Array([1]).buffer)),
+ Buffer.alloc(1),
+ ]);
+ const blocks = [];
+ for (let firstRow = 0; firstRow < height; firstRow += EXR_ZIP_ROWS_PER_BLOCK) {
+ const rowCount = Math.min(EXR_ZIP_ROWS_PER_BLOCK, height - firstRow);
+ const raw = Buffer.alloc(rowCount * width * bytesPerPixel);
+ let offset = 0;
+ for (let row = 0; row < rowCount; row++) {
+ for (const channel of ordered) {
+ for (let column = 0; column < width; column++) {
+ const value = channel.values[(firstRow + row) * width + column];
+ if (channel.pixelType === EXR_FLOAT) {
+ raw.writeFloatLE(value, offset);
+ offset += 4;
+ } else {
+ raw.writeUInt16LE(toHalfBits(value), offset);
+ offset += 2;
+ }
+ }
+ }
+ }
+ const data = zipBlock(raw);
+ const blockHeader = Buffer.alloc(8);
+ blockHeader.writeInt32LE(firstRow, 0);
+ blockHeader.writeInt32LE(data.length, 4);
+ blocks.push(Buffer.concat([blockHeader, data]));
+ }
+ const table = Buffer.alloc(blocks.length * 8);
+ let position = header.length + table.length;
+ blocks.forEach((block, index) => {
+ table.writeBigUInt64LE(BigInt(position), index * 8);
+ position += block.length;
+ });
+ return Buffer.concat([header, table, ...blocks]);
+}
+
+/** Interleaved bottom-up RGBA floats into one planar top-down Float32Array per requested component (0 to 3). */
+function planes(pixels, width, height, components) {
+ return components.map(component => {
+ const plane = new Float32Array(width * height);
+ for (let row = 0; row < height; row++) {
+ const sourceRow = height - 1 - row;
+ for (let column = 0; column < width; column++) plane[row * width + column] = pixels[(sourceRow * width + column) * 4 + component];
+ }
+ return plane;
+ });
+}
+
+function byteOf(value) {
+ return Math.round(Math.min(1, Math.max(0, value)) * 255);
+}
+
+/** The 16 bit depth step of a depth pass pixel: red is the high byte and green the low byte. */
+function depthSteps(pixels, offset) {
+ return byteOf(pixels[offset]) * 256 + byteOf(pixels[offset + 1]);
+}
+
+/** The 24 bit object ID of an object ID pass pixel, zero for no object. */
+function objectIdentifier(pixels, offset) {
+ return byteOf(pixels[offset]) + byteOf(pixels[offset + 1]) * 256 + byteOf(pixels[offset + 2]) * 65536;
+}
+
+function encodeExrPass({ pixels, width, height, pass, hasAlpha, depthRange }) {
+ if (pass === 'depth' || pass === 'objectId') {
+ const values = new Float32Array(width * height);
+ for (let row = 0; row < height; row++) {
+ const sourceRow = height - 1 - row;
+ for (let column = 0; column < width; column++) {
+ const offset = (sourceRow * width + column) * 4;
+ values[row * width + column] = pass === 'depth' ? depthSteps(pixels, offset) / 65535 * depthRange : objectIdentifier(pixels, offset);
+ }
+ }
+ return encodeExr(width, height, [{ name: pass === 'depth' ? 'Z' : 'id', type: 'float', values }]);
+ }
+ const [red, green, blue, alpha] = planes(pixels, width, height, [0, 1, 2, 3]);
+ const channels = [{ name: 'R', type: 'half', values: red }, { name: 'G', type: 'half', values: green }, { name: 'B', type: 'half', values: blue }];
+ if (hasAlpha) channels.push({ name: 'A', type: 'half', values: alpha });
+ return encodeExr(width, height, channels);
+}
+
+function encodePng16Pass({ pixels, width, height, pass, hasAlpha }) {
+ const pixelCount = width * height;
+ if (pass === 'depth' || pass === 'objectId') {
+ const samples = new Uint16Array(pixelCount);
+ for (let row = 0; row < height; row++) {
+ for (let column = 0; column < width; column++) {
+ const offset = ((height - 1 - row) * width + column) * 4;
+ const value = pass === 'depth' ? depthSteps(pixels, offset) : objectIdentifier(pixels, offset);
+ if (value > MAXIMUM_PNG16_OBJECT_ID && pass === 'objectId') throw new RangeError('A 16-bit PNG holds object IDs up to 65535. Use format exr for this scene.');
+ samples[row * width + column] = value;
+ }
+ }
+ return encodePng16(samples, width, height, 1);
+ }
+ const channels = hasAlpha ? 4 : 3;
+ const samples = new Uint16Array(pixelCount * channels);
+ for (let row = 0; row < height; row++) {
+ for (let column = 0; column < width; column++) {
+ const offset = ((height - 1 - row) * width + column) * 4;
+ const target = (row * width + column) * channels;
+ const alpha = pixels[offset + 3];
+ // The lit frame is premultiplied by alpha; a PNG stores straight colour.
+ const scale = hasAlpha && alpha > 0 ? 1 / alpha : 1;
+ for (let component = 0; component < 3; component++) samples[target + component] = toSixteenBits(pixels[offset + component] * scale);
+ if (hasAlpha) samples[target + 3] = toSixteenBits(alpha);
+ }
+ }
+ return encodePng16(samples, width, height, channels);
+}
+
+/**
+ * The file bytes of one rendered image in a high-precision format. `pixels` is the page's RGBA float readback
+ * (rows bottom to top), `pass` names the render pass or is undefined for the lit image, and `hasAlpha` keeps the alpha
+ * of a transparent lit image. Depth is written as metres over `depthRange` in EXR and as the raw 16-bit step in PNG.
+ */
+export function encodeRenderOutput({ format, pixels, width, height, pass, hasAlpha = false, depthRange = 100 }) {
+ if (pixels.length !== width * height * 4) throw new RangeError('The pixel count does not match the image size.');
+ const options = { pixels, width, height, pass, hasAlpha: hasAlpha && pass === undefined, depthRange };
+ if (format === 'png16') return encodePng16Pass(options);
+ if (format === 'exr') return encodeExrPass(options);
+ throw new TypeError('format must be png16 or exr.');
+}
diff --git a/scripts/smoke-agent-render-jobs.mjs b/scripts/smoke-agent-render-jobs.mjs
index 149dbf52..2a2a32df 100644
--- a/scripts/smoke-agent-render-jobs.mjs
+++ b/scripts/smoke-agent-render-jobs.mjs
@@ -57,7 +57,7 @@ async function close() {
}
try {
await connect();
- assert.equal((await client.listTools()).tools.length, 22);
+ assert.equal((await client.listTools()).tools.length, 23);
const catalog = (await call('scene_asset_catalog')).structuredContent;
const created = (await call('scene_create_asset', { recipe: {
format: '67js.asset', schemaVersion: 1, assetId: catalog.assets[0].assetId,
@@ -105,6 +105,31 @@ try {
}
imageJobs.push(submitted.jobId);
}
+ // Output options (issue #611): each one writes a file of its own kind from a real job.
+ async function optionFiles(settings) {
+ const submitted = (await call('scene_render_submit', { sessionId, expectedRevision: 1, settings })).structuredContent;
+ assert.equal((await status(submitted.jobId)).state, 'completed');
+ imageJobs.push(submitted.jobId);
+ const { artifacts } = (await call('scene_render_result', { jobId: submitted.jobId })).structuredContent;
+ return Promise.all(artifacts.map(async artifact => ({ ...artifact, data: await readFile(artifact.path) })));
+ }
+ const pngColorType = data => data[25];
+ const pngBitDepth = data => data[24];
+ const [transparentFile] = await optionFiles({ kind: 'image', width: 128, height: 128, transparent: true });
+ assert.equal(pngColorType(transparentFile.data), 6);
+ const [sixteenBit, depthSixteen] = await optionFiles({ kind: 'image', width: 128, height: 128, format: 'png16', passes: ['depth'] });
+ assert.deepEqual([pngBitDepth(sixteenBit.data), pngColorType(sixteenBit.data), pngBitDepth(depthSixteen.data), pngColorType(depthSixteen.data)], [16, 2, 16, 0]);
+ const [exrFile, exrDepth] = await optionFiles({ kind: 'image', width: 128, height: 128, format: 'exr', passes: ['depth'] });
+ for (const file of [exrFile, exrDepth]) {
+ assert.equal(file.mimeType, 'image/x-exr');
+ assert.equal(file.data.subarray(0, 4).toString('hex'), '762f3101');
+ }
+ const [cropped] = await optionFiles({ kind: 'image', width: 128, height: 128, region: { x: 16, y: 8, width: 64, height: 48 } });
+ assert.deepEqual([cropped.data.readUInt32BE(16), cropped.data.readUInt32BE(20)], [64, 48]);
+ const [croppedSixteen] = await optionFiles({ kind: 'image', width: 128, height: 128, format: 'png16', region: { x: 16, y: 8, width: 64, height: 48 } });
+ assert.deepEqual([croppedSixteen.data.readUInt32BE(16), croppedSixteen.data.readUInt32BE(20)], [64, 48]);
+ const [fastVideo] = await optionFiles({ kind: 'video', width: 128, height: 128, fps: 60, duration: 0.25, animation: { clip: 0, start: 0 }, region: { x: 16, y: 8, width: 64, height: 48 } });
+ assert.equal(fastVideo.data.subarray(4, 8).toString('latin1'), 'ftyp');
const longJob = (await call('scene_render_submit', { sessionId, expectedRevision: 1,
settings: { kind: 'video', width: 320, height: 320, fps: 30, duration: 30 } })).structuredContent;
await status(longJob.jobId, record => record.progress.frames > 0);
diff --git a/skills/render-67js-media/SKILL.md b/skills/render-67js-media/SKILL.md
index c835cd4b..7df8bf29 100644
--- a/skills/render-67js-media/SKILL.md
+++ b/skills/render-67js-media/SKILL.md
@@ -13,6 +13,7 @@ Use the repository pipeline rather than constructing a parallel exporter.
For MCP snapshot jobs, read the checkout's `docs/agent-render-jobs.md` and verify source versus published support.
Configure a private job store. Poll status after cancellation before resuming preview edits.
For a video with cuts, sound or an encoder choice, use the recipe `timeline`, `settings.timeline`, `settings.audio` (a file in the server's `--audio` folder) and `settings.encoder` described in the same document; the job reports measured sync.
+For compositing output, add `settings.transparent: true` (PNG with alpha; no sky or auto ground), `settings.format` `png16` or `exr` (depth in metres, object IDs as float) and `settings.region` `{x, y, width, height}` (a crop of the whole frame, also for video); video runs up to 60 fps. These stay off unless set, so defaults are unchanged. Details in `docs/agent-render-jobs.md`, section Output options.
For procedural motion, read `docs/agent-motion.md` and select `settings.motion.start`. Preview seeks preserve revisions and authored recipes.
For imported character motion from `0.8.0`, read `docs/agent-character-motion.md`. Use independent `settings.animation` and `settings.motion` clocks.
diff --git a/src/agent-scene-capabilities.js b/src/agent-scene-capabilities.js
index efc33124..9cb8b090 100644
--- a/src/agent-scene-capabilities.js
+++ b/src/agent-scene-capabilities.js
@@ -100,6 +100,9 @@ export const RENDER_PASS_NAMES = Object.freeze(['albedo', 'normal', 'emission',
// extension, no drive or protocol syntax.
export const VIDEO_ENCODERS = Object.freeze(['cpu', 'nvenc']);
export const MAXIMUM_VIDEO_SECONDS = 60;
+// File formats of an image or stills job (8-bit PNG by default, 16-bit PNG, OpenEXR) and the highest video frame rate.
+export const RENDER_OUTPUT_FORMATS = Object.freeze(['png', 'png16', 'exr']);
+export const MAXIMUM_JOB_FPS = 60;
/** Recipe bake presets and ranges. A bake inline on the page fails the edit after 20 s, before the 30 s page deadline; on a worker it has 90 s. */
export const BAKE_PRESETS = Object.freeze({
draft: Object.freeze({ samples: 32, bounces: 1, texelsPerUnit: 3 }),
@@ -193,6 +196,13 @@ export const AGENT_SCENE_CAPABILITIES = Object.freeze({
materialPatterns: MATERIAL_PATTERN_TYPES, maxMaterialPatterns: MAX_MATERIAL_PATTERNS,
materialMappings: MATERIAL_MAPPINGS,
renderPasses: Object.freeze({ names: RENDER_PASS_NAMES, files: 'render-<pass>.png or frame-N-<pass>.png, raw and unantialiased; opaque and cutout geometry only', depth: `16 bits in red and green over ${RENDER_PASS_DEPTH_RANGE} m`, objectId: '24 bits in RGB; the job result lists the part names' }),
+ renderOutput: Object.freeze({
+ transparent: 'true writes the lit image with alpha (PNG, 16-bit PNG or EXR); image and stills only. The sky, environment background and automatic ground are not drawn, so the scene needs its own ground or a shadowCatcher; volume fog is refused.',
+ format: `${RENDER_OUTPUT_FORMATS.join(', ')}: png is 8-bit (default); png16 is 16-bit from a half-float frame; exr holds the same display colour as the PNG in half floats (alpha stays premultiplied); its depth pass is metres in one float32 channel Z and objectId one float32 channel id. image and stills only.`,
+ region: '{x, y, width, height} pixels from the top left, at least 16 each; renders the whole frame and keeps the crop. Works for image, stills and video (even sizes).',
+ fps: `1 to ${MAXIMUM_JOB_FPS}; a video is at most 900 frames`,
+ pathTracer: 'renderer jobs write the whole opaque 8-bit PNG: transparent, format and region are refused.',
+ }),
// Recipe timeline (src/film-timeline.js), rendered with settings.timeline; the rest is in docs/agent-render-jobs.md.
filmTimeline: Object.freeze({
maxShots: 64, maxTotalSeconds: 60, maxShotSeconds: 30,
diff --git a/src/export-png.js b/src/export-png.js
index 576cbd67..e0688eec 100644
--- a/src/export-png.js
+++ b/src/export-png.js
@@ -59,6 +59,8 @@ function chunk(type, data) {
return output;
}
+export { PNG_SIGNATURE, chunk as pngChunk };
+
function hasTransparentPixel(pixels) {
for (let offset = 3; offset < pixels.length; offset += 4) {
if (pixels[offset] !== 255) return true;
diff --git a/src/float-output.js b/src/float-output.js
new file mode 100644
index 00000000..c4c72e47
--- /dev/null
+++ b/src/float-output.js
@@ -0,0 +1,54 @@
+// A half-float colour target for the frame's last pass, and a float readback of any framebuffer, so a render job can
+// write 16-bit PNG and EXR files from more than the 8 bits the canvas keeps. Needs WebGL 2 and EXT_color_buffer_float.
+
+const FLOAT_EXTENSION = 'EXT_color_buffer_float';
+
+export function requireFloatTargets(context) {
+ if (!context.getExtension(FLOAT_EXTENSION)) {
+ throw new Error('16-bit PNG and EXR output need EXT_color_buffer_float, which this browser or GPU does not offer. Use format png.');
+ }
+}
+
+/** One RGBA16F framebuffer of the given size. The frame's post pass draws into it instead of the canvas. */
+export function createFloatOutput(context, width, height) {
+ requireFloatTargets(context);
+ const texture = context.createTexture();
+ context.bindTexture(context.TEXTURE_2D, texture);
+ context.texImage2D(context.TEXTURE_2D, 0, context.RGBA16F, width, height, 0, context.RGBA, context.HALF_FLOAT, null);
+ context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST);
+ context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST);
+ const framebuffer = context.createFramebuffer();
+ context.bindFramebuffer(context.FRAMEBUFFER, framebuffer);
+ context.framebufferTexture2D(context.FRAMEBUFFER, context.COLOR_ATTACHMENT0, context.TEXTURE_2D, texture, 0);
+ const complete = context.checkFramebufferStatus(context.FRAMEBUFFER) === context.FRAMEBUFFER_COMPLETE;
+ context.bindFramebuffer(context.FRAMEBUFFER, null);
+ if (!complete) {
+ context.deleteFramebuffer(framebuffer);
+ context.deleteTexture(texture);
+ throw new Error('The float output framebuffer could not be allocated. Use format png.');
+ }
+ return {
+ width,
+ height,
+ framebuffer,
+ dispose() {
+ context.deleteFramebuffer(framebuffer);
+ context.deleteTexture(texture);
+ },
+ };
+}
+
+/**
+ * Reads a region of a framebuffer as RGBA float32, rows bottom to top as WebGL stores them. `region` counts pixels
+ * from the top left, like an image, and defaults to the whole frame.
+ */
+export function readFloatPixels(context, framebuffer, frameWidth, frameHeight, region = { x: 0, y: 0, width: frameWidth, height: frameHeight }) {
+ if (region.x < 0 || region.y < 0 || region.width < 1 || region.height < 1 || region.x + region.width > frameWidth || region.y + region.height > frameHeight) {
+ throw new RangeError('The region lies outside the rendered frame.');
+ }
+ const pixels = new Float32Array(region.width * region.height * 4);
+ context.bindFramebuffer(context.READ_FRAMEBUFFER, framebuffer);
+ context.readPixels(region.x, frameHeight - region.y - region.height, region.width, region.height, context.RGBA, context.FLOAT, pixels);
+ context.bindFramebuffer(context.READ_FRAMEBUFFER, null);
+ return pixels;
+}
diff --git a/src/render-passes.js b/src/render-passes.js
index 4ec9a8f8..db58c1f5 100644
--- a/src/render-passes.js
+++ b/src/render-passes.js
@@ -1,4 +1,5 @@
import { compileProgram, fullscreenVertex } from './gl-program.js';
+import { requireFloatTargets } from './float-output.js';
// Render passes draw raw material data (albedo, normal, emission, depth, object ID) instead of the lit image.
// The shader branch lives in the main fragment shader behind the `renderPass` feature, which compiles on the first
@@ -17,10 +18,13 @@ uniform sampler2D sourceTexture;
out vec4 outputColor;
void main() { outputColor = vec4(texelFetch(sourceTexture, ivec2(gl_FragCoord.xy), 0).rgb, 1.0); }`;
-function createPassTarget(context, width, height) {
+// A float target keeps the pass values as half floats (frame.passFloat) for 16-bit PNG and EXR output; the shader writes the same values.
+function createPassTarget(context, width, height, float = false) {
+ if (float) requireFloatTargets(context);
const texture = context.createTexture();
context.bindTexture(context.TEXTURE_2D, texture);
- context.texImage2D(context.TEXTURE_2D, 0, context.RGBA8, width, height, 0, context.RGBA, context.UNSIGNED_BYTE, null);
+ if (float) context.texImage2D(context.TEXTURE_2D, 0, context.RGBA16F, width, height, 0, context.RGBA, context.HALF_FLOAT, null);
+ else context.texImage2D(context.TEXTURE_2D, 0, context.RGBA8, width, height, 0, context.RGBA, context.UNSIGNED_BYTE, null);
context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MIN_FILTER, context.NEAREST);
context.texParameteri(context.TEXTURE_2D, context.TEXTURE_MAG_FILTER, context.NEAREST);
const depth = context.createRenderbuffer();
@@ -33,7 +37,7 @@ function createPassTarget(context, width, height) {
if (context.checkFramebufferStatus(context.FRAMEBUFFER) !== context.FRAMEBUFFER_COMPLETE) {
throw new Error('Render pass framebuffer allocation failed. Choose a lower quality preset.');
}
- return { texture, depth, framebuffer };
+ return { texture, depth, framebuffer, float };
}
/**
@@ -74,10 +78,12 @@ export function installRenderPasses(renderer) {
copyProgram = compileProgram(context, fullscreenVertex, passCopyFragment);
copyTextureUniform = context.getUniformLocation(copyProgram, 'sourceTexture');
}
+ const float = frame.passFloat === true;
+ if (target && target.float !== float) releaseTargets(context);
if (!target) {
const { width, height } = renderer.canvas;
- target = createPassTarget(context, width, height);
- renderer.renderTargets.record('passColor', { width, height, format: 'RGBA8' });
+ target = createPassTarget(context, width, height, float);
+ renderer.renderTargets.record('passColor', { width, height, format: float ? 'RGBA16F' : 'RGBA8' });
renderer.renderTargets.record('passDepth', { width, height, format: 'DEPTH_COMPONENT24' });
}
return mode;
diff --git a/src/renderer.js b/src/renderer.js
index 9ce1514a..efcc204d 100644
--- a/src/renderer.js
+++ b/src/renderer.js
@@ -1837,7 +1837,8 @@ export class Renderer67 {
if (effects.bloom.strength > 0) this.drawBloom(effects.bloom, sourceTexture);
const occluded = effects.ambientOcclusion > 0 && Boolean(this.depthTexture);
if (occluded) this.drawAmbientOcclusion(frame, effects);
- context.bindFramebuffer(context.FRAMEBUFFER, null);
+ // A frame that names an outputFramebuffer (src/float-output.js) draws its final pass there, not on the canvas.
+ context.bindFramebuffer(context.FRAMEBUFFER, frame.outputFramebuffer ?? null);
context.viewport(0, 0, this.canvas.width, this.canvas.height);
context.disable(context.DEPTH_TEST);
context.disable(context.CULL_FACE);
diff --git a/tests/agent-render-jobs.test.js b/tests/agent-render-jobs.test.js
index da72dad7..a7af9cee 100644
--- a/tests/agent-render-jobs.test.js
+++ b/tests/agent-render-jobs.test.js
@@ -205,7 +205,7 @@ test('validation enforces range, geometry-independent storage limits and one pen
{ kind: 'image', width: 1921, height: 64 },
{ kind: 'image', width: 1920, height: 1081 },
{ kind: 'image', width: 1920, height: 1920 },
- { kind: 'video', width: 64, height: 64, fps: 31 },
+ { kind: 'video', width: 64, height: 64, fps: 61 },
{ kind: 'video', width: 64, height: 64, duration: 61 },
]) await assert.rejects(controller.submit({ ...request(), settings }));
await assert.rejects(controller.submit({ ...request(), snapshot: { recipe: {}, clips: [{ duration: 1 }] }, settings: { kind: 'video', width: 64, height: 64, duration: 1, animation: { clip: 0, start: 0.1 } } }), /clip duration/);
diff --git a/tests/agent-render-output.test.js b/tests/agent-render-output.test.js
new file mode 100644
index 00000000..6fefaa35
--- /dev/null
+++ b/tests/agent-render-output.test.js
@@ -0,0 +1,299 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+import { mkdtemp, rm, writeFile } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import { inflateSync } from 'node:zlib';
+
+import { createAgentRenderJobs, artifactNames } from '../scripts/agent-render-jobs.mjs';
+import { encodePng16, encodeExr, encodeRenderOutput, toHalfBits } from '../scripts/render-output-encoders.mjs';
+
+const limits = { maxOutputBytes: 1024 ** 2, maxFrameBytes: 128, maxStoreBytes: 8 * 1024 ** 2, cleanupMs: 100 };
+const submission = settings => ({ sourceRevision: 1, snapshot: { recipe: { format: 'fixture' } }, settings });
+
+async function withController(context, run) {
+ const root = await mkdtemp(join(tmpdir(), '67js-render-output-'));
+ const controller = await createAgentRenderJobs({ root, limits, run });
+ context.after(async () => {
+ await controller.close();
+ await rm(root, { recursive: true, force: true });
+ });
+ return controller;
+}
+
+// A runner that records the settings it got and writes the artifacts those settings name, with the right MIME types.
+function recordingRunner(seen) {
+ return async ({ directory, settings, progress }) => {
+ seen.push(settings);
+ const mimeType = settings.kind === 'video' ? 'video/mp4' : settings.format === 'exr' ? 'image/x-exr' : 'image/png';
+ const artifacts = [];
+ for (const filename of artifactNames(settings.kind, settings.passes, settings.format)) {
+ await writeFile(join(directory, `${filename}.partial`), Buffer.alloc(100, 1));
+ artifacts.push({ temporary: `${filename}.partial`, filename, mimeType });
+ }
+ await progress({ frames: settings.frameCount, maximumPendingFrames: 1 });
+ return { artifacts, metrics: {} };
+ };
+}
+
+async function finished(controller, settings) {
+ const { jobId } = await controller.submit(submission(settings));
+ while (controller.status(jobId).state === 'queued' || controller.status(jobId).state === 'running' || controller.status(jobId).state === 'finalizing') {
+ await new Promise(resolve => setTimeout(resolve, 2));
+ }
+ return { status: controller.status(jobId), result: controller.status(jobId).state === 'completed' ? controller.result(jobId) : undefined };
+}
+
+test('defaults leave transparent, format and region out of the settings, so a plain job is unchanged', async context => {
+ const seen = [];
+ const controller = await withController(context, recordingRunner(seen));
+ const { status, result } = await finished(controller, { kind: 'stills', width: 64, height: 64, passes: ['depth'] });
+ assert.deepEqual(Object.keys(status.settings).sort(), ['duration', 'fps', 'frameCount', 'height', 'kind', 'passes', 'rendererTime', 'width']);
+ assert.equal(result.artifacts.length, 16);
+ assert.ok(result.artifacts.every(artifact => artifact.mimeType === 'image/png' && artifact.filename.endsWith('.png')));
+ const explicit = await finished(controller, { kind: 'image', width: 64, height: 64, transparent: false, format: 'png' });
+ assert.equal('transparent' in explicit.status.settings || 'format' in explicit.status.settings, false);
+});
+
+test('transparent is kept for an image job and refused for video and the path tracer', async context => {
+ const seen = [];
+ const controller = await withController(context, recordingRunner(seen));
+ const { status } = await finished(controller, { kind: 'image', width: 64, height: 64, transparent: true });
+ assert.equal(status.state, 'completed');
+ assert.equal(status.settings.transparent, true);
+ await assert.rejects(controller.submit(submission({ kind: 'video', width: 64, height: 64, transparent: true })), /no alpha/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, transparent: true, renderer: { type: 'pathTracer' } })), /transparent cannot combine/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, transparent: 1 })), /transparent must be boolean/);
+});
+
+test('png16 and exr jobs name their files and MIME types, and pass files follow the format', async context => {
+ const seen = [];
+ const controller = await withController(context, recordingRunner(seen));
+ const exr = await finished(controller, { kind: 'image', width: 64, height: 64, format: 'exr', passes: ['depth', 'objectId'] });
+ assert.deepEqual(exr.result.artifacts.map(artifact => artifact.filename), ['render.exr', 'render-depth.exr', 'render-objectId.exr']);
+ assert.ok(exr.result.artifacts.every(artifact => artifact.mimeType === 'image/x-exr'));
+ const png16 = await finished(controller, { kind: 'stills', width: 64, height: 64, format: 'png16' });
+ assert.equal(png16.status.settings.format, 'png16');
+ assert.equal(png16.result.artifacts.length, 8);
+ assert.ok(png16.result.artifacts.every(artifact => artifact.mimeType === 'image/png' && artifact.filename.endsWith('.png')));
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, format: 'tiff' })), /format must be one of png, png16, exr/);
+ await assert.rejects(controller.submit(submission({ kind: 'video', width: 64, height: 64, format: 'exr' })), /format needs kind image or stills/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, format: 'exr', renderer: { type: 'pathTracer' } })), /format cannot combine/);
+});
+
+test('a high-precision frame may be four times the 8-bit frame limit', async context => {
+ const controller = await withController(context, async ({ directory, settings, progress }) => {
+ const filename = `render.${settings.format === 'exr' ? 'exr' : 'png'}`;
+ await writeFile(join(directory, `${filename}.partial`), Buffer.alloc(limits.maxFrameBytes * 3, 1));
+ await progress({ frames: 1, maximumPendingFrames: 1 });
+ return { artifacts: [{ temporary: `${filename}.partial`, filename, mimeType: settings.format === 'exr' ? 'image/x-exr' : 'image/png' }], metrics: {} };
+ });
+ assert.equal((await finished(controller, { kind: 'image', width: 64, height: 64, format: 'exr' })).status.state, 'completed');
+ assert.equal((await finished(controller, { kind: 'image', width: 64, height: 64 })).status.state, 'failed');
+});
+
+test('a region must lie inside the frame, and a video region keeps even sides', async context => {
+ const seen = [];
+ const controller = await withController(context, recordingRunner(seen));
+ const region = { x: 8, y: 4, width: 32, height: 24 };
+ const { status } = await finished(controller, { kind: 'image', width: 64, height: 64, region });
+ assert.deepEqual(status.settings.region, region);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, region: { x: 40, y: 0, width: 32, height: 32 } })), /inside the frame/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, region: { x: 0, y: 0, width: 8, height: 32 } })), /region width/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, region: { x: 0, y: 0, width: 32, height: 32, extra: 1 } })), /Unknown region field/);
+ await assert.rejects(controller.submit(submission({ kind: 'video', width: 64, height: 64, region: { x: 0, y: 0, width: 31, height: 32 } })), /even width and height/);
+ await assert.rejects(controller.submit(submission({ kind: 'image', width: 64, height: 64, region, renderer: { type: 'pathTracer' } })), /region cannot combine/);
+ const video = await finished(controller, { kind: 'video', width: 64, height: 64, region });
+ assert.deepEqual(video.status.settings.region, region);
+});
+
+test('a video may run at 60 fps and no faster', async context => {
+ const seen = [];
+ const controller = await withController(context, recordingRunner(seen));
+ const { status } = await finished(controller, { kind: 'video', width: 64, height: 64, fps: 60, duration: 0.5 });
+ assert.equal(status.settings.fps, 60);
+ assert.equal(status.settings.frameCount, 30);
+ await assert.rejects(controller.submit(submission({ kind: 'video', width: 64, height: 64, fps: 61 })), /fps must be within 1 and 60/);
+ await assert.rejects(controller.submit(submission({ kind: 'video', width: 64, height: 64, fps: 60, duration: 16 })), /900 frames/);
+});
+
+test('options survive a restart of the job store', async context => {
+ const root = await mkdtemp(join(tmpdir(), '67js-render-output-'));
+ const first = await createAgentRenderJobs({ root, limits, run: recordingRunner([]) });
+ const settings = { kind: 'image', width: 64, height: 64, format: 'exr', transparent: true, region: { x: 0, y: 0, width: 32, height: 32 } };
+ const { jobId } = await first.submit(submission(settings));
+ while (!['completed', 'failed'].includes(first.status(jobId).state)) await new Promise(resolve => setTimeout(resolve, 2));
+ await first.close();
+ const second = await createAgentRenderJobs({ root, limits, run: recordingRunner([]) });
+ context.after(async () => {
+ await second.close();
+ await rm(root, { recursive: true, force: true });
+ });
+ assert.equal(second.status(jobId).state, 'completed');
+ assert.equal(second.status(jobId).settings.format, 'exr');
+ assert.equal(second.result(jobId).artifacts[0].mimeType, 'image/x-exr');
+});
+
+// Encoders
+
+function decodePng16(bytes) {
+ assert.deepEqual([...bytes.subarray(0, 8)], [137, 80, 78, 71, 13, 10, 26, 10]);
+ const width = bytes.readUInt32BE(16);
+ const height = bytes.readUInt32BE(20);
+ assert.equal(bytes[24], 16);
+ const channels = { 0: 1, 2: 3, 6: 4 }[bytes[25]];
+ const data = [];
+ let offset = 8;
+ while (offset < bytes.length) {
+ const length = bytes.readUInt32BE(offset);
+ if (bytes.toString('latin1', offset + 4, offset + 8) === 'IDAT') data.push(bytes.subarray(offset + 8, offset + 8 + length));
+ offset += 12 + length;
+ }
+ const raw = inflateSync(Buffer.concat(data));
+ const rowBytes = width * channels * 2;
+ const samples = [];
+ for (let row = 0; row < height; row++) {
+ const line = Buffer.from(raw.subarray(row * (rowBytes + 1) + 1, (row + 1) * (rowBytes + 1)));
+ assert.equal(raw[row * (rowBytes + 1)], 1);
+ for (let index = channels * 2; index < rowBytes; index++) line[index] = (line[index] + line[index - channels * 2]) & 0xff;
+ for (let index = 0; index < rowBytes; index += 2) samples.push(line.readUInt16BE(index));
+ }
+ return { width, height, channels, samples };
+}
+
+test('encodePng16 writes 16-bit samples that decode back exactly', () => {
+ const samples = Uint16Array.from([0, 1, 255, 256, 65535, 12345, 40000, 7, 500, 65534, 3, 9]);
+ for (const [channels, width, height] of [[1, 4, 3], [3, 2, 2], [4, 3, 1]]) {
+ const decoded = decodePng16(encodePng16(samples.subarray(0, width * height * channels), width, height, channels));
+ assert.deepEqual([decoded.width, decoded.height, decoded.channels], [width, height, channels]);
+ assert.deepEqual(decoded.samples, [...samples.subarray(0, width * height * channels)]);
+ }
+ assert.throws(() => encodePng16(samples, 2, 2, 2), /1, 3 or 4 channels/);
+});
+
+test('toHalfBits rounds to IEEE half floats and clamps to the finite range', () => {
+ assert.equal(toHalfBits(0), 0);
+ assert.equal(toHalfBits(1), 0x3c00);
+ assert.equal(toHalfBits(-2), 0xc000);
+ assert.equal(toHalfBits(0.5), 0x3800);
+ assert.equal(toHalfBits(65504), 0x7bff);
+ assert.equal(toHalfBits(1e9), 0x7bff);
+ assert.equal(toHalfBits(Number.NaN), 0);
+ assert.equal(toHalfBits(1 / 3), 0x3555);
+});
+
+/** Reads an EXR written by encodeExr back into one float array per channel, undoing the ZIP step. */
+function decodeExr(bytes) {
+ assert.deepEqual([...bytes.subarray(0, 4)], [0x76, 0x2f, 0x31, 0x01]);
+ let offset = 8;
+ const attributes = {};
+ while (bytes[offset] !== 0) {
+ const nameEnd = bytes.indexOf(0, offset);
+ const name = bytes.toString('latin1', offset, nameEnd);
+ const typeEnd = bytes.indexOf(0, nameEnd + 1);
+ const size = bytes.readInt32LE(typeEnd + 1);
+ attributes[name] = bytes.subarray(typeEnd + 5, typeEnd + 5 + size);
+ offset = typeEnd + 5 + size;
+ }
+ offset += 1;
+ const window = attributes.dataWindow;
+ const width = window.readInt32LE(8) + 1;
+ const height = window.readInt32LE(12) + 1;
+ const channels = [];
+ let position = 0;
+ while (attributes.channels[position] !== 0) {
+ const nameEnd = attributes.channels.indexOf(0, position);
+ channels.push({ name: attributes.channels.toString('latin1', position, nameEnd), pixelType: attributes.channels.readInt32LE(nameEnd + 1) });
+ position = nameEnd + 17;
+ }
+ const rowsPerBlock = 16;
+ const blockCount = Math.ceil(height / rowsPerBlock);
+ const output = Object.fromEntries(channels.map(channel => [channel.name, new Float32Array(width * height)]));
+ const bytesPerPixel = channels.reduce((total, channel) => total + (channel.pixelType === 2 ? 4 : 2), 0);
+ for (let block = 0; block < blockCount; block++) {
+ const start = Number(bytes.readBigUInt64LE(offset + block * 8));
+ const firstRow = bytes.readInt32LE(start);
+ const dataSize = bytes.readInt32LE(start + 4);
+ const rowCount = Math.min(rowsPerBlock, height - firstRow);
+ const expected = rowCount * width * bytesPerPixel;
+ let raw = bytes.subarray(start + 8, start + 8 + dataSize);
+ if (dataSize < expected) {
+ const delta = Buffer.from(inflateSync(raw));
+ for (let index = 1; index < delta.length; index++) delta[index] = (delta[index - 1] + delta[index] - 128) & 0xff;
+ raw = Buffer.alloc(delta.length);
+ const half = (delta.length + 1) >> 1;
+ for (let index = 0; index < delta.length; index++) raw[index] = delta[index % 2 === 0 ? index / 2 : half + (index - 1) / 2];
+ }
+ let read = 0;
+ for (let row = 0; row < rowCount; row++) {
+ for (const channel of channels) {
+ for (let column = 0; column < width; column++) {
+ if (channel.pixelType === 2) {
+ output[channel.name][(firstRow + row) * width + column] = raw.readFloatLE(read);
+ read += 4;
+ } else {
+ const bits = raw.readUInt16LE(read);
+ const exponent = (bits >> 10) & 0x1f;
+ const mantissa = bits & 0x3ff;
+ const magnitude = exponent === 0 ? mantissa * 2 ** -24 : (1 + mantissa / 1024) * 2 ** (exponent - 15);
+ output[channel.name][(firstRow + row) * width + column] = bits & 0x8000 ? -magnitude : magnitude;
+ read += 2;
+ }
+ }
+ }
+ }
+ }
+ return { width, height, channels: channels.map(channel => channel.name), compression: attributes.compression[0], values: output };
+}
+
+test('encodeExr writes ZIP scanline blocks that decode back, in channel name order', () => {
+ const width = 37;
+ const height = 35;
+ const red = Float32Array.from({ length: width * height }, (_, index) => (index % 11) / 4);
+ const alpha = Float32Array.from({ length: width * height }, (_, index) => (index % 3 === 0 ? 0 : 1));
+ const depth = Float32Array.from({ length: width * height }, (_, index) => index * 0.37);
+ const colour = decodeExr(encodeExr(width, height, [{ name: 'R', type: 'half', values: red }, { name: 'A', type: 'half', values: alpha }]));
+ assert.deepEqual(colour.channels, ['A', 'R']);
+ assert.equal(colour.compression, 3);
+ assert.deepEqual([...colour.values.R], [...red]);
+ assert.deepEqual([...colour.values.A], [...alpha]);
+ const metres = decodeExr(encodeExr(width, height, [{ name: 'Z', type: 'float', values: depth }]));
+ assert.deepEqual([...metres.values.Z], [...depth]);
+});
+
+function floatPixels(width, height, make) {
+ const pixels = new Float32Array(width * height * 4);
+ for (let row = 0; row < height; row++) {
+ for (let column = 0; column < width; column++) pixels.set(make(column, height - 1 - row), (row * width + column) * 4);
+ }
+ return pixels;
+}
+
+test('encodeRenderOutput flips the bottom-up readback and decodes depth, object IDs and premultiplied alpha', () => {
+ // Pixel (column, topRow); the readback holds the bottom row first.
+ const pixels = floatPixels(2, 2, (column, topRow) => {
+ const steps = topRow * 2 + column + 1;
+ return [Math.floor(steps / 256) / 255, (steps % 256) / 255, 0, 1];
+ });
+ const depth = decodeExr(encodeRenderOutput({ format: 'exr', pixels, width: 2, height: 2, pass: 'depth', depthRange: 65535 }));
+ assert.deepEqual([...depth.values.Z], [1, 2, 3, 4]);
+ const depthPng = decodePng16(encodeRenderOutput({ format: 'png16', pixels, width: 2, height: 2, pass: 'depth' }));
+ assert.deepEqual([depthPng.channels, depthPng.samples], [1, [1, 2, 3, 4]]);
+
+ const identifiers = floatPixels(2, 1, column => [(column + 1) / 255, 1 / 255, 0, 1]);
+ const identifierExr = decodeExr(encodeRenderOutput({ format: 'exr', pixels: identifiers, width: 2, height: 1, pass: 'objectId' }));
+ assert.deepEqual([...identifierExr.values.id], [257, 258]);
+ const tooMany = floatPixels(1, 1, () => [0, 0, 1 / 255, 1]);
+ assert.throws(() => encodeRenderOutput({ format: 'png16', pixels: tooMany, width: 1, height: 1, pass: 'objectId' }), /exr/);
+
+ const lit = floatPixels(2, 1, column => (column === 0 ? [0.25, 0.25, 0.25, 0.5] : [0, 0, 0, 0]));
+ const litPng = decodePng16(encodeRenderOutput({ format: 'png16', pixels: lit, width: 2, height: 1, hasAlpha: true }));
+ assert.deepEqual(litPng.samples, [32768, 32768, 32768, 32768, 0, 0, 0, 0]);
+ const opaquePng = decodePng16(encodeRenderOutput({ format: 'png16', pixels: lit, width: 2, height: 1 }));
+ assert.equal(opaquePng.channels, 3);
+ const litExr = decodeExr(encodeRenderOutput({ format: 'exr', pixels: lit, width: 2, height: 1, hasAlpha: true }));
+ assert.deepEqual(litExr.channels, ['A', 'B', 'G', 'R']);
+ assert.deepEqual([...litExr.values.R], [0.25, 0]);
+ assert.throws(() => encodeRenderOutput({ format: 'bmp', pixels: lit, width: 2, height: 1 }), /png16 or exr/);
+ assert.throws(() => encodeRenderOutput({ format: 'exr', pixels: lit, width: 3, height: 1 }), /pixel count/);
+});