3D engine that AI agents drive
Merge main, keeping session jobs and whole materials (#1036)
Nguyen Xuan And64a05ba8e07
diff --cc docs/agent-render-jobs.mdindex dd35aef3,60c9a647..a7c5d736--- a/docs/agent-render-jobs.md+++ b/docs/agent-render-jobs.md@@@ -185,7 -185,7 +185,7 @@@ An `image`, `stills` or `video` job ca"renderer": { "type": "pathTracer", "backend": "gpu", "samples": 64, "bounces": 8, "denoise": true, "seed": 0, "maxSeconds": 120 } }```- Every field after `type` is optional. `backend` is `cpu` (the default, byte-identical on SwiftShader smokes), `gpu`, which traces in a WebGL2 shader and needs a runtime started on the GPU backend (a `--backend cpu` runtime refuses it), or `native`, which traces on hardware ray queries (Vulkan) through the optional `@codepawlhq/67js-render-native` package. A native job exports the scene from the job page, traces it in the job runner with the Cycles-matching kernel (`JS67_NATIVE_KERNEL=legacy` keeps the older one), then denoises (OIDN runs on a device kept alive between frames when the OIDN library is installed, so only the first frame pays for the device and the filter commit; `metrics.pathTracer.oidnMode` says `in-process` or `cli`) and writes the PNG as the other backends do: `denoise: true` means OIDN with the albedo and normal guides when `@codepawlhq/67js-denoise-oidn` and its binary are installed, else (or when OIDN fails) the GPU a-trous filter of the native tracer with a note in `metrics.pathTracer.denoiserNote` (`denoiser: "atrous"`; it is biased, it smooths; `denoise: "wavelet"` asks for the CPU filter). The report carries `kernel` and `averageSamples`. On the Cycles-matching kernel the job sends each material whole (FORMAT-v2 tables, instanced or not): blend opacity (`alphaMode: "blend"` with `opacity` below 1), sheen, `specular` and `specularColor`, anisotropy, iridescence, attenuation, and the clearcoat, transmission, sheen, anisotropy and coat normal maps all reach the kernel, and `metrics.pathTracer.ignored` lists only what it does not shade (the legacy kernel keeps the v1 tables and the old notes). A scene whose materials cannot be exported whole is traced from the v1 tables for that frame with the reason in `materialsReason`; the native tracer stays usable. It never fails for lack of the package: without it, without its built binary or without a Vulkan ray query adapter the job runs on `gpu` (`cpu` on a cpu runtime) and `metrics.pathTracer.backendNote` says why; `metrics.pathTracer.backend` always names what ran. The look up costs one import and is cached per process. Native traces volume fog and subsurface scattering with the same walk as the WebGPU megakernel, honours `maxSeconds`, and honours `noiseThreshold` on the GPU (after 16 samples, and then every 8, a converged pixel draws no more; `report.averageSamples` is the mean a pixel drew). The `gpu` fallback ignores `noiseThreshold`. The report in `metrics.pathTracer` names the `backend` that ran, the `gpu`, `gpuCompileSeconds` and `gpuKernelCompiles`. A `stills` or `video` job traces every frame with the same options: the kernel compiles once per job page and each frame reuses it, so a film shows `gpuKernelCompiles: 1`. A video of traced frames goes through the ffmpeg encoder (the browser H.264 encoder has no drawn frame to read), `maxSeconds` limits each frame, and the job deadline is `maxSeconds` times the frame count plus 60 seconds, up to the job ceiling. Limits: `samples` 1 to 1024, `bounces` 1 to 16, `seed` 0 to 2147483647, `maxSeconds` 1 to 600 (the job's deadline is `maxSeconds` plus 60 seconds; a 1920 by 1080 still of a 180,000 triangle room at 64 samples took 246 s, so give a large still `maxSeconds` 600). `passes` are refused with it. The tracer loads in the job page on demand, uses eight threads at below-normal priority, stops pixels that have converged (`report.averageSamples`), and writes `render.png`; `scene_render_result` metrics carry `pathTracer` with the samples, threads, seconds and what the scene asked for that was not drawn. The same recipe and seed give the same PNG on the cpu and gpu backends; the native Cycles-matching kernel is not always byte-identical between runs (on the demo room at 960 by 540 and 256 samples two jobs differed in a few bytes; a 480 by 270 job with OIDN did not). A job that passes `maxSeconds` fails and writes no image.-Every field after `type` is optional. `backend` is `cpu` (the default, byte-identical on SwiftShader smokes), `gpu`, which traces in a WebGL2 shader and needs a runtime started on the GPU backend (a `--backend cpu` runtime refuses it), or `native`, which traces on hardware ray queries (Vulkan) through the optional `@codepawlhq/67js-render-native` package. A native job exports the scene from the job page, traces it in the job runner with the Cycles-matching kernel (`JS67_NATIVE_KERNEL=legacy` keeps the older one), then denoises (OIDN runs on a device kept alive between frames when the OIDN library is installed, so only the first frame pays for the device and the filter commit; `metrics.pathTracer.oidnMode` says `in-process` or `cli`) and writes the PNG as the other backends do: `denoise: true` means OIDN with the albedo and normal guides when `@codepawlhq/67js-denoise-oidn` and its binary are installed, else (or when OIDN fails) the GPU a-trous filter of the native tracer with a note in `metrics.pathTracer.denoiserNote` (`denoiser: "atrous"`; it is biased, it smooths; `denoise: "wavelet"` asks for the CPU filter). The report carries `kernel` and `averageSamples`. It never fails for lack of the package: without it, without its built binary or without a Vulkan ray query adapter the job runs on `gpu` (`cpu` on a cpu runtime) and `metrics.pathTracer.backendNote` says why; `metrics.pathTracer.backend` always names what ran. The look up costs one import and is cached per process. Native traces volume fog and subsurface scattering with the same walk as the WebGPU megakernel, honours `maxSeconds`, and honours `noiseThreshold` on the GPU (after 16 samples, and then every 8, a converged pixel draws no more; `report.averageSamples` is the mean a pixel drew). The `gpu` fallback ignores `noiseThreshold`. The report in `metrics.pathTracer` names the `backend` that ran, the `gpu`, `gpuCompileSeconds` and `gpuKernelCompiles`. A `stills` or `video` job traces every frame with the same options: the kernel compiles once per job page and each frame reuses it, so a film shows `gpuKernelCompiles: 1`. A native `stills` or `video` job (the Cycles-matching kernel) also keeps one native session for the whole job: a frame that only moves the camera reuses the uploaded scene, the compiled kernel and the acceleration structures; changed materials, lights or instance transforms are sent to the session; a frame whose counts, geometry, textures or environment changed gets a new session, and a failed session update or render falls back to a new session or a one-shot trace for that frame. `metrics.pathTracer.nativeSession` says which one each frame took (`path` is `new-session`, `reused`, `updated` or `one-shot`, with `reason`, `changedTables`, `pipelineReused` and `accelerationReused`); `metrics.nativeSessionRoutes` lists the `path` of every frame. A job reuses the session only when the tables the page exports are equal to the previous frame's apart from the camera (`reused`), or differ in material values, lights or instance transforms only (`updated`). Anything that rebuilds geometry from the time, such as particle systems (sparks, swaying hair), a moved node in a scene exported as flattened triangles, or a changed emissive triangle count, gives every frame a new session, which costs what a one-shot trace costs and is byte-identical to it; `reason` names the first table that differed. A single-frame job and `JS67_NATIVE_KERNEL=legacy` trace one-shot and carry no `nativeSession`; `JS67_NATIVE_SESSIONS=off` in the server's environment traces every frame one-shot (default on). The session is closed when the job ends or fails, and every frame equals the one-shot render byte for byte. A video of traced frames goes through the ffmpeg encoder (the browser H.264 encoder has no drawn frame to read), `maxSeconds` limits each frame, and the job deadline is `maxSeconds` times the frame count plus 60 seconds, up to the job ceiling. Limits: `samples` 1 to 1024, `bounces` 1 to 16, `seed` 0 to 2147483647, `maxSeconds` 1 to 600 (the job's deadline is `maxSeconds` plus 60 seconds; a 1920 by 1080 still of a 180,000 triangle room at 64 samples took 246 s, so give a large still `maxSeconds` 600). `passes` are refused with it. The tracer loads in the job page on demand, uses eight threads at below-normal priority, stops pixels that have converged (`report.averageSamples`), and writes `render.png`; `scene_render_result` metrics carry `pathTracer` with the samples, threads, seconds and what the scene asked for that was not drawn. The same recipe and seed give the same PNG on the cpu and gpu backends; the native Cycles-matching kernel is not always byte-identical between runs (on the demo room at 960 by 540 and 256 samples two jobs differed in a few bytes; a 480 by 270 job with OIDN did not). A job that passes `maxSeconds` fails and writes no image.++Every field after `type` is optional. `backend` is `cpu` (the default, byte-identical on SwiftShader smokes), `gpu`, which traces in a WebGL2 shader and needs a runtime started on the GPU backend (a `--backend cpu` runtime refuses it), or `native`, which traces on hardware ray queries (Vulkan) through the optional `@codepawlhq/67js-render-native` package. A native job exports the scene from the job page, traces it in the job runner with the Cycles-matching kernel (`JS67_NATIVE_KERNEL=legacy` keeps the older one), then denoises (OIDN runs on a device kept alive between frames when the OIDN library is installed, so only the first frame pays for the device and the filter commit; `metrics.pathTracer.oidnMode` says `in-process` or `cli`) and writes the PNG as the other backends do: `denoise: true` means OIDN with the albedo and normal guides when `@codepawlhq/67js-denoise-oidn` and its binary are installed, else (or when OIDN fails) the GPU a-trous filter of the native tracer with a note in `metrics.pathTracer.denoiserNote` (`denoiser: "atrous"`; it is biased, it smooths; `denoise: "wavelet"` asks for the CPU filter). The report carries `kernel` and `averageSamples`. On the Cycles-matching kernel the job sends each material whole (FORMAT-v2 tables, instanced or not): blend opacity (`alphaMode: "blend"` with `opacity` below 1), sheen, `specular` and `specularColor`, anisotropy, iridescence, attenuation, and the clearcoat, transmission, sheen, anisotropy and coat normal maps all reach the kernel, and `metrics.pathTracer.ignored` lists only what it does not shade (the legacy kernel keeps the v1 tables and the old notes). A scene whose materials cannot be exported whole is traced from the v1 tables for that frame with the reason in `materialsReason`; the native tracer stays usable. It never fails for lack of the package: without it, without its built binary or without a Vulkan ray query adapter the job runs on `gpu` (`cpu` on a cpu runtime) and `metrics.pathTracer.backendNote` says why; `metrics.pathTracer.backend` always names what ran. The look up costs one import and is cached per process. Native traces volume fog and subsurface scattering with the same walk as the WebGPU megakernel, honours `maxSeconds`, and honours `noiseThreshold` on the GPU (after 16 samples, and then every 8, a converged pixel draws no more; `report.averageSamples` is the mean a pixel drew). The `gpu` fallback ignores `noiseThreshold`. The report in `metrics.pathTracer` names the `backend` that ran, the `gpu`, `gpuCompileSeconds` and `gpuKernelCompiles`. A `stills` or `video` job traces every frame with the same options: the kernel compiles once per job page and each frame reuses it, so a film shows `gpuKernelCompiles: 1`. A native `stills` or `video` job (the Cycles-matching kernel) also keeps one native session for the whole job: a frame that only moves the camera reuses the uploaded scene, the compiled kernel and the acceleration structures; changed materials, lights or instance transforms are sent to the session; a frame whose counts, geometry, textures or environment changed gets a new session, and a failed session update or render falls back to a new session or a one-shot trace for that frame. `metrics.pathTracer.nativeSession` says which one each frame took (`path` is `new-session`, `reused`, `updated` or `one-shot`, with `reason`, `changedTables`, `pipelineReused` and `accelerationReused`); `metrics.nativeSessionRoutes` lists the `path` of every frame. A job reuses the session only when the tables the page exports are equal to the previous frame's apart from the camera (`reused`), or differ in material values, lights or instance transforms only (`updated`). Anything that rebuilds geometry from the time, such as particle systems (sparks, swaying hair), a moved node in a scene exported as flattened triangles, or a changed emissive triangle count, gives every frame a new session, which costs what a one-shot trace costs and is byte-identical to it; `reason` names the first table that differed. A single-frame job and `JS67_NATIVE_KERNEL=legacy` trace one-shot and carry no `nativeSession`; `JS67_NATIVE_SESSIONS=off` in the server's environment traces every frame one-shot (default on). The session is closed when the job ends or fails, and every frame equals the one-shot render byte for byte. A video of traced frames goes through the ffmpeg encoder (the browser H.264 encoder has no drawn frame to read), `maxSeconds` limits each frame, and the job deadline is `maxSeconds` times the frame count plus 60 seconds, up to the job ceiling. Limits: `samples` 1 to 1024, `bounces` 1 to 16, `seed` 0 to 2147483647, `maxSeconds` 1 to 600 (the job's deadline is `maxSeconds` plus 60 seconds; a 1920 by 1080 still of a 180,000 triangle room at 64 samples took 246 s, so give a large still `maxSeconds` 600). `passes` are refused with it. The tracer loads in the job page on demand, uses eight threads at below-normal priority, stops pixels that have converged (`report.averageSamples`), and writes `render.png`; `scene_render_result` metrics carry `pathTracer` with the samples, threads, seconds and what the scene asked for that was not drawn. The same recipe and seed give the same PNG on the cpu and gpu backends; the native Cycles-matching kernel is not always byte-identical between runs (on the demo room at 960 by 540 and 256 samples two jobs differed in a few bytes; a 480 by 270 job with OIDN did not). A job that passes `maxSeconds` fails and writes no image.## Controller APIdiff --cc scripts/agent-native-trace.mjsindex 8eb5bdd8,251908c8..31ffc860--- a/scripts/agent-native-trace.mjs+++ b/scripts/agent-native-trace.mjs@@@ -205,8 -210,8 +210,9 @@@ export async function traceFrameNative(gpu: traced.adapter, nativeBackend: traced.backend, nativeFeatures: traced.features, setupSeconds: traced.setupSeconds, compileSeconds: traced.compileSeconds,uploadSeconds: traced.uploadSeconds, accelerationBuildSeconds: traced.accelerationBuildSeconds,kernel, averageSamples: traced.averageSamples ?? samples, noiseThreshold, triangles: exported.triangles, inputBytes: tableBytes(tables), ...(exported.instances === undefined ? {} : { instances: exported.instances }),- ...(exported.instancing === undefined ? {} : { instancing: exported.instancing, instancingReason: exported.instancingReason }), tracedSeconds, ignored: exported.ignored,+ ...(exported.instancing === undefined ? {} : { instancing: exported.instancing, instancingReason: exported.instancingReason }),- ...(exported.materialsReason === undefined ? {} : { materialsReason: exported.materialsReason }), tracedSeconds, ignored: exported.ignored };++ ...(exported.materialsReason === undefined ? {} : { materialsReason: exported.materialsReason }), tracedSeconds, ignored: exported.ignored,+ ...(route ? { nativeSession: route } : {}) };let linear = features.color;const denoiseStarted = performance.now();if (denoise === 'oidn') {diff --cc scripts/agent-runtime.mjsindex e3ec7627,c5caa7d3..cbf4d269--- a/scripts/agent-runtime.mjs+++ b/scripts/agent-runtime.mjs@@@ -15,7 -15,8 +15,8 @@@ import { MATERIAL_SNIPPET_CAPABILITIESimport { AGENT_SCENE_CAPABILITIES, AGENT_SCENE_DEFAULT_LIMITS, RENDER_PASS_NAMES, RENDER_PASS_DEPTH_RANGE, VIDEO_ENCODERS, MAXIMUM_VIDEO_SECONDS, MAXIMUM_JOB_SIDE, MAXIMUM_JOB_PIXELS, MAXIMUM_MEMORY_MIB, BAKE_WORKER_MAXIMUM_SECONDS } from '../src/agent-scene-capabilities.js';import { PATH_TRACER_CAPABILITIES } from '../src/path-tracer-capabilities.js';import { chooseDenoiser, finishWithOidn } from './agent-path-trace-denoise.mjs';+ import { createNativeSequence } from './agent-native-sequence.mjs';-import { chooseNativeInstancing, chooseTraceBackend, createNativeTracerLocator, isNativeTimeLimitError, traceFrameNative, traceTexelsNative } from './agent-native-trace.mjs';+import { chooseNativeInstancing, chooseNativeKernel, chooseTraceBackend, createNativeTracerLocator, isNativeTimeLimitError, traceFrameNative, traceTexelsNative } from './agent-native-trace.mjs';import { createAgentAssetCatalog } from './agent-asset-catalog.mjs';import { geometryOnlyGlb } from './agent-query-asset.mjs';import { AGENT_ASSET_LIMITS, assertCombinedAssetBytes } from '../src/agent-asset-limits.js';@@@ -566,9 -569,10 +569,10 @@@ export function createAgentRuntime({ keif (trace.backend === 'native') {let exportedScene = false;try {- const exported = await jobPage.evaluate(input => window.__agentPreview.exportNativeScene(input), { ...frameInput, options: { width, height, bounces: tracerOptions.bounces, seed: tracerOptions.seed, fireflyClamp: 20, noiseThreshold: tracerOptions.noiseThreshold, instancing: chooseNativeInstancing() } });+ const exported = await jobPage.evaluate(input => window.__agentPreview.exportNativeScene(input), { ...frameInput, options: { width, height, bounces: tracerOptions.bounces, seed: tracerOptions.seed, fireflyClamp: 20, noiseThreshold: tracerOptions.noiseThreshold, samples: tracerOptions.samples, instancing: chooseNativeInstancing(), kernel: chooseNativeKernel() } });exportedScene = true;- traced = await traceFrameNative({ tracer: trace.tracer, exported, width, height, samples: tracerOptions.samples, bounces: tracerOptions.bounces,+ if (settings.frameCount > 1 && process.env.JS67_NATIVE_SESSIONS !== 'off') nativeSequence ??= createNativeSequence({ tracer: trace.tracer });+ traced = await traceFrameNative({ tracer: trace.tracer, sequence: nativeSequence, exported, width, height, samples: tracerOptions.samples, bounces: tracerOptions.bounces,seed: tracerOptions.seed, denoise: choice.denoise === true ? 'wavelet' : choice.denoise === false ? 'none' : choice.denoise, denoiser: choice.denoiser,maxSeconds: tracerOptions.maxSeconds, noiseThreshold: tracerOptions.noiseThreshold });} catch (error) {