3D engine that AI agents drive
Document the persistent OIDN device, add the change note and record the measured saving (#1025)
Nguyen Xuan An3c778710ff7a
diff --git a/changes/1025.md b/changes/1025.mdnew file mode 100644index 00000000..baf6bc50--- /dev/null+++ b/changes/1025.md@@ -0,0 +1 @@+- Native render jobs and the Cook comparison keep one OIDN device and filter loaded in the render-native addon, so a denoise after the first skips the tool start, device creation and PFM files, and `createOidnDenoiser` and `findOidnLibrary` in the denoise package expose it (#1025)diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.mdindex 674d7d55..2ee94f4b 100644--- a/docs/agent-render-jobs.md+++ b/docs/agent-render-jobs.md@@ -185,7 +185,7 @@ An `image`, `stills` or `video` job can ask for the offline [path tracer](https:"renderer": { "type": "pathTracer", "backend": "gpu", "samples": 64, "bounces": 8, "denoise": true, "seed": 0, "maxSeconds": 120 } }```-Every field after `type` is optional. `backend` is `cpu` (the default, byte-identical on SwiftShader smokes), `gpu`, which traces in a WebGL2 shader and needs a runtime started on the GPU backend (a `--backend cpu` runtime refuses it), or `native`, which traces on hardware ray queries (Vulkan) through the optional `@codepawlhq/67js-render-native` package. A native job exports the scene from the job page, traces it in the job runner with the Cycles-matching kernel (`JS67_NATIVE_KERNEL=legacy` keeps the older one), then denoises and writes the PNG as the other backends do: `denoise: true` means OIDN with the albedo and normal guides when `@codepawlhq/67js-denoise-oidn` and its binary are installed, else (or when OIDN fails) the GPU a-trous filter of the native tracer with a note in `metrics.pathTracer.denoiserNote` (`denoiser: "atrous"`; it is biased, it smooths; `denoise: "wavelet"` asks for the CPU filter). The report carries `kernel` and `averageSamples`. It never fails for lack of the package: without it, without its built binary or without a Vulkan ray query adapter the job runs on `gpu` (`cpu` on a cpu runtime) and `metrics.pathTracer.backendNote` says why; `metrics.pathTracer.backend` always names what ran. The look up costs one import and is cached per process. Native traces volume fog and subsurface scattering with the same walk as the WebGPU megakernel, honours `maxSeconds`, and honours `noiseThreshold` on the GPU (after 16 samples, and then every 8, a converged pixel draws no more; `report.averageSamples` is the mean a pixel drew). The `gpu` fallback ignores `noiseThreshold`. The report in `metrics.pathTracer` names the `backend` that ran, the `gpu`, `gpuCompileSeconds` and `gpuKernelCompiles`. A `stills` or `video` job traces every frame with the same options: the kernel compiles once per job page and each frame reuses it, so a film shows `gpuKernelCompiles: 1`. A video of traced frames goes through the ffmpeg encoder (the browser H.264 encoder has no drawn frame to read), `maxSeconds` limits each frame, and the job deadline is `maxSeconds` times the frame count plus 60 seconds, up to the job ceiling. Limits: `samples` 1 to 1024, `bounces` 1 to 16, `seed` 0 to 2147483647, `maxSeconds` 1 to 600 (the job's deadline is `maxSeconds` plus 60 seconds; a 1920 by 1080 still of a 180,000 triangle room at 64 samples took 246 s, so give a large still `maxSeconds` 600). `passes` are refused with it. The tracer loads in the job page on demand, uses eight threads at below-normal priority, stops pixels that have converged (`report.averageSamples`), and writes `render.png`; `scene_render_result` metrics carry `pathTracer` with the samples, threads, seconds and what the scene asked for that was not drawn. The same recipe and seed give the same PNG on the cpu and gpu backends; the native Cycles-matching kernel is not always byte-identical between runs (on the demo room at 960 by 540 and 256 samples two jobs differed in a few bytes; a 480 by 270 job with OIDN did not). A job that passes `maxSeconds` fails and writes no image.+Every field after `type` is optional. `backend` is `cpu` (the default, byte-identical on SwiftShader smokes), `gpu`, which traces in a WebGL2 shader and needs a runtime started on the GPU backend (a `--backend cpu` runtime refuses it), or `native`, which traces on hardware ray queries (Vulkan) through the optional `@codepawlhq/67js-render-native` package. A native job exports the scene from the job page, traces it in the job runner with the Cycles-matching kernel (`JS67_NATIVE_KERNEL=legacy` keeps the older one), then denoises (OIDN runs on a device kept alive between frames when the OIDN library is installed, so only the first frame pays for the device and the filter commit; `metrics.pathTracer.oidnMode` says `in-process` or `cli`) and writes the PNG as the other backends do: `denoise: true` means OIDN with the albedo and normal guides when `@codepawlhq/67js-denoise-oidn` and its binary are installed, else (or when OIDN fails) the GPU a-trous filter of the native tracer with a note in `metrics.pathTracer.denoiserNote` (`denoiser: "atrous"`; it is biased, it smooths; `denoise: "wavelet"` asks for the CPU filter). The report carries `kernel` and `averageSamples`. It never fails for lack of the package: without it, without its built binary or without a Vulkan ray query adapter the job runs on `gpu` (`cpu` on a cpu runtime) and `metrics.pathTracer.backendNote` says why; `metrics.pathTracer.backend` always names what ran. The look up costs one import and is cached per process. Native traces volume fog and subsurface scattering with the same walk as the WebGPU megakernel, honours `maxSeconds`, and honours `noiseThreshold` on the GPU (after 16 samples, and then every 8, a converged pixel draws no more; `report.averageSamples` is the mean a pixel drew). The `gpu` fallback ignores `noiseThreshold`. The report in `metrics.pathTracer` names the `backend` that ran, the `gpu`, `gpuCompileSeconds` and `gpuKernelCompiles`. A `stills` or `video` job traces every frame with the same options: the kernel compiles once per job page and each frame reuses it, so a film shows `gpuKernelCompiles: 1`. A video of traced frames goes through the ffmpeg encoder (the browser H.264 encoder has no drawn frame to read), `maxSeconds` limits each frame, and the job deadline is `maxSeconds` times the frame count plus 60 seconds, up to the job ceiling. Limits: `samples` 1 to 1024, `bounces` 1 to 16, `seed` 0 to 2147483647, `maxSeconds` 1 to 600 (the job's deadline is `maxSeconds` plus 60 seconds; a 1920 by 1080 still of a 180,000 triangle room at 64 samples took 246 s, so give a large still `maxSeconds` 600). `passes` are refused with it. The tracer loads in the job page on demand, uses eight threads at below-normal priority, stops pixels that have converged (`report.averageSamples`), and writes `render.png`; `scene_render_result` metrics carry `pathTracer` with the samples, threads, seconds and what the scene asked for that was not drawn. The same recipe and seed give the same PNG on the cpu and gpu backends; the native Cycles-matching kernel is not always byte-identical between runs (on the demo room at 960 by 540 and 256 samples two jobs differed in a few bytes; a 480 by 270 job with OIDN did not). A job that passes `maxSeconds` fails and writes no image.## Controller APIdiff --git a/docs/cycles-parity.md b/docs/cycles-parity.mdindex f98f5dc4..ecc9fe9c 100644--- a/docs/cycles-parity.md+++ b/docs/cycles-parity.md@@ -819,6 +819,23 @@ uv run --no-project python bench/cycles-parity/summarize-denoised-comparison.pyuv run --no-project --with numpy --with pillow python bench/cycles-parity/make-denoised-comparison-figure.py --rows <rows.json> --images <folder> --reference <reference.npy> --crop-x 300 --crop-y 255 --output docs/cook-denoised-comparison.jpg```+## Persistent in-process OIDN (#1025)++The `oidnDenoise` tool pays for a process start, CUDA initialisation, the weights and three PFM files on every call. `createNativeOidnDenoiser` (render-native) and `createOidnDenoiser` (denoise package) keep one OIDN device, four buffers and the committed filter alive, and recommit only when the size or the set of guides changes. `trace-native.mjs --oidn-devices cuda,cuda-cli,cpu,cpu-cli` runs the in-process device (`cuda`) beside the tool (`cuda-cli`) on one trace; `node bench/oidn-cost/measure-oidn-cost.mjs --out <file>` writes the split below. Every number is measured on one Modal container with an NVIDIA A10 (driver 610.57) and a 24 thread Xeon, OIDN 2.5.1, quality high, colour plus albedo plus normal, seeded synthetic input, median of seven calls.++| size | device | tool call s | of which process start to exit | in-process first call s (plus device creation) | in-process warm call s | warm: write / execute / read ms |+| --- | --- | ---: | ---: | ---: | ---: | --- |+| 960 x 540 | cuda | 0.345 | 0.320 | 0.015 (0.103) | 0.0072 | 1.3 / 3.5 / 2.2 |+| 960 x 540 | cpu | 0.218 | 0.194 | 0.075 (0.010) | 0.048 | 1.8 / 43.5 / 2.8 |+| 1920 x 1080 | cuda | 0.787 | 0.671 | 0.033 (0.090) | 0.029 | 7.2 / 13.6 / 7.7 |+| 1920 x 1080 | cpu | 0.732 | 0.624 | 0.223 (0.002) | 0.178 | 6.5 / 162.5 / 8.3 |++The tool call also spends 0.010 / 0.007 s encoding and writing PFM files and 0.007 s reading one back at 960 x 540 (0.034 / 0.054 / 0.023 s at 1920 x 1080). A bare `oidnDenoise --version` takes 0.007 s, and `--verbose 2` shows 0.110 s for the CUDA device, 0.005 s for the filter commit and 0.007 s for the filter at 960 x 540, so about 0.2 s of the 0.32 s process is CUDA runtime start and teardown that the verbose lines do not count. The three images cost 0.007 s (960 x 540) and 0.024 s (1920 x 1080) to copy in JS, which is the floor left by not sharing memory between the Vulkan tracer and CUDA.++Output: the in-process image equals the tool's image byte for byte (maximum absolute difference 0) on all four rows, the first call equals a warm call, and so does the call after a size change and back. Resizing recommits the filter; warm calls never do.++Equal total time on `lit-interior` and `many-lights-500` (same container, 3 rounds, seed 3, medians): the in-process CUDA denoise costs 0.008 s against 0.34 to 0.38 s for the tool, so the total of Cook + OIDN falls from 0.56 to 0.23 s at 16 spp on `lit-interior`, with the same relMSE (3.09e-4) as the tool path. The in-process CPU device costs 0.047 s against 0.22 s. The table is in the pull request.+## Depth and grey-diffuse comparison (#896, slice C1a)The Done-when item of #896 for the Classroom scene asks for two checks against Cycles with every material overridden to grey diffuse: the image-match gates and a depth pass that agrees on 99.9 percent of pixels. This slice adds the harness for both and proves it on two existing scenes; the Classroom scene itself is a later slice. Only `bench/`, `tests/` and this guide change.diff --git a/packages/denoise-oidn/README.md b/packages/denoise-oidn/README.mdindex 1a9acaca..3e13aa38 100644--- a/packages/denoise-oidn/README.md+++ b/packages/denoise-oidn/README.md@@ -9,4 +9,16 @@ npm run fetch --prefix node_modules/@codepawlhq/67js-denoise-oidn # downloadsThen ask a render job for it: `settings.renderer: { type: "pathTracer", denoise: "oidn" }`. Samples default to 16 for `oidn`. If the package or its binary is missing, the job uses the wavelet filter (the GPU a-trous filter of the native tracer on `backend: "native"`) and says so in `metrics.pathTracer.denoiserNote`. `OIDN_DENOISE_PATH` points at an `oidnDenoise` you installed yourself.+A render job on `backend: "native"` keeps one OIDN device and filter alive in the render-native addon, so the second and later frames skip the process start, the device creation, the weights load and the PFM files. It needs the shared library from the same release (`findOidnLibrary()` looks next to the tool, in `OIDN_LIBRARY_PATH` and in ./vendor) and a built addon; without either, the job runs the `oidnDenoise` tool as before and `metrics.pathTracer.denoiserNote` says why. The report's `oidnMode` is `"in-process"` or `"cli"`.++```js+import { createNativeOidnDenoiser } from '@codepawlhq/67js-render-native';+import { createOidnDenoiser } from '@codepawlhq/67js-denoise-oidn';++const denoiser = createOidnDenoiser({ createNative: createNativeOidnDenoiser, device: 'cuda' }); // throws, with what to install, when the library or addon is missing+const filtered = await denoiser.denoise({ width, height, color, albedo, normal }); // the first call commits the filter; a changed size recommits+denoiser.lastTimings; // { recommitted, commitSeconds, writeSeconds, executeSeconds, readSeconds } of the last call+denoiser.close();+```+`denoiseWithOidn({ width, height, color, albedo, normal })` takes row-major `Float32Array`s with three floats per pixel and returns the filtered colour. It runs on the CPU by default, so the GPU stays free and the result does not depend on a graphics driver. On one machine the same input gave the same bytes twice (measured).diff --git a/packages/render-native/README.md b/packages/render-native/README.mdindex 2b34db14..f0cca0e6 100644--- a/packages/render-native/README.md+++ b/packages/render-native/README.md@@ -17,6 +17,8 @@ const input = buildNativeInput(traceScene, camera, options, pack);const { rgba, adapter, tracedSeconds } = traceNative(input, options);```+`createNativeOidnDenoiser({ libraryPath, device = 'cpu', quality = 'high', cleanAux = false, threads = 0 })` opens a persistent Open Image Denoise device, four image buffers and a committed "RT" filter inside the addon. The OIDN shared library is loaded at run time from `libraryPath` (`findOidnLibrary()` of the denoise package), so the addon builds and loads without it. The object has `denoise({ width, height, color, albedo, normal })` (`albedo` optional, `normal` only with `albedo`; returns the colour and the seconds of the commit, buffer writes, execute and read), `close()` and `deviceSeconds`. The filter is recommitted only when the width, height or set of guides changes. The image goes to the device with `oidnWriteBuffer` and back with `oidnReadBuffer`, because GPU devices take no host pointers; the tracer's Vulkan buffers are not shared with CUDA. Most jobs use the wrapper `createOidnDenoiser` of `@codepawlhq/67js-denoise-oidn`.+`rgba` is linear float RGBA, rows from the top, averaged over the samples. `albedo` and `normal` (three floats per pixel, normalise the normal), `depth` and `variance` (one float) are the first-hit features the wavelet and OIDN denoisers read. `pack-input.mjs` holds `buildNativeInput` without Node imports, so the job page packs the scene itself. `bench/native-tracer` compares it with the browser kernel.Fog (`scene.volume`), subsurface scattering and `options.noiseThreshold` (the CPU tracer's stop rule: converged pixels draw no more samples, `averageSamples` in the result is the mean drawn) are in the kernel. `traceNative` also takes `maxSeconds` (stopping with an error once the trace has run that long) and `kernel` (`"legacy"` by default, or `"cycles"`, with `JS67_NATIVE_KERNEL` as environment fallback; see [FORMAT-v2.md](./FORMAT-v2.md)). Alpha cutouts are traced by stepping through transparent texels in the kernel (the acceleration structure is opaque).