3D engine that AI agents drive
Document the WebSocket raw capture and its checks
Nguyen Xuan An4b86d59ffccc
diff --git a/changes/301.md b/changes/301.mdindex c0353434..286e6f71 100644--- a/changes/301.md+++ b/changes/301.md@@ -1 +1 @@-- Add `--capture raw` to `scripts/render-media.mjs` (issue #301), which reads each video frame from the drawing buffer with `readPixels` and pipes RGBA to ffmpeg instead of taking a JPEG screenshot, and make the script pick a free port and read Vite's coloured ready line. It is opt in until it survives a full film (a 20 s clip ran 97 frames at 55 fps against 22 fps with screenshots, then a post failed).+- Make raw capture the default for video in `scripts/render-media.mjs` and the MCP video job (issue #301). Each frame is read with `readPixels` in the draw task and sent to Node over one binary WebSocket with a one byte acknowledgement, which replaces the Blob post that failed after about 100 frames. FILM_NUMBERS `--capture jpeg` keeps the screenshot path, PNG stills are unchanged, and `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.mdindex 0478a154..5d491d4d 100644--- a/docs/agent-render-jobs.md+++ b/docs/agent-render-jobs.md@@ -36,8 +36,9 @@ The source revision, pose and checkpoints survive completion or cancellation.Cancellation becomes terminal after the owned encoder and job page stop and the preview recovers.Video uses H.264 (CPU by default, see `settings.encoder`), even dimensions and one streamed frame at a time.-MCP video jobs still take a JPEG screenshot per frame. Measured at 1280 by 720 on an RTX 3060, capture is 28.8 ms and the draw 7.4 ms a frame, and `metrics.captureSeconds` and `metrics.renderSeconds` report both.-A raw read of the drawing buffer exists in `scripts/render-media.mjs` as `--capture raw` and is not yet in the MCP job (issue #301).+MCP video frames are read from the drawing buffer with `readPixels` in the same task as the draw and sent to Node over one binary WebSocket on the preview server port, with a one byte acknowledgement per frame so one frame is in flight and the encoder sets the pace (issue #301).+PNG stills and render passes still use a DevTools screenshot, byte for byte as before.+`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs. A screenshot cost 28.8 ms a frame to capture and 7.4 ms to draw at 1280 by 720 on an RTX 3060. FILM_NUMBERSThe render backend can use GPU or CPU. Memory limits can reduce the actual output dimensions.Results report actual dimensions and measured timings. They do not embed video bytes.diff --git a/skills/render-67js-media/references/rendering.md b/skills/render-67js-media/references/rendering.mdindex 818dfbf8..d3da10db 100644--- a/skills/render-67js-media/references/rendering.md+++ b/skills/render-67js-media/references/rendering.md@@ -38,12 +38,11 @@ The hardware encoder requires NVENC. A failed encoder stops the export and reporThe renderer does not silently replace a failed hardware export with a lower-quality output.Video capture cost (measured on an RTX 3060 at 1280x720 on a loaded machine, 20 s of the film, 480 frames, NVENC).-Screenshots (`--capture jpeg`, the default) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.+Screenshots (`--capture jpeg`) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.Every report has `stageTimings.captureSeconds` and `renderSeconds`.-`--capture raw` reads the drawing buffer with `readPixels` (3.3 ms a frame) and posts it to a local receiver as a binary body.-It ran 97 frames at 55 fps, about 18 ms a frame in all, and then a post failed, so it is not the default yet and a full film is not proven.-Returning the pixels through Playwright instead cost 55 ms a frame, worse than a screenshot.-A typed-array post makes Playwright keep 3.7 MB per frame in Node. A Blob post does not.+`--capture raw`, the default for video, reads the drawing buffer with `readPixels` (3.3 ms a frame) and sends it to Node over a binary WebSocket, one frame in flight, acknowledged by one byte. FILM_NUMBERS+Returning the pixels through Playwright instead cost 55 ms a frame, and a fetch with a Blob body failed after about 100 frames, so neither is used.+A decoded raw frame matches the PNG seek of the same time at 41.0 dB PSNR on SwiftShader (`npm run smoke:capture-parity`).`scripts/score.mjs` generates the original film score.The `--audio <path>` option selects an existing score.