Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Document the WebSocket raw capture and its checks

Nguyen Xuan An4b86d59ffccc
diff --git a/changes/301.md b/changes/301.md
index c0353434..286e6f71 100644
--- a/changes/301.md
+++ b/changes/301.md
@@ -1 +1 @@
-- Add `--capture raw` to `scripts/render-media.mjs` (issue #301), which reads each video frame from the drawing buffer with `readPixels` and pipes RGBA to ffmpeg instead of taking a JPEG screenshot, and make the script pick a free port and read Vite's coloured ready line. It is opt in until it survives a full film (a 20 s clip ran 97 frames at 55 fps against 22 fps with screenshots, then a post failed).
+- Make raw capture the default for video in `scripts/render-media.mjs` and the MCP video job (issue #301). Each frame is read with `readPixels` in the draw task and sent to Node over one binary WebSocket with a one byte acknowledgement, which replaces the Blob post that failed after about 100 frames. FILM_NUMBERS `--capture jpeg` keeps the screenshot path, PNG stills are unchanged, and `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.
diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.md
index 0478a154..5d491d4d 100644
--- a/docs/agent-render-jobs.md
+++ b/docs/agent-render-jobs.md
@@ -36,8 +36,9 @@ The source revision, pose and checkpoints survive completion or cancellation.
Cancellation becomes terminal after the owned encoder and job page stop and the preview recovers.
Video uses H.264 (CPU by default, see `settings.encoder`), even dimensions and one streamed frame at a time.
-MCP video jobs still take a JPEG screenshot per frame. Measured at 1280 by 720 on an RTX 3060, capture is 28.8 ms and the draw 7.4 ms a frame, and `metrics.captureSeconds` and `metrics.renderSeconds` report both.
-A raw read of the drawing buffer exists in `scripts/render-media.mjs` as `--capture raw` and is not yet in the MCP job (issue #301).
+MCP video frames are read from the drawing buffer with `readPixels` in the same task as the draw and sent to Node over one binary WebSocket on the preview server port, with a one byte acknowledgement per frame so one frame is in flight and the encoder sets the pace (issue #301).
+PNG stills and render passes still use a DevTools screenshot, byte for byte as before.
+`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs. A screenshot cost 28.8 ms a frame to capture and 7.4 ms to draw at 1280 by 720 on an RTX 3060. FILM_NUMBERS
The render backend can use GPU or CPU. Memory limits can reduce the actual output dimensions.
Results report actual dimensions and measured timings. They do not embed video bytes.
diff --git a/skills/render-67js-media/references/rendering.md b/skills/render-67js-media/references/rendering.md
index 818dfbf8..d3da10db 100644
--- a/skills/render-67js-media/references/rendering.md
+++ b/skills/render-67js-media/references/rendering.md
@@ -38,12 +38,11 @@ The hardware encoder requires NVENC. A failed encoder stops the export and repor
The renderer does not silently replace a failed hardware export with a lower-quality output.
Video capture cost (measured on an RTX 3060 at 1280x720 on a loaded machine, 20 s of the film, 480 frames, NVENC).
-Screenshots (`--capture jpeg`, the default) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.
+Screenshots (`--capture jpeg`) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.
Every report has `stageTimings.captureSeconds` and `renderSeconds`.
-`--capture raw` reads the drawing buffer with `readPixels` (3.3 ms a frame) and posts it to a local receiver as a binary body.
-It ran 97 frames at 55 fps, about 18 ms a frame in all, and then a post failed, so it is not the default yet and a full film is not proven.
-Returning the pixels through Playwright instead cost 55 ms a frame, worse than a screenshot.
-A typed-array post makes Playwright keep 3.7 MB per frame in Node. A Blob post does not.
+`--capture raw`, the default for video, reads the drawing buffer with `readPixels` (3.3 ms a frame) and sends it to Node over a binary WebSocket, one frame in flight, acknowledged by one byte. FILM_NUMBERS
+Returning the pixels through Playwright instead cost 55 ms a frame, and a fetch with a Blob body failed after about 100 frames, so neither is used.
+A decoded raw frame matches the PNG seek of the same time at 41.0 dB PSNR on SwiftShader (`npm run smoke:capture-parity`).
`scripts/score.mjs` generates the original film score.
The `--audio <path>` option selects an existing score.