Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Keep screenshots as the video default until raw capture is faster

Nguyen Xuan An5bef4eef0b89
diff --git a/changes/301.md b/changes/301.md
index 92d20789..f98cd607 100644
--- a/changes/301.md
+++ b/changes/301.md
@@ -1 +1 @@
-- Make raw capture the default for video in `scripts/render-media.mjs` and the MCP video job (issue #301). Each frame is read with `readPixels` in the draw task and sent to Node over one binary WebSocket with a one byte acknowledgement, which replaces the Blob post that failed after about 100 frames. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. `--capture jpeg` keeps the screenshot path, PNG stills are unchanged, and `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.
+- Add raw capture, opt in, to `scripts/render-media.mjs` and the MCP video job (issue #301). Each frame is read with `readPixels` in the draw task and sent to Node over one binary WebSocket with a one byte acknowledgement, which replaces the Blob post that failed after about 100 frames. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw. `--capture jpeg` keeps the screenshot path, PNG stills are unchanged, and `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.
diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.md
index 2005c72b..c178a34b 100644
--- a/docs/agent-render-jobs.md
+++ b/docs/agent-render-jobs.md
@@ -38,7 +38,7 @@ Cancellation becomes terminal after the owned encoder and job page stop and the
Video uses H.264 (CPU by default, see `settings.encoder`), even dimensions and one streamed frame at a time.
MCP video frames are read from the drawing buffer with `readPixels` in the same task as the draw and sent to Node over one binary WebSocket on the preview server port, with a one byte acknowledgement per frame so one frame is in flight and the encoder sets the pace (issue #301).
PNG stills and render passes still use a DevTools screenshot, byte for byte as before.
-`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs. A screenshot cost 28.8 ms a frame to capture and 7.4 ms to draw at 1280 by 720 on an RTX 3060. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met.
+`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs. A screenshot cost 28.8 ms a frame to capture and 7.4 ms to draw at 1280 by 720 on an RTX 3060. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw.
The render backend can use GPU or CPU. Memory limits can reduce the actual output dimensions.
Results report actual dimensions and measured timings. They do not embed video bytes.
diff --git a/scripts/render-media.mjs b/scripts/render-media.mjs
index ff1ca347..6db8af8d 100644
--- a/scripts/render-media.mjs
+++ b/scripts/render-media.mjs
@@ -53,7 +53,7 @@ export function parseRenderOptions(values) {
const start = numberOption('start', 0, 0, 3600);
const memory = numberOption('memory', preset.memoryBudgetMiB, 32, 1024);
const assembly = numberOption('assembly', 0, 0, 1);
- const captureFormat = options.capture ?? (kind === 'video' ? 'raw' : 'png');
+ const captureFormat = options.capture ?? (kind === 'video' ? 'jpeg' : 'png');
if (!['jpeg', 'png', 'raw'].includes(captureFormat)) throw new Error('Capture must be raw, jpeg or png.');
if (captureFormat === 'raw' && kind !== 'video') throw new Error('Raw capture is for video only. Stills are PNG or JPEG screenshots.');
const jpegQuality = numberOption('jpeg-quality', 95, 1, 100);
diff --git a/skills/render-67js-media/references/rendering.md b/skills/render-67js-media/references/rendering.md
index 2de4f7c5..69638530 100644
--- a/skills/render-67js-media/references/rendering.md
+++ b/skills/render-67js-media/references/rendering.md
@@ -40,7 +40,7 @@ The renderer does not silently replace a failed hardware export with a lower-qua
Video capture cost (measured on an RTX 3060 at 1280x720 on a loaded machine, 20 s of the film, 480 frames, NVENC).
Screenshots (`--capture jpeg`) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.
Every report has `stageTimings.captureSeconds` and `renderSeconds`.
-`--capture raw`, the default for video, reads the drawing buffer with `readPixels` (3.3 ms a frame) and sends it to Node over a binary WebSocket, one frame in flight, acknowledged by one byte. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met.
+`--capture raw`, the default for video, reads the drawing buffer with `readPixels` (3.3 ms a frame) and sends it to Node over a binary WebSocket, one frame in flight, acknowledged by one byte. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw.
Returning the pixels through Playwright instead cost 55 ms a frame, and a fetch with a Blob body failed after about 100 frames, so neither is used.
A decoded raw frame matches the PNG seek of the same time at 41.0 dB PSNR on SwiftShader (`npm run smoke:capture-parity`).
diff --git a/tests/export-options.test.js b/tests/export-options.test.js
index 46b1e4fc..e460c793 100644
--- a/tests/export-options.test.js
+++ b/tests/export-options.test.js
@@ -55,12 +55,12 @@ test('invalid export options fail before starting resources', () => {
]) assert.throws(() => parseRenderOptions(args));
});
-test('video defaults to raw drawing buffer reads and auto encoding while stills retain lossless PNG', () => {
+test('video defaults to JPEG 95 and auto encoding while stills retain lossless PNG', () => {
const video = parseRenderOptions(['--kind', 'video']);
- assert.equal(video.captureFormat, 'raw');
+ assert.equal(video.captureFormat, 'jpeg');
assert.equal(video.jpegQuality, 95);
assert.throws(() => parseRenderOptions(['--capture', 'raw']));
- assert.equal(parseRenderOptions(['--kind', 'video', '--capture', 'jpeg']).captureFormat, 'jpeg');
+ assert.equal(parseRenderOptions(['--kind', 'video', '--capture', 'raw']).captureFormat, 'raw');
assert.equal(video.encoderBackend, 'auto');
assert.equal(parseRenderOptions([]).captureFormat, 'png');
assert.equal(parseRenderOptions(['--kind', 'stills']).captureFormat, 'png');