Skip to content

armgit / 67.js

Public Imported from GitHub
0

3D engine that AI agents drive

Make in-page WebCodecs encoding the default video capture (part of #301)

Nguyen Xuan An92369b7c2356
diff --git a/changes/301.md b/changes/301.md
index f98cd607..159156c7 100644
--- a/changes/301.md
+++ b/changes/301.md
@@ -1 +1 @@
-- Add raw capture, opt in, to `scripts/render-media.mjs` and the MCP video job (issue #301). Each frame is read with `readPixels` in the draw task and sent to Node over one binary WebSocket with a one byte acknowledgement, which replaces the Blob post that failed after about 100 frames. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw. `--capture jpeg` keeps the screenshot path, PNG stills are unchanged, and `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.
+- Encode video frames in the page with WebCodecs (issue #301). `scripts/render-media.mjs` and the MCP video job draw a frame, hand the canvas to the browser's hardware H.264 `VideoEncoder` in the same task, and send only the encoded chunks (5 to 60 kB a frame) to Node over one binary WebSocket, where `ffmpeg -c copy` muxes them with the audio. The full 4,488 frame film at 1280 by 720 on the RTX 3060 took 20.6 s in all, 2.2 ms a frame of capture and 2.0 ms of draw, against 139.6 s for screenshots (25.1 ms a frame of capture) and 405.1 s for reading raw pixels over a WebSocket (84.8 ms a frame, 94 ms in a 10 s clip). Profiling showed ffmpeg was not the cost (3.7 ms a frame) and `readPixels` was 6.6 ms, but Chrome moved the 3.7 MB frame to Node at only 100 to 140 MB a second. The output matches the previous NVENC path at 44.8 dB PSNR and is tagged BT.709 limited range. `--capture auto` is the default for video and falls back to screenshots when the browser cannot hardware encode, or for `--encoder cpu` and `--backend cpu`. `--capture webcodecs` fails instead of falling back, `--capture raw` keeps the `readPixels` path as opt in, and PNG stills are unchanged. `npm run smoke:capture-parity` checks a decoded raw frame against its PNG seek. `ws` is a new dev dependency, bundled into the MCP server.
diff --git a/docs/agent-render-jobs.md b/docs/agent-render-jobs.md
index c178a34b..0c9fbcd3 100644
--- a/docs/agent-render-jobs.md
+++ b/docs/agent-render-jobs.md
@@ -36,9 +36,9 @@ The source revision, pose and checkpoints survive completion or cancellation.
Cancellation becomes terminal after the owned encoder and job page stop and the preview recovers.
Video uses H.264 (CPU by default, see `settings.encoder`), even dimensions and one streamed frame at a time.
-MCP video frames are read from the drawing buffer with `readPixels` in the same task as the draw and sent to Node over one binary WebSocket on the preview server port, with a one byte acknowledgement per frame so one frame is in flight and the encoder sets the pace (issue #301).
+With `settings.encoder` `nvenc`, MCP video frames are encoded in the page by the browser's hardware H.264 encoder in the same task as the draw, and only the encoded chunks travel to Node over one binary WebSocket on the preview server port, where `ffmpeg -c copy` muxes them with the audio (issue #301). Without NVENC, or when the browser cannot encode the size, frames are JPEG screenshots encoded by ffmpeg.
PNG stills and render passes still use a DevTools screenshot, byte for byte as before.
-`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs. A screenshot cost 28.8 ms a frame to capture and 7.4 ms to draw at 1280 by 720 on an RTX 3060. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw.
+`metrics.captureSeconds` and `metrics.renderSeconds` report the two costs, and `metrics.videoEncoder` is `browser-h264`, `h264_nvenc` or `libx264`. On the 4,488 frame film at 1280 by 720 on an RTX 3060 (loaded machine, one evening), in-page encoding took 20.6 s in all with 2.2 ms a frame of capture and 2.0 ms of draw, screenshots took 139.6 s (25.1 ms a frame of capture) and reading raw pixels over a WebSocket took 405.1 s (84.8 ms a frame), so pixels never travel to Node.
The render backend can use GPU or CPU. Memory limits can reduce the actual output dimensions.
Results report actual dimensions and measured timings. They do not embed video bytes.
diff --git a/scripts/agent-runtime.mjs b/scripts/agent-runtime.mjs
index c859039a..2fa55831 100644
--- a/scripts/agent-runtime.mjs
+++ b/scripts/agent-runtime.mjs
@@ -12,7 +12,7 @@ import { chooseDenoiser, finishWithOidn } from './agent-path-trace-denoise.mjs';
import { createAgentAssetCatalog } from './agent-asset-catalog.mjs';
import { createAgentRenderJobs, EXPORT_DIRECTORY_NAME } from './agent-render-jobs.mjs';
import { createFrameEncoder } from './frame-encoder.mjs';
-import { createFrameReceiver, installFrameReaderInPage } from './gpu-frame-reader.mjs';
+import { DEFAULT_PAGE_ENCODER_QUANTIZER, createFrameReceiver, startVideoEncoderInPage } from './gpu-frame-reader.mjs';
import { audioInputArguments, createAudioLibrary, measureStreamSeconds } from './job-audio.mjs';
import { filmTimelineDuration, filmTimelineShotStarts, sampleFilmTimeline } from '../src/film-timeline.js';
import { isPreviewHost } from './preview-host.mjs';
@@ -323,6 +323,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
let jobPage;
let encoder;
let frameReceiver;
+ let pageEncodes = false;
let suspended = false;
const sessionId = activeSessionId;
const started = performance.now();
@@ -357,14 +358,21 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
const timeline = settings.timeline ? snapshot.recipe.timeline : undefined;
const audio = settings.audio ? await prepareAudio(settings) : undefined;
if (settings.kind === 'video') {
- // Video frames are read from the drawing buffer and sent over a socket on the preview's own port (issue #301).
+ // With NVENC the page encodes each frame in the browser and sends only the H.264 chunks over a socket on the
+ // preview's own port (issue #301). Otherwise, or when the browser cannot encode this size, frames are screenshots.
frameReceiver = createFrameReceiver({ maximumBytes: width * height * 4 });
activeFrameReceiver = frameReceiver;
- await installFrameReaderInPage(jobPage, `ws://127.0.0.1:${httpServer.address().port}${frameReceiver.path}`);
+ const frameSocketUrl = `ws://127.0.0.1:${httpServer.address().port}${frameReceiver.path}`;
+ if (settings.encoder === 'nvenc') {
+ const pageEncoder = await startVideoEncoderInPage(jobPage, frameSocketUrl,
+ { width, height, fps: settings.fps, quantizer: DEFAULT_PAGE_ENCODER_QUANTIZER });
+ pageEncodes = pageEncoder.supported;
+ }
encoder = createFrameEncoder({ executable: ffmpeg, output: join(directory, 'render.mp4.partial'),
- fps: settings.fps, format: 'rawvideo', width, height, signal, maximumFrameBytes: MAXIMUM_JOB_FRAME_BYTES, backend: settings.encoder === 'nvenc' ? 'gpu' : 'cpu',
+ fps: settings.fps, format: pageEncodes ? 'h264' : 'jpeg', width, height, signal, maximumFrameBytes: MAXIMUM_JOB_FRAME_BYTES, backend: settings.encoder === 'nvenc' ? 'gpu' : 'cpu',
audioArguments: settings.audio ? audioInputArguments(audio.path, settings.audio.offset) : [] });
- metrics.videoEncoder = settings.encoder === 'nvenc' ? 'h264_nvenc' : 'libx264';
+ if (pageEncodes) frameReceiver.onChunk(chunk => encoder.writeChunk(chunk));
+ metrics.videoEncoder = pageEncodes ? 'browser-h264' : settings.encoder === 'nvenc' ? 'h264_nvenc' : 'libx264';
}
if (timeline) metrics.shots = filmTimelineShotStarts(timeline);
for (let frame = 0; frame < settings.frameCount; frame++) {
@@ -402,16 +410,16 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
continue;
}
// A video frame is read in the same task as its draw, because the browser clears the drawing buffer after presenting.
- const rendered = await jobPage.evaluate(async ({ input, readFrame }) => {
+ const rendered = await jobPage.evaluate(async ({ input, encodeIndex }) => {
const drawStarted = performance.now();
const result = await window.__agentPreview.frame(input);
- if (!readFrame) return result;
+ if (encodeIndex === null) return result;
window.__finishDraw();
const drawMilliseconds = performance.now() - drawStarted;
- return { ...result, drawMilliseconds, rawFrame: await window.__readFrame() };
- }, { input: frameInput, readFrame: settings.kind === 'video' });
+ return { ...result, drawMilliseconds, pageEncoded: await window.__encodeFrame(encodeIndex) };
+ }, { input: frameInput, encodeIndex: pageEncodes ? frame : null });
const roundTripSeconds = (performance.now() - renderStarted) / 1000;
- const drawSeconds = rendered.rawFrame ? rendered.drawMilliseconds / 1000 : roundTripSeconds;
+ const drawSeconds = rendered.pageEncoded ? rendered.drawMilliseconds / 1000 : roundTripSeconds;
metrics.renderSeconds += drawSeconds;
metrics.captureSeconds += roundTripSeconds - drawSeconds;
if (backend === 'cpu' && !/swiftshader/i.test(rendered.graphics)) throw new Error('CPU job did not select SwiftShader.');
@@ -420,9 +428,9 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
metrics.height = height;
metrics.estimatedBytes = rendered.estimatedBytes;
async function captureFrameBytes() {
- if (rendered.rawFrame) {
+ if (rendered.pageEncoded) {
signal.throwIfAborted();
- return frameReceiver.take({ width, height }, rendered.rawFrame);
+ return undefined;
}
const captureStarted = performance.now();
const captured = await session.send('Page.captureScreenshot', { fromSurface: true,
@@ -439,7 +447,9 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
artifacts.push({ temporary: `${filename}.partial`, filename, mimeType: 'image/png' });
}
const bytes = await captureFrameBytes();
- if (encoder) await encoder.writeFrame(bytes);
+ if (encoder) {
+ if (bytes) await encoder.writeFrame(bytes);
+ }
else {
const stem = settings.kind === 'image' ? 'render' : `frame-${frame}`;
await writeArtifact(`${stem}.png`, bytes);
@@ -464,6 +474,7 @@ export function createAgentRuntime({ kernelWorkers = true, backend = 'gpu', brow
await progress({ frames: frame + 1, maximumPendingFrames: 1 });
}
if (encoder) {
+ if (pageEncodes) await jobPage.evaluate(() => window.__finishVideoEncoder());
metrics.encoder = await encoder.finish();
if (audio) metrics.sync = await measureSync(join(directory, 'render.mp4.partial'));
artifacts.push({ temporary: 'render.mp4.partial', filename: 'render.mp4', mimeType: 'video/mp4' });
diff --git a/scripts/render-media.mjs b/scripts/render-media.mjs
index 9b8949a2..9adff56a 100644
--- a/scripts/render-media.mjs
+++ b/scripts/render-media.mjs
@@ -199,13 +199,13 @@ export async function renderMedia(values = process.argv.slice(2)) {
}
if (['auto', 'webcodecs'].includes(captureFormat)) {
// The page encodes with the browser's hardware H.264 encoder and sends only the small encoded chunks. Software
- // rendering or a CPU encoder request keeps the readback path, which feeds ffmpeg's libx264.
+ // rendering or a CPU encoder request uses screenshots, which feed ffmpeg's libx264, because raw readback is slower.
const wantsCpu = backend === 'cpu' || options.encoder === 'cpu';
const started = wantsCpu ? { supported: false, reason: 'A CPU render or encoder was requested.' }
: await startVideoEncoderInPage(page, frameServer.url, { width, height, fps, quantizer: DEFAULT_PAGE_ENCODER_QUANTIZER });
if (started.supported) captureFormat = 'webcodecs';
else if (captureFormat === 'webcodecs') throw new Error(`Capture webcodecs is not available: ${started.reason}`);
- else captureFormat = 'raw';
+ else captureFormat = 'jpeg';
}
if (captureFormat === 'raw') await installFrameReaderInPage(page, frameServer.url);
// A raw frame is read from the drawing buffer in the same task as the draw. The round trip beyond the draw is capture.
diff --git a/skills/render-67js-media/references/rendering.md b/skills/render-67js-media/references/rendering.md
index 69638530..1ede64a6 100644
--- a/skills/render-67js-media/references/rendering.md
+++ b/skills/render-67js-media/references/rendering.md
@@ -40,7 +40,8 @@ The renderer does not silently replace a failed hardware export with a lower-qua
Video capture cost (measured on an RTX 3060 at 1280x720 on a loaded machine, 20 s of the film, 480 frames, NVENC).
Screenshots (`--capture jpeg`) cost 28.8 ms a frame to capture and 7.4 ms to draw, 21.5 s for the clip.
Every report has `stageTimings.captureSeconds` and `renderSeconds`.
-`--capture raw`, the default for video, reads the drawing buffer with `readPixels` (3.3 ms a frame) and sends it to Node over a binary WebSocket, one frame in flight, acknowledged by one byte. The full 4,488 frame film at 1280 by 720 on the RTX 3060 survived raw capture (measured once, loaded machine): 405 s in all, draw 2.9 ms and readPixels 4.4 ms a frame, but 85 ms a frame in capture, so the WebSocket round trip is too slow and the 40 s and 5 ms targets are not met. The same job with screenshots took 140 s (draw 20 s, capture 113 s, 25 ms a frame), so raw is about 3 times slower today and stays opt in with --capture raw.
+`--capture auto`, the default for video, uses WebCodecs and falls back to screenshots (never raw readback, which is slower) when the browser cannot hardware encode the size or `--encoder cpu` or `--backend cpu` is requested. `--capture webcodecs` fails instead of falling back, `--capture jpeg` and `--capture png` keep screenshots, and `--capture raw` keeps the `readPixels` and WebSocket path as opt in. The hardware encoder is the browser's, so the bitstream can differ between machines and drivers even though the frames are deterministic, which is why `--encoder cpu` keeps libx264 on screenshots.
+Profile of the first readback design (10 s of the film, 240 frames, RTX 3060, 1280 by 720): 94 ms a frame in capture, of which `readPixels` was 6.6 ms and ffmpeg was nothing, because ffmpeg alone with vflip, the BT.709 conversion and NVENC took 3.7 ms a frame (3.0 ms with no filter). The cost was moving 3.7 MB of pixels from Chrome to Node over a WebSocket: 27 to 52 ms a frame depending on frames in flight and sockets, about 100 to 140 MB a second, and still 9 to 16 ms for the 1.4 MB a YUV 4:2:0 frame would need. Chrome's WebCodecs `VideoEncoder` removes the pixels from the trip. The page encodes each frame with the browser's hardware H.264 encoder (fixed quantizer 20, no B-frames), so only about 5 to 60 kB of encoded chunks a frame cross the socket, and `ffmpeg -c copy` muxes them with the audio. Measured on the full 4,488 frame film at 1280 by 720: 20.6 s in all, draw 2.0 ms and capture 2.2 ms a frame, no encoder backpressure. The same machine, same evening, screenshots took 139.6 s (capture 25.1 ms a frame) and the first raw readback took 405.1 s (capture 84.8 ms a frame). On a 10 s clip within one hour, capture was 4.2 ms with WebCodecs, 22.1 ms with screenshots and 93.9 ms with raw readback. The result matches the NVENC raw path of the same clip at 44.8 dB PSNR (quantizer 22 when measured), is tagged BT.709 limited range, and has all 4,488 frames.
Returning the pixels through Playwright instead cost 55 ms a frame, and a fetch with a Blob body failed after about 100 frames, so neither is used.
A decoded raw frame matches the PNG seek of the same time at 41.0 dB PSNR on SwiftShader (`npm run smoke:capture-parity`).
diff --git a/tests/export-options.test.js b/tests/export-options.test.js
index e460c793..3a578b37 100644
--- a/tests/export-options.test.js
+++ b/tests/export-options.test.js
@@ -55,11 +55,13 @@ test('invalid export options fail before starting resources', () => {
]) assert.throws(() => parseRenderOptions(args));
});
-test('video defaults to JPEG 95 and auto encoding while stills retain lossless PNG', () => {
+test('video defaults to automatic capture, JPEG 95 for screenshots and auto encoding while stills retain lossless PNG', () => {
const video = parseRenderOptions(['--kind', 'video']);
- assert.equal(video.captureFormat, 'jpeg');
+ assert.equal(video.captureFormat, 'auto');
assert.equal(video.jpegQuality, 95);
assert.throws(() => parseRenderOptions(['--capture', 'raw']));
+ assert.equal(parseRenderOptions(['--kind', 'video', '--capture', 'webcodecs']).captureFormat, 'webcodecs');
+ assert.throws(() => parseRenderOptions(['--capture', 'webcodecs']));
assert.equal(parseRenderOptions(['--kind', 'video', '--capture', 'raw']).captureFormat, 'raw');
assert.equal(video.encoderBackend, 'auto');
assert.equal(parseRenderOptions([]).captureFormat, 'png');