Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

6.4 KB
Raw Blame

Storyboard paused-video color and frame capture

The storyboard keeps continuous playback in a native muted video element. On pause, each visible footage layer presents its decoded frame in an explicitly sRGB Canvas2D surface. Save frame freezes that same canvas, then rasterizes the actual preview layout, transitions, exact paused film look, lyrics and outro. Native playback resumes without a JavaScript per-frame rasterization loop. Hidden preloaded clips do not allocate decoded canvas frames.

PreviewVideo.jsx owns the shared surface for ordinary footage and incoming transition footage. The wrapper owns fitting/background and opacity; the video and canvas have the same object fitting. Seeks invalidate the previous surface, and presentation waits two animation frames after decoded data/seek completion. The existing song-to-source mapping still owns offsets, speed, reverse and loop. Save frame waits for the visible paused surfaces (bounded to two seconds) and rejects a changed, seeking or unavailable surface instead of exporting native video through a different color path. Canvas storage is capped at 16 megapixels, matching capture's limit. Unsupported Canvas2D or SVG foreignObject capture reports an error; it does not upload a blank image.

Interpretation and evidence

This is a shared SDR preview/capture interpretation, not a color-specific correction. The browser decodes the source's matrix and range into sRGB Canvas2D. The composed renderer (server/services/musicVideo/render.js) processes source YUV through FFmpeg scale/format and encodes yuv420p. The regression independently checks encoded YUV against the source matrix and browser raw video planes. FFmpeg's matrix-to-RGB decode remains a diagnostic comparison. The layered document renderer also uses Canvas2D drawImage. This change does not add an HDR tone map, a wide-gamut mastering workflow, or a new primaries/transfer conversion to the final renderer; agreement with a calibrated native-video display is not claimed.

The synthetic browser fixture generates the original 120×240 cyan VP8 clip and 480×240 VP9 charts with eight saturated/neutral patches, a sixteen-step grayscale ramp and an alternating, frame-specific marker. RGB is actually converted into limited-range SMPTE170M (BT601) or BT709 YUV before encoding. VideoFrame.colorSpace assertions verify matrix, primaries, transfer and range at the decoder boundary. Metadata-only BT709 tags on the original VP8 clip still reported SMPTE170M in Chrome, so that clip cannot stand in for a correctly decoded BT709 test.

macOS arm64, headless Google Chrome 155.0.8059.80, hardware-decoded NV12, forced sRGB display profile:

Source / patch Native video screenshot RGB Decoded canvas / paused preview / PNG RGB FFmpeg matrix decode RGB
BT601 cyan 70, 253, 255 1, 255, 255 0, 255, 255
BT601 red 248, 36, 0 254, 0, 0 254, 0, 0
BT601 magenta 249, 52, 255 255, 0, 254 255, 0, 254
BT709 cyan 0, 254, 255 0, 254, 255 0, 254, 255
BT709 red 255, 1, 0 255, 1, 0 255, 0, 0
Both grayscale ramps 0, 17, …, 255 0, 17, …, 255 0, 17, …, 255

The original standalone VP8 cyan reproduction gave native RGBA [70,253,255,255] versus Canvas2D [1,255,255,255], before any PortOS compositor. Prior probes with ImageBitmap default/none conversion, VideoFrame, neutral CSS filters, extra animation frames, and a display-P3 canvas converted back to sRGB did not eliminate it. The VP9 matrix checks above distinguish that presentation boundary from an incorrectly tagged source.

The hosted Linux x64 diagnostic (Chrome 155.0.8059.39, FFmpeg 6.1.1) exposed a separate CPU conversion limit: BT709 magenta was [254,0,244] in native video, paused canvas and PNG, while FFmpeg produced [253,0,253]. Yellow, green and blue also differed in the blue channel. On the same local Chrome 155, disabling GPU and hardware video decoding changed NV12 to I420 and reproduced those blue values while browser raw YUV still matched FFmpeg exactly. FFmpeg 6.1.1 arm64/x64 and FFmpeg 9.0.2 synthetic probes independently verified the encoded matrix.

This is consistent with libyuv's default BT709 blue coefficient being capped at 128/64 rather than 135/64; see the pinned conversion source. The source YUV [78,214,229] predicts blue 244 with that approximation. Thus a universal three-level bound between FFmpeg RGB and browser RGB is invalid even when source tagging and decoding are correct. PortOS does not compensate for individual colors or promise identical RGB across CPU/GPU decoder paths.

server/services/musicVideo/animaticCapture.browser.test.js validates source and presentation separately. Each saturated patch, grayscale step and moving marker must encode within two YUV levels of the mathematical limited-range 601/709 matrix (the lossy chart's quantization bound). The browser's raw I420/NV12 planes must exactly match FFmpeg's decoded YUV for the requested frame, and matrix, primaries, transfer and range metadata must match. The paused product canvas must exactly match a fresh, independent sRGB draw of the source video. This catches stale canvas frames without treating the product canvas as its own reference.

The real paused preview screenshot is compared to the saved PNG without replacing DOM nodes: mean channel error remains below 1, and patch readback/export differences remain at most 1. Native video and FFmpeg RGB remain diagnostic evidence; failures include complete synthetic samples and versions even when CI silences console output. Forward/backward and adjacent-frame seeks check the alternating marker, and pause/resume checks native playback restoration. Fitting, incoming footage with film look, still motion/effects, lyrics, title and outro remain covered.

Run the fixture with:

npm test --prefix server -- services/musicVideo/animaticCapture.browser.test.js

The CI planner selects it when the capture helper, Save frame control or paused video surface changes. Hosted Linux Chrome runs the same fixture; failing assertions retain its browser/version and numeric evidence. Local macOS results do not substitute for that CI result. Safari was present but could not create a WebDriver session because Allow remote automation was disabled; Safari pixels remain unverified. No browser settings were changed for this investigation.