Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

perf(preview): stop decoding pictures the hidden media elements never show

EtienneLescote73097c84f62
diff --git a/electron/windows.ts b/electron/windows.ts
index 0faaef02f..72f2fe938 100644
--- a/electron/windows.ts
+++ b/electron/windows.ts
@@ -505,6 +505,10 @@ export function createEditorWindow(query: Record<string, string> = {}): BrowserW
contextIsolation: true,
webSecurity: false,
backgroundThrottling: false,
+ // Media track lists: the preview's hidden <video> and <audio> deselect their video
+ // track, so Chromium stops decoding pictures the native compositor already draws
+ // (`dropVideoTrack` in VirtualPreview.tsx).
+ enableBlinkFeatures: "AudioVideoTracks",
},
});
diff --git a/src/components/ai-edition/NewEditorShell.module.css b/src/components/ai-edition/NewEditorShell.module.css
index 1fde13eda..30b3c0947 100644
--- a/src/components/ai-edition/NewEditorShell.module.css
+++ b/src/components/ai-edition/NewEditorShell.module.css
@@ -726,7 +726,8 @@
position: relative;
z-index: 1;
/* Pixels come from the native canvas now; this <video> stays mounted (and
- playing) purely as the decode/playback-clock/metadata source. A plain
+ playing) purely as the playback clock and metadata source, its picture
+ undecoded (`dropVideoTrack` in VirtualPreview.tsx). A plain
descendant selector (not a fixed child-depth chain) is required here:
VirtualPreview nests the actual <video> THREE levels down
(.screenStage > .container > .videoFrame > video), not one or two —
@@ -744,9 +745,8 @@
border-radius: 18px !important;
}
-/* Webcam overlay: real <video> driven by the cameraTrack. The slot is
- transparent so clip-path / border-radius carve the camera without leaving
- a rectangular placeholder peeking through. */
+/* Webcam slot: the drag-to-reposition hitbox over the camera the native canvas
+ draws. Transparent, so nothing of it shows. */
.webcamSlot {
background: transparent;
overflow: hidden;
@@ -757,8 +757,8 @@
width: 100%;
height: 100%;
object-fit: cover;
- /* Native canvas already draws the composited webcam PiP; this <video> stays
- mounted (playing) only to drive the camera's own decode/sync clock. The
+ /* Native canvas already draws the composited webcam PiP; this <video> is
+ never played, only read for the camera's size (WebcamOverlay.tsx). The
slot div itself stays visible/clickable for the drag-to-reposition hitbox. */
visibility: hidden;
display: block;
diff --git a/src/components/ai-edition/PreviewCanvas.tsx b/src/components/ai-edition/PreviewCanvas.tsx
index 79d95e5ed..c817fde3c 100644
--- a/src/components/ai-edition/PreviewCanvas.tsx
+++ b/src/components/ai-edition/PreviewCanvas.tsx
@@ -48,7 +48,6 @@ import type {
import { useProjectStore } from "@/lib/ai-edition/store/projectStore";
import { useEditorSettings } from "@/lib/ai-edition/store/useEditorSettings";
import { resolveActiveCameraTrack } from "@/lib/ai-edition/timeline/camera";
-import { createPlaybackClockRef } from "@/lib/ai-edition/timeline/playback-clock";
import type { SpeedRegion } from "@/lib/ai-edition/timeline/speed";
import { locateVirtualPosition } from "@/lib/ai-edition/timeline/virtual-preview";
import {
@@ -135,11 +134,6 @@ export function PreviewCanvas(props: PreviewCanvasProps) {
const assets = document?.assets ?? [];
const frameRef = useRef<HTMLDivElement | null>(null);
const webcamSlotRef = useRef<HTMLDivElement | null>(null);
- // One clock per mounted canvas, shared between the screen preview (writer)
- // and the webcam overlay (reader) — see playback-clock.ts.
- const clockRefHolder = useRef<ReturnType<typeof createPlaybackClockRef>>();
- if (!clockRefHolder.current) clockRefHolder.current = createPlaybackClockRef();
- const clockRef = clockRefHolder.current;
// Real dimensions of the active source, from the <video>'s own
// onLoadedMetadata (videoWidth/videoHeight) — null until the first source
// loads, then falls back to SCREEN_SOURCE_SIZE.
@@ -384,7 +378,6 @@ export function PreviewCanvas(props: PreviewCanvasProps) {
onVideoElement: relayIsPlaying,
onLoadedMetadata: relayLoadedMetadata,
cropRegion,
- clockRef,
};
const handleWebcamPointerDown = (event: ReactPointerEvent<HTMLDivElement>) => {
@@ -445,8 +438,8 @@ export function PreviewCanvas(props: PreviewCanvasProps) {
>
{/* Sole pixel source: the D3D-composited frame (wallpaper + screen + webcam +
cursor), streamed into a canvas. The <video> elements below are CSS-hidden
- (visibility only — they stay mounted for decode/playback-clock/metadata
- duties, since the native compositor doesn't drive playback itself), and the
+ (visibility only — they stay mounted as the playback clock and for metadata,
+ since the native compositor doesn't drive playback itself), and the
interactive-only layers (ZoomFocusOverlay, AnnotationLayer, webcam drag
hitbox) still render on top as normal DOM so they stay clickable. No more
dual preview path. */}
@@ -467,7 +460,6 @@ export function PreviewCanvas(props: PreviewCanvasProps) {
// behind a flag with its own tests.
// See technical-documentation/architecture/preview.md
// (todo) for the failure write-up.
- void relayProps.clockRef;
return (
<VirtualPreview
{...relayProps}
@@ -497,18 +489,7 @@ export function PreviewCanvas(props: PreviewCanvasProps) {
onPointerDown={isPipGrab ? handleWebcamPointerDown : undefined}
aria-label={te("preview.webcamPreview")}
>
- <WebcamOverlay
- clips={props.clips}
- currentTimeSec={props.currentTimeSec}
- onTimeChange={props.onTimeChange}
- isPlaying={isPlaying}
- clockRef={clockRef}
- borderRadius={
- effectiveLayout?.webcamRect?.borderRadius ?? layout.webcamRect.borderRadius
- }
- webcamMaskShape={effectiveLayout?.webcamRect?.maskShape ?? settings.webcamMaskShape}
- layoutPreset={settings.webcamLayoutPreset}
- />
+ <WebcamOverlay clips={props.clips} currentTimeSec={props.currentTimeSec} />
</div>
) : null}
{/* Last, so a selected annotation over the camera takes the pointer before the
diff --git a/src/components/ai-edition/VirtualPreview.tracks.test.ts b/src/components/ai-edition/VirtualPreview.tracks.test.ts
new file mode 100644
index 000000000..731861265
--- /dev/null
+++ b/src/components/ai-edition/VirtualPreview.tracks.test.ts
@@ -0,0 +1,46 @@
+import { describe, expect, it } from "vitest";
+import { dropVideoTrack } from "./VirtualPreview";
+
+/** A media element as Chromium shows it under `AudioVideoTracks`: track lists, nothing else. */
+function element(tracks: { audio?: number; video?: number }) {
+ const videoTracks = Array.from({ length: tracks.video ?? 0 }, (_, index) => ({
+ selected: index === 0,
+ }));
+ return {
+ element: {
+ audioTracks: tracks.audio === undefined ? undefined : { length: tracks.audio },
+ videoTracks: tracks.video === undefined ? undefined : videoTracks,
+ } as unknown as HTMLMediaElement,
+ videoTracks,
+ };
+}
+
+describe("dropVideoTrack", () => {
+ it("deselects the picture of an element that has sound to keep time with", () => {
+ const { element: media, videoTracks } = element({ audio: 1, video: 1 });
+
+ expect(dropVideoTrack(media)).toBe(true);
+ expect(videoTracks[0].selected).toBe(false);
+ });
+
+ // With neither track selected an element has no stream left to keep time with: measured,
+ // its clock races to the end instead of playing.
+ it("leaves a recording without sound decoding, so its clock still runs", () => {
+ const { element: media, videoTracks } = element({ audio: 0, video: 1 });
+
+ expect(dropVideoTrack(media)).toBe(false);
+ expect(videoTracks[0].selected).toBe(true);
+ });
+
+ it("does nothing where the track lists are absent (the Blink feature is off)", () => {
+ const { element: media } = element({});
+
+ expect(dropVideoTrack(media)).toBe(false);
+ });
+
+ it("does nothing to an element with no picture, the extracted second audio track", () => {
+ const { element: media } = element({ audio: 1, video: 0 });
+
+ expect(dropVideoTrack(media)).toBe(false);
+ });
+});
diff --git a/src/components/ai-edition/VirtualPreview.tsx b/src/components/ai-edition/VirtualPreview.tsx
index a6777d8f4..beda87868 100644
--- a/src/components/ai-edition/VirtualPreview.tsx
+++ b/src/components/ai-edition/VirtualPreview.tsx
@@ -21,7 +21,6 @@ import type {
} from "@/lib/ai-edition/schema";
import { audioGainScalar } from "@/lib/ai-edition/store/editorSettings";
import { useEditorSettings } from "@/lib/ai-edition/store/useEditorSettings";
-import type { PlaybackClockRef } from "@/lib/ai-edition/timeline/playback-clock";
import { removedRawSpans } from "@/lib/ai-edition/timeline/programme-time";
import { findActiveSpeedRegion, type SpeedRegion } from "@/lib/ai-edition/timeline/speed";
import {
@@ -178,6 +177,37 @@ export function shouldResyncAudio(
return Math.abs(driftSec) > (freeRunning ? playingLeashSec : AUDIO_PARKED_LEASH_SEC);
}
+/** The track lists Chromium exposes on a media element under the `AudioVideoTracks` Blink
+ * feature, which the editor window turns on (`createEditorWindow`). Absent elsewhere. */
+interface MediaTrackLists {
+ audioTracks?: { length: number };
+ videoTracks?: { length: number; [index: number]: { selected: boolean } };
+}
+
+/**
+ * Stops Chromium decoding the picture of a media element that is only here for its clock or
+ * its sound: the native compositor draws every pixel of the preview. Measured on a 1080p60
+ * recording: a hidden `<video>` that plays keeps ~6.7 % of an RTX 4070 Ti's decode engine
+ * busy, and an `<audio>` on the same mp4 decodes its picture just as much — a second and a
+ * third decode of the recording next to the compositor's own. With the video track
+ * deselected the element decodes no frame, and its clock, `playbackRate` and seeks behave
+ * the same.
+ *
+ * Only when the element has a sound track: with neither track selected it has no stream
+ * left to keep time with, and races to its end. A recording without sound keeps decoding,
+ * as before. Returns whether the track was dropped.
+ */
+export function dropVideoTrack(element: HTMLMediaElement): boolean {
+ const { audioTracks, videoTracks } = element as HTMLMediaElement & MediaTrackLists;
+ if (!videoTracks?.length || !audioTracks?.length) {
+ return false;
+ }
+ for (let index = 0; index < videoTracks.length; index++) {
+ videoTracks[index].selected = false;
+ }
+ return true;
+}
+
/**
* Where an audio element should sit to track the video, and whether it should be playing.
*
@@ -442,12 +472,6 @@ interface VirtualPreviewProps {
* identity ({x:0,y:0,width:1,height:1}) renders the full frame, unchanged
* from before crop support existed. */
cropRegion?: CropRegion | null;
- /**
- * Written every rAF tick with this video's live position/rate so other
- * media elements (the webcam overlay) can read it directly instead of
- * waiting for a React state round trip. See playback-clock.ts.
- */
- clockRef?: PlaybackClockRef;
}
export function VirtualPreview({
@@ -467,7 +491,6 @@ export function VirtualPreview({
onVideoRecovered,
retryToken,
cropRegion,
- clockRef,
}: VirtualPreviewProps) {
const { settings } = useEditorSettings();
// ponytail: an oversized, offset video inside .videoFrame's overflow:hidden
@@ -1117,15 +1140,6 @@ export function VirtualPreview({
el.pause();
}
}
- // Publish this frame's live position/rate for other media elements
- // (webcam) to read directly — see playback-clock.ts for why this
- // bypasses React state entirely.
- if (clockRef) {
- clockRef.current.sourceTimeSec = v.currentTime;
- clockRef.current.isPlaying = !v.paused;
- clockRef.current.playbackRate = v.playbackRate;
- clockRef.current.virtualTimeSec = virtualTimeSecRef.current;
- }
// A reload is in flight: the decoder is dead and `currentTime` is
// frozen (or already reset to 0), so every decision below — trim
// skipping, the clip-boundary advance, the unmapped-position
@@ -1210,9 +1224,9 @@ export function VirtualPreview({
// `seekToVirtualTimeRef(nextClip.timelineStartSec)` plus bas renvoyait la tête au
// DÉBUT du clip voisin — le tressaillement observé au passage d'un clip à l'autre.
//
- // `clockRef` et `setSourceTimeSec` ci-dessus continuent d'être publiés : la webcam
- // et le calque curseur ont besoin du temps source même à l'arrêt. Seule la
- // position de la TIMELINE cesse d'être dictée par le média.
+ // `setSourceTimeSec` ci-dessus continue d'être publié : le calque curseur a besoin du
+ // temps source même à l'arrêt. Seule la position de la TIMELINE cesse d'être
+ // dictée par le média.
if (v.paused) {
return;
}
@@ -1657,6 +1671,8 @@ export function VirtualPreview({
e.currentTarget.videoWidth,
e.currentTarget.videoHeight,
);
+ // Its size is read: from here on only its clock is needed.
+ dropVideoTrack(e.currentTarget);
if (pendingSeekRef.current) {
const { sourceTimeSec, play } = pendingSeekRef.current;
pendingSeekRef.current = null;
@@ -1795,6 +1811,8 @@ export function VirtualPreview({
preload="metadata"
aria-hidden="true"
data-testid="preview-audio-primary"
+ // The recording's own mp4: without this its picture is decoded too.
+ onLoadedMetadata={(e) => dropVideoTrack(e.currentTarget)}
/>
{supplementalAudioSrc ? (
<audio
@@ -1803,6 +1821,7 @@ export function VirtualPreview({
src={supplementalAudioSrc}
preload="metadata"
aria-hidden="true"
+ onLoadedMetadata={(e) => dropVideoTrack(e.currentTarget)}
data-testid="preview-audio-supplemental"
/>
) : null}
diff --git a/src/components/ai-edition/WebcamOverlay.test.tsx b/src/components/ai-edition/WebcamOverlay.test.tsx
index d00f017ab..6275198e8 100644
--- a/src/components/ai-edition/WebcamOverlay.test.tsx
+++ b/src/components/ai-edition/WebcamOverlay.test.tsx
@@ -1,7 +1,7 @@
// @vitest-environment jsdom
import "@testing-library/jest-dom";
import { cleanup, render } from "@testing-library/react";
-import { afterEach, describe, expect, it } from "vitest";
+import { afterEach, describe, expect, it, vi } from "vitest";
import type { AxcutClip, AxcutDocument } from "@/lib/ai-edition/schema";
import { axcutSchemaVersion } from "@/lib/ai-edition/schema";
import { useProjectStore } from "@/lib/ai-edition/store/projectStore";
@@ -83,11 +83,6 @@ function baseProps(currentTimeSec: number) {
return {
clips: [CLIP_WITH_CAMERA, CLIP_WITHOUT_CAMERA],
currentTimeSec,
- onTimeChange: () => undefined,
- isPlaying: false,
- borderRadius: 0,
- webcamMaskShape: "rectangle" as const,
- layoutPreset: "picture-in-picture" as const,
};
}
@@ -205,4 +200,32 @@ describe("WebcamOverlay (per-clip camera resolution)", () => {
unmount();
}
});
+
+ // The native compositor draws the camera. Playing this element decoded the whole camera
+ // recording a second time, alongside the compositor, for pixels CSS hides.
+ it("reads the camera for its size and never plays it", () => {
+ useProjectStore.setState({
+ projectId: "proj_test",
+ document: makeDocument(),
+ revision: 1,
+ status: "ready",
+ error: null,
+ sourceDurationSec: 0,
+ currentTimeSec: 2,
+ dirty: false,
+ lastSavedAt: new Date(),
+ });
+ const play = vi.spyOn(HTMLMediaElement.prototype, "play").mockResolvedValue(undefined);
+ try {
+ const { container, rerender } = render(<WebcamOverlay {...baseProps(2)} />);
+ rerender(<WebcamOverlay {...baseProps(3)} />);
+
+ const video = container.querySelector("video");
+ expect(video?.getAttribute("preload")).toBe("metadata");
+ expect(video?.autoplay).toBe(false);
+ expect(play).not.toHaveBeenCalled();
+ } finally {
+ play.mockRestore();
+ }
+ });
});
diff --git a/src/components/ai-edition/WebcamOverlay.tsx b/src/components/ai-edition/WebcamOverlay.tsx
index d9b256149..d65984148 100644
--- a/src/components/ai-edition/WebcamOverlay.tsx
+++ b/src/components/ai-edition/WebcamOverlay.tsx
@@ -1,209 +1,53 @@
-// Live webcam preview overlay. Reads the ACTIVE clip's asset `cameraTrack`
-// (P4 — the camera link lives per-asset, not on the document, since a
-// project can hold multiple recordings each with their own camera or none)
-// and drives a real <video> element at the right source-time. The webcam is
-// a derived stream — cuts/zoom/speed come from the main timeline. This
-// component only reads; it does not write.
+// The active clip's camera file, read for its size. Resolves the ACTIVE clip's asset
+// `cameraTrack` (P4 — the camera link lives per-asset, not on the document, since a project
+// can hold multiple recordings each with their own camera or none), so the element
+// disappears when the playhead moves onto a clip whose asset has no camera, and reappears
+// when it moves onto one that does.
//
-// ponytail: the camera plays in parallel with the screen. Source-time mapping
-// cameraTime = clip.sourceStartSec + (currentTimeSec − clip.timelineStartSec)
-// adjustment = (cameraTrack.startMs + cameraTrack.offsetMs) / 1000
-// final = max(0, cameraTime − adjustment)
-// (startMs is when the camera comes online; offsetMs is the early/late delay).
-// Because this is resolved from the active clip's asset, the overlay
-// naturally disappears when the playhead moves onto a clip whose asset has
-// no camera, and reappears when it moves onto one that does.
+// The native compositor draws the camera; this element never shows a pixel. What it is for
+// is `loadedmetadata`: the camera's real width and height shape the PiP box the scene asks
+// the compositor for (`webcamSizeCache`). It is never played, because playing it decoded the
+// whole camera recording a second time, alongside the compositor, for pixels CSS hid.
-import { useEffect, useMemo, useRef, useState } from "react";
+import { useMemo } from "react";
import { toFileUrl } from "@/components/video-editor/projectPersistence";
-import type { WebcamLayoutPreset, WebcamMaskShape } from "@/components/video-editor/types";
import type { AxcutClip } from "@/lib/ai-edition/schema";
import { useProjectStore } from "@/lib/ai-edition/store/projectStore";
-import { useEditorSettings } from "@/lib/ai-edition/store/useEditorSettings";
import { resolveActiveCameraTrack } from "@/lib/ai-edition/timeline/camera";
-import {
- CAMERA_SYNC_TOLERANCE_PAUSED_SEC,
- type PlaybackClockRef,
- resolveCameraSyncTarget,
-} from "@/lib/ai-edition/timeline/playback-clock";
-import { locateVirtualPosition } from "@/lib/ai-edition/timeline/virtual-preview";
-import { getCssClipPath } from "@/lib/webcamMaskShapes";
import { setWebcamNativeSize } from "@/native/webcamSizeCache";
import styles from "./NewEditorShell.module.css";
interface WebcamOverlayProps {
clips: AxcutClip[];
currentTimeSec: number;
- onTimeChange: (sec: number) => void;
- isPlaying: boolean;
- // ponytail: container renders without a frame; the <video> is the only
- // thing the user actually sees. Border radius + clip-path therefore
- // belong on the video so they actually round the camera content.
- borderRadius: number;
- webcamMaskShape: WebcamMaskShape;
- layoutPreset: WebcamLayoutPreset;
- // The screen preview's live clock (see playback-clock.ts). When present,
- // sync is driven from this ref on our own rAF tick instead of the
- // currentTimeSec/isPlaying PROPS above — those props still gate whether
- // the camera element renders at all (see cameraTrack below), but the
- // numeric sync target comes straight from the screen's own rAF, this
- // frame, with no React round trip in between.
- clockRef?: PlaybackClockRef;
}
export function WebcamOverlay(props: WebcamOverlayProps) {
- const { settings } = useEditorSettings();
const assets = useProjectStore((s) => s.document?.assets ?? null);
- const [videoEl, setVideoEl] = useState<HTMLVideoElement | null>(null);
- const [hasError, setHasError] = useState(false);
-
- // Fallback (pre-clockRef / first paint) position from props, used only for
- // the initial correction on loadedmetadata before the rAF loop below has
- // had a chance to run.
- const position = useMemo(
- () => locateVirtualPosition(props.clips, props.currentTimeSec),
- [props.clips, props.currentTimeSec],
- );
-
const cameraTrack = useMemo(
() => resolveActiveCameraTrack(assets ?? [], props.clips, props.currentTimeSec),
[assets, props.clips, props.currentTimeSec],
);
- const cameraTime = useMemo(() => {
- if (!cameraTrack?.visible || !position) return null;
- const offsetSec = (cameraTrack.startMs + cameraTrack.offsetMs) / 1000;
- return Math.max(0, position.sourceTimeSec - offsetSec);
- }, [cameraTrack, position]);
-
- // Refs so the rAF tick below always reads the latest clips/assets without
- // re-creating the loop on every document mutation.
- const clipsRef = useRef(props.clips);
- clipsRef.current = props.clips;
- const assetsRef = useRef(assets);
- assetsRef.current = assets;
-
- // Drive the camera <video> directly off the shared playback clock: read
- // it every rAF tick, resolve which clip/camera is active THIS frame, and
- // correct time/rate/play-state in one place. This replaces two separate
- // prop-driven effects (time correction + play/pause mirroring), both of
- // which depended on a React state round trip from the screen preview.
- useEffect(() => {
- if (!videoEl || !props.clockRef) return;
- const clockRef = props.clockRef;
- let raf = 0;
- const tick = () => {
- raf = window.requestAnimationFrame(tick);
- const clock = clockRef.current;
- const clipsNow = clipsRef.current;
- const positionNow = locateVirtualPosition(clipsNow, clock.virtualTimeSec);
- const trackNow = resolveActiveCameraTrack(
- assetsRef.current ?? [],
- clipsNow,
- clock.virtualTimeSec,
- );
- const target = resolveCameraSyncTarget(
- clock,
- trackNow,
- positionNow ? positionNow.sourceTimeSec : null,
- );
- if (!target) return;
-
- if (videoEl.playbackRate !== target.playbackRate) {
- videoEl.playbackRate = target.playbackRate;
- }
-
- if (Math.abs(videoEl.currentTime - target.targetTimeSec) > target.toleranceSec) {
- try {
- videoEl.currentTime = target.targetTimeSec;
- } catch {
- // ponytail: silent — video not ready yet
- }
- }
-
- if (target.isPlaying && videoEl.paused) {
- void videoEl.play().catch(() => setHasError(true));
- } else if (!target.isPlaying && !videoEl.paused) {
- videoEl.pause();
- }
- };
- raf = window.requestAnimationFrame(tick);
- return () => window.cancelAnimationFrame(raf);
- }, [videoEl, props.clockRef]);
-
- // Fallback for when no clockRef is wired up (defensive — all current call
- // sites pass one): keep the old prop-driven correction so the overlay
- // still works, just with the previously-reported latency.
- useEffect(() => {
- if (!videoEl || props.clockRef) return;
- if (cameraTime === null) return;
- if (Math.abs(videoEl.currentTime - cameraTime) > CAMERA_SYNC_TOLERANCE_PAUSED_SEC) {
- try {
- videoEl.currentTime = cameraTime;
- } catch {
- // ponytail: silent — video not ready yet
- }
- }
- }, [videoEl, cameraTime, props.clockRef]);
-
- useEffect(() => {
- if (!videoEl || props.clockRef) return;
- if (props.isPlaying) {
- void videoEl.play().catch(() => setHasError(true));
- } else {
- videoEl.pause();
- }
- }, [videoEl, props.isPlaying, props.clockRef]);
-
if (!cameraTrack?.sourcePath || !cameraTrack.visible) {
return null;
}
- // ponytail: the layout computes the final borderRadius (preset fraction for
- // dual-frame/overlay, 0 for stack, half-circle for circle PiP, etc.). Push it onto
- // the element that actually carries the pixels so it clips the camera content; the
- // container stays a transparent, overflow:hidden wrapper.
- const videoStyle: React.CSSProperties = {
- display: hasError ? "none" : "block",
- transform: settings.webcamMirrored ? "scaleX(-1)" : undefined,
- clipPath: getCssClipPath(props.webcamMaskShape) ?? undefined,
- borderRadius: `${props.borderRadius}px`,
- };
-
return (
<video
key={cameraTrack.sourcePath}
- ref={(el) => {
- setVideoEl(el);
- setHasError(false);
- }}
src={toFileUrl(cameraTrack.sourcePath)}
className={styles.webcamVideo}
muted
playsInline
preload="metadata"
- onError={() => setHasError(true)}
onLoadedMetadata={(event) => {
- const target = event.currentTarget;
- const w = target.videoWidth;
- const h = target.videoHeight;
- if (w > 0 && h > 0) {
- setWebcamNativeSize(cameraTrack.sourcePath, { width: w, height: h });
- }
- if (
- cameraTime !== null &&
- videoEl &&
- Math.abs(videoEl.currentTime - cameraTime) > CAMERA_SYNC_TOLERANCE_PAUSED_SEC
- ) {
- try {
- videoEl.currentTime = cameraTime;
- } catch {
- // silent
- }
+ const { videoWidth, videoHeight } = event.currentTarget;
+ if (videoWidth > 0 && videoHeight > 0) {
+ setWebcamNativeSize(cameraTrack.sourcePath, { width: videoWidth, height: videoHeight });
}
}}
- style={videoStyle}
/>
);
}
diff --git a/src/lib/ai-edition/timeline/playback-clock.test.ts b/src/lib/ai-edition/timeline/playback-clock.test.ts
deleted file mode 100644
index 2db6308d0..000000000
--- a/src/lib/ai-edition/timeline/playback-clock.test.ts
+++ /dev/null
@@ -1,82 +0,0 @@
-import { describe, expect, it } from "vitest";
-import {
- CAMERA_SYNC_TOLERANCE_PAUSED_SEC,
- CAMERA_SYNC_TOLERANCE_PLAYING_SEC,
- createPlaybackClockRef,
- type PlaybackClockSnapshot,
- resolveCameraSyncTarget,
-} from "./playback-clock";
-
-function clock(overrides: Partial<PlaybackClockSnapshot> = {}): PlaybackClockSnapshot {
- return { virtualTimeSec: 0, sourceTimeSec: 0, isPlaying: false, playbackRate: 1, ...overrides };
-}
-
-describe("createPlaybackClockRef", () => {
- it("starts at a stopped, rate-1 zero position", () => {
- expect(createPlaybackClockRef().current).toEqual({
- virtualTimeSec: 0,
- sourceTimeSec: 0,
- isPlaying: false,
- playbackRate: 1,
- });
- });
-});
-
-describe("resolveCameraSyncTarget", () => {
- it("returns null when there is no visible camera track", () => {
- expect(resolveCameraSyncTarget(clock(), null, 5)).toBeNull();
- expect(
- resolveCameraSyncTarget(clock(), { startMs: 0, offsetMs: 0, visible: false }, 5),
- ).toBeNull();
- });
-
- it("returns null when the active clip's source position couldn't be resolved", () => {
- expect(
- resolveCameraSyncTarget(clock(), { startMs: 0, offsetMs: 0, visible: true }, null),
- ).toBeNull();
- });
-
- it("subtracts the camera's start/offset adjustment from the clip's source time", () => {
- const target = resolveCameraSyncTarget(
- clock({ sourceTimeSec: 10 }),
- { startMs: 2000, offsetMs: 500, visible: true },
- 10,
- );
- // adjustment = (2000 + 500) / 1000 = 2.5s
- expect(target?.targetTimeSec).toBeCloseTo(7.5);
- });
-
- it("clamps the target time to zero instead of going negative", () => {
- const target = resolveCameraSyncTarget(
- clock(),
- { startMs: 5000, offsetMs: 0, visible: true },
- 1,
- );
- expect(target?.targetTimeSec).toBe(0);
- });
-
- it("carries the clock's playbackRate through so speed regions apply to the camera too", () => {
- const target = resolveCameraSyncTarget(
- clock({ playbackRate: 2 }),
- { startMs: 0, offsetMs: 0, visible: true },
- 3,
- );
- expect(target?.playbackRate).toBe(2);
- });
-
- it("uses the looser playing tolerance while playing, and the tighter paused tolerance while paused", () => {
- const playing = resolveCameraSyncTarget(
- clock({ isPlaying: true }),
- { startMs: 0, offsetMs: 0, visible: true },
- 1,
- );
- const paused = resolveCameraSyncTarget(
- clock({ isPlaying: false }),
- { startMs: 0, offsetMs: 0, visible: true },
- 1,
- );
- expect(playing?.toleranceSec).toBe(CAMERA_SYNC_TOLERANCE_PLAYING_SEC);
- expect(paused?.toleranceSec).toBe(CAMERA_SYNC_TOLERANCE_PAUSED_SEC);
- expect(paused?.toleranceSec).toBeLessThan(playing?.toleranceSec ?? Number.POSITIVE_INFINITY);
- });
-});
diff --git a/src/lib/ai-edition/timeline/playback-clock.ts b/src/lib/ai-edition/timeline/playback-clock.ts
deleted file mode 100644
index 8d5884f71..000000000
--- a/src/lib/ai-edition/timeline/playback-clock.ts
+++ /dev/null
@@ -1,70 +0,0 @@
-// Shared, ref-based playback clock written by the screen preview's rAF loop
-// and read directly by any other media element that needs to stay in lockstep
-// with it (currently: the webcam overlay).
-//
-// Why this exists: previously the webcam derived its sync target from a
-// `currentTimeSec` PROP that traveled screen-video-rAF -> React state
-// (VirtualPreview) -> onTimeChange callback -> React state (NewEditorShell)
-// -> prop drilling back down through Preview/PreviewCanvas -> WebcamOverlay.
-// Each hop adds a render pass of latency and makes the exact update cadence
-// dependent on React's scheduling rather than the video's actual frame rate.
-// Writing to a plain ref every rAF tick and reading it from the webcam's OWN
-// rAF tick removes every one of those hops — the webcam sees the same value
-// the screen just computed, this frame, with no React round trip at all.
-export interface PlaybackClockSnapshot {
- /** Position on the edited/virtual timeline, in seconds. */
- virtualTimeSec: number;
- /** Position in the active screen source's own media time, in seconds. */
- sourceTimeSec: number;
- isPlaying: boolean;
- /** Active `<video>.playbackRate` for the screen source (speed regions). */
- playbackRate: number;
-}
-
-export type PlaybackClockRef = { current: PlaybackClockSnapshot };
-
-export function createPlaybackClockRef(): PlaybackClockRef {
- return {
- current: { virtualTimeSec: 0, sourceTimeSec: 0, isPlaying: false, playbackRate: 1 },
- };
-}
-
-// ponytail: mirrors main's VideoPlayback.tsx webcam-sync tolerances — tighter
-// while paused (a still frame's exact position is visually obvious and a
-// seek is cheap when nothing's moving) and looser while playing (a little
-// slack avoids constant re-seeking/stutter that a viewer won't perceive as
-// precisely during motion).
-export const CAMERA_SYNC_TOLERANCE_PAUSED_SEC = 0.05;
-export const CAMERA_SYNC_TOLERANCE_PLAYING_SEC = 0.15;
-
-export interface CameraSyncTarget {
- targetTimeSec: number;
- playbackRate: number;
- isPlaying: boolean;
- toleranceSec: number;
-}
-
-/**
- * Pure core of the camera-sync tick: given this frame's clock snapshot and
- * the currently-active camera track/position (already resolved by the
- * caller from clips+assets), compute where the camera video should be and
- * how strict to be about correcting it. Kept separate from the rAF/DOM glue
- * in WebcamOverlay so the actual sync math is unit-testable without mounting
- * a component or a real <video> element.
- */
-export function resolveCameraSyncTarget(
- clock: PlaybackClockSnapshot,
- cameraTrack: { startMs: number; offsetMs: number; visible: boolean } | null,
- sourceTimeSecForActiveClip: number | null,
-): CameraSyncTarget | null {
- if (!cameraTrack?.visible || sourceTimeSecForActiveClip === null) return null;
- const offsetSec = (cameraTrack.startMs + cameraTrack.offsetMs) / 1000;
- return {
- targetTimeSec: Math.max(0, sourceTimeSecForActiveClip - offsetSec),
- playbackRate: clock.playbackRate,
- isPlaying: clock.isPlaying,
- toleranceSec: clock.isPlaying
- ? CAMERA_SYNC_TOLERANCE_PLAYING_SEC
- : CAMERA_SYNC_TOLERANCE_PAUSED_SEC,
- };
-}
diff --git a/technical-documentation/architecture/preview.md b/technical-documentation/architecture/preview.md
index 0a83ac5fd..08f730cd0 100644
--- a/technical-documentation/architecture/preview.md
+++ b/technical-documentation/architecture/preview.md
@@ -32,9 +32,20 @@ offscreen `compositor_view` view sized to the canvas's device-pixel rect
and pushes a `SceneDescription` JSON every time the document or the editor
settings change. Wallpaper, screen video, webcam, cursor, zoom regions, annotations
— every visible pixel comes from this view. The DOM neighbours it (the `.screenStage`
-wrapper, the `<video>` for screen decode, the `WebcamOverlay` `<video>`, the
-`AnnotationLayer`, the `ZoomFocusOverlay`, the webcam drag hitbox) are *interactive
-overlays*: their pixels are hidden in CSS, only their pointer-event geometry counts.
+wrapper, the screen `<video>`, the `WebcamOverlay` `<video>`, the `AnnotationLayer`, the
+`ZoomFocusOverlay`, the webcam drag hitbox) are *interactive overlays*: their pixels are
+hidden in CSS, only their pointer-event geometry counts.
+
+The two `<video>` elements decode no picture either. The screen `<video>` is the playback
+clock, and an `<audio>` on the same file plays its sound; both deselect their video track
+once their metadata is in (`dropVideoTrack`, under the `AudioVideoTracks` Blink feature the
+editor window enables). Left alone, each decoded the whole recording a second and a third
+time beside the compositor: ~6.7 % of an RTX 4070 Ti's decode engine apiece on a 1080p60
+take, measured, and zero once the track is dropped, with the clock, `playbackRate` and seeks
+unchanged. A recording without sound keeps its picture: with neither track selected the
+element has nothing left to keep time with and races to its end. The `WebcamOverlay`
+`<video>` is never played at all — it is read for the camera's size (`loadedmetadata`),
+which shapes the PiP box in the scene.
The path is enabled by the *presence of the native addon* — there is no flag,
no capability probe, no per-document switch. The compositing service loads