Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
fix(preview): stop re-seeking the native view at every cut and in speed regions
EtienneLescote358d7710049
diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rsindex 338beaa80..905a6a5e5 100644--- a/crates/compositor-view-napi/src/lib.rs+++ b/crates/compositor-view-napi/src/lib.rs@@ -204,6 +204,10 @@ pub struct FramePacket {/// Le métrage passe de ses coins à l'image par leur homographie (caméra réelle), pas par/// leur interpolation bilinéaire.pub footage_projective: bool,+ /// Clip actif de la scène quand la frame a été composée (`FramePosition`).+ pub clip_index: u32,+ /// Temps source de la frame écran composée, en secondes.+ pub source_time_sec: f64,}/// Renvoie la dernière frame readback du thread de rendu SI elle est plus récente que@@ -243,7 +247,7 @@ pub fn read_frame(id: i32, since_gen: f64) -> Result<Option<FramePacket>> {v.latest_frame_since(since_gen.max(0.0) as u64)}};- Ok(slot.map(|(gen, w, h, pixels, footage)| {+ Ok(slot.map(|(gen, w, h, pixels, footage, position)| {debug_assert_eq!(pixels.len(), (w as usize) * (h as usize) * 4);FramePacket {gen: gen as f64,@@ -252,6 +256,8 @@ pub fn read_frame(id: i32, since_gen: f64) -> Result<Option<FramePacket>> {data: Buffer::from(pixels),footage: footage_corners(footage),footage_projective: footage.is_some_and(|q| q.projective),+ clip_index: position.clip_index,+ source_time_sec: position.source_time_sec,}}))}@@ -276,6 +282,8 @@ pub struct SharedFramePacket {pub height: u32,pub footage: Option<Vec<f64>>,pub footage_projective: bool,+ pub clip_index: u32,+ pub source_time_sec: f64,}/// Livrer les frames de la vue par textures partagées plutôt que par `read_frame`. Rend@@ -314,6 +322,8 @@ pub fn read_shared_frame(id: i32, since_gen: f64) -> Result<Option<SharedFramePaheight: f.height,footage: footage_corners(f.footage),footage_projective: f.footage.is_some_and(|q| q.projective),+ clip_index: f.position.clip_index,+ source_time_sec: f.position.source_time_sec,}))}diff --git a/crates/compositor/src/live.rs b/crates/compositor/src/live.rsindex 912741b2c..5a7a6d737 100644--- a/crates/compositor/src/live.rs+++ b/crates/compositor/src/live.rs@@ -759,15 +759,34 @@ fn scene_for_clip(scene: &Scene, clip_index: usize) -> Scene {/// Dernière frame readback vers CPU, prête pour le napi `read_frame`.///-/// `(gen, w, h, vec)` où `vec.len() == w*h*4` octets RGBA8 tightly-packed (R, G, B, A-/// en mémoire — cf. `Compositor::readback_resized`). `gen` est une génération monotone+/// `(gen, w, h, vec, métrage, position)` où `vec.len() == w*h*4` octets RGBA8 tightly-packed+/// (R, G, B, A en mémoire — cf. `Compositor::readback_resized`). `gen` est une génération monotone/// (≥ 1, `0` réservé à « le consommateur n'a encore rien vu ») incrémentée à CHAQUE/// publication, càd uniquement quand une nouvelle frame a réellement été composée (le/// thread de rendu ne republie pas une frame identique — cf. `stepped || first`). Elle/// est l'IDENTITÉ de la frame : le consommateur (`read_frame`) ne repaie le clone + l'IPC/// que lorsqu'elle change. `None` = "aucune frame composée pour l'instant" (toutes les/// lectures avant la 1re frame composée retournent `None` côté napi, jamais un buffer vide).-pub type LatestFrame = (u64, u32, u32, Vec<u8>, Option<crate::frame_geometry::FootageQuad>);+pub type LatestFrame = (+ u64,+ u32,+ u32,+ Vec<u8>,+ Option<crate::frame_geometry::FootageQuad>,+ FramePosition,+);++/// Où en est la vue quand elle compose une frame : le clip actif de la scène et le temps+/// source de la frame écran. Voyage avec chaque frame publiée, quel que soit le transport,+/// pour que l'app compare la position RÉELLE de la vue à sa propre tête de lecture au lieu+/// de la deviner à partir de l'horloge murale.+#[derive(Debug, Clone, Copy, PartialEq)]+pub struct FramePosition {+ /// Index du clip actif dans `scene.clips`.+ pub clip_index: u32,+ /// pts de la frame écran composée, en secondes du fichier source.+ pub source_time_sec: f64,+}/// État partagé thread appelant → thread de rendu (commandes sans blocage).struct Shared {@@ -1840,7 +1859,11 @@ unsafe fn render_thread(if stepped || first || publish_pending {if pw > 0 && ph > 0 {- match publish_shared(&shared, &gpu, &comp, &mut ring) {+ let position = FramePosition {+ clip_index: active_clip_index as u32,+ source_time_sec: player.screen_time_sec(),+ };+ match publish_shared(&shared, &gpu, &comp, &mut ring, position) {SharedPublish::Published => {first = false;publish_pending = false;@@ -1874,7 +1897,7 @@ unsafe fn render_thread(// l'ancienne dérivation tant que le slot n'est pas vidé.let next_gen = shared.frame_gen.fetch_add(1, Ordering::Relaxed) + 1;if let Ok(mut slot) = shared.latest_frame.lock() {- *slot = Some((next_gen, rw, rh, rgba, comp.footage_quad()));+ *slot = Some((next_gen, rw, rh, rgba, comp.footage_quad(), position));}first = false;publish_pending = false;@@ -1918,6 +1941,7 @@ unsafe fn publish_shared(gpu: &Gpu,comp: &Compositor,ring: &mut Ring,+ position: FramePosition,) -> SharedPublish {if !shared.shared_frames.load(Ordering::Relaxed) {return SharedPublish::Off;@@ -1967,6 +1991,7 @@ unsafe fn publish_shared(width,height,footage: comp.footage_quad(),+ position,});}SharedPublish::Published@@ -1976,7 +2001,13 @@ unsafe fn publish_shared(}#[cfg(not(windows))]-unsafe fn publish_shared(_: &Shared, _: &Gpu, _: &Compositor, _: &mut Ring) -> SharedPublish {+unsafe fn publish_shared(+ _: &Shared,+ _: &Gpu,+ _: &Compositor,+ _: &mut Ring,+ _: FramePosition,+) -> SharedPublish {SharedPublish::Off}@@ -2113,7 +2144,7 @@ pub fn run_standalone(screen: &str, webcam: &str, cursor_json: &str) -> Result<(// standalone n'affiche pas réellement les pixels ici (l'embed Electron est// le consumer réel). On imprime juste une frame de temps en temps pour// confirmer que la chaîne fonctionne.- if let Some((_gen, fw, fh, _pixels, _)) = view.latest_frame() {+ if let Some((_gen, fw, fh, _pixels, _, _)) = view.latest_frame() {if (fw, fh) != (w, h) {// garde-fou : la staging de readback suit `set_rect` côté thread// de rendu, donc ce serait une désynchro transitoire — acceptable.diff --git a/crates/compositor/src/shared_frames.rs b/crates/compositor/src/shared_frames.rsindex 421897131..a6b4f8415 100644--- a/crates/compositor/src/shared_frames.rs+++ b/crates/compositor/src/shared_frames.rs@@ -13,6 +13,7 @@//! macOS (IOSurface) et Linux (dmabuf) restent sur le readback.use crate::frame_geometry::FootageQuad;+use crate::live::FramePosition;use std::time::{Duration, Instant};/// Cases de l'anneau : une prête, une en transit vers le renderer (tenue jusqu'à ce que@@ -37,6 +38,7 @@ pub struct SharedFrame {pub width: u32,pub height: u32,pub footage: Option<FootageQuad>,+ pub position: FramePosition,}/// Une case prise par JS : la génération qu'elle porte et quand elle est partie.@@ -244,7 +246,15 @@ mod tests {use super::*;fn frame(gen: u64, slot: u32) -> SharedFrame {- SharedFrame { gen, slot, handle: 0, width: 2, height: 2, footage: None }+ SharedFrame {+ gen,+ slot,+ handle: 0,+ width: 2,+ height: 2,+ footage: None,+ position: FramePosition { clip_index: 0, source_time_sec: 0.0 },+ }}/// Toutes les cases prises par JS, la case `n` portant la génération `n + 1`.diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.tsindex 51e562979..6dbfb1461 100644--- a/electron/native-bridge/services/compositorViewService.ts+++ b/electron/native-bridge/services/compositorViewService.ts@@ -703,6 +703,8 @@ export class CompositorViewService {height: frame.height,footage: frame.footage ?? null,footageProjective: frame.footageProjective ?? false,+ clipIndex: frame.clipIndex,+ sourceTimeSec: frame.sourceTimeSec,};const api = this.sharedTextureApi();let imported: Electron.SharedTextureImported | undefined;diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.tsindex 631d70fdf..e87b830b5 100644--- a/electron/native/compositor-view/addon.d.ts+++ b/electron/native/compositor-view/addon.d.ts@@ -39,6 +39,10 @@ export interface NativeFramePacket {footage?: number[] | null;/** The footage maps from those corners by their homography (real camera), not bilinearly. */footageProjective?: boolean;+ /** Where the view was when it composed this frame: the active clip in `scene.clips` and the+ * screen frame's time in its source file. Absent from an older `.node`. */+ clipIndex?: number;+ sourceTimeSec?: number;}/** A preview frame left in a shared GPU texture instead of copied into RAM (Windows, hardware@@ -53,6 +57,10 @@ export interface NativeSharedFramePacket {height: number;footage?: number[] | null;footageProjective?: boolean;+ /** Where the view was when it composed this frame: the active clip in `scene.clips` and the+ * screen frame's time in its source file. Absent from an older `.node`. */+ clipIndex?: number;+ sourceTimeSec?: number;}export interface ExportStats {diff --git a/src/components/ai-edition/NativeCompositorOverlay.test.tsx b/src/components/ai-edition/NativeCompositorOverlay.test.tsxnew file mode 100644index 000000000..43d389e62--- /dev/null+++ b/src/components/ai-edition/NativeCompositorOverlay.test.tsx@@ -0,0 +1,195 @@+// @vitest-environment jsdom+/**+ * How the overlay steers the native view while the transport plays.+ *+ * The view runs its own clock and crosses clip boundaries by itself. The overlay used to+ * re-send the clip at every cut anyway, which made the view seek back to a place it had just+ * left, or drop the next clip it had preloaded: a hitch at every cut of an edited take. Now it+ * reads where the view is from the position each frame reports, and only steps in when the+ * view is somewhere else.+ */+import { act, cleanup, render } from "@testing-library/react";+import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";+import type { AxcutClip, AxcutDocument } from "@/lib/ai-edition/schema";+import { axcutSchemaVersion } from "@/lib/ai-edition/schema";+import { useProjectStore } from "@/lib/ai-edition/store/projectStore";+import { publishNativePosition } from "@/native/nativeSync";++const native = vi.hoisted(() => ({+ setActiveClip: vi.fn(async () => ({ ok: true })),+ setNativePlaying: vi.fn(),+}));++vi.mock("@/native", () => ({+ pushAllNativeParams: vi.fn(),+ setActiveClip: native.setActiveClip,+ setCurrentNativeViewId: vi.fn(),+ setNativePlaying: native.setNativePlaying,+ setNativeScene: vi.fn(),+ subscribeNativeCompositor: () => () => undefined,+ useIsCpuCompositor: () => false,+ useNativeCompositorView: () => ({ viewId: 7, error: null }),+}));++vi.mock("@/contexts/I18nContext", () => ({+ useScopedT: () => (key: string) => key,+}));++import { NativeCompositorOverlay } from "./NativeCompositorOverlay";++/** One take cut in two: source 0-5 s, then 7-12 s after a 2 s cut. */+const FIRST: AxcutClip = {+ id: "clip_a",+ assetId: "asset_take",+ sourceStartSec: 0,+ sourceEndSec: 5,+ timelineStartSec: 0,+ timelineEndSec: 5,+ wordRefs: [],+ origin: "system",+ reason: "",+};+const SECOND: AxcutClip = {+ ...FIRST,+ id: "clip_b",+ sourceStartSec: 7,+ sourceEndSec: 12,+ timelineStartSec: 5,+ timelineEndSec: 10,+};++function makeDocument(): AxcutDocument {+ return {+ schemaVersion: axcutSchemaVersion,+ project: {+ id: "proj_sync",+ title: "Sync",+ createdAt: "2026-10-02T08:00:00.000Z",+ updatedAt: "2026-10-02T08:00:00.000Z",+ primaryAssetId: "asset_take",+ },+ assets: [+ {+ id: "asset_take",+ kind: "video",+ label: "take",+ originalPath: "/take.mp4",+ cameraTrack: null,+ },+ ],+ transcript: null,+ transcripts: [],+ timeline: {+ clips: [FIRST, SECOND],+ gaps: [],+ trimRanges: [],+ muteRanges: [],+ speedRanges: [],+ captionRanges: [],+ },+ annotations: [],+ zoomRanges: [],+ audioTracks: [],+ legacyEditor: null,+ };+}++function setPlayhead(currentTimeSec: number, playing: boolean) {+ act(() => {+ useProjectStore.setState({ currentTimeSec, playing });+ });+}++describe("NativeCompositorOverlay while playing", () => {+ let now = 1000;++ beforeEach(() => {+ vi.clearAllMocks();+ vi.spyOn(performance, "now").mockImplementation(() => now);+ useProjectStore.setState({+ projectId: "proj_sync",+ document: makeDocument(),+ revision: 1,+ status: "ready",+ error: null,+ sourceDurationSec: 12,+ currentTimeSec: 4.9,+ playing: true,+ dirty: false,+ lastSavedAt: new Date(),+ });+ });++ afterEach(() => {+ cleanup();+ vi.restoreAllMocks();+ publishNativePosition(null);+ useProjectStore.getState().clear();+ });++ async function mountAtFirstClip() {+ render(<NativeCompositorOverlay />);+ // The first mount sends the starting clip, and its reply resumes the view: let both+ // settle, so only what follows is under test.+ await act(async () => {+ await Promise.resolve();+ });+ native.setActiveClip.mockClear();+ native.setNativePlaying.mockClear();+ }++ it("lets the view cross a cut by itself", async () => {+ await mountAtFirstClip();+ publishNativePosition({ clipIndex: 0, sourceTimeSec: 4.99 }, now);++ setPlayhead(5.02, true);++ expect(native.setActiveClip).not.toHaveBeenCalled();+ expect(native.setNativePlaying).not.toHaveBeenCalled();+ });++ it("follows a jump to another clip at once, without pausing the view", async () => {+ await mountAtFirstClip();+ publishNativePosition({ clipIndex: 0, sourceTimeSec: 4.9 }, now);++ setPlayhead(8, true);++ expect(native.setActiveClip).toHaveBeenCalledWith(7, "/take.mp4", "", 0, 1, 10);+ expect(native.setNativePlaying).not.toHaveBeenCalledWith(false);+ });++ it("re-anchors a view that stays behind, once the gap holds", async () => {+ await mountAtFirstClip();+ // The view stalled 0.4 s behind the playhead, inside the first clip.+ publishNativePosition({ clipIndex: 0, sourceTimeSec: 2 }, now);+ setPlayhead(2.4, true);+ expect(native.setActiveClip).not.toHaveBeenCalled();++ now += 120;+ publishNativePosition({ clipIndex: 0, sourceTimeSec: 2.12 }, now);+ setPlayhead(2.52, true);++ expect(native.setActiveClip).toHaveBeenCalledTimes(1);+ expect(native.setActiveClip).toHaveBeenCalledWith(7, "/take.mp4", "", 0, 0, 2.52);+ });++ it("still sends the clip on a change while paused", async () => {+ useProjectStore.setState({ playing: false });+ await mountAtFirstClip();+ publishNativePosition({ clipIndex: 0, sourceTimeSec: 4.9 }, now);++ setPlayhead(6, false);++ expect(native.setActiveClip).toHaveBeenCalledWith(7, "/take.mp4", "", 0, 1, 8);+ });++ // An addon that reports no position cannot be read, so it is driven as before.+ it("pauses the view across the swap when it reports no position", async () => {+ await mountAtFirstClip();++ setPlayhead(5.02, true);++ expect(native.setNativePlaying).toHaveBeenCalledWith(false);+ expect(native.setActiveClip).toHaveBeenCalledTimes(1);+ });+});diff --git a/src/components/ai-edition/NativeCompositorOverlay.tsx b/src/components/ai-edition/NativeCompositorOverlay.tsxindex e4af64102..e98e9e45b 100644--- a/src/components/ai-edition/NativeCompositorOverlay.tsx+++ b/src/components/ai-edition/NativeCompositorOverlay.tsx@@ -15,6 +15,14 @@ import {useIsCpuCompositor,useNativeCompositorView,} from "@/native";+import {+ type DriftWatch,+ getNativePosition,+ IDLE_DRIFT_WATCH,+ NATIVE_DRIFT_TOLERANCE_SEC,+ nativeLeadSec,+ watchDrift,+} from "@/native/nativeSync";import { buildSceneDescription, resolveVisibleClips } from "@/native/sceneDescription";import {getWebcamNativeSize,@@ -186,9 +194,18 @@ export function NativeCompositorOverlay() {const pendingTargetClipIdRef = useRef<string | null>(null);const playing = useProjectStore((s) => s.playing);+ const wasPlayingRef = useRef(playing);- // Change les décodeurs screen/webcam uniquement quand le playhead entre dans un autre clip.+ // Change les décodeurs screen/webcam quand le playhead entre dans un autre clip, à l'arrêt.+ //+ // En lecture, la vue enchaîne seule les clips, préchargement compris. Lui renvoyer le clip à+ // chaque coupe la faisait rechercher une position qu'elle avait déjà, ou jeter le+ // préchargement qu'elle s'apprêtait à utiliser : un recul de quelques images et un arrêt à+ // chaque coupe. Elle n'est recalée que si elle est réellement ailleurs (guet de dérive+ // ci-dessous), d'après la position que porte chaque frame.useEffect(() => {+ const justPaused = wasPlayingRef.current && !playing;+ wasPlayingRef.current = playing;if (viewId === null ||!document ||@@ -199,7 +216,13 @@ export function NativeCompositorOverlay() {) {return;}- if (previousActiveClipIdRef.current === activeClipId) {+ const native = getNativePosition();+ const clipChanged = previousActiveClipIdRef.current !== activeClipId;+ // Mise en pause pendant que la vue était sur un autre clip que la tête de lecture : elle+ // y retourne, sans quoi le recalage en temps (`setNativeTime`) chercherait dans le+ // mauvais fichier.+ const nativeElsewhere = justPaused && native !== null && native.clipIndex !== activeClipIndex;+ if (!clipChanged && !nativeElsewhere) {return;}const asset = document.assets.find((candidate) => candidate.id === activeClip.assetId);@@ -210,13 +233,30 @@ export function NativeCompositorOverlay() {const targetClipId = activeClipId;// Sonde de fluidité (diagnostic) : sépare les mesures d'avant et d'après un// franchissement de clip, qui se sont déjà révélées non comparables.- noteUiProbeClipSwitch(previousActiveClipIdRef.current, activeClipId);+ if (clipChanged) {+ noteUiProbeClipSwitch(previousActiveClipIdRef.current, activeClipId);+ }pendingTargetClipIdRef.current = targetClipId;previousActiveClipIdRef.current = targetClipId;+ if (playing && native !== null) {+ const lead = nativeLeadSec(+ native,+ { clipIndex: activeClipIndex, sourceTimeSec: activeSourceTimeSec },+ nativeClips,+ performance.now(),+ );+ // Une coupe franchie en lecture : la vue la franchit d'elle-même, au même endroit du+ // programme. Un saut (clic ailleurs sur la timeline) la met loin : suivi tout de suite,+ // sans attendre le délai du guet de dérive.+ if (lead !== null && Math.abs(lead) <= NATIVE_DRIFT_TOLERANCE_SEC) {+ return;+ }+ }- // Pause native across the decoder swap. Deliberately NOT kept in a variable to- // resume from later — see the `.then` below, which re-reads the live transport.- if (playing) {+ // Sans position rapportée (addon antérieur), la vue est encore pilotée comme avant : en+ // pause pendant le changement de décodeurs. Deliberately NOT kept in a variable to resume+ // from later — see the `.then` below, which re-reads the live transport.+ if (playing && native === null) {setNativePlaying(false);}@@ -248,7 +288,63 @@ export function NativeCompositorOverlay() {previousActiveClipIdRef.current = null;}});- }, [viewId, document, activeClipId, activeClip, activeClipIndex, activeSourceTimeSec, playing]);+ }, [+ viewId,+ document,+ activeClipId,+ activeClip,+ activeClipIndex,+ activeSourceTimeSec,+ playing,+ nativeClips,+ ]);++ // Guet de dérive, en lecture : la vue tourne sur sa propre horloge, et n'est recalée que si+ // la position que porte sa dernière frame s'écarte de la tête de lecture et le reste. Couvre+ // un saut de l'utilisateur pendant la lecture comme un blocage du thread de rendu. L'ancien+ // recalage devinait la dérive à l'horloge murale, à vitesse 1 : dans une région à 2× il+ // relançait une recherche dix fois par seconde.+ const driftRef = useRef<DriftWatch>(IDLE_DRIFT_WATCH);+ useEffect(() => {+ if (+ viewId === null ||+ !document ||+ !playing ||+ !activeClip ||+ activeClipIndex === null ||+ activeSourceTimeSec === null+ ) {+ driftRef.current = { ...driftRef.current, outSinceMs: null };+ return;+ }+ const now = performance.now();+ const lead = nativeLeadSec(+ getNativePosition(),+ { clipIndex: activeClipIndex, sourceTimeSec: activeSourceTimeSec },+ nativeClips,+ now,+ );+ const { watch, resync } = watchDrift(driftRef.current, lead, now);+ driftRef.current = watch;+ if (!resync) {+ return;+ }+ const asset = document.assets.find((candidate) => candidate.id === activeClip.assetId);+ if (!asset?.originalPath) {+ return;+ }+ const camera = assetCameraSource(asset);+ setActiveClip(+ viewId,+ asset.originalPath,+ camera.path,+ camera.offsetSec,+ activeClipIndex,+ activeSourceTimeSec,+ ).catch((error: unknown) => {+ console.warn("[compositor-view] re-anchoring the preview failed:", error);+ });+ }, [viewId, document, playing, activeClip, activeClipIndex, activeSourceTimeSec, nativeClips]);if (!ready) {return null;diff --git a/src/native/contracts.ts b/src/native/contracts.tsindex d13a95f5c..5421b994a 100644--- a/src/native/contracts.ts+++ b/src/native/contracts.ts@@ -165,6 +165,10 @@ export interface CompositorFramePacket {footage?: number[] | null;/** The footage maps from those corners by their homography (real camera), not bilinearly. */footageProjective?: boolean;+ /** Where the view was when it composed this frame: the active clip in the scene's clips and+ * the screen frame's time in its source file. Absent from an older addon. */+ clipIndex?: number;+ sourceTimeSec?: number;}/** What travels with a preview frame handed over as a shared GPU texture (Windows): all a@@ -177,6 +181,9 @@ export interface CompositorSharedFrameMeta {height: number;footage: number[] | null;footageProjective: boolean;+ /** See {@link CompositorFramePacket.clipIndex}. */+ clipIndex?: number;+ sourceTimeSec?: number;}/** `readFrame`'s answer for a frame sent as a shared texture. The texture reacheddiff --git a/src/native/hooks/useNativeCompositorView.ts b/src/native/hooks/useNativeCompositorView.tsindex 94688c2bf..e11ec5938 100644--- a/src/native/hooks/useNativeCompositorView.ts+++ b/src/native/hooks/useNativeCompositorView.ts@@ -37,6 +37,7 @@ import {} from "../compositorViewClient";import type { CompositorParamValue, CompositorViewRect } from "../contracts";import { publishFootageQuad } from "../footageQuadStore";+import { publishNativePosition } from "../nativeSync";import { computeDeviceRect, rectsEqual } from "../nativeViewRect";export interface UseNativeCompositorViewOptions {@@ -245,6 +246,7 @@ export function useNativeCompositorView(sharedTransport = true;lastGen = Math.max(lastGen, meta.gen);publishFootageQuad(meta.footage, meta.footageProjective);+ publishNativePosition(meta);const fresh = canvas.dataset.painted === undefined;const drawn = paint(meta.gen, meta.width, meta.height, () => ctx.drawImage(frame, 0, 0));if (drawn && fresh && !sharedChecked) {@@ -314,6 +316,7 @@ export function useNativeCompositorView(}// Where the footage lies in this frame: a privacy blur's gimbal follows it.publishFootageQuad(frame.footage, frame.footageProjective);+ publishNativePosition(frame);// Wrap the received buffer DIRECTLY — no intermediate copy. `data` is a// fresh per-frame Buffer from IPC (never pooled or reused across frames),// so a view over it is valid for the lifetime of this paint, and nothing@@ -399,6 +402,7 @@ export function useNativeCompositorView(disposed = true;unsubscribeShared();publishFootageQuad(null);+ publishNativePosition(null);if (rectRafHandle !== 0) {cancelAnimationFrame(rectRafHandle);}diff --git a/src/native/nativeSync.test.ts b/src/native/nativeSync.test.tsnew file mode 100644index 000000000..224dd067f--- /dev/null+++ b/src/native/nativeSync.test.ts@@ -0,0 +1,115 @@+import { describe, expect, it } from "vitest";+import {+ getNativePosition,+ IDLE_DRIFT_WATCH,+ NATIVE_DRIFT_PERSIST_MS,+ NATIVE_RESYNC_COOLDOWN_MS,+ nativeLeadSec,+ programmeTimeSec,+ publishNativePosition,+ watchDrift,+} from "./nativeSync";++/** Two segments of one recording around a cut: source 0-5 s, then 7-12 s from programme 5 s. */+const SEGMENTS = [+ { timelineStartSec: 0, sourceStartSec: 0 },+ { timelineStartSec: 5, sourceStartSec: 7 },+];++describe("programmeTimeSec", () => {+ it("places a source time on the trim-compressed timeline", () => {+ expect(programmeTimeSec(SEGMENTS, { clipIndex: 1, sourceTimeSec: 8 })).toBe(6);+ });++ it("has no answer for a clip the layout does not have", () => {+ expect(programmeTimeSec(SEGMENTS, { clipIndex: 4, sourceTimeSec: 8 })).toBeNull();+ });+});++describe("nativeLeadSec", () => {+ // Across a cut the source time jumps by the trimmed 2 s; on the programme it does not.+ it("sees no gap when the view and the playhead sit on either side of a cut", () => {+ const lead = nativeLeadSec(+ { clipIndex: 0, sourceTimeSec: 4.99, receivedAtMs: 1000 },+ { clipIndex: 1, sourceTimeSec: 7.02 },+ SEGMENTS,+ 1000,+ );+ expect(lead).toBeCloseTo(-0.03, 5);+ });++ it("ages the view's last frame by the time since it arrived", () => {+ const lead = nativeLeadSec(+ { clipIndex: 1, sourceTimeSec: 8, receivedAtMs: 1000 },+ { clipIndex: 1, sourceTimeSec: 8.05 },+ SEGMENTS,+ 1050,+ );+ expect(lead).toBeCloseTo(0, 5);+ });++ it("measures a view left behind by a stall", () => {+ const lead = nativeLeadSec(+ { clipIndex: 0, sourceTimeSec: 2, receivedAtMs: 1000 },+ { clipIndex: 0, sourceTimeSec: 2.4 },+ SEGMENTS,+ 1000,+ );+ expect(lead).toBeCloseTo(-0.4, 5);+ });++ it("says nothing without a reported position", () => {+ expect(nativeLeadSec(null, { clipIndex: 0, sourceTimeSec: 1 }, SEGMENTS, 0)).toBeNull();+ });+});++describe("watchDrift", () => {+ it("leaves a view within tolerance alone", () => {+ const { resync, watch } = watchDrift(IDLE_DRIFT_WATCH, 0.1, 1000);+ expect(resync).toBe(false);+ expect(watch.outSinceMs).toBeNull();+ });++ // Right at a cut, or after an edit re-indexes the clips, the gap lasts a frame or two.+ it("re-anchors only a gap that holds", () => {+ let state = watchDrift(IDLE_DRIFT_WATCH, 0.5, 1000);+ expect(state.resync).toBe(false);+ state = watchDrift(state.watch, 0.5, 1000 + NATIVE_DRIFT_PERSIST_MS - 1);+ expect(state.resync).toBe(false);+ state = watchDrift(state.watch, 0.5, 1000 + NATIVE_DRIFT_PERSIST_MS);+ expect(state.resync).toBe(true);+ });++ it("forgets a gap that closed before it held", () => {+ let state = watchDrift(IDLE_DRIFT_WATCH, 0.5, 1000);+ state = watchDrift(state.watch, 0.01, 1050);+ state = watchDrift(state.watch, 0.5, 1100);+ expect(state.resync).toBe(false);+ });++ it("waits for a re-anchor to land before the next", () => {+ let state = watchDrift(IDLE_DRIFT_WATCH, 0.5, 0);+ state = watchDrift(state.watch, 0.5, NATIVE_DRIFT_PERSIST_MS);+ expect(state.resync).toBe(true);+ const at = NATIVE_DRIFT_PERSIST_MS;+ state = watchDrift(state.watch, 0.5, at + 10);+ state = watchDrift(state.watch, 0.5, at + NATIVE_RESYNC_COOLDOWN_MS - 1);+ expect(state.resync).toBe(false);+ state = watchDrift(state.watch, 0.5, at + NATIVE_RESYNC_COOLDOWN_MS);+ expect(state.resync).toBe(true);+ });+});++describe("publishNativePosition", () => {+ it("keeps the position a frame reports, with when it arrived", () => {+ publishNativePosition({ clipIndex: 2, sourceTimeSec: 3.5 }, 42);+ expect(getNativePosition()).toEqual({ clipIndex: 2, sourceTimeSec: 3.5, receivedAtMs: 42 });+ });++ // An addon that predates positions must not be steered on a stale one.+ it("forgets the position when a frame reports none", () => {+ publishNativePosition({ clipIndex: 2, sourceTimeSec: 3.5 }, 42);+ publishNativePosition({});+ expect(getNativePosition()).toBeNull();+ });+});diff --git a/src/native/nativeSync.ts b/src/native/nativeSync.tsnew file mode 100644index 000000000..84539572a--- /dev/null+++ b/src/native/nativeSync.ts@@ -0,0 +1,120 @@+/**+ * Where the native preview view is, against where the app's playhead is.+ *+ * While playing, the view runs its own clock and crosses clip boundaries by itself, so the+ * app does not drive it frame by frame. It used to guess whether the view had drifted from+ * the wall clock, assuming 1× speed: inside a 2× speed region the guess was 100 ms off every+ * 100 ms, and the view was re-seeked ten times a second. Each frame now says where the view+ * really is (`clipIndex`, `sourceTimeSec`), and this compares that with the playhead on the+ * one timeline both share: programme time, the trim-compressed one, where a cut is no jump.+ */++/** A segment of the trim-compressed programme, as `resolveVisibleClips` lays them out. */+export interface ProgrammeSegment {+ timelineStartSec: number;+ sourceStartSec: number;+}++/** Where a frame is: the active clip in the scene and the frame's time in its source file. */+export interface NativePosition {+ clipIndex: number;+ sourceTimeSec: number;+}++/** The position the view last reported, and when it reached the renderer. */+export interface ReportedNativePosition extends NativePosition {+ receivedAtMs: number;+}++/** Beyond this the picture is visibly off the sound: re-anchor the view. Lip sync is noticed+ * from ~45 ms when the picture leads and ~125 ms when it lags (ITU-R BT.1359); frames reach+ * the renderer up to a frame or two late, so a tighter bound would chase measurement noise. */+export const NATIVE_DRIFT_TOLERANCE_SEC = 0.15;+/** The gap must hold this long: right at a cut the view and the playhead cross it a frame or+ * two apart, and a scene edit re-indexes the clips for a frame before both agree again. */+export const NATIVE_DRIFT_PERSIST_MS = 100;+/** After a re-anchor the view needs a seek and a first frame before it reports the new place. */+export const NATIVE_RESYNC_COOLDOWN_MS = 500;++/** `sourceTimeSec` of segment `clipIndex`, on the programme timeline. `null` for a segment the+ * layout does not have (the scene and the document briefly disagree after an edit). */+export function programmeTimeSec(+ segments: readonly ProgrammeSegment[],+ position: NativePosition,+): number | null {+ const segment = segments[position.clipIndex];+ if (!segment) {+ return null;+ }+ return segment.timelineStartSec + (position.sourceTimeSec - segment.sourceStartSec);+}++/** How far ahead of the playhead the view is now, in programme seconds (negative: behind).+ * Its last frame is aged by the wall time since it arrived. `null` when either position+ * cannot be placed. */+export function nativeLeadSec(+ native: ReportedNativePosition | null,+ app: NativePosition,+ segments: readonly ProgrammeSegment[],+ nowMs: number,+): number | null {+ if (!native) {+ return null;+ }+ const nativeSec = programmeTimeSec(segments, native);+ const appSec = programmeTimeSec(segments, app);+ if (nativeSec === null || appSec === null) {+ return null;+ }+ return nativeSec + Math.max(0, nowMs - native.receivedAtMs) / 1000 - appSec;+}++export interface DriftWatch {+ /** Since when the view has been out of tolerance, `null` while it is in. */+ outSinceMs: number | null;+ lastResyncMs: number;+}++export const IDLE_DRIFT_WATCH: DriftWatch = {+ outSinceMs: null,+ lastResyncMs: Number.NEGATIVE_INFINITY,+};++/** Whether to re-anchor the view now, given its lead. Only for a gap that holds, and not again+ * before the last re-anchor has had time to land. */+export function watchDrift(+ watch: DriftWatch,+ leadSec: number | null,+ nowMs: number,+): { watch: DriftWatch; resync: boolean } {+ if (leadSec === null || Math.abs(leadSec) <= NATIVE_DRIFT_TOLERANCE_SEC) {+ return { watch: { ...watch, outSinceMs: null }, resync: false };+ }+ const outSinceMs = watch.outSinceMs ?? nowMs;+ if (+ nowMs - outSinceMs < NATIVE_DRIFT_PERSIST_MS ||+ nowMs - watch.lastResyncMs < NATIVE_RESYNC_COOLDOWN_MS+ ) {+ return { watch: { ...watch, outSinceMs }, resync: false };+ }+ return { watch: { outSinceMs: null, lastResyncMs: nowMs }, resync: true };+}++let reported: ReportedNativePosition | null = null;++/** Records where the view's latest frame is, as it reaches the renderer. `null` forgets it: the+ * view is gone, or its addon predates positions — then nothing here may steer it. */+export function publishNativePosition(+ position: Partial<NativePosition> | null | undefined,+ receivedAtMs = performance.now(),+): void {+ reported =+ position && typeof position.clipIndex === "number" && typeof position.sourceTimeSec === "number"+ ? { clipIndex: position.clipIndex, sourceTimeSec: position.sourceTimeSec, receivedAtMs }+ : null;+}++/** The position the view last reported, `null` before its first frame or without one. */+export function getNativePosition(): ReportedNativePosition | null {+ return reported;+}diff --git a/src/native/useNativePlaybackSync.ts b/src/native/useNativePlaybackSync.tsindex 64e877b63..3ade18269 100644--- a/src/native/useNativePlaybackSync.ts+++ b/src/native/useNativePlaybackSync.ts@@ -13,9 +13,10 @@* over free-run). So discrete seeks are only sent while *paused* — i.e. real* scrub/step interactions. Pausing also re-snaps native to the app playhead.*- * Known POC limitation: during free-run the native clock and the app clock can- * drift (independent tickers); acceptable for the fixture (~6 s loop). A pause- * re-aligns them.+ * While playing, the two clocks are compared by `NativeCompositorOverlay`, from the+ * position each native frame reports (`nativeSync.ts`), not here: this hook used to+ * guess the drift from the wall clock at 1× speed, and re-seeked the view ten times a+ * second inside a 2× speed region.*/import { useEffect, useMemo, useRef, useSyncExternalStore } from "react";import type { AxcutClip } from "@/lib/ai-edition/schema";@@ -57,43 +58,19 @@ export function useNativePlaybackSync(setNativePlaying(playing);}, [active, playing]);- // Scrub/step while paused OR periodic resync during playback when drift > 100ms- const lastSyncedSourceTimeRef = useRef<number | null>(null);- const lastSyncedWallTimeRef = useRef<number>(0);+ // Scrub/step while paused. A clip change is `setActiveClip`'s, in the overlay.const lastActiveClipIdRef = useRef<string | null>(null);useEffect(() => {if (!active || sourceTimeSec === null || !activeClipId) {return;}- const now = performance.now();-- // When clip changes, let setActiveClip handle the atomic clip-switch-and-seek.if (lastActiveClipIdRef.current !== activeClipId) {lastActiveClipIdRef.current = activeClipId;- lastSyncedSourceTimeRef.current = sourceTimeSec;- lastSyncedWallTimeRef.current = now;return;}-if (!playing) {setNativeTime(sourceTimeSec);- lastSyncedSourceTimeRef.current = sourceTimeSec;- lastSyncedWallTimeRef.current = now;- return;- }- // While playing: periodically verify master clock alignment to prevent drift- if (lastSyncedSourceTimeRef.current === null || lastSyncedWallTimeRef.current === 0) {- lastSyncedSourceTimeRef.current = sourceTimeSec;- lastSyncedWallTimeRef.current = now;- return;- }- const wallElapsedSec = (now - lastSyncedWallTimeRef.current) / 1000;- const expectedSourceTimeSec = lastSyncedSourceTimeRef.current + wallElapsedSec;- if (Math.abs(sourceTimeSec - expectedSourceTimeSec) > 0.1) {- setNativeTime(sourceTimeSec);- lastSyncedSourceTimeRef.current = sourceTimeSec;- lastSyncedWallTimeRef.current = now;}}, [active, playing, activeClipId, sourceTimeSec]);}diff --git a/technical-documentation/architecture/preview.md b/technical-documentation/architecture/preview.mdindex 8fa2d6fd5..6df1823e1 100644--- a/technical-documentation/architecture/preview.md+++ b/technical-documentation/architecture/preview.md@@ -245,12 +245,15 @@ a new clip). The mapping sits in`clipIndex` + source time. Without this bridge a RAW playhead against acompressed clip list pointed at the wrong clip after a trim — wrong camera,misaligned screen.-- **Drift re-anchor.** During free-run the two clocks can drift; once the- additive error exceeds 100 ms (`Math.abs(sourceTimeSec - expectedSourceTimeSec) > 0.1`,- [`useNativePlaybackSync.ts:94`](../../src/native/useNativePlaybackSync.ts:94))- the hook re-issues `setNativeTime`. `useNativePlaybackSync:18` calls this a- known limitation acceptable for the ~6 s fixture it's measured on; a pause- resets the drift by construction.+- **Drift, measured rather than guessed.** Every frame carries where the view was when it+ composed it (`clipIndex`, `sourceTimeSec`, both transports), and+ `NativeCompositorOverlay` compares that with the playhead on the one timeline both share:+ programme time, the trim-compressed one, where a cut is no jump+ ([`nativeSync.ts`](../../src/native/nativeSync.ts)). A gap over 150 ms that holds for+ 100 ms re-anchors the view with `setActiveClip`, at most every 500 ms. That covers a stall+ of the render thread and a jump by the user while playing. It replaced a guess from the+ wall clock at 1× speed, which inside a 2× speed region re-seeked the view ten times a+ second.The overlay's rect is kept aligned with the DOM via the same primitives usedelsewhere in the renderer:@@ -272,12 +275,21 @@ elsewhere in the renderer:Clip changes across the playhead boundary are atomic at the`setActiveClip(viewId, screenPath, webcamPath, webcamOffsetSec, clipIndex, sourceTimeSec)`RPC-([`compositorViewClient.ts:93`](../../src/native/compositorViewClient.ts:93)):-when the playhead crosses into a clip whose `assetId` / `webcamPath` differs-from the previous one, `NativeCompositorOverlay.tsx:175-203` pauses native across-the decoder swap, awaits `setActiveClip`, and re-reads the live transport *now*-(not from a captured `isPlaying`) before resuming — so a user pause that lands-in the middle of a clip transition is honoured, not silently undone.+([`compositorViewClient.ts`](../../src/native/compositorViewClient.ts)), and who sends it+depends on the transport:++- **Paused** (scrub, step): `NativeCompositorOverlay` sends it whenever the playhead enters+ another clip.+- **Playing**: the render thread crosses into the next clip by itself, preloading it ahead+ of the cut. The overlay used to send the clip again at every cut, which made the view+ seek back to a place it had just left, or drop the clip it had preloaded: a hitch at+ every cut of an edited take. It now only sends it when the view is elsewhere — a jump+ (a click on the timeline, followed at once) or a gap the drift watch above catches.+ On a pause it also brings back a view left on another clip, before `setNativeTime`+ seeks in it.+- **An addon that reports no position** is driven as before: native is paused across the+ decoder swap, and the live transport is re-read *now* (not from a captured `isPlaying`)+ before resuming, so a user pause that lands mid-transition is honoured.## When the decode clock fails@@ -341,14 +353,9 @@ was thrown away.path and not for the live view. Editing playback is therefore silent againstthe exported file; users hear audio only when the export runs. There is noflag in this branch that re-routes live audio.-- **Long-recording scrub drift.** `useNativePlaybackSync:18` documents the- accepted-at-fixture-time drift between the app's rAF playhead and the addon's- free-run clock as a known limitation; a pause re-aligns them. A scrub further- than 100 ms past expected position triggers an explicit re-anchor; below that- the two clocks run independently until something forces a sync. Long recordings- measured at the bench in- [engineering/rendering-performance.md](../engineering/rendering-performance.md)- stay below the threshold in practice, but no systematic measurement exists.+- **Drift under 150 ms is left alone.** The view and the app's clock run independently+ inside the drift watch's tolerance, and a correction is a seek, not a change of pace: a+ view drifting slowly is re-anchored with a visible step rather than eased back.- **Shared textures are Windows-only.** macOS (an `IOSurface`-backed Metal texture) andLinux (a dmabuf exported from Vulkan) still read back: `sharedTexture` imports both, thenative halves are not written.