Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
perf(preview): hand composed frames to the canvas as shared GPU textures
EtienneLescot58f7fdf5daa0
diff --git a/crates/Cargo.toml b/crates/Cargo.tomlindex 3caa7edfd..a65f9aaef 100644--- a/crates/Cargo.toml+++ b/crates/Cargo.toml@@ -68,6 +68,7 @@ features = ["Win32_Graphics_Direct2D_Common", # D2D1_COLOR_F / PIXEL_FORMAT"Win32_Graphics_DirectWrite", # mise en page + rastérisation du texte"Win32_Graphics_Gdi", # brosses/police GUI de la preview (poc-d3d/src/app.rs)+ "Win32_Security", # IDXGIResource1::CreateSharedHandle (shared_frames.rs)"Win32_System_Performance", # QueryPerformanceCounter / Frequency (§10)"Win32_System_LibraryLoader", # GetModuleHandleW (hInstance)"Win32_UI_WindowsAndMessaging", # fenêtres (harnais live + GUI du POC)diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rsindex 185f48d43..338beaa80 100644--- a/crates/compositor-view-napi/src/lib.rs+++ b/crates/compositor-view-napi/src/lib.rs@@ -11,6 +11,7 @@ use napi::{Env, JsFunction, Task};use napi_derive::napi;use openscreen_compositor::compositor::{live_params_from_scene, Compositor};use openscreen_compositor::d3d::{Backend, Gpu};+use openscreen_compositor::frame_geometry::FootageQuad;use openscreen_compositor::gif_export::{GifExportParams, GifStats};use openscreen_compositor::gif_export_control::{GifExportCancelled, GifExportControl};use openscreen_compositor::live::{LiveView, PausedPreviews};@@ -249,12 +250,82 @@ pub fn read_frame(id: i32, since_gen: f64) -> Result<Option<FramePacket>> {width: w,height: h,data: Buffer::from(pixels),- footage: footage.map(|q| q.corners.iter().flatten().map(|&v| v as f64).collect()),+ footage: footage_corners(footage),footage_projective: footage.is_some_and(|q| q.projective),}}))}+/// Les coins du métrage (TL, TR, BR, BL) aplatis en huit nombres, tels que JS les lit.+fn footage_corners(footage: Option<FootageQuad>) -> Option<Vec<f64>> {+ footage.map(|q| q.corners.iter().flatten().map(|&v| v as f64).collect())+}++/// Une frame de preview posée dans une texture partagée : de quoi l'importer côté GPU+/// (`sharedTexture.importSharedTexture`) au lieu d'en recevoir les pixels.+#[napi(object)]+pub struct SharedFramePacket {+ pub gen: f64,+ /// Case de l'anneau qui porte la frame, à rendre avec `gen` par `release_shared_frame`+ /// quand Chromium l'a relâchée.+ pub slot: u32,+ /// Handle NT de la texture, 8 octets little-endian : la forme qu'attend Electron+ /// (`SharedTextureHandle.ntHandle`). Valable dans ce processus seulement.+ pub handle: Buffer,+ pub width: u32,+ pub height: u32,+ pub footage: Option<Vec<f64>>,+ pub footage_projective: bool,+}++/// Livrer les frames de la vue par textures partagées plutôt que par `read_frame`. Rend+/// `false` quand la machine ne le peut pas (hors Windows, backend logiciel) : la vue reste+/// alors au readback.+#[napi]+pub fn set_shared_frames(id: i32, enabled: bool) -> bool {+ registry()+ .lock()+ .unwrap()+ .get(&id)+ .is_some_and(|v| v.set_shared_frames(enabled))+}++/// Le pendant de `read_frame` pour une vue en textures partagées : la dernière frame posée+/// dans l'anneau, si elle est plus récente que `since_gen`. Sa case reste tenue jusqu'à+/// `release_shared_frame`. `Ok(None)` aussi quand la vue relit en RAM.+#[napi]+pub fn read_shared_frame(id: i32, since_gen: f64) -> Result<Option<SharedFramePacket>> {+ let frame = match registry().lock().unwrap().get(&id) {+ None => return Ok(None),+ Some(v) => {+ // Même relais que `read_frame` : sans lui, un thread de rendu mort ne se verrait+ // que par un canvas noir.+ if let Some(fatal) = v.fatal_error() {+ return Err(Error::from_reason(fatal));+ }+ v.take_shared_frame(since_gen.max(0.0) as u64)+ }+ };+ Ok(frame.map(|f| SharedFramePacket {+ gen: f.gen as f64,+ slot: f.slot,+ handle: f.handle.to_le_bytes().to_vec().into(),+ width: f.width,+ height: f.height,+ footage: footage_corners(f.footage),+ footage_projective: f.footage.is_some_and(|q| q.projective),+ }))+}++/// Chromium a relâché la frame `gen` de la case `slot` (`allReferencesReleased` côté+/// Electron) : le thread de rendu peut y réécrire. Sans effet sur une vue détruite.+#[napi]+pub fn release_shared_frame(id: i32, slot: u32, gen: f64) {+ if let Some(v) = registry().lock().unwrap().get(&id) {+ v.release_shared_frame(slot, gen.max(0.0) as u64);+ }+}+/// Param live (inspector). Le type de valeur route vers le bon setter :/// bool = switch (webcamMirror…), number = slider (shadow/roundness/motionBlur/backgroundBlur),/// string = sélection (backgroundColor "#rrggbb").diff --git a/crates/compositor/src/compositor_windows.rs b/crates/compositor/src/compositor_windows.rsindex cfe335143..1875d4a34 100644--- a/crates/compositor/src/compositor_windows.rs+++ b/crates/compositor/src/compositor_windows.rs@@ -3008,6 +3008,12 @@ impl Compositor {Ok((rw, rh, out))}+ /// Le RT de la dernière composition, à `render_size()`, pour qui le copie ailleurs sans+ /// passer par la RAM : l'anneau de textures partagées de la preview (`shared_frames`).+ pub fn render_target(&self) -> &ID3D11Texture2D {+ &self.rt+ }+/// Comme `rgb_to_nv12`, mais vers `target_w`×`target_h` : si la cible diffère de la taille/// de rendu, le RT composé est d'abord redimensionné (bilinéaire, `ps_tex`/`sampler` déjà/// utilisés partout ailleurs dans le fichier, cf. `blit_resized`). Le RT suivant la sortie,diff --git a/crates/compositor/src/lib.rs b/crates/compositor/src/lib.rsindex 8e5f63e17..767abac8a 100644--- a/crates/compositor/src/lib.rs+++ b/crates/compositor/src/lib.rs@@ -51,6 +51,8 @@ pub mod sculpt;// `segmentation` ses deux entrées échouent proprement, ce qui garde le reste du crate// indépendant du choix de packaging d'ONNX Runtime.pub mod segmentation;+// Livraison de la preview par textures partagées (Windows) ; la tenue des cases est portable.+pub mod shared_frames;pub mod text_anim;pub mod text_fonts;pub mod text_plate;diff --git a/crates/compositor/src/live.rs b/crates/compositor/src/live.rsindex af586f262..cbed49464 100644--- a/crates/compositor/src/live.rs+++ b/crates/compositor/src/live.rs@@ -27,9 +27,12 @@ use crate::regions::{speed_at, ProgrammeClock};use crate::scene::Scene;use crate::config::{self, Cfg};use crate::cursor::CursorTrack;-use crate::d3d::Gpu;+use crate::d3d::{Backend, Gpu};use crate::frame_geometry::webcam_is_real;use crate::pipeline::Decoder;+#[cfg(windows)]+use crate::shared_frames::SharedRing;+use crate::shared_frames::{SharedFrame, SlotBook};use crate::timeline_walk::{frame_step, FrameStep, NextFrameTime};use anyhow::Result;use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};@@ -800,6 +803,15 @@ struct Shared {/// vide la ferait repartir à 1 — donc rejouer des générations déjà peintes. Monotone,/// jamais remise à zéro.frame_gen: AtomicU64,+ /// La vue livre ses frames dans l'anneau de textures partagées (`take_shared_frame`) au+ /// lieu de les relire en RAM (`latest_frame_since`). Posé par JS ; le thread de rendu le+ /// rabat à `false` si l'anneau ne peut pas servir, et les frames repassent par la RAM.+ shared_frames: AtomicBool,+ /// Tenue des cases de l'anneau, entre le thread de rendu et le thread Node.+ slot_book: Mutex<SlotBook>,+ /// Republier la frame courante au prochain tour, même en pause : posé quand le transport+ /// change, pour que le consommateur ait une image sans attendre que quelque chose bouge.+ republish: AtomicBool,/// Erreur fatale du thread de rendu (device D3D11 introuvable, décodeur qui refuse/// le fichier…). Le thread meurt sur la première erreur ; sans ce champ, elle/// finissait dans un `eprintln!` que personne ne lit et l'utilisateur n'avait@@ -853,6 +865,9 @@ impl LiveView {stop: AtomicBool::new(false),latest_frame: Mutex::new(None),frame_gen: AtomicU64::new(0),+ shared_frames: AtomicBool::new(false),+ slot_book: Mutex::new(SlotBook::default()),+ republish: AtomicBool::new(false),fatal: Mutex::new(None),});let sh = shared.clone();@@ -929,6 +944,35 @@ impl LiveView {}}+ /// Livrer les frames par textures partagées plutôt que par readback. Seul le backend+ /// matériel de Windows sait le faire : ailleurs la demande est refusée et la vue continue+ /// de relire en RAM. Rend l'état obtenu.+ pub fn set_shared_frames(&self, enabled: bool) -> bool {+ let on = enabled && cfg!(windows) && Gpu::probe() == Some(Backend::Hardware);+ if self.shared.shared_frames.swap(on, Ordering::Relaxed) != on {+ if !on {+ if let Ok(mut book) = self.shared.slot_book.lock() {+ book.clear_ready();+ }+ }+ self.shared.republish.store(true, Ordering::Relaxed);+ }+ on+ }++ /// La dernière frame posée dans l'anneau, si elle est plus récente que `since_gen`. Sa+ /// case reste tenue jusqu'à `release_shared_frame`. `None` aussi quand la vue relit en RAM.+ pub fn take_shared_frame(&self, since_gen: u64) -> Option<SharedFrame> {+ self.shared.slot_book.lock().ok()?.take(since_gen, Instant::now())+ }++ /// Chromium a relâché la frame `gen` de la case `slot` : le thread de rendu peut y réécrire.+ pub fn release_shared_frame(&self, slot: u32, gen: u64) {+ if let Ok(mut book) = self.shared.slot_book.lock() {+ book.release(slot, gen);+ }+ }+/// Switch inspector (booléen).pub fn set_param_bool(&self, key: &str, value: bool) {if let Ok(mut p) = self.shared.inspector.lock() {@@ -1426,8 +1470,15 @@ unsafe fn render_thread(// appliquée, on refuse de jouer le layout fixture (POC) : un fallback fixture ne ferait que// MASQUER un scene-push cassé. On attend la scène avant de produire le 1er frame.let mut scene_applied = false;+ // Textures partagées de la preview, créées au premier besoin (voir `publish_shared`).+ let mut ring = Ring::default();while !shared.stop.load(Ordering::SeqCst) {+ // Le transport vient de changer : le consommateur doit recevoir la frame courante+ // sans attendre qu'une autre soit composée (en pause, il n'y en aurait pas).+ if shared.republish.swap(false, Ordering::Relaxed) {+ first = true;+ }// params inspector : booléens/taps → cfg ; valeurs continues → live_paramslet ip = *shared.inspector.lock().unwrap();let mut clip_changed = false;@@ -1785,37 +1836,43 @@ unsafe fn render_thread(if stepped || first {if pw > 0 && ph > 0 {- // Step complet : `compose_frame` (déjà appelé par `step`/`present_frame`/- // `recompose`) a rastérisé le RT à la géométrie de sortie ramenée au panneau.- // On lit ce RT DIRECTEMENT à sa résolution de rendu (`readback_direct` : copy- // rt → staging → Map/Unmap), sans le resize `blit_resized` qui, depuis la- // refonte ratio, n'était plus qu'une copie identité + une alloc NV12 inutile.- match comp.readback_direct() {- Ok((rw, rh, rgba)) => {- // Publie dans `latest_frame` : on remplace le buffer précédent- // (le canvas ne montre que la dernière frame, peu importe combien- // le renderer en a raté entre deux lectures napi). On incrémente- // la génération sous le MÊME lock que l'écriture du buffer, pour- // qu'un lecteur ne puisse jamais voir un `gen` neuf appairé à un- // buffer périmé (ou l'inverse). `+ 1` depuis la précédente, `1` au- // premier publish. Les dims publiées sont celles du RENDU (`rw`×`rh`) :- // le canvas JS s'y dimensionne (packet auto-descriptif) puis CSS met à- // l'échelle vers la boîte du panneau — plus de resize GPU intermédiaire.- // La génération vient d'un compteur atomique et non du slot : la- // livraison sans copie VIDE le slot en le lisant, et un- // `unwrap_or(1)` repartirait alors de 1 — le consommateur recevrait- // des générations déjà peintes et boucherait. Séquence identique à- // l'ancienne dérivation tant que le slot n'est pas vidé.- let next_gen = shared.frame_gen.fetch_add(1, Ordering::Relaxed) + 1;- if let Ok(mut slot) = shared.latest_frame.lock() {- *slot = Some((next_gen, rw, rh, rgba, comp.footage_quad()));+ match publish_shared(&shared, &gpu, &comp, &mut ring) {+ SharedPublish::Published => first = false,+ // Chromium tient toutes les cases : la frame est sautée, et la boucle ne+ // tourne pas à vide le temps qu'il en relâche une.+ SharedPublish::NoFreeSlot => std::thread::sleep(Duration::from_millis(2)),+ // Step complet : `compose_frame` (déjà appelé par `step`/`present_frame`/+ // `recompose`) a rastérisé le RT à la géométrie de sortie ramenée au panneau.+ // On lit ce RT DIRECTEMENT à sa résolution de rendu (`readback_direct` : copy+ // rt → staging → Map/Unmap), sans le resize `blit_resized` qui, depuis la+ // refonte ratio, n'était plus qu'une copie identité + une alloc NV12 inutile.+ SharedPublish::Off => match comp.readback_direct() {+ Ok((rw, rh, rgba)) => {+ // Publie dans `latest_frame` : on remplace le buffer précédent+ // (le canvas ne montre que la dernière frame, peu importe combien+ // le renderer en a raté entre deux lectures napi). On incrémente+ // la génération sous le MÊME lock que l'écriture du buffer, pour+ // qu'un lecteur ne puisse jamais voir un `gen` neuf appairé à un+ // buffer périmé (ou l'inverse). `+ 1` depuis la précédente, `1` au+ // premier publish. Les dims publiées sont celles du RENDU (`rw`×`rh`) :+ // le canvas JS s'y dimensionne (packet auto-descriptif) puis CSS met à+ // l'échelle vers la boîte du panneau — plus de resize GPU intermédiaire.+ // La génération vient d'un compteur atomique et non du slot : la+ // livraison sans copie VIDE le slot en le lisant, et un+ // `unwrap_or(1)` repartirait alors de 1 — le consommateur recevrait+ // des générations déjà peintes et boucherait. Séquence identique à+ // l'ancienne dérivation tant que le slot n'est pas vidé.+ let next_gen = shared.frame_gen.fetch_add(1, Ordering::Relaxed) + 1;+ if let Ok(mut slot) = shared.latest_frame.lock() {+ *slot = Some((next_gen, rw, rh, rgba, comp.footage_quad()));+ }+ first = false;}- first = false;- }- Err(e) => {- eprintln!("[live] readback_direct: {e:#}");- std::thread::sleep(Duration::from_millis(8));- }+ Err(e) => {+ eprintln!("[live] readback_direct: {e:#}");+ std::thread::sleep(Duration::from_millis(8));+ }+ },}}} else {@@ -1825,6 +1882,88 @@ unsafe fn render_thread(Ok(())}+/// Ce que la publication dans l'anneau partagé a fait de la frame composée.+enum SharedPublish {+ /// Posée dans une case de l'anneau.+ Published,+ /// Chromium tient toutes les cases : frame sautée.+ NoFreeSlot,+ /// Transport partagé coupé ou indisponible : la frame repasse par le readback.+ Off,+}++#[cfg(windows)]+type Ring = Option<SharedRing>;+#[cfg(not(windows))]+type Ring = ();++/// Pose la frame composée dans l'anneau de textures partagées, si la vue livre ainsi.+///+/// Toute panne coupe le transport (`shared_frames` à `false`) et rend `Off` : la frame part+/// par le readback, et le service, qui lit les deux, n'a rien à décider.+#[cfg(windows)]+unsafe fn publish_shared(+ shared: &Shared,+ gpu: &Gpu,+ comp: &Compositor,+ ring: &mut Ring,+) -> SharedPublish {+ if !shared.shared_frames.load(Ordering::Relaxed) {+ return SharedPublish::Off;+ }+ let turn_off = |why: String| {+ eprintln!("[live] textures partagées coupées ({why}) — retour au readback");+ shared.shared_frames.store(false, Ordering::Relaxed);+ SharedPublish::Off+ };+ // WARP ne partage rien avec le device de Chromium.+ if gpu.backend != Backend::Hardware {+ return turn_off("backend logiciel".into());+ }+ if ring.is_none() {+ match SharedRing::new(gpu) {+ Ok(created) => *ring = Some(created),+ Err(e) => return turn_off(format!("{e:#}")),+ }+ }+ let Some(ring) = ring.as_mut() else {+ return SharedPublish::Off;+ };+ let claimed = shared+ .slot_book+ .lock()+ .ok()+ .and_then(|mut book| book.claim(crate::shared_frames::RING_SLOTS, Instant::now()));+ let Some(slot) = claimed else {+ return SharedPublish::NoFreeSlot;+ };+ let (width, height) = comp.render_size();+ match ring.write(slot, comp.render_target(), width, height) {+ Ok(handle) => {+ if let Ok(mut book) = shared.slot_book.lock() {+ // Le compteur du readback : une seule suite de générations quel que soit le+ // transport, incrémentée sous le lock de la publication comme là-bas.+ let gen = shared.frame_gen.fetch_add(1, Ordering::Relaxed) + 1;+ book.publish(SharedFrame {+ gen,+ slot,+ handle,+ width,+ height,+ footage: comp.footage_quad(),+ });+ }+ SharedPublish::Published+ }+ Err(e) => turn_off(format!("{e:#}")),+ }+}++#[cfg(not(windows))]+unsafe fn publish_shared(_: &Shared, _: &Gpu, _: &Compositor, _: &mut Ring) -> SharedPublish {+ SharedPublish::Off+}+// ---------- harnais standalone (poc-d3d.exe --live) ----------//// Le harnais crée une fenêtre Win32 simple qui héberge la preview live ; c'est undiff --git a/crates/compositor/src/shared_frames.rs b/crates/compositor/src/shared_frames.rsnew file mode 100644index 000000000..bf0e1763a--- /dev/null+++ b/crates/compositor/src/shared_frames.rs@@ -0,0 +1,329 @@+//! Livraison de la preview par textures partagées : le thread de rendu copie chaque frame+//! composée dans l'une des textures d'un petit anneau, et Electron l'importe côté GPU+//! (`sharedTexture.importSharedTexture`) au lieu d'en recevoir les pixels. Plus de `Map`+//! qui attend le GPU puis recopie l'image en RAM, plus de `Vec<u8>` de plusieurs Mo par frame+//! à travers l'IPC : mesuré, ce transport coûtait à lui seul 37 à 55 % du thread principal+//! du renderer à 30 images/s.+//!+//! Deux moitiés :+//! - `SlotBook`, portable et sans GPU : quelle case porte la frame prête, lesquelles+//! Chromium tient encore. Partagé entre le thread de rendu (`claim`, `publish`) et le+//! thread Node (`take`, `release`).+//! - `SharedRing` (Windows, backend matériel) : les textures D3D11 et leurs handles NT.+//! macOS (IOSurface) et Linux (dmabuf) restent sur le readback.++use crate::frame_geometry::FootageQuad;+use std::time::{Duration, Instant};++/// Cases de l'anneau : une prête, une en transit vers le renderer (tenue jusqu'à ce que+/// Chromium la relâche), une en écriture, et une de marge pour la latence de la libération.+pub const RING_SLOTS: u32 = 4;++/// Une case tenue plus longtemps que ça a perdu sa libération (renderer rechargé, import+/// échoué en route) : elle est reprise plutôt que de laisser l'anneau se vider et la preview+/// se figer. Même ordre que le délai d'Electron sur `sendSharedTexture` (1 s).+pub const LOST_SLOT_AFTER: Duration = Duration::from_secs(1);++/// Une frame composée posée dans une case de l'anneau, telle que JS la reçoit.+#[derive(Debug, Clone, Copy, PartialEq)]+pub struct SharedFrame {+ /// Même compteur que les frames lues en RAM : le consommateur ne voit qu'une suite de+ /// générations, quel que soit le transport.+ pub gen: u64,+ pub slot: u32,+ /// Valeur du handle NT de la texture de la case, valable dans CE processus. Electron le+ /// duplique à l'import : la case garde le sien.+ pub handle: u64,+ pub width: u32,+ pub height: u32,+ pub footage: Option<FootageQuad>,+}++/// Une case prise par JS : la génération qu'elle porte et quand elle est partie.+#[derive(Debug, Clone, Copy)]+struct Held {+ slot: u32,+ gen: u64,+ since: Instant,+}++/// Qui tient quelle case de l'anneau.+#[derive(Debug, Default)]+pub struct SlotBook {+ /// La dernière frame publiée, que JS n'a pas encore prise.+ ready: Option<SharedFrame>,+ /// Les cases prises par JS, que Chromium n'a pas encore relâchées, plus anciennes d'abord.+ held: Vec<Held>,+}++impl SlotBook {+ /// La case où écrire la prochaine frame. Une case libre d'abord ; à défaut, celle de la+ /// frame prête que personne n'a prise, retirée puisqu'elle est périmée ; à défaut, une+ /// case tenue depuis `LOST_SLOT_AFTER`. `None` quand Chromium tient légitimement toutes+ /// les cases : la frame est sautée, la suivante passera.+ pub fn claim(&mut self, slots: u32, now: Instant) -> Option<u32> {+ let ready = self.ready.map(|frame| frame.slot);+ let is_held = |slot: u32| self.held.iter().any(|held| held.slot == slot);+ if let Some(free) = (0..slots).find(|slot| !is_held(*slot) && Some(*slot) != ready) {+ return Some(free);+ }+ if let Some(stale) = ready {+ self.ready = None;+ return Some(stale);+ }+ let lost = self.held.first().filter(|held| now.duration_since(held.since) >= LOST_SLOT_AFTER)?;+ let slot = lost.slot;+ self.held.remove(0);+ Some(slot)+ }++ pub fn publish(&mut self, frame: SharedFrame) {+ self.ready = Some(frame);+ }++ /// La frame prête, si elle est plus récente que `since_gen`. Sa case reste tenue jusqu'à+ /// `release` : le thread de rendu n'y écrira plus d'ici là.+ pub fn take(&mut self, since_gen: u64, now: Instant) -> Option<SharedFrame> {+ let frame = self.ready.filter(|frame| frame.gen > since_gen)?;+ self.ready = None;+ self.held.push(Held { slot: frame.slot, gen: frame.gen, since: now });+ Some(frame)+ }++ /// Chromium a relâché la frame `gen` de la case `slot`. La génération évite qu'une+ /// libération tardive, arrivée après que la case a été reprise, ne libère la frame+ /// suivante.+ pub fn release(&mut self, slot: u32, gen: u64) {+ self.held.retain(|held| held.slot != slot || held.gen != gen);+ }++ /// Oublie la frame prête : la vue repasse au readback, personne ne viendra la prendre.+ pub fn clear_ready(&mut self) {+ self.ready = None;+ }+}++#[cfg(windows)]+pub use ring::SharedRing;++#[cfg(windows)]+mod ring {+ use super::RING_SLOTS;+ use crate::d3d::Gpu;+ use anyhow::{bail, Result};+ use std::time::{Duration, Instant};+ use windows::core::{Interface, PCWSTR};+ use windows::Win32::Foundation::{CloseHandle, BOOL, HANDLE};+ use windows::Win32::Graphics::Direct3D11::{+ ID3D11Device, ID3D11DeviceContext, ID3D11Query, ID3D11Texture2D,+ D3D11_ASYNC_GETDATA_DONOTFLUSH, D3D11_BIND_RENDER_TARGET, D3D11_BIND_SHADER_RESOURCE,+ D3D11_QUERY_DESC, D3D11_QUERY_EVENT, D3D11_RESOURCE_MISC_SHARED,+ D3D11_RESOURCE_MISC_SHARED_NTHANDLE, D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT,+ };+ use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_R8G8B8A8_UNORM, DXGI_SAMPLE_DESC};+ use windows::Win32::Graphics::Dxgi::{+ IDXGIResource1, DXGI_SHARED_RESOURCE_READ, DXGI_SHARED_RESOURCE_WRITE,+ };++ /// Au-delà, le GPU est bloqué ou perdu : on rend une erreur plutôt que de geler la vue.+ const COPY_TIMEOUT: Duration = Duration::from_millis(500);++ struct SlotTexture {+ tex: ID3D11Texture2D,+ handle: HANDLE,+ width: u32,+ height: u32,+ }++ impl Drop for SlotTexture {+ fn drop(&mut self) {+ // Electron importe un duplicata du handle : fermer le nôtre ne retire rien à une+ // frame que Chromium affiche encore.+ unsafe {+ let _ = CloseHandle(self.handle);+ }+ }+ }++ /// Les textures partagées d'une vue live, créées à la demande à la taille du rendu.+ pub struct SharedRing {+ device: ID3D11Device,+ context: ID3D11DeviceContext,+ copied: ID3D11Query,+ slots: Vec<Option<SlotTexture>>,+ }++ impl SharedRing {+ pub fn new(gpu: &Gpu) -> Result<SharedRing> {+ let desc = D3D11_QUERY_DESC { Query: D3D11_QUERY_EVENT, MiscFlags: 0 };+ let mut copied = None;+ unsafe { gpu.device.CreateQuery(&desc, Some(&mut copied))? };+ Ok(SharedRing {+ device: gpu.device.clone(),+ context: gpu.context.clone(),+ copied: copied.expect("CreateQuery a réussi sans requête"),+ slots: (0..RING_SLOTS).map(|_| None).collect(),+ })+ }++ /// Copie `src` (`width`×`height`, RGBA8) dans la case `slot` et rend le handle de sa+ /// texture une fois le GPU arrivé au bout de la copie. Chromium la lit depuis son propre+ /// device, sans keyed mutex : quand le handle part, plus rien ne doit y écrire. L'attente+ /// est celle que faisait déjà le `Map` du readback, la recopie en RAM en moins.+ ///+ /// La case ne doit être ni prête ni tenue (`SlotBook::claim`) : sa texture peut être+ /// recréée ici quand la taille du rendu a changé.+ pub unsafe fn write(+ &mut self,+ slot: u32,+ src: &ID3D11Texture2D,+ width: u32,+ height: u32,+ ) -> Result<u64> {+ let entry = &mut self.slots[slot as usize];+ if !matches!(entry, Some(texture) if texture.width == width && texture.height == height) {+ *entry = Some(create_slot(&self.device, width, height)?);+ }+ let target = entry.as_ref().expect("case créée juste au-dessus");+ self.context.CopyResource(&target.tex, src);+ self.context.End(&self.copied);+ self.context.Flush();+ let deadline = Instant::now() + COPY_TIMEOUT;+ loop {+ let mut done = BOOL(0);+ self.context.GetData(+ &self.copied,+ Some(&mut done as *mut BOOL as *mut core::ffi::c_void),+ std::mem::size_of::<BOOL>() as u32,+ D3D11_ASYNC_GETDATA_DONOTFLUSH.0 as u32,+ )?;+ if done.as_bool() {+ break;+ }+ if Instant::now() > deadline {+ bail!("le GPU n'a pas fini de copier la frame en {COPY_TIMEOUT:?}");+ }+ std::thread::yield_now();+ }+ Ok(target.handle.0 as u64)+ }+ }++ unsafe fn create_slot(device: &ID3D11Device, width: u32, height: u32) -> Result<SlotTexture> {+ let desc = D3D11_TEXTURE2D_DESC {+ Width: width,+ Height: height,+ MipLevels: 1,+ ArraySize: 1,+ // Le format du RT composé : la copie est un `CopyResource`, sans conversion.+ Format: DXGI_FORMAT_R8G8B8A8_UNORM,+ SampleDesc: DXGI_SAMPLE_DESC { Count: 1, Quality: 0 },+ Usage: D3D11_USAGE_DEFAULT,+ BindFlags: (D3D11_BIND_RENDER_TARGET.0 | D3D11_BIND_SHADER_RESOURCE.0) as u32,+ CPUAccessFlags: 0,+ // Handle NT, le seul qu'Electron importe, sans keyed mutex : `write` attend la fin+ // de la copie avant de le livrer.+ MiscFlags: (D3D11_RESOURCE_MISC_SHARED.0 | D3D11_RESOURCE_MISC_SHARED_NTHANDLE.0) as u32,+ };+ let mut tex = None;+ device.CreateTexture2D(&desc, None, Some(&mut tex))?;+ let tex = tex.expect("CreateTexture2D a réussi sans texture");+ let resource: IDXGIResource1 = tex.cast()?;+ let handle = resource.CreateSharedHandle(+ None,+ DXGI_SHARED_RESOURCE_READ.0 | DXGI_SHARED_RESOURCE_WRITE.0,+ PCWSTR::null(),+ )?;+ Ok(SlotTexture { tex, handle, width, height })+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn frame(gen: u64, slot: u32) -> SharedFrame {+ SharedFrame { gen, slot, handle: 0, width: 2, height: 2, footage: None }+ }++ /// Toutes les cases prises par JS, la case `n` portant la génération `n + 1`.+ fn all_held(now: Instant) -> SlotBook {+ let mut book = SlotBook::default();+ for slot in 0..RING_SLOTS {+ book.publish(frame(u64::from(slot) + 1, slot));+ assert!(book.take(u64::from(slot), now).is_some());+ }+ book+ }++ #[test]+ fn claims_a_free_slot_before_a_ready_one() {+ let now = Instant::now();+ let mut book = SlotBook::default();+ book.publish(frame(1, 0));+ assert_eq!(book.claim(RING_SLOTS, now), Some(1));+ // La frame prête n'a pas bougé : JS peut toujours la prendre.+ assert_eq!(book.take(0, now), Some(frame(1, 0)));+ }++ #[test]+ fn a_taken_slot_is_never_written_until_released() {+ let now = Instant::now();+ let mut book = all_held(now);+ assert_eq!(book.claim(RING_SLOTS, now), None, "Chromium tient toutes les cases");+ book.release(2, 3);+ assert_eq!(book.claim(RING_SLOTS, now), Some(2));+ }++ #[test]+ fn a_frame_nobody_took_gives_its_slot_back_when_nothing_else_is_free() {+ let now = Instant::now();+ let mut book = SlotBook::default();+ for slot in 0..RING_SLOTS - 1 {+ book.publish(frame(u64::from(slot) + 1, slot));+ book.take(u64::from(slot), now);+ }+ book.publish(frame(9, 3));+ assert_eq!(book.claim(RING_SLOTS, now), Some(3));+ // Retirée : la livrer maintenant, pendant qu'on la réécrit, donnerait une image déchirée.+ assert_eq!(book.take(0, now), None);+ }++ #[test]+ fn take_hands_out_only_newer_generations() {+ let now = Instant::now();+ let mut book = SlotBook::default();+ book.publish(frame(5, 1));+ assert_eq!(book.take(5, now), None);+ assert_eq!(book.take(4, now), Some(frame(5, 1)));+ assert_eq!(book.take(0, now), None, "déjà prise");+ }++ #[test]+ fn a_release_only_frees_the_generation_it_names() {+ let now = Instant::now();+ let mut book = SlotBook::default();+ book.publish(frame(1, 0));+ book.take(0, now);+ book.release(3, 1);+ book.release(0, 7);+ assert_eq!(book.claim(1, now), None, "ni la case 3 ni la génération 7 ne sont tenues");+ book.release(0, 1);+ assert_eq!(book.claim(1, now), Some(0));+ }++ #[test]+ fn a_slot_whose_release_never_came_is_taken_back_after_a_second() {+ let start = Instant::now();+ let mut book = all_held(start);+ assert_eq!(book.claim(RING_SLOTS, start + LOST_SLOT_AFTER / 2), None);+ // La plus ancienne d'abord.+ assert_eq!(book.claim(RING_SLOTS, start + LOST_SLOT_AFTER), Some(0));+ // Sa libération tardive ne doit plus rien libérer : la case est déjà reprise.+ book.release(0, 1);+ book.publish(frame(10, 0));+ assert!(book.take(9, start + LOST_SLOT_AFTER).is_some());+ book.release(0, 1);+ assert_eq!(book.claim(RING_SLOTS, start + LOST_SLOT_AFTER), Some(1));+ }+}diff --git a/crates/compositor/tests/shared_frame_handoff.rs b/crates/compositor/tests/shared_frame_handoff.rsnew file mode 100644index 000000000..2465f2b98--- /dev/null+++ b/crates/compositor/tests/shared_frame_handoff.rs@@ -0,0 +1,135 @@+//! La texture qu'une vue live livre par handle NT, rouverte par un AUTRE device D3D11 — la+//! place qu'occupe le processus GPU de Chromium quand Electron l'importe. Ce que le+//! consommateur lit doit être exactement ce que le compositeur a composé : pas une image+//! à moitié copiée, pas une case périmée, et une case redimensionnée doit livrer un handle+//! neuf à la nouvelle taille.+//!+//! Demande un device matériel : saute sans lui (runner sans adaptateur).++#![cfg(windows)]++use openscreen_compositor::d3d::Gpu;+use openscreen_compositor::shared_frames::SharedRing;+use windows::core::Interface;+use windows::Win32::Foundation::{HANDLE, HMODULE};+use windows::Win32::Graphics::Direct3D::{D3D_DRIVER_TYPE_HARDWARE, D3D_FEATURE_LEVEL_11_1};+use windows::Win32::Graphics::Direct3D11::{+ D3D11CreateDevice, ID3D11Device, ID3D11Device1, ID3D11DeviceContext, ID3D11Texture2D,+ D3D11_BIND_SHADER_RESOURCE, D3D11_CPU_ACCESS_READ, D3D11_CREATE_DEVICE_FLAG,+ D3D11_MAPPED_SUBRESOURCE, D3D11_MAP_READ, D3D11_SDK_VERSION, D3D11_SUBRESOURCE_DATA,+ D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT, D3D11_USAGE_STAGING,+};+use windows::Win32::Graphics::Dxgi::Common::{DXGI_FORMAT_R8G8B8A8_UNORM, DXGI_SAMPLE_DESC};++fn pattern(width: u32, height: u32, seed: u8) -> Vec<u8> {+ (0..width * height)+ .flat_map(|i| {+ let (x, y) = (i % width, i / width);+ [x as u8 ^ seed, y as u8, seed, 255]+ })+ .collect()+}++fn texture_with(gpu: &Gpu, width: u32, height: u32, pixels: &[u8]) -> ID3D11Texture2D {+ let desc = D3D11_TEXTURE2D_DESC {+ Width: width,+ Height: height,+ MipLevels: 1,+ ArraySize: 1,+ Format: DXGI_FORMAT_R8G8B8A8_UNORM,+ SampleDesc: DXGI_SAMPLE_DESC { Count: 1, Quality: 0 },+ Usage: D3D11_USAGE_DEFAULT,+ BindFlags: D3D11_BIND_SHADER_RESOURCE.0 as u32,+ CPUAccessFlags: 0,+ MiscFlags: 0,+ };+ let init = D3D11_SUBRESOURCE_DATA {+ pSysMem: pixels.as_ptr().cast(),+ SysMemPitch: width * 4,+ SysMemSlicePitch: 0,+ };+ let mut tex = None;+ unsafe { gpu.device.CreateTexture2D(&desc, Some(&init), Some(&mut tex)) }.expect("texture");+ tex.expect("texture")+}++/// Le second device, sans rien de commun avec celui du compositeur que l'adaptateur.+fn consumer() -> (ID3D11Device, ID3D11DeviceContext) {+ let mut device = None;+ let mut context = None;+ unsafe {+ D3D11CreateDevice(+ None,+ D3D_DRIVER_TYPE_HARDWARE,+ HMODULE::default(),+ D3D11_CREATE_DEVICE_FLAG(0),+ Some(&[D3D_FEATURE_LEVEL_11_1]),+ D3D11_SDK_VERSION,+ Some(&mut device),+ None,+ Some(&mut context),+ )+ }+ .expect("second device");+ (device.unwrap(), context.unwrap())+}++/// Ce que le consommateur lit à travers le handle, en RGBA serré.+fn read_through(handle: u64, width: u32, height: u32) -> Vec<u8> {+ let (device, context) = consumer();+ let device1: ID3D11Device1 = device.cast().expect("ID3D11Device1");+ let shared: ID3D11Texture2D =+ unsafe { device1.OpenSharedResource1(HANDLE(handle as *mut core::ffi::c_void)) }+ .expect("le handle doit s'ouvrir depuis un autre device");+ let mut desc = D3D11_TEXTURE2D_DESC::default();+ unsafe { shared.GetDesc(&mut desc) };+ assert_eq!((desc.Width, desc.Height), (width, height));+ desc.Usage = D3D11_USAGE_STAGING;+ desc.BindFlags = 0;+ desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ.0 as u32;+ desc.MiscFlags = 0;+ let mut staging = None;+ unsafe { device.CreateTexture2D(&desc, None, Some(&mut staging)) }.expect("staging");+ let staging = staging.unwrap();+ let mut mapped = D3D11_MAPPED_SUBRESOURCE::default();+ let mut out = vec![0u8; (width * height * 4) as usize];+ unsafe {+ context.CopyResource(&staging, &shared);+ context.Map(&staging, 0, D3D11_MAP_READ, 0, Some(&mut mapped)).expect("map");+ for y in 0..height as usize {+ std::ptr::copy_nonoverlapping(+ (mapped.pData as *const u8).add(y * mapped.RowPitch as usize),+ out.as_mut_ptr().add(y * width as usize * 4),+ width as usize * 4,+ );+ }+ context.Unmap(&staging, 0);+ }+ out+}++#[test]+fn another_device_reads_exactly_the_composed_frame() {+ let Ok(gpu) = Gpu::create(false) else {+ eprintln!("pas de device D3D11 matériel — saute");+ return;+ };+ let mut ring = SharedRing::new(&gpu).expect("anneau");++ let (w, h) = (64, 36);+ let first = pattern(w, h, 0x11);+ let handle = unsafe { ring.write(0, &texture_with(&gpu, w, h, &first), w, h) }.expect("écriture");+ assert_eq!(read_through(handle, w, h), first);++ // La même case réécrite : même texture, même handle, nouveau contenu.+ let second = pattern(w, h, 0x22);+ let again = unsafe { ring.write(0, &texture_with(&gpu, w, h, &second), w, h) }.expect("réécriture");+ assert_eq!(again, handle);+ assert_eq!(read_through(again, w, h), second);++ // Le rendu change de taille : la case est recréée, avec un handle à la nouvelle taille.+ let (w2, h2) = (32, 18);+ let resized = pattern(w2, h2, 0x33);+ let fresh = unsafe { ring.write(0, &texture_with(&gpu, w2, h2, &resized), w2, h2) }.expect("taille");+ assert_eq!(read_through(fresh, w2, h2), resized);+}diff --git a/electron/electron-env.d.ts b/electron/electron-env.d.tsindex e0fddd35c..26f2f5308 100644--- a/electron/electron-env.d.ts+++ b/electron/electron-env.d.ts@@ -37,6 +37,15 @@ interface Window {* `compositor.export`/`compositor.exportMulti` runs. Distinct from `exportOnFrameAck`,* the OLD web/CPU pipeline's per-frame ack, not a progress signal. */onNativeExportProgress?: (callback: (frames: number, exportId?: string) => void) => () => void;+ /** Preview frames the main process hands over as shared GPU textures (Windows). The+ * listener draws `frame` and closes it; one listener at a time. Returns the+ * unsubscribe. Optional: shim/web contexts have no bridge. */+ onCompositorFrame?: (+ listener: (+ frame: VideoFrame,+ meta: import("../src/native/contracts").CompositorSharedFrameMeta,+ ) => void,+ ) => () => void;getSources: (opts: Electron.SourcesOptions) => Promise<ProcessedDesktopSource[]>;switchToEditor: () => Promise<void>;switchToHud: () => Promise<void>;diff --git a/electron/ipc/nativeBridge.ts b/electron/ipc/nativeBridge.tsindex 452f42701..61ed8ef86 100644--- a/electron/ipc/nativeBridge.ts+++ b/electron/ipc/nativeBridge.ts@@ -384,18 +384,23 @@ export function registerNativeBridgeHandlers(context: NativeBridgeContext) {compositorViewService.setRect(request.payload.id, request.payload.rect);return createSuccessResponse(requestId, { ok: true });case "readFrame": {- // The renderer polls this every rAF tick (~30fps). It passes the- // generation it last painted as `sinceGen`; native returns `null` when- // nothing newer exists (idle path — no buffer copy). On a new frame it- // returns `{ gen, width, height, data }`. The response wrapper does NOT- // JSON-stringify — `ipcMain.handle` round-trips via structured clone,- // which preserves the nested `Buffer` in `.data` as binary.- const frame = compositorViewService.readFrame(+ // The renderer polls this every rAF tick. It passes the generation it+ // last painted as `sinceGen`; native returns `null` when nothing newer+ // exists (idle path — no buffer copy). On a new frame it returns+ // `{ gen, width, height, data }`, or — for a view on shared textures —+ // sends the texture to the asking frame and returns its receipt. The+ // response wrapper does NOT JSON-stringify — `ipcMain.handle` round-trips+ // via structured clone, which preserves the nested `Buffer` as binary.+ const frame = await compositorViewService.readFrame(request.payload.id,request.payload.sinceGen,+ event.senderFrame,);return createSuccessResponse(requestId, frame);}+ case "stopSharedFrames":+ compositorViewService.stopSharedFrames(request.payload.id);+ return createSuccessResponse(requestId, { ok: true });case "setParam":compositorViewService.setParam(request.payload.id,diff --git a/electron/native-bridge/services/compositorViewService.test.ts b/electron/native-bridge/services/compositorViewService.test.tsindex eee8fe305..4a099a7f9 100644--- a/electron/native-bridge/services/compositorViewService.test.ts+++ b/electron/native-bridge/services/compositorViewService.test.ts@@ -1,6 +1,7 @@import fs from "node:fs";import os from "node:os";import path from "node:path";+import type { WebFrameMain } from "electron";import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";import { CURSOR_THEMES, DEFAULT_CURSOR_SPRITES } from "../../../src/lib/cursor/cursorThemes";import type { CompositorViewAddon, GifExportStats } from "../../native/compositor-view/addon";@@ -58,6 +59,166 @@ describe("native GIF cancellation capability", () => {});});+describe("CompositorViewService frames handed over as shared GPU textures", () => {+ const target = { frameTreeNodeId: 1 } as unknown as WebFrameMain;+ const sharedFrame = {+ gen: 3,+ slot: 1,+ handle: Buffer.from([1, 2, 3, 4, 5, 6, 7, 8]),+ width: 4,+ height: 2,+ footage: null,+ footageProjective: false,+ };++ function setup(options: { addon?: Partial<CompositorViewAddon>; gpuCompositing?: boolean } = {}) {+ const addon = {+ createView: vi.fn(() => 7),+ setSharedFrames: vi.fn(() => true),+ readSharedFrame: vi.fn(() => sharedFrame),+ readFrame: vi.fn(() => null),+ releaseSharedFrame: vi.fn(),+ destroyView: vi.fn(),+ ...options.addon,+ };+ const imported = { release: vi.fn() };+ let allReferencesReleased: (() => void) | undefined;+ const sharedTexture = {+ importSharedTexture: vi.fn((importOptions: Electron.ImportSharedTextureOptions) => {+ allReferencesReleased = importOptions.allReferencesReleased as () => void;+ return imported as unknown as Electron.SharedTextureImported;+ }),+ sendSharedTexture: vi.fn(async () => undefined),+ };+ const service = new CompositorViewService({+ addon: addon as unknown as CompositorViewAddon,+ sharedTexture,+ gpuCompositing: () => options.gpuCompositing ?? true,+ });+ const id = service.createView({ x: 0, y: 0, width: 4, height: 2 });+ return {+ addon,+ id,+ imported,+ service,+ sharedTexture,+ releaseEverywhere: () => allReferencesReleased?.(),+ };+ }++ afterEach(() => {+ vi.unstubAllEnvs();+ });++ it("sends the texture to the frame that asked, and answers with its receipt", async () => {+ const { addon, id, imported, service, sharedTexture } = setup();+ expect(addon.setSharedFrames).toHaveBeenCalledWith(id, true);++ const receipt = await service.readFrame(id, 2, target);++ expect(addon.readSharedFrame).toHaveBeenCalledWith(id, 2);+ expect(sharedTexture.importSharedTexture).toHaveBeenCalledWith(+ expect.objectContaining({+ textureInfo: {+ pixelFormat: "rgba",+ codedSize: { width: 4, height: 2 },+ handle: { ntHandle: sharedFrame.handle },+ },+ }),+ );+ const meta = {+ viewId: id,+ gen: 3,+ width: 4,+ height: 2,+ footage: null,+ footageProjective: false,+ };+ expect(sharedTexture.sendSharedTexture).toHaveBeenCalledWith(+ { frame: target, importedSharedTexture: imported },+ meta,+ );+ expect(receipt).toEqual({ ...meta, shared: true });+ // This process's reference goes at once: the renderer holds its own.+ expect(imported.release).toHaveBeenCalled();+ expect(addon.releaseSharedFrame).not.toHaveBeenCalled();+ });++ it("hands the slot back once Chromium has let go of the texture everywhere", async () => {+ const { addon, id, releaseEverywhere, service } = setup();+ await service.readFrame(id, 0, target);++ releaseEverywhere();++ expect(addon.releaseSharedFrame).toHaveBeenCalledWith(id, 1, 3);+ });++ it("goes back to read-back when a delivery fails", async () => {+ const { addon, id, service, sharedTexture } = setup();+ sharedTexture.sendSharedTexture.mockRejectedValueOnce(new Error("timed out after 1000ms"));+ const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);+ try {+ expect(await service.readFrame(id, 0, target)).toBeNull();+ } finally {+ warn.mockRestore();+ }+ expect(addon.setSharedFrames).toHaveBeenLastCalledWith(id, false);++ vi.mocked(addon.readSharedFrame).mockClear();+ await service.readFrame(id, 3, target);+ expect(addon.readSharedFrame).not.toHaveBeenCalled();+ expect(addon.readFrame).toHaveBeenCalledWith(id, 3);+ });++ it("frees the slot itself when the texture never got imported", async () => {+ const { addon, id, service, sharedTexture } = setup();+ sharedTexture.importSharedTexture.mockImplementationOnce(() => {+ throw new TypeError("Invalid ntHandle value");+ });+ const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);+ try {+ await service.readFrame(id, 0, target);+ } finally {+ warn.mockRestore();+ }+ expect(addon.releaseSharedFrame).toHaveBeenCalledWith(id, 1, 3);+ });++ it("reads pixels for a view whose render thread went back to read-back on its own", async () => {+ const packet = { gen: 4, width: 4, height: 2, data: Buffer.alloc(32) };+ const { id, service, sharedTexture } = setup({+ addon: { readSharedFrame: vi.fn(() => null), readFrame: vi.fn(() => packet) },+ });++ expect(await service.readFrame(id, 3, target)).toBe(packet);+ expect(sharedTexture.sendSharedTexture).not.toHaveBeenCalled();+ });++ it("keeps a view on read-back when Chromium does not composite on the GPU", async () => {+ const { addon, id, service } = setup({ gpuCompositing: false });+ expect(addon.setSharedFrames).not.toHaveBeenCalled();++ await service.readFrame(id, 0, target);+ expect(addon.readSharedFrame).not.toHaveBeenCalled();+ expect(addon.readFrame).toHaveBeenCalledWith(id, 0);+ });++ it("keeps a view on read-back when OPENSCREEN_PREVIEW_READBACK=1", () => {+ vi.stubEnv("OPENSCREEN_PREVIEW_READBACK", "1");+ const { addon } = setup();+ expect(addon.setSharedFrames).not.toHaveBeenCalled();+ });++ it("keeps a view on read-back when the addon cannot share (not Windows, software backend)", async () => {+ const { id, service, sharedTexture } = setup({+ addon: { setSharedFrames: vi.fn(() => false) },+ });++ await service.readFrame(id, 0, target);+ expect(sharedTexture.importSharedTexture).not.toHaveBeenCalled();+ });+});+/** A source checkout's `crates/.cargo/config.toml`, with `FFMPEG_DIR` written as `body`. */function writeCargoConfig(root: string, body: string): void {const cargoDir = path.join(root, "crates", ".cargo");diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.tsindex 4a765db2b..51e562979 100644--- a/electron/native-bridge/services/compositorViewService.ts+++ b/electron/native-bridge/services/compositorViewService.ts@@ -2,12 +2,16 @@ import fs from "node:fs";import { createRequire } from "node:module";import path from "node:path";import { fileURLToPath } from "node:url";-import { app } from "electron";+import { app, sharedTexture, type WebFrameMain } from "electron";import {type CursorKind,readCursorAsArrow,resolveCursorSprites,} from "../../../src/lib/cursor/cursorThemes";+import type {+ CompositorSharedFrameMeta,+ CompositorSharedFrameReceipt,+} from "../../../src/native/contracts";import type { GifExportJob } from "../../ipc/gifExportJobs";import type {ClipInput,@@ -20,6 +24,7 @@ import type {GifExportStats,GifParamsInput,NativeFramePacket,+ NativeSharedFramePacket,RemuxStats,SegmentationSupport,} from "../../native/compositor-view/addon";@@ -215,6 +220,26 @@ export interface CompositorViewServiceOptions {*/appRoot?: string;isPackaged?: boolean;+ /** Electron's `sharedTexture` module, injectable for tests. `null` keeps every view on+ * read-back. */+ sharedTexture?: SharedTextureApi | null;+ /** Whether Chromium composites on the GPU, where a shared texture is imported. Injectable+ * for tests; defaults to `app.getGPUFeatureStatus()`. */+ gpuCompositing?: () => boolean;+}++/** The two calls a shared preview frame needs from Electron's `sharedTexture` module. */+export type SharedTextureApi = Pick<+ Electron.SharedTexture,+ "importSharedTexture" | "sendSharedTexture"+>;++function defaultGpuCompositing(): boolean {+ try {+ return String(app.getGPUFeatureStatus().gpu_compositing).startsWith("enabled");+ } catch {+ return false;+ }}function defaultAppRoot(): string {@@ -467,6 +492,8 @@ function tryLoadAddon(candidates: string[]): CompositorViewAddon | null {export class CompositorViewService {private readonly options: CompositorViewServiceOptions;private readonly rects = new Map<number, CompositorViewRect>();+ /** Views whose frames go out as shared GPU textures rather than RAM pixels. */+ private readonly sharedViews = new Set<number>();private addon: CompositorViewAddon | null = null;private loadAttempted = false;private syntheticIdCounter = 0;@@ -593,9 +620,41 @@ export class CompositorViewService {}const id = addon.createView(rect, paths?.screenPath, paths?.webcamPath, paths?.cursorPath);this.rects.set(id, rect);+ this.shareFrames(addon, id);return id;}+ /** Hands the view's frames over as shared GPU textures when this host can, instead of+ * copying them through IPC: measured, that transport alone kept 37 to 55 % of the+ * renderer's main thread busy at 30 fps (rendering-performance.md). Anything missing —+ * the Electron API, GPU compositing, Windows' hardware backend native-side — leaves the+ * view on read-back. `OPENSCREEN_PREVIEW_READBACK=1` forces read-back, to compare. */+ private shareFrames(addon: CompositorViewAddon, id: number): void {+ if (process.env.OPENSCREEN_PREVIEW_READBACK === "1" || !this.sharedTextureApi()) {+ return;+ }+ if (!(this.options.gpuCompositing ?? defaultGpuCompositing)()) {+ return;+ }+ if (addon.setSharedFrames?.(id, true)) {+ this.sharedViews.add(id);+ }+ }++ private sharedTextureApi(): SharedTextureApi | null {+ const api =+ this.options.sharedTexture === undefined ? sharedTexture : this.options.sharedTexture;+ return typeof api?.importSharedTexture === "function" ? api : null;+ }++ /** Takes the view back to read-back frames: a delivery failed, or the renderer saw a shared+ * frame land as nothing (Chromium could not open the texture). The render thread+ * republishes the current frame, so the canvas is not left empty. */+ stopSharedFrames(id: number): void {+ this.sharedViews.delete(id);+ this.ensureAddon()?.setSharedFrames?.(id, false);+ }+setRect(id: number, rect: CompositorViewRect): void {const addon = this.ensureAddon();this.rects.set(id, rect);@@ -605,19 +664,76 @@ export class CompositorViewService {addon.setRect(id, rect);}- /** Reads the most recently rendered frame for `id` as a self-describing packet- * (`{ gen, width, height, data }`), but only if its generation is newer than- * `sinceGen`. Returns `null` when the addon is absent, no frame is ready yet,- * OR the caller already holds the current generation — the idle path, where- * `null` comes back without any buffer copy. Byte order is RGBA. */- readFrame(id: number, sinceGen: number): NativeFramePacket | null {+ /** Reads the most recently rendered frame for `id`, but only if its generation is newer+ * than `sinceGen`. Returns `null` when the addon is absent, no frame is ready yet, OR the+ * caller already holds the current generation — the idle path, where nothing is copied.+ *+ * A view on shared textures sends the frame to `target` as a GPU texture, and answers with+ * its receipt (`shared: true`, no pixels): the texture reaches the renderer before this+ * reply does. Any other view answers with the RGBA pixels themselves. */+ async readFrame(+ id: number,+ sinceGen: number,+ target?: WebFrameMain | null,+ ): Promise<NativeFramePacket | CompositorSharedFrameReceipt | null> {const addon = this.ensureAddon();if (!addon) {return null;}+ if (target && this.sharedViews.has(id)) {+ const frame = addon.readSharedFrame?.(id, sinceGen);+ if (frame) {+ return this.sendSharedFrame(addon, id, frame, target);+ }+ }+ // Also the answer for a view whose render thread went back to read-back on its own.return addon.readFrame(id, sinceGen);}+ private async sendSharedFrame(+ addon: CompositorViewAddon,+ id: number,+ frame: NativeSharedFramePacket,+ target: WebFrameMain,+ ): Promise<CompositorSharedFrameReceipt | null> {+ const meta: CompositorSharedFrameMeta = {+ viewId: id,+ gen: frame.gen,+ width: frame.width,+ height: frame.height,+ footage: frame.footage ?? null,+ footageProjective: frame.footageProjective ?? false,+ };+ const api = this.sharedTextureApi();+ let imported: Electron.SharedTextureImported | undefined;+ try {+ if (!api) {+ throw new Error("the sharedTexture API is gone");+ }+ imported = api.importSharedTexture({+ textureInfo: {+ pixelFormat: "rgba",+ codedSize: { width: frame.width, height: frame.height },+ handle: { ntHandle: frame.handle },+ },+ // Chromium is done with the texture in every process: the slot may be written again.+ allReferencesReleased: () => addon.releaseSharedFrame?.(id, frame.slot, frame.gen),+ });+ await api.sendSharedTexture({ frame: target, importedSharedTexture: imported }, meta);+ return { ...meta, shared: true };+ } catch (error) {+ console.warn("[compositor-view] shared frame not delivered; reading frames back:", error);+ this.stopSharedFrames(id);+ if (!imported) {+ addon.releaseSharedFrame?.(id, frame.slot, frame.gen);+ }+ return null;+ } finally {+ // This process's reference only: the renderer holds its own until it has drawn.+ imported?.release();+ }+ }+setParam(id: number, key: string, value: CompositorParamValue): void {const addon = this.ensureAddon();if (!addon) {@@ -668,6 +784,7 @@ export class CompositorViewService {destroyView(id: number): void {const addon = this.ensureAddon();this.rects.delete(id);+ this.sharedViews.delete(id);if (!addon) {return;}diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.tsindex ee29cd80f..631d70fdf 100644--- a/electron/native/compositor-view/addon.d.ts+++ b/electron/native/compositor-view/addon.d.ts@@ -41,6 +41,20 @@ export interface NativeFramePacket {footageProjective?: boolean;}+/** A preview frame left in a shared GPU texture instead of copied into RAM (Windows, hardware+ * backend — see `setSharedFrames`). `handle` is the texture's NT handle the way Electron's+ * `sharedTexture.importSharedTexture` takes it: 8 bytes, little-endian, valid in this process+ * only. Its `slot` stays reserved until `releaseSharedFrame(id, slot, gen)`. */+export interface NativeSharedFramePacket {+ gen: number;+ slot: number;+ handle: Buffer;+ width: number;+ height: number;+ footage?: number[] | null;+ footageProjective?: boolean;+}+export interface ExportStats {frames: number;wallS: number;@@ -158,6 +172,17 @@ export interface CompositorViewAddon {* still frame): `null` comes back WITHOUT cloning the buffer or crossing IPC.* Pass `sinceGen = 0` to force delivery of the current frame. */readFrame(id: number, sinceGen: number): NativeFramePacket | null;+ /** Deliver this view's frames as shared GPU textures (`readSharedFrame`) rather than RAM+ * pixels (`readFrame`). Returns `false` where that cannot work (not Windows, software+ * backend): the view keeps reading back. Optional: an older `.node` predates it. */+ setSharedFrames?(id: number, enabled: boolean): boolean;+ /** The latest frame left in the view's shared texture ring, if newer than `sinceGen`.+ * `null` too while the view reads back to RAM. Throws the render thread's fatal error,+ * like `readFrame`. */+ readSharedFrame?(id: number, sinceGen: number): NativeSharedFramePacket | null;+ /** Chromium let go of frame `gen` in `slot`, in every process: the render thread may write+ * that slot again. A no-op for a destroyed view. */+ releaseSharedFrame?(id: number, slot: number, gen: number): void;setParam(id: number, key: string, value: CompositorParamValue): void;setPlaying(id: number, playing: boolean): void;/** Seeks the view to source-media `seconds` for the active clip. */diff --git a/electron/preload.ts b/electron/preload.tsindex a754c99b6..feb3942df 100644--- a/electron/preload.ts+++ b/electron/preload.ts@@ -1,4 +1,4 @@-import { contextBridge, ipcRenderer, webUtils } from "electron";+import { contextBridge, ipcRenderer, sharedTexture, webUtils } from "electron";import type { NativeLinuxRecordingRequest } from "../src/lib/nativeLinuxRecording";import type { NativeMacRecordingRequest } from "../src/lib/nativeMacRecording";import type { NativeWindowsRecordingRequest } from "../src/lib/nativeWindowsRecording";@@ -8,6 +8,7 @@ import type {AiEditionChatEvent,AiEditionMcpHostRequest,AiEditionMcpHostResponse,+ CompositorSharedFrameMeta,} from "../src/native/contracts";import {AI_EDITION_MCP_HOST_CHANNEL,@@ -34,6 +35,23 @@ const assetBaseUrl = assetBaseUrlArg ? assetBaseUrlArg.slice(ASSET_BASE_URL_ARG_// so a synchronous read here saves the renderer's every-call IPC round-trip.const PLATFORM = process.platform;+// Preview frames the main process hands over as shared GPU textures (see+// `compositorViewService`). Electron gives up on a send after 1 s without a receiver, so it is+// registered here, at load, and forwards to whichever listener the page has set.+type CompositorFrameListener = (frame: VideoFrame, meta: CompositorSharedFrameMeta) => void;+let compositorFrameListener: CompositorFrameListener | null = null;+sharedTexture?.setSharedTextureReceiver(async ({ importedSharedTexture }, meta) => {+ const frame = importedSharedTexture.getVideoFrame();+ try {+ compositorFrameListener?.(frame, meta as CompositorSharedFrameMeta);+ } finally {+ // The page drew from its own copy (contextBridge clones the frame) and closed it. Closing+ // ours and releasing the import is what lets the native ring write the slot again.+ frame.close();+ importedSharedTexture.release();+ }+});+contextBridge.exposeInMainWorld("electronAPI", {assetBaseUrl,@@ -85,6 +103,14 @@ contextBridge.exposeInMainWorld("electronAPI", {ipcRenderer.on("export:native-progress", handler);return () => ipcRenderer.off("export:native-progress", handler);},+ onCompositorFrame: (listener: CompositorFrameListener) => {+ compositorFrameListener = listener;+ return () => {+ if (compositorFrameListener === listener) {+ compositorFrameListener = null;+ }+ };+ },invokeNativeBridge: <TData>(request: NativeBridgeRequest) => {return ipcRenderer.invoke(NATIVE_BRIDGE_CHANNEL, request) as Promise<TData>;},diff --git a/src/native/compositorViewClient.ts b/src/native/compositorViewClient.tsindex 793521086..66f63ce56 100644--- a/src/native/compositorViewClient.ts+++ b/src/native/compositorViewClient.ts@@ -18,6 +18,8 @@ import type {CompositorExportResult,CompositorFramePacket,CompositorParamValue,+ CompositorSharedFrameMeta,+ CompositorSharedFrameReceipt,CompositorViewRect,CompositorViewResult,SegmentationSupport,@@ -97,18 +99,39 @@ export function setCompositorRect(id: number, rect: CompositorViewRect): Promise* {@link CompositorFramePacket} on a new frame, or `null` when the addon is absent,* no frame is ready yet, OR the caller already holds the current generation — the* idle path, where `null` returns without any buffer crossing IPC. Pass `sinceGen = 0`- * to force delivery of the current frame. */+ * to force delivery of the current frame.+ *+ * A view on shared textures answers with a {@link CompositorSharedFrameReceipt} instead:+ * its frame already went to {@link subscribeCompositorSharedFrames}, ahead of this reply. */export function readCompositorFrame(id: number,sinceGen: number,-): Promise<CompositorFramePacket | null> {- return requireNativeBridgeData<CompositorFramePacket | null>({+): Promise<CompositorFramePacket | CompositorSharedFrameReceipt | null> {+ return requireNativeBridgeData<CompositorFramePacket | CompositorSharedFrameReceipt | null>({domain: "compositor",action: "readFrame",payload: { id, sinceGen },});}+/** Preview frames handed over as shared GPU textures. The listener draws `frame` and closes+ * it. Returns the unsubscribe; without the Electron bridge (pure web, jsdom) nothing ever+ * arrives. */+export function subscribeCompositorSharedFrames(+ listener: (frame: VideoFrame, meta: CompositorSharedFrameMeta) => void,+): () => void {+ return window.electronAPI?.onCompositorFrame?.(listener) ?? (() => undefined);+}++/** Back to read-back frames for `id`: a shared frame reached the canvas as nothing. */+export function stopSharedCompositorFrames(id: number): Promise<{ ok: true }> {+ return requireNativeBridgeData<{ ok: true }>({+ domain: "compositor",+ action: "stopSharedFrames",+ payload: { id },+ });+}+export function setCompositorParam(id: number,key: string,diff --git a/src/native/contracts.ts b/src/native/contracts.tsindex 432a6ca69..d13a95f5c 100644--- a/src/native/contracts.ts+++ b/src/native/contracts.ts@@ -167,6 +167,24 @@ export interface CompositorFramePacket {footageProjective?: boolean;}+/** What travels with a preview frame handed over as a shared GPU texture (Windows): all a+ * {@link CompositorFramePacket} says but the pixels, plus the view the frame belongs to. The+ * main process sends it alongside the texture, to `electronAPI.onCompositorFrame`. */+export interface CompositorSharedFrameMeta {+ viewId: number;+ gen: number;+ width: number;+ height: number;+ footage: number[] | null;+ footageProjective: boolean;+}++/** `readFrame`'s answer for a frame sent as a shared texture. The texture reached+ * `onCompositorFrame` before this reply did, so only the generation is news here. */+export interface CompositorSharedFrameReceipt extends CompositorSharedFrameMeta {+ shared: true;+}+/** Un clip de la timeline pour l'export multiclip natif (fichiers screen+webcam + trim). */export interface CompositorClipInput {screenPath: string;@@ -820,6 +838,14 @@ export type NativeBridgeRequest =payload: { id: number; sinceGen: number };requestId?: string;}+ | {+ domain: "compositor";+ /** Back to read-back frames: a shared frame reached the canvas as nothing, i.e.+ * Chromium could not open the texture. */+ action: "stopSharedFrames";+ payload: { id: number };+ requestId?: string;+ }| {domain: "compositor";action: "setParam";diff --git a/src/native/hooks/useNativeCompositorView.test.ts b/src/native/hooks/useNativeCompositorView.test.tsindex 55b89174b..ac182aceb 100644--- a/src/native/hooks/useNativeCompositorView.test.ts+++ b/src/native/hooks/useNativeCompositorView.test.ts@@ -15,12 +15,18 @@import { renderHook, waitFor } from "@testing-library/react";import type { RefObject } from "react";import { beforeEach, describe, expect, it, vi } from "vitest";+import type { CompositorSharedFrameMeta } from "../contracts";++type SharedFrameListener = (frame: VideoFrame, meta: CompositorSharedFrameMeta) => void;const mocks = vi.hoisted(() => ({createCompositorView: vi.fn(),readCompositorFrame: vi.fn(),destroyCompositorView: vi.fn(),setCompositorRect: vi.fn(async () => undefined),+ stopSharedCompositorFrames: vi.fn(async () => ({ ok: true })),+ /** What the preload would call with each frame sent as a shared texture. */+ sharedListener: null as SharedFrameListener | null,}));vi.mock("../compositorViewClient", () => ({@@ -30,6 +36,15 @@ vi.mock("../compositorViewClient", () => ({setCompositorParam: vi.fn(),setCompositorPlaying: vi.fn(),setCompositorRect: mocks.setCompositorRect,+ stopSharedCompositorFrames: mocks.stopSharedCompositorFrames,+ subscribeCompositorSharedFrames: (listener: SharedFrameListener) => {+ mocks.sharedListener = listener;+ return () => {+ if (mocks.sharedListener === listener) {+ mocks.sharedListener = null;+ }+ };+ },}));import { useNativeCompositorView } from "./useNativeCompositorView";@@ -48,22 +63,50 @@ globalThis.ResizeObserver = class {}} as unknown as typeof ResizeObserver;-/** A canvas with a stubbed 2D context — jsdom has none, and the pull loop bails without it. */-function stubCanvasRef(): RefObject<HTMLCanvasElement> {+/** A canvas with a stubbed 2D context — jsdom has none, and the pull loop bails without it.+ * `centreAlpha` is what reading back the middle pixel answers: opaque, as a composed frame+ * always is, unless a test says otherwise. */+function stubCanvasRef(centreAlpha = 255): RefObject<HTMLCanvasElement> {const canvas = document.createElement("canvas");- canvas.getContext = vi.fn(() => ({+ const ctx = {drawImage: vi.fn(),putImageData: vi.fn(),- })) as unknown as HTMLCanvasElement["getContext"];+ getImageData: vi.fn(() => ({ data: new Uint8ClampedArray([0, 0, 0, centreAlpha]) })),+ };+ canvas.getContext = vi.fn(() => ctx) as unknown as HTMLCanvasElement["getContext"];return { current: canvas };}+function context(ref: RefObject<HTMLCanvasElement>) {+ return ref.current?.getContext("2d") as unknown as {+ drawImage: ReturnType<typeof vi.fn>;+ getImageData: ReturnType<typeof vi.fn>;+ };+}++function sharedMeta(overrides: Partial<CompositorSharedFrameMeta> = {}): CompositorSharedFrameMeta {+ return {+ viewId: 7,+ gen: 3,+ width: 4,+ height: 2,+ footage: null,+ footageProjective: false,+ ...overrides,+ };+}++function fakeVideoFrame() {+ return { close: vi.fn() } as unknown as VideoFrame & { close: ReturnType<typeof vi.fn> };+}+const DEVICE_FAILURE ="this display adapter has no D3D11 video decoder (0x887A0004). OpenScreen decodes every preview and export frame with D3D11VA";describe("useNativeCompositorView", () => {beforeEach(() => {vi.clearAllMocks();+ mocks.sharedListener = null;});it("surfaces the native message when the render thread dies", async () => {@@ -317,4 +360,86 @@ describe("useNativeCompositorView", () => {resolveSecond({ id: 8 });await waitFor(() => expect(result.current.viewId).toBe(8));});++ describe("frames handed over as shared GPU textures", () => {+ it("draws the frame on the canvas, sized to it, and closes it", async () => {+ mocks.createCompositorView.mockResolvedValue({ id: 7 });+ mocks.readCompositorFrame.mockResolvedValue(null);+ const ref = stubCanvasRef();+ const { result } = renderHook(() =>+ useNativeCompositorView(ref, { sources: { screenPath: "rec.mp4" } }),+ );+ await waitFor(() => expect(result.current.viewId).toBe(7));++ const frame = fakeVideoFrame();+ mocks.sharedListener?.(frame, sharedMeta());++ expect(context(ref).drawImage).toHaveBeenCalledWith(frame, 0, 0);+ expect(ref.current?.width).toBe(4);+ expect(ref.current?.height).toBe(2);+ expect(ref.current?.dataset.painted).toBe("true");+ expect(frame.close).toHaveBeenCalled();+ expect(mocks.stopSharedCompositorFrames).not.toHaveBeenCalled();+ });++ it("closes a frame meant for another view without drawing it", async () => {+ mocks.createCompositorView.mockResolvedValue({ id: 7 });+ mocks.readCompositorFrame.mockResolvedValue(null);+ const ref = stubCanvasRef();+ const { result } = renderHook(() =>+ useNativeCompositorView(ref, { sources: { screenPath: "rec.mp4" } }),+ );+ await waitFor(() => expect(result.current.viewId).toBe(7));++ const frame = fakeVideoFrame();+ mocks.sharedListener?.(frame, sharedMeta({ viewId: 8 }));++ expect(context(ref).drawImage).not.toHaveBeenCalled();+ expect(frame.close).toHaveBeenCalled();+ });++ // Chromium opens the texture in its own GPU process. One it cannot open — a hybrid+ // laptop's other adapter — draws as nothing, and the preview would stay empty.+ it("goes back to read-back frames when the first one lands as nothing", async () => {+ mocks.createCompositorView.mockResolvedValue({ id: 7 });+ mocks.readCompositorFrame.mockResolvedValue(null);+ const ref = stubCanvasRef(0);+ const { result } = renderHook(() =>+ useNativeCompositorView(ref, { sources: { screenPath: "rec.mp4" } }),+ );+ await waitFor(() => expect(result.current.viewId).toBe(7));++ mocks.sharedListener?.(fakeVideoFrame(), sharedMeta());++ expect(mocks.stopSharedCompositorFrames).toHaveBeenCalledWith(7);+ });++ it("checks only the view's first frame, not every frame", async () => {+ mocks.createCompositorView.mockResolvedValue({ id: 7 });+ mocks.readCompositorFrame.mockResolvedValue(null);+ const ref = stubCanvasRef();+ const { result } = renderHook(() =>+ useNativeCompositorView(ref, { sources: { screenPath: "rec.mp4" } }),+ );+ await waitFor(() => expect(result.current.viewId).toBe(7));++ mocks.sharedListener?.(fakeVideoFrame(), sharedMeta({ gen: 3 }));+ mocks.sharedListener?.(fakeVideoFrame(), sharedMeta({ gen: 4 }));++ // A readback stalls the GPU pipeline: one per view, not one per frame.+ expect(context(ref).getImageData).toHaveBeenCalledTimes(1);+ });++ it("takes a receipt as the generation to ask after, with nothing to draw", async () => {+ mocks.createCompositorView.mockResolvedValue({ id: 7 });+ mocks.readCompositorFrame+ .mockResolvedValueOnce({ ...sharedMeta({ gen: 5 }), shared: true })+ .mockResolvedValue(null);+ const ref = stubCanvasRef();+ renderHook(() => useNativeCompositorView(ref, { sources: { screenPath: "rec.mp4" } }));++ await waitFor(() => expect(mocks.readCompositorFrame).toHaveBeenCalledWith(7, 5));+ expect(context(ref).drawImage).not.toHaveBeenCalled();+ });+ });});diff --git a/src/native/hooks/useNativeCompositorView.ts b/src/native/hooks/useNativeCompositorView.tsindex 416b57c77..a138c8b92 100644--- a/src/native/hooks/useNativeCompositorView.ts+++ b/src/native/hooks/useNativeCompositorView.ts@@ -5,13 +5,17 @@* rect (measured via ResizeObserver + window resize/scroll, rAF-coalesced* — the exact same sync machinery as before, repurposed: it now drives* the offscreen render-target resolution instead of a window position).- * 2. Polls `readCompositorFrame` on every other rAF tick (~30fps), passing the- * generation it last painted. Native returns a self-describing packet- * (`{ gen, width, height, data }`) ONLY when a newer frame exists — otherwise- * `null`, and the canvas is left untouched. So while the preview sits still- * (paused editing) nothing is cloned, sent over IPC, or repainted. The canvas- * drawing buffer is sized from the packet's own dims, so pixels and canvas- * can never drift apart.+ * 2. Polls `readCompositorFrame` on rAF ticks, passing the generation it last+ * painted. Native answers ONLY when a newer frame exists — otherwise `null`, and+ * the canvas is left untouched, so while the preview sits still (paused editing)+ * nothing is cloned, sent over IPC, or repainted. A new frame comes one of two ways:+ * - as a shared GPU texture (Windows), sent to `subscribeCompositorSharedFrames`+ * ahead of the reply, which then only names its generation. Nothing is copied+ * through RAM, so it is pulled every tick;+ * - as RGBA pixels (`{ gen, width, height, data }`) everywhere else, pulled every+ * other tick (~30fps) to bound the copies.+ * Either way the canvas drawing buffer is sized from the frame's own dims, so pixels+ * and canvas can never drift apart.** Every native-bridge call is wrapped in a try/catch that swallows + warns,* because the renderer may run without the bridge (pure web `npm run dev`,@@ -28,6 +32,8 @@ import {setCompositorParam,setCompositorPlaying,setCompositorRect,+ stopSharedCompositorFrames,+ subscribeCompositorSharedFrames,} from "../compositorViewClient";import type { CompositorParamValue, CompositorViewRect } from "../contracts";import { publishFootageQuad } from "../footageQuadStore";@@ -78,9 +84,9 @@ function safelyCall(label: string, call: () => Promise<unknown>) {}}-/** Throttle the rAF pull loop to roughly 30fps: process every other animation- * frame. Keeps IPC + GPU readback + putImageData cheap on high-refresh- * displays (120/144 Hz) without changing perceived preview smoothness. */+/** Read-back frames are pulled on every other animation frame (~30fps on 60 Hz): each one+ * is a GPU readback, a structured clone across IPC and a canvas upload. Shared-texture+ * frames copy nothing through RAM and are pulled every tick. */const PULL_LOOP_TICK_DIVISOR = 2;export function useNativeCompositorView(@@ -189,17 +195,68 @@ export function useNativeCompositorView(// the next frame's. The generation actually on the canvas lets an older one be dropped,// instead of bringing back its pixels and its buffer size until native sends another.let paintedGen = 0;+ // Frames arrive as shared GPU textures (see `subscribeCompositorSharedFrames`): nothing+ // to bound, so the pull loop stops skipping ticks.+ let sharedTransport = false;+ // Whether this view's first shared frame was checked to have actually landed.+ let sharedChecked = false;- /** rAF pull loop: throttle to ~30fps and repaint ONLY when native reports a- * newer generation. The returned packet is self-describing (`gen` + dims +- * pixels), so the canvas is sized from the packet — pixels and canvas can- * never drift out of sync. Runs off the main thread so UI stays at 60/120fps. */+ /** Puts a frame on the canvas. The buffer takes the frame's size HERE, in the same task+ * as the draw that refills it, never when the canvas box changes: anything awaited+ * between the two (the rect's trip to native, `createImageBitmap`) is a frame the+ * browser presents empty. Meanwhile CSS stretches the previous frame over the new box.+ * An Auto format reshapes that box on every padding tick, so an early resize blinked+ * the footage out and back while the slider moved. */+ const paint = (gen: number, width: number, height: number, draw: () => void): boolean => {+ if (disposed || gen < paintedGen) {+ return false;+ }+ setBufferSize(width, height);+ draw();+ paintedGen = gen;+ markPainted();+ return true;+ };++ // Frames the main process hands over as shared GPU textures land here, ahead of the+ // `readCompositorFrame` reply that names their generation.+ const unsubscribeShared = subscribeCompositorSharedFrames((frame, meta) => {+ try {+ const id = viewIdRef.current;+ const ctx = canvas.getContext("2d");+ if (disposed || id == null || meta.viewId !== id || !ctx) {+ return;+ }+ sharedTransport = true;+ lastGen = Math.max(lastGen, meta.gen);+ publishFootageQuad(meta.footage, meta.footageProjective);+ const fresh = canvas.dataset.painted === undefined;+ const drawn = paint(meta.gen, meta.width, meta.height, () => ctx.drawImage(frame, 0, 0));+ if (drawn && fresh && !sharedChecked) {+ sharedChecked = true;+ // Chromium opens the texture in its own GPU process, and one it cannot open (a+ // hybrid laptop's other adapter) draws as nothing at all. The composed frame is+ // opaque everywhere, so a transparent pixel on this fresh canvas is that failure.+ if (ctx.getImageData(meta.width >> 1, meta.height >> 1, 1, 1).data[3] === 0) {+ sharedTransport = false;+ safelyCall("stopSharedFrames", () => stopSharedCompositorFrames(id));+ }+ }+ noteUiProbePreviewFrame();+ } finally {+ frame.close();+ }+ });++ /** rAF pull loop: repaint ONLY when native reports a newer generation. A pixel packet+ * is self-describing (`gen` + dims + pixels), so the canvas is sized from the packet —+ * pixels and canvas can never drift out of sync. */const pullLoop = () => {pullRafHandle = requestAnimationFrame(pullLoop);if (disposed || inFlight) {return;}- pullTick = (pullTick + 1) % PULL_LOOP_TICK_DIVISOR;+ pullTick = (pullTick + 1) % (sharedTransport ? 1 : PULL_LOOP_TICK_DIVISOR);if (pullTick !== 0) {return;}@@ -220,6 +277,11 @@ export function useNativeCompositorView(if (disposed || !frame) {return;}+ if ("shared" in frame) {+ // Already drawn by the shared-frame listener: only the generation is news.+ lastGen = Math.max(lastGen, frame.gen);+ return;+ }const { gen, width, height, data } = frame;// Defensive: the packet's byte count must match its own declared// dimensions. A mismatch would corrupt the image silently — bail.@@ -239,30 +301,15 @@ export function useNativeCompositorView(data.byteLength,);const image = new ImageData(pixels, width, height);- // The buffer takes the packet's size HERE, in the same task as the draw that- // refills it, never when the canvas box changes: anything awaited between the- // two (the rect's trip to native, `createImageBitmap`) is a frame the browser- // presents empty. Meanwhile CSS stretches the previous frame over the new box.- // An Auto format reshapes that box on every padding tick, so an early resize- // blinked the footage out and back while the slider moved.- const paint = (draw: () => void) => {- if (disposed || gen < paintedGen) {- return;- }- setBufferSize(width, height);- draw();- paintedGen = gen;- markPainted();- };// `createImageBitmap` decodes off the main thread (keeps UI at 60/120fps)// and snapshots `image`, so the view can be released after; `putImageData`// is the synchronous fallback if bitmap creation is unavailable.createImageBitmap(image).then((bitmap) => {- paint(() => ctx.drawImage(bitmap, 0, 0));+ paint(gen, width, height, () => ctx.drawImage(bitmap, 0, 0));bitmap.close();})- .catch(() => paint(() => ctx.putImageData(image, 0, 0)));+ .catch(() => paint(gen, width, height, () => ctx.putImageData(image, 0, 0)));// Advance only after a successful, validated frame — so a dropped/// malformed packet is retried rather than silently skipped.lastGen = gen;@@ -326,6 +373,7 @@ export function useNativeCompositorView(return () => {disposed = true;+ unsubscribeShared();publishFootageQuad(null);if (rectRafHandle !== 0) {cancelAnimationFrame(rectRafHandle);diff --git a/technical-documentation/architecture/preview.md b/technical-documentation/architecture/preview.mdindex 08f730cd0..6be0f7a54 100644--- a/technical-documentation/architecture/preview.md+++ b/technical-documentation/architecture/preview.md@@ -182,16 +182,45 @@ The contract, with the invariant on the consumer side:a mismatch would corrupt the image silently. The consumer never assumes asize — every draw is preceded by a resize of the canvas's drawing buffer tothe packet's declared `width`/`height`.-5. **Pull loop cadence.** The renderer pulls on every other rAF tick (`PULL_LOOP_TICK_DIVISOR = 2`,- [`useNativeCompositorView.ts:70`](../../src/native/hooks/useNativeCompositorView.ts:70)),- so IPC + GPU readback + `putImageData` run at roughly 30 fps on 60/120 Hz- displays without changing perceived smoothness.+5. **Pull loop cadence.** Read-back frames are pulled on every other rAF tick+ (`PULL_LOOP_TICK_DIVISOR = 2`), shared-texture frames (below) on every tick. Each+ read-back frame is a GPU readback, a structured clone across IPC and a canvas upload,+ and the tick counter does not advance while a read is in flight: an 8 MB frame whose+ round trip passes 16.7 ms is pulled one tick in three, about 20 fps.The renderer-side wrapper mirrors this verbatim in the Electron main process-([`compositorViewService.ts:339`](../../electron/native-bridge/services/compositorViewService.ts:339)),+([`compositorViewService.ts`](../../electron/native-bridge/services/compositorViewService.ts)),so when the addon is absent the IPC layer returns `null` too — the renderernever has to special-case "addon missing".+### Shared textures (Windows)++On Windows' hardware backend the pixels never leave the GPU. Copying them through RAM+was the preview's bottleneck, not the compositor: at the same ~57 composed frames per+second, read-back reached the canvas at 20-26 fps and kept 54-57 % of the renderer's main+thread busy (measurements in+[engineering/rendering-performance.md](../engineering/rendering-performance.md#preview-transport--2026-10-02)).++- **Native side.** The render thread copies each composed frame into one of four shared+ D3D11 textures (`shared_frames.rs`, NT handles, no keyed mutex), waits for the GPU to+ finish the copy, and publishes `{ gen, slot, handle }`. `SlotBook` tracks which slot+ holds the ready frame and which ones Chromium still holds; a frame nobody took gives+ its slot back, and a slot whose release never came is reclaimed after 1 s.+- **Main process.** `readFrame` takes the frame (`readSharedFrame`), imports it with+ `sharedTexture.importSharedTexture`, sends it with `sharedTexture.sendSharedTexture` to+ the frame that asked, drops its own reference and answers with a receipt+ (`{ …meta, shared: true }`, no pixels). `allReferencesReleased` returns the slot+ (`releaseSharedFrame(id, slot, gen)`) once Chromium is done with it in every process.+- **Renderer.** The preload's `setSharedTextureReceiver` hands the frame to+ `electronAPI.onCompositorFrame`, ahead of the receipt; the hook draws the `VideoFrame`+ with `drawImage` and closes it. The pixels are byte-identical to read-back.+- **Fallbacks, all to read-back.** Not Windows, the software backend, Chromium without GPU+ compositing, or `OPENSCREEN_PREVIEW_READBACK=1` never enable it. A failed import or send+ turns it off for the view, and so does a first frame that lands transparent (a texture+ Chromium could not open, e.g. on another adapter): the composed frame is cleared opaque,+ so a transparent pixel can only be that. The render thread republishes the current+ frame on every switch, so the canvas never waits for something to move.+## Playback syncThe native view runs its own clock while playing, so the renderer only pushes a@@ -318,6 +347,9 @@ was thrown away.measured at the bench in[engineering/rendering-performance.md](../engineering/rendering-performance.md)stay below the threshold in practice, but no systematic measurement exists.+- **Shared textures are Windows-only.** macOS (an `IOSurface`-backed Metal texture) and+ Linux (a dmabuf exported from Vulkan) still read back: `sharedTexture` imports both, the+ native halves are not written.- **Add-on absent = blank frame.** When `compositor_view.node` is missing(development with the addon not yet built, or a packaged build for anunsupported architecture) the overlay renders no pixels: only the DOM/CSSdiff --git a/technical-documentation/engineering/rendering-performance.md b/technical-documentation/engineering/rendering-performance.mdindex 6fa913214..33de33cb2 100644--- a/technical-documentation/engineering/rendering-performance.md+++ b/technical-documentation/engineering/rendering-performance.md@@ -212,6 +212,52 @@ cliffs above: fine for a real GPU, heavy for a per-pixel loop on the CPU rasteri> of the aurora export moves by up to 22/255 between frames 0 and 300, against 4/255 for the> still one.+## Preview transport — 2026-10-02++**The preview was bound by the trip of its pixels to the canvas, not by the compositor.**+The compositor composed ~57 frames per second of a 1080p60 recording in both arms below; what+reached the canvas, and what it cost the UI, depended only on how the frames travelled.++Two throwaway Electron benches (not kept in the tree), run in a hidden window on a desktop —+Ryzen 7 5800X, GeForce RTX 4070 Ti, Windows 11 — not the reference laptop: the copies are CPU+and memory work, so an iGPU laptop pays more for read-back, not less. A hidden window+throttles nothing here (`backgroundThrottling: false`, ticks on timers rather than rAF), but+it renders no React and decodes nothing alongside, so the absolutes are a floor, not the app.+Main-thread load is measured as the gaps in a back-to-back `MessageChannel` ping loop+(`setImmediate` in the main process).++**IPC alone** (synthetic buffers through `ipcRenderer.invoke` from a sandboxed preload and+`contextBridge`, as the app does), two runs:++| frame | MB | round trip p50 / p90 | renderer main thread busy at 30/s | main process busy at 30/s |+|---|---:|---:|---:|---:|+| 1920×1080 | 8.3 | 24 / 31 ms | 54-55 % | 31-37 % |+| 1650×930 | 6.1 | 18-20 / 22-25 ms | 37-39 % | 19-21 % |+| 1100×620 | 2.7 | 8-10 / 10-13 ms | 17-21 % | 9-12 % |++**End to end** (the built addon, a real 32 s 1080p60 recording with its 1080p30 webcam, the+hook's pull loop on 60 Hz ticks), two runs per arm:++| preview | arm | frames on the canvas /s | composed /s | renderer busy | main busy | main import + send p50 / p90 |+|---|---|---:|---:|---:|---:|---:|+| 1920×1080 | read-back | 20-21 | 57-59 | 57 % | 37-38 % | — |+| 1920×1080 | shared texture | 53-54 | 56-57 | 1.8-2.0 % | 0.1-0.5 % | 0.67 / 0.97 ms |+| 1650×928 | read-back | 26 | 57 | 54-56 % | 35 % | — |+| 1650×928 | shared texture | 52.5-53 | 57-58 | 2.0-2.6 % | 0.6-1.7 % | 0.67-0.73 / 1.02-1.18 ms |++The same paused frame, read back and drawn from the shared texture, differs in **0 of+8 294 400 bytes** at 1080p (0 of 6 124 800 at 1650×928).++- **Read-back stalls on its own round trip.** At 18-24 ms per 8 MB frame, the hook's pull loop+ (one read in flight, one tick in two) gets one frame every three ticks: ~20 fps on 60 Hz.+- **The transport was most of the renderer's load.** The ~44 % main-thread block measured+ during playback on 2026-09-29 (`VirtualPreview.tsx`, another machine) is the order of this+ cost alone. Every copy, structured clone and allocation landed on the thread React paints+ from.+- **Shared textures leave ~7 % of the composed frames undrawn.** Two ~60 Hz clocks — the+ render thread and the pull loop — beat against each other; a tick that finds nothing new+ is followed by one that finds two, and only the newer is drawn.+## The macOS export path — 2026-09-03/04Everything above is the Windows reference machine. This section is a **different machine and a different pipeline**: a Mac mini M1 (8 cores, 8 GiB, macOS 26.5), Metal compositor, VideoToolbox on both ends. Nothing here transfers to the Windows numbers, and the reverse held too — of the three levers that mattered on Windows and Linux, **none applied here**.