Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
feat(effects): getFrames renders from the document when no export is current
EtienneLescot8c896107b539
diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rsindex b25b0dfe0..f1e4bd8fa 100644--- a/crates/compositor-view-napi/src/lib.rs+++ b/crates/compositor-view-napi/src/lib.rs@@ -664,21 +664,7 @@ impl Task for ExportMultiTask {// libopenh264 au lieu d'AMF, plutôt que d'échouer.let gpu = Gpu::create_auto(false).map_err(|e| Error::from_reason(format!("{e:#}")))?;self.control.check().map_err(mp4_task_error)?;- let mut cfg = config::all().pop().expect("au moins une config"); // C8- cfg.zoom = false;- cfg.layout_anim = false;- cfg.mblur_n = 1; // layout statique → pas de motion blur de layout (pas de surcoût)-- // scène de l'app = même chemin que la preview live : fond, layout, webcam, curseur.- // JSON absent/invalide → pas de scène (fixture), pareil que si la preview n'en avait- // jamais reçu — jamais un fallback masquant, juste rien de configuré.- let scene = self.scene_json.as_deref().and_then(|j| Scene::from_json(j).ok());- if let Some(scene) = &scene {- cfg.bg_blur = scene.effects.blur;- cfg.cursor = scene.cursor.show;- } else {- cfg.cursor = false;- }+ let (cfg, scene) = export_cfg_and_scene(self.scene_json.as_deref());let mut export_params = pipeline::ExportParams::default();let (width, height) = resolve_output_size(@@ -763,6 +749,121 @@ fn mp4_task_error(error: anyhow::Error) -> Error {}}+/// La config et la scène d'un rendu d'export (MP4, GIF, frames) : la C8 en layout statique, et ce+/// que la scène de l'app règle. Scène = même chemin que la preview live : fond, layout, webcam,+/// curseur. JSON absent ou invalide → pas de scène (fixture), pareil que si la preview n'en avait+/// jamais reçu : jamais un fallback masquant, juste rien de configuré.+fn export_cfg_and_scene(scene_json: Option<&str>) -> (config::Cfg, Option<Scene>) {+ let mut cfg = config::all().pop().expect("au moins une config"); // C8+ cfg.zoom = false;+ cfg.layout_anim = false;+ cfg.mblur_n = 1; // layout statique → pas de motion blur de layout (pas de surcoût)+ let scene = scene_json.and_then(|j| Scene::from_json(j).ok());+ match &scene {+ Some(scene) => {+ cfg.bg_blur = scene.effects.blur;+ cfg.cursor = scene.cursor.show;+ }+ None => cfg.cursor = false,+ }+ (cfg, scene)+}++fn clip_sources(clips: Vec<ClipInput>) -> Vec<pipeline::ClipSource> {+ clips+ .into_iter()+ .map(|c| pipeline::ClipSource {+ screen: c.screen_path,+ webcam: c.webcam_path,+ source_start_sec: c.source_start_sec,+ source_end_sec: c.source_end_sec,+ webcam_offset_sec: c.webcam_offset_sec,+ has_audio: c.has_audio,+ })+ .collect()+}++/// Une frame de `render_frames`.+#[napi(object)]+pub struct RenderedFrameOutput {+ /// Son indice sur la grille de sortie.+ pub index: u32,+ pub width: u32,+ pub height: u32,+ /// JPEG, qualité 85.+ pub jpeg: Buffer,+}++pub struct RenderFramesTask {+ clips: Vec<pipeline::ClipSource>,+ scene_json: Option<String>,+ fps: u32,+ frames: Vec<u32>,+ max_width: u32,+}++impl Task for RenderFramesTask {+ type Output = Vec<(u32, u32, u32, Vec<u8>)>;+ type JsValue = Vec<RenderedFrameOutput>;++ fn compute(&mut self) -> Result<Self::Output> {+ let reason = |e: anyhow::Error| Error::from_reason(format!("{e:#}"));+ let gpu = Gpu::create_auto(false).map_err(reason)?;+ let (cfg, scene) = export_cfg_and_scene(self.scene_json.as_deref());+ let default = pipeline::ExportParams::default();+ let (width, height) = resolve_output_size(scene.as_ref(), None, None, (default.width, default.height))?;+ // Un compositeur neuf attend la compilation des effets shader, comme celui de l'export.+ let comp = Compositor::new_sized(&gpu, width, height).map_err(reason)?;+ if let Some(scene) = &scene {+ comp.set_live_params(live_params_from_scene(scene));+ }+ comp.set_scene(scene);+ let wanted: Vec<u64> = self.frames.iter().map(|&f| u64::from(f)).collect();+ let frames = unsafe {+ openscreen_compositor::timeline_walk::render_frames(+ &self.clips,+ &gpu,+ &comp,+ &cfg,+ self.fps.max(1) as i32,+ &wanted,+ )+ }+ .map_err(reason)?;+ frames+ .into_iter()+ .map(|frame| {+ let index = frame.index as u32;+ let (w, h, jpeg) = frame.to_jpeg(self.max_width).map_err(reason)?;+ Ok((index, w, h, jpeg))+ })+ .collect()+ }++ fn resolve(&mut self, _env: Env, out: Self::Output) -> Result<Self::JsValue> {+ Ok(out+ .into_iter()+ .map(|(index, width, height, jpeg)| RenderedFrameOutput { index, width, height, jpeg: jpeg.into() })+ .collect())+ }+}++/// Rend des frames de la timeline sans l'exporter : `getFrames` quand l'export manque ou date+/// (`docs/shader-effects.md`, E.4). Mêmes clips et même scène qu'`export_multi`, mêmes pixels :+/// la marche est celle de l'export (`timeline_walk::render_frames`), sans encodage. `frames` sont+/// des indices sur la grille `fps` ; ceux au-delà de la timeline ne rendent rien. Les images sont+/// en JPEG, réduites à `max_width` de large.+#[napi]+pub fn render_frames(+ clips: Vec<ClipInput>,+ scene_json: Option<String>,+ fps: u32,+ frames: Vec<u32>,+ max_width: u32,+) -> AsyncTask<RenderFramesTask> {+ AsyncTask::new(RenderFramesTask { clips: clip_sources(clips), scene_json, fps, frames, max_width })+}+/// Lance un export multiclip natif (vraie timeline → MP4) et résout `Promise<ExportStats>`./// `scene_json` : même `SceneDescription` que la preview (fond/layout/webcam/effets/curseur)./// `params` : taille/cadence/codec de sortie voulus (absent → 1920x1080/fps du 1er clip/h264).@@ -779,17 +880,7 @@ pub fn export_multi(on_progress: Option<JsFunction>,control: Option<External<ExportControl>>,) -> Result<AsyncTask<ExportMultiTask>> {- let clips = clips- .into_iter()- .map(|c| pipeline::ClipSource {- screen: c.screen_path,- webcam: c.webcam_path,- source_start_sec: c.source_start_sec,- source_end_sec: c.source_end_sec,- webcam_offset_sec: c.webcam_offset_sec,- has_audio: c.has_audio,- })- .collect();+ let clips = clip_sources(clips);Ok(AsyncTask::new(ExportMultiTask {out_path,clips,@@ -883,18 +974,7 @@ impl Task for ExportGifTask {// where the CPU backend exists specifically so the export still completes.let gpu = Gpu::create_auto(false).map_err(|e| Error::from_reason(format!("{e:#}")))?;self.control.check().map_err(gif_task_error)?;- let mut cfg = config::all().pop().expect("au moins une config"); // C8- cfg.zoom = false;- cfg.layout_anim = false;- cfg.mblur_n = 1;-- let scene = self.scene_json.as_deref().and_then(|j| Scene::from_json(j).ok());- if let Some(scene) = &scene {- cfg.bg_blur = scene.effects.blur;- cfg.cursor = scene.cursor.show;- } else {- cfg.cursor = false;- }+ let (cfg, scene) = export_cfg_and_scene(self.scene_json.as_deref());let (width, height) = resolve_output_size(scene.as_ref(),@@ -961,17 +1041,7 @@ pub fn export_gif(// Deliberately the same argument shape as `export_multi`: the caller builds// one clip list and one scene, and picks the container. Cursor comes from// the scene like every other effect — there is no GIF-specific input left.- let clips = clips- .into_iter()- .map(|c| pipeline::ClipSource {- screen: c.screen_path,- webcam: c.webcam_path,- source_start_sec: c.source_start_sec,- source_end_sec: c.source_end_sec,- webcam_offset_sec: c.webcam_offset_sec,- has_audio: c.has_audio,- })- .collect();+ let clips = clip_sources(clips);let gif_params = params.map(|p| GifExportParams {width: p.width,diff --git a/crates/compositor/src/lib.rs b/crates/compositor/src/lib.rsindex 4aed79293..16ebde9d0 100644--- a/crates/compositor/src/lib.rs+++ b/crates/compositor/src/lib.rs@@ -58,7 +58,7 @@ pub mod shared_frames;pub mod text_anim;pub mod text_fonts;pub mod text_plate;-pub(crate) mod timeline_walk;+pub mod timeline_walk;// GPU backend : Windows → d3d_windows, macOS → d3d_macos. Ré-exporté sous le nom `d3d`// pour que `crate::d3d::Gpu`/`Backend` reste portable. Le module sous-jacent estdiff --git a/crates/compositor/src/timeline_walk.rs b/crates/compositor/src/timeline_walk.rsindex 0953656ce..5556e577b 100644--- a/crates/compositor/src/timeline_walk.rs+++ b/crates/compositor/src/timeline_walk.rs@@ -157,6 +157,119 @@ pub(crate) unsafe fn walk_composited_timeline(webcam_decs: &mut HashMap<String, Decoder>,on_frame: &mut dyn FnMut(u64) -> Result<()>,on_clip_end: &mut dyn FnMut(usize, f64, u64, &[SpeedSegment]) -> Result<()>,+) -> Result<u64> {+ walk_timeline(clips, gpu, comp, cfg, out_fps, scene, screen_decs, webcam_decs, None, on_frame, on_clip_end)+}++/// Une frame de sortie relue par `render_frames`.+pub struct RenderedFrame {+ /// Son indice sur la grille de sortie (`out_fps`).+ pub index: u64,+ pub width: u32,+ pub height: u32,+ /// RGBA8, `width * height * 4` octets.+ pub rgba: Vec<u8>,+}++impl RenderedFrame {+ /// En JPEG (qualité 85), réduite à `max_width` pixels de large au plus : la forme d'une image+ /// de `getFrames`. Rend (largeur, hauteur, octets).+ pub fn to_jpeg(self, max_width: u32) -> Result<(u32, u32, Vec<u8>)> {+ let (index, width, height) = (self.index, self.width, self.height);+ let img = image::RgbaImage::from_raw(width, height, self.rgba)+ .ok_or_else(|| anyhow::anyhow!("frame {index}: the readback does not hold {width}x{height} pixels"))?;+ let img = if max_width > 0 && width > max_width {+ let h = ((height as u64 * max_width as u64) / width as u64).max(1) as u32;+ image::imageops::resize(&img, max_width, h, image::imageops::FilterType::Triangle)+ } else {+ img+ };+ let rgb = image::DynamicImage::ImageRgba8(img).to_rgb8();+ let mut jpeg = Vec::new();+ image::codecs::jpeg::JpegEncoder::new_with_quality(&mut jpeg, 85).encode_image(&rgb)?;+ Ok((rgb.width(), rgb.height(), jpeg))+ }+}++/// Les frames de sortie `wanted` (indices sur la grille `out_fps`), composées comme l'export les+/// compose et relues en RGBA, par indice croissant. C'est la marche de l'export : mêmes clips,+/// mêmes vitesses, même frame source à chaque instant. Seules les autres frames ne sont ni+/// décodées ni composées. Un indice au-delà de la timeline ne rend rien.+pub unsafe fn render_frames(+ clips: &[ClipSource],+ gpu: &Gpu,+ comp: &Compositor,+ cfg: &Cfg,+ out_fps: i32,+ wanted: &[u64],+) -> Result<Vec<RenderedFrame>> {+ let mut only = wanted.to_vec();+ only.sort_unstable();+ only.dedup();+ let scene = comp.scene_snapshot();+ let mut out = Vec::new();+ walk_timeline(+ clips,+ gpu,+ comp,+ cfg,+ out_fps,+ &scene,+ &mut HashMap::new(),+ &mut HashMap::new(),+ Some(&only),+ &mut |index| {+ let (width, height, rgba) = comp.readback_direct()?;+ out.push(RenderedFrame { index, width, height, rgba });+ Ok(())+ },+ &mut |_, _, _, _| Ok(()),+ )?;+ Ok(out)+}++/// Combien de secondes de vidéo un décodeur déroule, plutôt que de repartir d'une image clé, pour+/// rattraper les frames qu'une marche partielle (`only`) a sautées.+const CATCH_UP_DECODE_SEC: f64 = 2.0;++/// Rattrape un décodeur laissé derrière `target` par les frames sautées, sans changer la frame que+/// la sémantique de hold y montrera. `seek_to` rend la première frame à ou après le point demandé :+/// on recule ce point jusqu'à tomber au plus sur `target`, ou sur le début du clip, là où la marche+/// s'est placée. `advance_decoder_to` adopte ensuite la dernière frame dont le pts est ≤ `target`,+/// celle qu'une marche sans saut aurait montrée. `target` et `clip_start` sont dans le temps du+/// décodeur (la webcam retranche son offset).+unsafe fn catch_up(dec: &mut Decoder, target: f64, clip_start: f64) -> Result<()> {+ if target - dec.cur_time_sec() <= CATCH_UP_DECODE_SEC {+ return Ok(());+ }+ let mut back = CATCH_UP_DECODE_SEC;+ loop {+ let from = (target - back).max(clip_start);+ let frame = dec.seek_to(from)?;+ if from <= clip_start || (!frame.is_null() && dec.cur_time_sec() <= target) {+ return Ok(());+ }+ back *= 2.0;+ }+}++/// `walk_composited_timeline`, qui ne compose que les frames de sortie `only` (triées) quand il y+/// en a : les autres avancent le compteur sans décoder ni composer, et la marche s'arrête après+/// la dernière. Un flux sans pts fiable avance d'une frame par frame de sortie : sauter le rend+/// inexact, le cas est assez rare pour ne pas le traiter.+#[allow(clippy::too_many_arguments)]+unsafe fn walk_timeline(+ clips: &[ClipSource],+ gpu: &Gpu,+ comp: &Compositor,+ cfg: &Cfg,+ out_fps: i32,+ scene: &Option<Scene>,+ screen_decs: &mut HashMap<String, Decoder>,+ webcam_decs: &mut HashMap<String, Decoder>,+ only: Option<&[u64]>,+ on_frame: &mut dyn FnMut(u64) -> Result<()>,+ on_clip_end: &mut dyn FnMut(usize, f64, u64, &[SpeedSegment]) -> Result<()>,) -> Result<u64> {// La piste se charge même curseur MASQUÉ : le focus auto des zooms la suit, comme en preview// (qui la charge toujours). Sans elle, un zoom auto suivait en preview et restait figé à@@ -180,7 +293,7 @@ pub(crate) unsafe fn walk_composited_timeline(// donc avec le vrai arrière-plan de la webcam gravé dans le fichier.comp.set_segmentation_deterministic(true);- for (clip_index, clip) in clips.iter().enumerate() {+ 'clips: for (clip_index, clip) in clips.iter().enumerate() {// Le preset de layout est GLOBAL (un seul panneau pour toute la timeline) mais la// caméra est PAR CLIP : un projet mélange sans problème un enregistrement avec webcam// et un import qui n'en a pas. Le preset ne doit donc s'appliquer qu'aux clips qui ont@@ -313,6 +426,21 @@ pub(crate) unsafe fn walk_composited_timeline('clip_frames: for segment in &speed_segments {for segment_frame in 0..segment.frame_count {let target_source_time = segment.source_time(segment_frame, out_fps as f64);+ if let Some(only) = only {+ if only.last().map_or(true, |&last| frames > last) {+ break 'clips;+ }+ if only.binary_search(&frames).is_err() {+ frames += 1;+ continue;+ }+ catch_up(sdec, target_source_time, clip.source_start_sec)?;+ catch_up(+ wdec,+ target_source_time - clip.webcam_offset_sec,+ (clip.source_start_sec - clip.webcam_offset_sec).max(0.0),+ )?;+ }{let _p = crate::export_probe::scope(crate::export_probe::Stage::DecodeScreen);if !advance_decoder_to(sdec, target_source_time, 0.0)? {@@ -374,7 +502,7 @@ pub(crate) unsafe fn walk_composited_timeline(#[cfg(test)]mod tests {- use super::{frame_step, FrameStep, NextFrameTime};+ use super::{frame_step, FrameStep, NextFrameTime, RenderedFrame};/// La cadence de lecture, en une phrase : à 24 fps, une seconde réelle doit adopter 24/// frames et pas une de plus. Le bug d'origine (un pas fixe de 1/60 s, une frame par@@ -464,4 +592,18 @@ mod tests {FrameStep::CommitAndStop);}++ /// La frame rendue devient l'image de getFrames : un JPEG réduit à la largeur demandée, au+ /// ratio de la sortie, et seulement réduit.+ #[test]+ fn a_rendered_frame_becomes_a_jpeg_no_wider_than_asked() {+ let frame = |w: u32, h: u32| RenderedFrame { index: 3, width: w, height: h, rgba: vec![200; (w * h * 4) as usize] };+ let (w, h, jpeg) = frame(64, 32).to_jpeg(16).expect("jpeg");+ assert_eq!((w, h), (16, 8));+ let decoded = image::load_from_memory(&jpeg).expect("decode");+ assert_eq!((decoded.width(), decoded.height()), (16, 8));+ assert_eq!(frame(8, 4).to_jpeg(16).expect("jpeg").0, 8, "jamais agrandie");+ let short = RenderedFrame { index: 0, width: 8, height: 4, rgba: vec![0; 10] };+ assert!(short.to_jpeg(16).is_err(), "une relecture incomplète est une erreur");+ }}diff --git a/crates/compositor/tests/render_frames_match_export.rs b/crates/compositor/tests/render_frames_match_export.rsnew file mode 100644index 000000000..8e765fc32--- /dev/null+++ b/crates/compositor/tests/render_frames_match_export.rs@@ -0,0 +1,117 @@+//! Une frame rendue seule (`timeline_walk::render_frames`, ce que getFrames montre sans export)+//! est celle de l'export, au pixel près. La marche est la même ; seules les frames sautées+//! diffèrent, et `catch_up` rattrape les décodeurs sans changer la frame source montrée. Comparé+//! ici à la marche complète, où toutes les frames sont demandées et aucune n'est sautée.+//!+//! La scène met deux coupes du même fichier, une région 2× dans la seconde, un effet shader qui+//! lit le temps programme, et le curseur de la prise s'il a sa piste à côté.+//!+//! La sortie est à 24 i/s : sur une source à 30 i/s, ses instants tombent entre deux frames+//! source, là où un rattrapage qui viserait la cible au lieu de la précéder montrerait la frame+//! suivante. Encore faut-il que deux frames voisines diffèrent : une prise d'écran immobile ne+//! prouve rien, d'où une mire dont chaque frame change.+//!+//! La CI ne le joue pas : il faut un device (lavapipe convient) et la vidéo, d'au moins 20 s.+//!+//! ```powershell+//! ffmpeg -f lavfi -i testsrc2=size=640x360:rate=30 -t 25 -c:v libopenh264 -g 60 counter30.mp4+//! $env:OPENSCREEN_RENDER_SOURCE = "...\counter30.mp4"+//! cargo test -p openscreen-compositor --test render_frames_match_export -- --nocapture+//! ```++use openscreen_compositor::compositor::{live_params_from_scene, Compositor};+use openscreen_compositor::config;+use openscreen_compositor::d3d::Gpu;+use openscreen_compositor::pipeline::ClipSource;+use openscreen_compositor::scene::Scene;+use openscreen_compositor::timeline_walk::{render_frames, RenderedFrame};++const FPS: i32 = 24;+const CLIPS: [(f64, f64); 2] = [(1.0, 4.0), (6.0, 20.0)];++fn scene_json(source: &str) -> String {+ let clips: Vec<String> = CLIPS+ .iter()+ .map(|(start, end)| {+ format!(+ r#"{{"screenPath":"{source}","webcamPath":"","sourceStartSec":{start},"sourceEndSec":{end},"webcamOffsetSec":0,"hasAudio":false}}"#+ )+ })+ .collect();+ format!(+ r##"{{+ "clips": [{clips}],+ "layout": {{"preset":"no-webcam","webcamSize":1.0,"webcamShape":"rounded","webcamMirror":false,"webcamPosition":null,"webcamReactiveZoom":false}},+ "effects": {{"padding":0.1,"blur":false,"shadow":0.0,"roundnessFrac":0,"motionBlur":0.0}},+ "background": {{"kind":"color","color":"#303030"}},+ "zoomRegions": [],+ "speedRegions": [{{"clipIndex":1,"startSec":8,"endSec":10,"speed":2}}],+ "cursor": {{"show":true,"size":1,"smoothing":0,"motionBlur":0,"clickBounce":0,"clipToBounds":false,"theme":"default"}},+ "cropByClip": [null, null],+ "output": {{"width":640,"height":360,"fps":null}},+ "shaderEffects": [{{+ "clipIndex": 1, "startSec": 6.0, "endSec": 20.0, "target": "image", "lane": 0, "contract": 1,+ "code": "fn effect(uv: vec2<f32>) -> vec4<f32> {{ let c = frame(uv); return vec4<f32>(c.rgb * (0.6 + 0.4 * sin(ctx.time * 3.0)), 1.0); }}",+ "params": [], "values": {{}}+ }}]+ }}"##,+ clips = clips.join(",")+ )+}++#[test]+fn a_frame_rendered_alone_is_the_frame_of_the_full_walk() {+ let Some(source) = std::env::var_os("OPENSCREEN_RENDER_SOURCE") else {+ eprintln!("OPENSCREEN_RENDER_SOURCE absent — test sauté");+ return;+ };+ let source = source.to_string_lossy().replace('\\', "/");+ let gpu = Gpu::create_auto(false).expect("device");+ let clips: Vec<ClipSource> = CLIPS+ .iter()+ .map(|&(start, end)| ClipSource {+ screen: source.clone(),+ webcam: String::new(),+ source_start_sec: start,+ source_end_sec: end,+ webcam_offset_sec: 0.0,+ has_audio: false,+ })+ .collect();+ // La config de l'export (`export_cfg_and_scene` dans l'addon).+ let mut cfg = config::all().pop().expect("C8");+ cfg.zoom = false;+ cfg.layout_anim = false;+ cfg.mblur_n = 1;+ cfg.bg_blur = 0.0;+ cfg.cursor = true;+ let render = |wanted: &[u64]| -> Vec<RenderedFrame> {+ let scene = Scene::from_json(&scene_json(&source)).expect("scène");+ let comp = Compositor::new_sized(&gpu, 640, 360).expect("compositeur");+ comp.set_live_params(live_params_from_scene(&scene));+ comp.set_scene(Some(scene));+ unsafe { render_frames(&clips, &gpu, &comp, &cfg, FPS, wanted) }.expect("render_frames")+ };++ let all: Vec<u64> = (0..2_000).collect();+ let dense = render(&all);+ let total = dense.len() as u64;+ eprintln!("marche complète : {total} frames");+ // 3 s, puis 2 s + 1 s (2×) + 10 s : 16 s de sortie.+ assert_eq!(total, 16 * FPS as u64, "la marche complète ne couvre pas la timeline");+ assert!(dense.iter().enumerate().all(|(i, f)| f.index == i as u64));++ // Des sauts de plus de `CATCH_UP_DECODE_SEC`, dans les deux clips, dont un dans la région 2×.+ let sparse_at = [2, 70, 130, 140, 260, total - 1, total + 50];+ let sparse = render(&sparse_at);+ assert_eq!(+ sparse.iter().map(|f| f.index).collect::<Vec<_>>(),+ &sparse_at[..6],+ "une frame demandée manque, ou une au-delà de la timeline a été rendue"+ );+ for frame in &sparse {+ let reference = &dense[frame.index as usize];+ let differing = frame.rgba.iter().zip(&reference.rgba).filter(|(a, b)| a != b).count();+ assert_eq!(differing, 0, "frame {} : {differing} octets diffèrent de la marche complète", frame.index);+ }+}diff --git a/docs/shader-effects.md b/docs/shader-effects.mdindex c7c55341a..8bdfdc1e5 100644--- a/docs/shader-effects.md+++ b/docs/shader-effects.md@@ -579,15 +579,30 @@ lecture :### E.4 Voir le résultat sans exporter-Aujourd'hui `getFrames` (`media-tools.ts`) ne montre que le dernier export : vérifier un effet-coûte un export complet. Sur un projet de dix minutes, la boucle de l'agent devient trop lente.--**`getFrames` rend désormais depuis le document quand l'export manque ou est périmé.**-- Il passe par le même compositeur, la même scène et les mêmes secondes de règle, et rend les- frames demandées à la taille d'export.-- C'est le chemin que le spike a utilisé pour comparer la vue live à l'export : une vue sans- fenêtre, `setScene`, `presentTime`, `readFrame`.-- Avec un export à jour, l'outil garde son comportement actuel.+Avant, `getFrames` (`media-tools.ts`) ne montrait que le dernier export : vérifier un effet+coûtait un export complet. Sur un projet de dix minutes, la boucle de l'agent devenait trop+lente.++**`getFrames` rend désormais depuis le document quand l'export manque ou est périmé** (PR 5).+- **Avec un export à jour,** l'outil lit toujours ce fichier.+- **Sinon,** le processus principal construit les clips et la scène comme l'export+ (`buildNativeClipList`, `buildSceneDescription`). Il passe les secondes de règle en indices de+ frame, sur la grille à 60 i/s de `exportVideo`, et demande ces frames à l'addon+ (`renderFrames`).+- **L'addon rend par la marche de l'export,** `timeline_walk::render_frames` : mêmes clips, mêmes+ vitesses, même frame source à chaque instant. Les frames non demandées ne sont ni décodées ni+ composées.+- **Un saut de plus de 2 s repart d'une image clé** (`catch_up`). `seek_to` rend la première+ frame à ou après le point visé ; on recule donc ce point jusqu'à tomber avant l'instant voulu,+ puis on avance comme l'export : la frame montrée est la dernière dont le pts le précède.+- **Le compositeur attend la compilation des effets,** comme celui de l'export.+- **Les images sont des JPEG** de 1 280 px de large au plus, comme les frames d'un export.+- **Prouvé au pixel** par `tests/render_frames_match_export.rs` : une frame rendue seule est+ identique, octet pour octet, à la même frame de la marche complète. La source est une mire dont+ chaque frame change, sur une grille à 24 i/s qui tombe entre ses frames. Un seek droit sur+ l'instant visé échoue à ce test.+- **Ce n'est pas la vue live** que le spike pilotait (`presentTime`, `readFrame`) : elle saute un+ effet qui compile encore, et il faut la sonder en boucle.### E.5 Ce que l'agent lit avant d'écrirediff --git a/electron/ipc/handlers.ts b/electron/ipc/handlers.tsindex bb2a68a7b..ebee7416f 100644--- a/electron/ipc/handlers.ts+++ b/electron/ipc/handlers.ts@@ -5172,6 +5172,7 @@ export function registerIpcHandlers(runners: mcpRunners,documents: aiEditionDocuments,openVideo: openVideoInHiddenWindow,+ renderFrames: (...args) => loudnessService.renderFrames(...args),readCursor: async (videoPath) => (await readCursorTelemetryFile(videoPath)).samples,// Never overriding the switch: setPointerTrack is refused while MCP edits is off.runTool: mcpRunTool,@@ -5265,6 +5266,8 @@ export function registerIpcHandlers(agentServices: {mcpToolDeps,documents: aiEditionDocuments,+ // getFrames without an export: this process's compositor renders them.+ renderFrames: loudnessService.renderFrames.bind(loudnessService),// The reader the export runner's auto zoom reads through.readCursorTelemetry: async (videoPath: string) =>(await readCursorTelemetryFile(videoPath)).samples,diff --git a/electron/mcp/media-tools.test.ts b/electron/mcp/media-tools.test.tsindex b86ee8a90..cf3561763 100644--- a/electron/mcp/media-tools.test.ts+++ b/electron/mcp/media-tools.test.ts@@ -120,6 +120,8 @@ let openVideoFails: boolean;let client: Client;/** The stdio server's media deps: the in-app tests reuse its stand-ins. */let stdioDeps: MediaToolDeps;+/** The addon's frame renderer, when a test gives it one: none by default, as an older addon. */+let renderFrames: MediaToolDeps["renderFrames"];/** The pointer recorded with the take, as the sidecar beside the video. */async function readSidecar(video: string): Promise<CursorTelemetryPoint[]> {@@ -140,6 +142,7 @@ beforeEach(async () => {runners = new FakeRunners(videoPath);closed = 0;openVideoFails = false;+ renderFrames = undefined;const toolDeps: McpToolDeps = {transport: "stdio",@@ -170,6 +173,7 @@ beforeEach(async () => {};},readCursor: readSidecar,+ renderFrames: async (...args) => (renderFrames ? renderFrames(...args) : null),runTool,now: () => 1_000,};@@ -774,6 +778,66 @@ describe("getFrames on a project", () => {});});+describe("getFrames without an export that shows the project", () => {+ let asked: Array<{ clips: number; fps: number; frames: number[]; maxWidth: number }>;+ beforeEach(() => {+ asked = [];+ renderFrames = async (clips, _scene, fps, frames, maxWidth) => {+ asked.push({ clips: clips.length, fps, frames, maxWidth });+ return frames.map((index) => ({+ index,+ width: 1280,+ height: 720,+ jpeg: Buffer.from(`render@${index}`),+ }));+ };+ });+ const images = (result: CallToolResult) =>+ result.content.flatMap((part) =>+ part.type === "image" ? [Buffer.from(part.data, "base64").toString()] : [],+ );++ it("renders them from the project, on the export's frame grid, when it has no export", async () => {+ const { projectId } = await recordTake();+ // A cut of 2 s early in the take: ruler 6 s is 4 s into what the export would play.+ await call("addTrim", { projectId, startSec: 2, endSec: 4 });+ const frames = await call("getFrames", { projectId, atSec: [1, 6] });+ expect(frames.isError).toBeFalsy();+ expect(textOf(frames)).toContain("as its export would show them: it has no export yet");+ expect(asked).toEqual([{ clips: 2, fps: 60, frames: [60, 240], maxWidth: 1280 }]);+ expect(images(frames)).toEqual(["render@60", "render@240"]);+ });++ it("reads a current export, and renders again once the project changed after it", async () => {+ const { projectId } = await recordTake();+ runners.results.export = { success: true, outputPath: path.join(dir, "demo.mp4") };+ await call("exportVideo", { projectId, path: path.join(dir, "demo.mp4") });+ const fresh = await call("getFrames", { projectId, atSec: [1] });+ expect(fresh.content).toContainEqual({+ type: "text",+ text: "1.00s on the ruler, 1.00s into the video",+ });+ expect(asked).toEqual([]);++ await call("addTrim", { projectId, startSec: 2, endSec: 4 });+ const stale = await call("getFrames", { projectId, atSec: [6] });+ expect(textOf(stale)).toContain("it changed after its last export");+ expect(images(stale)).toEqual(["render@240"]);+ });++ it("says why it could not render, and asks for an export when the addon cannot", async () => {+ const { projectId } = await recordTake();+ renderFrames = async () => {+ throw new Error("no GPU");+ };+ const failed = await call("getFrames", { projectId, atSec: [1] });+ expect(failed.isError).toBe(true);+ expect(textOf(failed)).toContain("could not be rendered: no GPU");+ renderFrames = async () => null;+ expect(textOf(await call("getFrames", { projectId, atSec: [1] }))).toContain("no export yet");+ });+});+describe("getFrames", () => {it("returns each frame as an image, labelled with its time", async () => {const result = await call("getFrames", { path: "take.mp4", atSec: [1, 5] });diff --git a/electron/mcp/media-tools.ts b/electron/mcp/media-tools.tsindex eb45333aa..720ebcd5b 100644--- a/electron/mcp/media-tools.ts+++ b/electron/mcp/media-tools.ts@@ -34,7 +34,12 @@ import {MP4_FRAME_RATES,VALID_GIF_FRAME_RATES,} from "../../src/lib/exporter/types";-import type { CursorRecordingData, CursorRecordingSample } from "../../src/native/contracts";+import type {+ CompositorClipInput,+ CursorRecordingData,+ CursorRecordingSample,+} from "../../src/native/contracts";+import { buildNativeClipList, buildSceneDescription } from "../../src/native/sceneDescription";import { runDocumentTool } from "../ai-edition/deep-agent/service";import type { DocumentService } from "../ai-edition/document-service";import type { RunnerHandlers } from "../cli/runnerHost";@@ -105,6 +110,16 @@ export interface MediaToolDeps {"createProject" | "addAsset" | "saveProject" | "filePathFor" | "getProject">;openVideo(videoPath: string): Promise<OpenedVideo>;+ /** Frames of a timeline composed as its export would compose them, without exporting: the+ * addon's `renderFrames` (`frames` are indices on the `fps` grid, JPEG at most `maxWidth`+ * wide). `null`, or absent, when the addon cannot: getFrames then needs an export. */+ renderFrames?(+ clips: CompositorClipInput[],+ sceneJson: string,+ fps: number,+ frames: number[],+ maxWidth: number,+ ): Promise<Array<{ index: number; width: number; height: number; jpeg: Buffer }> | null>;/** The pointer samples recorded with a video; empty when there are none. */readCursor(videoPath: string): Promise<CursorTelemetryPoint[]>;/** The document tools' runner: a fresh project's checkpoint and every edit go through it. */@@ -115,6 +130,8 @@ export interface MediaToolDeps {export const MAX_FRAMES = 12;// Wide enough to read UI text, small enough that a dozen frames stay cheap.const FRAME_WIDTH = 1280;+/** exportVideo's frame rate when none is asked: the grid getFrames renders on without an export. */+const EXPORT_FPS = 60;// A seek to the exact end shows no frame in Chromium; stop one frame short of it.const END_MARGIN_SEC = 1 / 60;@@ -496,6 +513,54 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {/** Each project's last export of this session, and when it was made: what getFrames reads. */const lastExports = new Map<string, { path: string; at: Date; rendered: AxcutDocument }>();+ /** getFrames when no export shows the project as it is (`docs/shader-effects.md`, E.4): its+ * frames composed as its export would compose them, on the export's default frame grid,+ * without encoding a video. `null` when the addon cannot render them: the export decides. */+ async function renderProjectFrames(+ document: AxcutDocument,+ atSec: number[] | null,+ count: number,+ why: string,+ ): Promise<CallToolResult | null> {+ if (!deps.renderFrames) return null;+ const clips = buildNativeClipList(document);+ if (clips.length === 0) return text("This project's timeline has no clip to show.", true);+ const rulerEndSec = Math.max(0, ...document.timeline.clips.map((clip) => clip.timelineEndSec));+ const times = frameTimes(rulerEndSec, atSec, count);+ // The output frame showing at that moment: the last one that starts at or before it.+ const indices = times.map((sec) =>+ Math.floor(programmeSecOf(document, sec) * EXPORT_FPS + 1e-6),+ );+ let frames: Awaited<ReturnType<NonNullable<MediaToolDeps["renderFrames"]>>>;+ try {+ const scene = JSON.stringify(buildSceneDescription(document));+ frames = await deps.renderFrames(clips, scene, EXPORT_FPS, indices, FRAME_WIDTH);+ } catch (error) {+ const message = error instanceof Error ? error.message : String(error);+ return text(`The frames could not be rendered: ${message}`, true);+ }+ if (!frames) return null;+ const byIndex = new Map(frames.map((frame) => [frame.index, frame]));+ const content: CallToolResult["content"] = [+ {+ type: "text",+ text: `${times.length} frame(s) rendered from the project as its export would show them: ${why}.`,+ },+ ];+ times.forEach((sec, i) => {+ content.push({ type: "text", text: `${sec.toFixed(2)}s on the ruler` });+ const frame = byIndex.get(indices[i]);+ if (frame) {+ content.push({+ type: "image",+ data: frame.jpeg.toString("base64"),+ mimeType: "image/jpeg",+ });+ }+ });+ return { content };+ }+/** A fresh project's checkpoint, so a client can edit it at once and take it back in one step. */async function checkpointOf(projectId: string): Promise<string | null> {// Never fails the call: the project exists either way, and its id is the answer.@@ -767,7 +832,7 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {// The Export dialog's defaults, not whatever an older editor stored.// A GIF carries no quality: the export runner refuses one, and gifSize sizes it.quality: gif ? null : (args.quality ?? "good"),- mp4FrameRate: args.fps ?? 60,+ mp4FrameRate: args.fps ?? EXPORT_FPS,gifFrameRate: args.gifFrameRate ?? 15,gifSizePreset: args.gifSize ?? "medium",gifLoop: args.gifLoop ?? true,@@ -828,10 +893,13 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {{description: [`Look at a video as still images, at most ${MAX_FRAMES} per call. Ask for the moments you need to check, such as each zoom and each callout.`,- "To check an edit, pass projectId: you get the project's last export, and atSec are the ruler seconds the document tools use, mapped through the trims and speed regions to the moment the export plays them. With path instead, you get any video file (a take's videoPath, say) and atSec are that file's own seconds.",+ "To check an edit, pass projectId: atSec are the ruler seconds the document tools use, mapped through the trims and speed regions to the moment the export plays them. You get the project's last export, or, when it has none or has changed since, the frames its export would show, rendered from the project without exporting it. With path instead, you get any video file (a take's videoPath, say) and atSec are that file's own seconds.",].join("\n"),inputSchema: z.object({- projectId: z.string().optional().describe("Look at this project's last export."),+ projectId: z+ .string()+ .optional()+ .describe("Look at this project: its last export, or the frames an export would show."),path: z.string().optional()@@ -866,6 +934,23 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {let note = "";if (args.projectId !== undefined) {const exported = lastExports.get(args.projectId);+ // A project deleted since still left its export on disk: show it, unflagged.+ const current = await deps.documents.getProject(args.projectId).catch(() => null);+ const stale =+ exported !== undefined &&+ current !== null &&+ Date.parse(current.project.updatedAt) > exported.at.getTime();+ // No export shows the project as it is: render its frames as an export would.+ if (current && (!exported || stale)) {+ const why = exported ? "it changed after its last export" : "it has no export yet";+ const rendered = await renderProjectFrames(+ current,+ args.atSec ?? null,+ args.count ?? 6,+ why,+ );+ if (rendered) return rendered;+ }if (!exported) {return text("This project has no export yet: call exportVideo first.", true);}@@ -875,9 +960,7 @@ export function createMediaTools(deps: MediaToolDeps): MediaTools {videoPath = exported.path;toVideoSec = (sec) => programmeSecOf(rendered, sec);rulerEndSec = Math.max(0, ...rendered.timeline.clips.map((clip) => clip.timelineEndSec));- // A project deleted since still left its export on disk: show it, unflagged.- const current = await deps.documents.getProject(args.projectId).catch(() => null);- if (current && Date.parse(current.project.updatedAt) > exported.at.getTime()) {+ if (stale) {note = " The project changed after this export: export it again to see the changes.";}} else {diff --git a/electron/mcp/stdio-server.ts b/electron/mcp/stdio-server.tsindex 54dbfdc63..e38e66ece 100644--- a/electron/mcp/stdio-server.ts+++ b/electron/mcp/stdio-server.ts@@ -31,6 +31,7 @@ export async function serveMcpOverStdio(options: {readCursorTelemetry: MediaToolDeps["readCursor"];runners: MediaToolDeps["runners"];openVideo: MediaToolDeps["openVideo"];+ renderFrames?: MediaToolDeps["renderFrames"];/** The client went away: nothing is left to serve. */onClose(): void;}): Promise<void> {@@ -52,6 +53,7 @@ export async function serveMcpOverStdio(options: {runners: options.runners,documents: options.documents,openVideo: options.openVideo,+ renderFrames: options.renderFrames,readCursor: options.readCursorTelemetry,runTool,});diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.tsindex 95bff7ad0..34f4004d8 100644--- a/electron/native-bridge/services/compositorViewService.ts+++ b/electron/native-bridge/services/compositorViewService.ts@@ -850,6 +850,20 @@ export class CompositorViewService {);}+ /** Frames of a timeline as an export composes them, without the export: see `renderFrames`+ * in the addon. `null` when this addon cannot render them. */+ async renderFrames(+ clips: ClipInput[],+ sceneJson: string,+ fps: number,+ frames: number[],+ maxWidth: number,+ ): Promise<Array<{ index: number; width: number; height: number; jpeg: Buffer }> | null> {+ const addon = this.ensureAddon();+ if (!addon || typeof addon.renderFrames !== "function") return null;+ return addon.renderFrames(clips, resolveSceneAssetPaths(sceneJson), fps, frames, maxWidth);+ }+/** `startGifExport` for MP4, except that an addon built before MP4 cancellation still* exports: the main export must not fail over a missing Cancel. That job runs to the end* and refuses to cancel. */diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.tsindex 501a41c73..1b1ad5e14 100644--- a/electron/native/compositor-view/addon.d.ts+++ b/electron/native/compositor-view/addon.d.ts@@ -248,6 +248,17 @@ export interface CompositorViewAddon {* a control: that addon's MP4 export cannot be cancelled. */createMp4ExportControl?(): object;cancelMp4Export?(control: object): boolean;+ /** Frames of the timeline composed as `exportMulti` composes them, without encoding a video:+ * getFrames when the export is missing or stale. Same clips and scene as `exportMulti`, sized+ * by the scene's output. `frames` are indices on the `fps` grid, and one past the timeline's+ * end renders nothing. JPEG, at most `maxWidth` wide. Absent from older builds. */+ renderFrames?(+ clips: ClipInput[],+ sceneJson: string | undefined,+ fps: number,+ frames: number[],+ maxWidth: number,+ ): Promise<Array<{ index: number; width: number; height: number; jpeg: Buffer }>>;/** Native GIF export. Identical inputs to `exportMulti` — same clips, same* scene — because it is the same render: both drive `walk_composited_timeline`* in the compositor crate and differ only in the encoder. Cursor, background,diff --git a/skills/openscreen-demo/SKILL.md b/skills/openscreen-demo/SKILL.mdindex b6450dabb..2bfdfb242 100644--- a/skills/openscreen-demo/SKILL.md+++ b/skills/openscreen-demo/SKILL.md@@ -103,7 +103,7 @@ Keep the rest as it is: the background, padding, shadow and cursor style are cho## 7. Export and review1. `exportVideo` with the `projectId` and an absolute path: resolve, say, `demo/demo.mp4` against your working directory first, since the server in the app refuses a relative path. For a pull request video, pass `quality: "medium"` (720p) to keep the file small. For a GIF (a README, say), end the path in `.gif` and pass `gifSize: "small"` (480p).-2. `getFrames` with the `projectId`, at the ruler seconds of your beats.+2. `getFrames` with the `projectId`, at the ruler seconds of your beats. While you edit, you can call it before exporting: without a current export it renders the frames the export would show.3. Check every frame:- Is each callout readable and clear of the action?- Is each zoom on the thing that changes?diff --git a/src/cli/CliExportRunner.tsx b/src/cli/CliExportRunner.tsxindex 685f80f14..97d3e6efa 100644--- a/src/cli/CliExportRunner.tsx+++ b/src/cli/CliExportRunner.tsx@@ -13,22 +13,18 @@ import {pickExtremeDims,resolveAspectRatioValue,} from "@/lib/ai-edition/document/outputFormat";-import type { AxcutDocument } from "@/lib/ai-edition/schema";import { getEditorSettings } from "@/lib/ai-edition/store/editorSettings";import {appendAutoZoomSuggestions,collectAutoZoomSuggestionsForDocument,} from "@/lib/ai-edition/timeline/apply-auto-zooms";-import { assetCameraSource } from "@/lib/ai-edition/timeline/camera";-import { resolveClipSourceEndSec } from "@/lib/ai-edition/timeline/clipDuration";import type { CliDoneResult, CliExportRequest } from "@/lib/cliContracts";import { GIF_SIZE_PRESETS, type GifSizePreset } from "@/lib/exporter";import { calculateMp4ExportSettings } from "@/lib/exporter/mp4ExportSettings";import { outputFrameCount } from "@/lib/exporter/outputFrameCount";import { mixVoiceoverIntoVideo } from "@/lib/exporter/voiceoverMix";import { exportGifNative, exportMultiNative, nativeBridgeClient } from "@/native";-import type { CompositorClipInput } from "@/native/contracts";-import { buildSceneDescription, resolveVisibleClips } from "@/native/sceneDescription";+import { buildNativeClipList, buildSceneDescription } from "@/native/sceneDescription";import { type ExportProject, loadDocumentProject } from "./exportProject";import { loadCliProject, probeVideoDimensions } from "./loadCliProject";@@ -36,31 +32,6 @@ function replaceExtension(filePath: string, newExtension: string): string {return filePath.replace(/\.(openscreen|json)$/i, "") + newExtension;}-/** Mirrors ExportDialog.buildNativeClipList: trim-narrowed visible clips mapped- * onto the native multiclip contract. Kept in lock-step with- * buildSceneDescription so export and scene agree on the clip stream. */-function buildNativeClipList(axcutDocument: AxcutDocument): CompositorClipInput[] {- const assetById = new Map(axcutDocument.assets.map((asset) => [asset.id, asset]));- return resolveVisibleClips(axcutDocument).flatMap((clip) => {- const asset = assetById.get(clip.assetId);- if (!asset?.originalPath) {- return [];- }- const camera = assetCameraSource(asset);- const sourceEndSec = resolveClipSourceEndSec(clip, asset);- return [- {- screenPath: asset.originalPath,- webcamPath: camera.path,- sourceStartSec: clip.sourceStartSec,- sourceEndSec,- webcamOffsetSec: camera.offsetSec,- hasAudio: true,- },- ];- });-}-/** Mirrors ExportDialog.gifOutputDims: from the "Source" size, cap height at the preset, keep even. */function gifOutputDims(preset: GifSizePreset,diff --git a/src/native/sceneDescription.ts b/src/native/sceneDescription.tsindex 8c8727ee7..ae1cdbe4c 100644--- a/src/native/sceneDescription.ts+++ b/src/native/sceneDescription.ts@@ -687,6 +687,40 @@ export function resolveVisibleClips(document: AxcutDocument): PlaybackSegment[].filter((clip) => clipAssetIsResolvable(clip, assetById));}+/**+ * The clip list a native export renders: the trim-narrowed visible clips, mapped onto the native+ * multiclip contract. `openscreen export` and getFrames share it, beside `buildSceneDescription`+ * so that the clip stream and the scene agree.+ * ponytail: the Export dialog keeps an identical copy, which its tests' module mocks rely on.+ */+export function buildNativeClipList(document: AxcutDocument): CompositorClipInput[] {+ const assetById = new Map(document.assets.map((a) => [a.id, a]));+ return resolveVisibleClips(document).flatMap((clip) => {+ const asset = assetById.get(clip.assetId);+ if (!asset?.originalPath) {+ return [];+ }+ const camera = assetCameraSource(asset);+ // sourceEndSec is optional in the schema (unknown until probed) — fall back through+ // the single canonical precedence used by every consumer (clip.probe → asset.duration+ // → timeline-length guess). See `resolveClipSourceEndSec` for the full order.+ const sourceEndSec = resolveClipSourceEndSec(clip, asset);+ // ponytail: `hasAudio` stays optimistic for the same reason as in+ // `buildSceneDescription` — nothing populates `asset.audio` yet, and the+ // native side degrades cleanly on a stream-less file.+ return [+ {+ screenPath: asset.originalPath,+ webcamPath: camera.path,+ sourceStartSec: clip.sourceStartSec,+ sourceEndSec,+ webcamOffsetSec: camera.offsetSec,+ hasAudio: true,+ },+ ];+ });+}+/** A clip's screen source size in pixels: its recording × its crop (see `screenSourceSizeOf`). */function screenSourceSize(video: { width?: number; height?: number } | null | undefined,diff --git a/technical-documentation/architecture/export-pipeline.md b/technical-documentation/architecture/export-pipeline.mdindex e43778772..8d773d99f 100644--- a/technical-documentation/architecture/export-pipeline.md+++ b/technical-documentation/architecture/export-pipeline.md@@ -29,6 +29,8 @@ flowchart LR## Segment loop+The walk itself is `walk_composited_timeline` ([`timeline_walk.rs`](../../crates/compositor/src/timeline_walk.rs)), shared by the MP4 and GIF exports. A third caller renders chosen frames without an export: `render_frames`, behind the addon's `renderFrames`, which MCP `getFrames` uses when a project has no current export. It walks the same timeline and composes only the frames asked for. The decoders skip the rest and catch up with a keyframe seek, which never changes the source frame shown (see [docs/shader-effects.md](../../docs/shader-effects.md), E.4).+The renderer hands the addon `exportMulti(clips, outPath, sceneJson, params)`through `compositorViewService.exportMulti`([`compositorViewService.ts:425`](../../electron/native-bridge/services/compositorViewService.ts)).diff --git a/technical-documentation/architecture/mcp-server.md b/technical-documentation/architecture/mcp-server.mdindex 528562d92..e9ac92564 100644--- a/technical-documentation/architecture/mcp-server.md+++ b/technical-documentation/architecture/mcp-server.md@@ -175,6 +175,7 @@ Then both servers carry on alike:- `exportVideo` renders in a hidden `cli-export` runner, the same in both processes: in the app, `listSources` and `exportVideo` run the CLI's runners (`createRunnerHost` with `createRunnerWindow`), whose IPC the app registers on first use. The runner reads the project by id (`CliExportRequest.projectId`, the editor's own `aiEdition.get`), not by path: opening a file by path moves the main process's current project and recording-session slot, which in the app is the hand-off from the recorder to the editor.- `exportVideo` takes the Export dialog's options and choices: `quality` and `fps` (24, 30, 60) for an MP4, `gifFrameRate`, `gifSize` and `gifLoop` for a GIF, the dialog's defaults when left out. An option of the other format is refused rather than ignored. Hiding the desktop icons is not a `startRecording` option: the HUD has no such control, so it follows the user's Record mode setting on both servers. Nor are the CLI's `--auto-zoom` and `--audio`: zooms and a voiceover are edits of the project, and an export that added them would not be the project `getFrames` maps through.- `exportVideo` and `getFrames` read projects; they write none. Over HTTP their paths must be absolute.+- `getFrames` with a `projectId` reads the project's last export while it is current. With no export, or one the project changed after, it renders the frames from the project instead (`renderFrames` in the addon, through the export's own walk), so an agent checks an edit without exporting. See [docs/shader-effects.md](../../docs/shader-effects.md), E.4.- `createProject` makes a project with `DocumentService.createProject`, so it starts in the look new projects take (the preset marked for them, else the last edited project's), as the editor's New does. With a `videoPath` it lays the video out with `createProjectFromVideo`, the path a stdio take goes through: one clip, measured, and the zooms on its clicks when a pointer track sits beside it. Either way it returns a checkpoint, as `stopRecording` does, so the client can edit the closed project at once. It runs where the media tools run (in the CLI process for `openscreen mcp`): a new project is one no editor holds. **MCP edits** gates it.## `openscreen mcp` and the running app