Skip to content

A virtual camera films an AI agent's screen while a tracker locks onto every glyph. Two fframes passes, one SkSL shader, all sound synthesized.

docs: drafts for fframes issues (bug report, feature idea)

Mustafa0d6c3a4c3e21
diff --git a/docs/contrib/2026-10-09-fframes-taslaklar.md b/docs/contrib/2026-10-09-fframes-taslaklar.md
new file mode 100644
index 0000000..d1bf6ce
--- /dev/null
+++ b/docs/contrib/2026-10-09-fframes-taslaklar.md
@@ -0,0 +1,180 @@
+# fframes katkı taslakları (2026-10-09)
+
+Hedef: github.com/dmtrKovalenko/fframes, hesap @mrsarac. Durum: **taslak, gönderilmedi.**
+Sıra: önce A (hata), sonra B (özellik önerisi, içinde C örnek teklifi).
+C için ff-tracking reposunun public olması ve bir lisans (MIT) alması gerekir.
+
+---
+
+## A. Issue: hata
+
+**Başlık:**
+Skia render: an error late in the render surfaces only after unfinished segments drain their x264 lookahead (looks like a hang)
+
+**Gövde:**
+
+### What happens
+
+When `render_frame` fails late in a full render (here: a frame without a root `<svg>`, so
+`ParserError(NoRootNode)`), the Skia pipeline stops, but the error is printed only after
+`SegmentWriter` is dropped. Dropping every unfinished segment flushes its encoder on the main
+thread, and with a slow x264 preset that drains the whole lookahead of each segment, one
+segment after another. Meanwhile one core sits at ~99 %, the progress output has stopped and
+nothing is printed, so it looks like a hang.
+
+| Case (fframes 1.2.0, Skia Metal, M3 Pro) | Time to exit |
+|---|---|
+| Repro below, no bad frame | 12.5 s (success) |
+| Repro, `BAD_FRAME=290` (of 300) | 22 s, then `ParserError(NoRootNode)` |
+| Repro, `BAD_FRAME=30` or `150` | < 1 s (fine) |
+| Real project (1080p, heavy SkSL pass, `preset=slow`, `tune=grain`), no bad frame | 28 s (success) |
+| Same project, bad frame at 170 of 180 | **86 s**, then `ParserError(NoRootNode)` |
+
+A short range render containing the same frame (`render 160..180`) fails within seconds.
+
+### Repro
+
+```toml
+[dependencies]
+fframes = { version = "=1.2.0", features = ["cli", "h264", "libav-agree-gpl"] }
+fframes_skia_renderer = { version = "=1.2.0", features = ["metal"] }
+```
+
+```rust
+use fframes::{AudioMap, Color, Duration, EncoderOptions, FFramesContext, Frame, RenderOptions, Svgr, Video, cli};
+use fframes_skia_renderer::{SkiaFFramesRenderer, SkiaPipelineConfig, metal::SkiaMetalCtx};
+
+struct Repro { bad_frame: Option<usize> }
+
+impl Video for Repro {
+ const FPS: usize = 30;
+ const WIDTH: usize = 1920;
+ const HEIGHT: usize = 1080;
+ const BACKGROUND_COLOR: Color = Color::BLACK;
+ fn duration(&self) -> Duration<'_> { Duration::Frames(300) }
+ fn audio(&self) -> AudioMap<'_> { AudioMap::none() }
+
+ fn render_frame<'a>(&'a self, frame: Frame, _ctx: &FFramesContext<'a, '_>) -> Svgr<'a> {
+ if self.bad_frame.is_some_and(|n| frame.index >= n) {
+ return Svgr::empty(); // no root <svg>
+ }
+ // busy, changing content so the encoder has real work
+ let dots: Vec<Svgr> = (0..1500usize).map(|i| {
+ let h = (i * 7919 + frame.index * 104_729) % 99_991;
+ let (x, y) = ((h * 31) % 1920, (h * 17) % 1080);
+ let fill = format!("#{:06x}", h.wrapping_mul(2_654_435_761) & 0xff_ffff);
+ fframes::svgr!(<rect x={x} y={y} width="14" height="14" fill={fill} />)
+ }).collect();
+ fframes::svgr!(<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080">
+ <rect width="1920" height="1080" fill="#101820" />
+ {dots}
+ </svg>)
+ }
+}
+
+fn main() -> std::process::ExitCode {
+ let video = Repro { bad_frame: std::env::var("BAD_FRAME").ok().and_then(|v| v.parse().ok()) };
+ let options = RenderOptions {
+ video_encoder_options: EncoderOptions {
+ preferred_encoder: Some("libx264"),
+ codec_params: Some(&[("crf", "16"), ("preset", "slow")]),
+ ..Default::default()
+ },
+ ..Default::default()
+ };
+ let gpu = SkiaMetalCtx::new(1920, 1080).expect("GPU context");
+ let backend = SkiaFFramesRenderer::new_metal(&gpu, SkiaPipelineConfig::default()).expect("skia");
+ cli::new(&video, options).backend(backend).run()
+}
+```
+
+```bash
+cargo run --release -- render -o ok.mp4 # 12.5 s
+BAD_FRAME=290 cargo run --release -- render -o bad.mp4 # 22 s, then the error
+```
+
+### Where the time goes
+
+`sample` of the stuck process (main thread, 2462/2462 samples):
+
+```
+fframes_skia_renderer::skia_pipeline::render
+ core::ptr::drop_in_place<fframes::renderer::segment_writer::SegmentWriter>
+ alloc::sync::Arc<T,A>::drop_slow
+ core::ptr::drop_in_place<UnsafeCell<Option<segment_writer::Segment>>>
+ libx264.165.dylib (encoding)
+```
+
+`Encoder::drop` (`fframes/src/renderer/encoder.rs`) calls `avcodec_flush_buffers` on the video
+encoder. For libx264 that drains every delayed frame. A segment that finished normally was
+already drained by `flush_stream` in `SegmentWriter::submit_with`, so the drop is cheap there.
+Only a segment that never finished, whose file is thrown away anyway, pays for the drain.
+
+### Possible fix
+
+On the error path, discard open segments without draining them, for example a
+`SegmentWriter::abort()` that drops the open encoders with a flag that skips
+`avcodec_flush_buffers` and `av_write_trailer` and removes the temp files. Then
+`skia_pipeline::render` calls it before returning `Err`. I'm happy to send a PR if this
+direction is OK.
+
+### Related docs note
+
+`skills/fframes-video/references/api.md` shows
+`let Some(clip) = frame.get_synced_video_frame(..) else { return Svgr::empty() };`.
+That's fine inside a `Scene`, but at the `Video::render_frame` level it produces exactly
+this `NoRootNode` error. A one-line hint ("at the video level return a root `<svg>`, e.g.
+a black rect") would save the next person the search.
+
+Environment: fframes 1.2.0, fframes_skia_renderer 1.2.0 (Metal), macOS 26.6.2, Apple M3
+Pro, rustc 1.91.1, Homebrew FFmpeg 9.0.1, libx264.165. On 1.0.0 I saw the same stop with
+0 % CPU and no exit after 10 min; I did not re-check 1.0.0 after finding the cause.
+
+---
+
+## B. Issue: özellik önerisi (içinde C)
+
+**Başlık:**
+Feature idea: feed a shader with an SVG subtree rendered in the same frame (multi-pass in one video)
+
+**Gövde:**
+
+### Motivation
+
+I recreated @mnowakdesign's tracking-HUD look (the same one you posted) with a camera that
+films the screen: perspective tilt, depth of field focused on the tracked glyph, LED grid,
+bloom, chromatic aberration, per-scene palette, all in one SkSL pass.
+
+The shader needs the flat screen as its input, so today it takes two fframes projects:
+
+1. `flat` renders the terminal + tracker at 3840×2160 to an mp4 (crf 8).
+2. `lens` reads it with `get_synced_video_frame(..).into_image()` and binds it as `iChannel0`.
+
+That works (thanks for the video-frames-in-shaders support), but it costs a 4K h264 round
+trip per frame (disk, decode, compression artefacts under 3× magnification), two binaries,
+and a shared crate just so both passes agree on the timeline.
+
+### Idea
+
+Let a shader take an SVG subtree of the same frame as a child image, rendered offscreen
+first:
+
+```rust
+let screen = fframes::svgr!(<g>{terminal}{tracker}</g>);
+let layer = self.lens.draw(&frame, ShaderUniforms::new()
+ .layer("iChannel0", screen, 3840, 2160) // rendered offscreen, bound as `uniform shader`
+ .float("uZoom", zoom));
+fframes::svgr!(<image href={layer.href()} width="1920" height="1080" />)
+```
+
+On Skia this could stay on the GPU: draw the subtree into an offscreen surface and bind its
+image snapshot as the child shader, with no CPU readback. On the CPU backend it could fall
+back to rasterizing the subtree. Is this something you'd want in fframes? I can help with
+the API, an example or tests.
+
+### Example (C)
+
+The project is at https://github.com/mrsarac/ff-tracking (MIT): 6 s, 12 scenes, a two-pass
+render plus a synthesized sound track (~40 s end to end on an M3 Pro). If it's useful, I can
+contribute a trimmed version as `examples/screen-camera`, now as a two-pass example or later
+as the single-pass demo of the feature above. Video: https://x.com/0xsarac/status/2108515856544317489