Skip to content

A virtual camera films an AI agent's screen while a tracker locks onto every glyph. Two fframes passes, one SkSL shader, all sound synthesized.

7.7 KB
Raw Blame

Contributions to fframes (2026-10-09)

Opened from @mrsarac on dmtrKovalenko/fframes:

  • A. #209, bug report.
  • B. #210, feature idea, with C, the offer to contribute this project as an example.

The texts below are the submitted versions (hard-wrapped here, unwrapped on GitHub).


A. Bug report (#209)

Title: Skia render: an error late in the render surfaces only after unfinished segments drain their x264 lookahead (looks like a hang)

Body:

What happens

When render_frame fails late in a full render (here: a frame without a root <svg>, so ParserError(NoRootNode)), the Skia pipeline stops, but the error is printed only after SegmentWriter is dropped. Dropping every unfinished segment flushes its encoder on the main thread, and with a slow x264 preset that drains the whole lookahead of each segment, one segment after another. Meanwhile one core sits at ~99 %, the progress output has stopped and nothing is printed, so it looks like a hang.

Case (fframes 1.2.0, Skia Metal, M3 Pro) Time to exit
Repro below, no bad frame 12.5 s (success)
Repro, BAD_FRAME=290 (of 300) 22 s, then ParserError(NoRootNode)
Repro, BAD_FRAME=30 or 150 < 1 s (fine)
Real project (1080p, heavy SkSL pass, preset=slow, tune=grain), no bad frame 28 s (success)
Same project, bad frame at 170 of 180 86 s, then ParserError(NoRootNode)

A short range render containing the same frame (render 160..180) fails within seconds.

Repro

[dependencies]
fframes = { version = "=1.2.0", features = ["cli", "h264", "libav-agree-gpl"] }
fframes_skia_renderer = { version = "=1.2.0", features = ["metal"] }
use fframes::{AudioMap, Color, Duration, EncoderOptions, FFramesContext, Frame, RenderOptions, Svgr, Video, cli};
use fframes_skia_renderer::{SkiaFFramesRenderer, SkiaPipelineConfig, metal::SkiaMetalCtx};

struct Repro { bad_frame: Option<usize> }

impl Video for Repro {
    const FPS: usize = 30;
    const WIDTH: usize = 1920;
    const HEIGHT: usize = 1080;
    const BACKGROUND_COLOR: Color = Color::BLACK;
    fn duration(&self) -> Duration<'_> { Duration::Frames(300) }
    fn audio(&self) -> AudioMap<'_> { AudioMap::none() }

    fn render_frame<'a>(&'a self, frame: Frame, _ctx: &FFramesContext<'a, '_>) -> Svgr<'a> {
        if self.bad_frame.is_some_and(|n| frame.index >= n) {
            return Svgr::empty(); // no root <svg>
        }
        // busy, changing content so the encoder has real work
        let dots: Vec<Svgr> = (0..1500usize).map(|i| {
            let h = (i * 7919 + frame.index * 104_729) % 99_991;
            let (x, y) = ((h * 31) % 1920, (h * 17) % 1080);
            let fill = format!("#{:06x}", h.wrapping_mul(2_654_435_761) & 0xff_ffff);
            fframes::svgr!(<rect x={x} y={y} width="14" height="14" fill={fill} />)
        }).collect();
        fframes::svgr!(<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080">
            <rect width="1920" height="1080" fill="#101820" />
            {dots}
        </svg>)
    }
}

fn main() -> std::process::ExitCode {
    let video = Repro { bad_frame: std::env::var("BAD_FRAME").ok().and_then(|v| v.parse().ok()) };
    let options = RenderOptions {
        video_encoder_options: EncoderOptions {
            preferred_encoder: Some("libx264"),
            codec_params: Some(&[("crf", "16"), ("preset", "slow")]),
            ..Default::default()
        },
        ..Default::default()
    };
    let gpu = SkiaMetalCtx::new(1920, 1080).expect("GPU context");
    let backend = SkiaFFramesRenderer::new_metal(&gpu, SkiaPipelineConfig::default()).expect("skia");
    cli::new(&video, options).backend(backend).run()
}
cargo run --release -- render -o ok.mp4                  # 12.5 s
BAD_FRAME=290 cargo run --release -- render -o bad.mp4   # 22 s, then the error

Where the time goes

sample of the stuck process (main thread, 2462/2462 samples):

fframes_skia_renderer::skia_pipeline::render
  core::ptr::drop_in_place<fframes::renderer::segment_writer::SegmentWriter>
    alloc::sync::Arc<T,A>::drop_slow
      core::ptr::drop_in_place<UnsafeCell<Option<segment_writer::Segment>>>
        libx264.165.dylib  (encoding)

Encoder::drop (fframes/src/renderer/encoder.rs) calls avcodec_flush_buffers on the video encoder. For libx264 that drains every delayed frame. A segment that finished normally was already drained by flush_stream in SegmentWriter::submit_with, so the drop is cheap there. Only a segment that never finished, whose file is thrown away anyway, pays for the drain.

Possible fix

On the error path, discard open segments without draining them, for example a SegmentWriter::abort() that drops the open encoders with a flag that skips avcodec_flush_buffers and av_write_trailer and removes the temp files. Then skia_pipeline::render calls it before returning Err. I'm happy to send a PR if this direction is OK.

Related docs note

skills/fframes-video/references/api.md shows let Some(clip) = frame.get_synced_video_frame(..) else { return Svgr::empty() };. That's fine inside a Scene, but at the Video::render_frame level it produces exactly this NoRootNode error. A one-line hint ("at the video level return a root <svg>, e.g. a black rect") would save the next person the search.

Environment: fframes 1.2.0, fframes_skia_renderer 1.2.0 (Metal), macOS 26.6.2, Apple M3 Pro, rustc 1.91.1, Homebrew FFmpeg 9.0.1, libx264.165. On 1.0.0 I saw the same stop with 0 % CPU and no exit after 10 min; I did not re-check 1.0.0 after finding the cause.


B. Feature idea (#210)

Title: Feature idea: feed a shader with an SVG subtree rendered in the same frame (multi-pass in one video)

Body:

Motivation

I recreated @mnowakdesign's tracking-HUD look (the same one you posted) with a camera that films the screen: perspective tilt, depth of field focused on the tracked glyph, LED grid, bloom, chromatic aberration, per-scene palette, all in one SkSL pass.

The shader needs the flat screen as its input, so today it takes two fframes projects:

  1. flat renders the terminal + tracker at 3840×2160 to an mp4 (crf 8).
  2. lens reads it with get_synced_video_frame(..).into_image() and binds it as iChannel0.

That works (thanks for the video-frames-in-shaders support), but it costs a 4K h264 round trip per frame (disk, decode, compression artefacts under 3× magnification), two binaries, and a shared crate just so both passes agree on the timeline.

Idea

Let a shader take an SVG subtree of the same frame as a child image, rendered offscreen first:

let screen = fframes::svgr!(<g>{terminal}{tracker}</g>);
let layer = self.lens.draw(&frame, ShaderUniforms::new()
    .layer("iChannel0", screen, 3840, 2160)   // rendered offscreen, bound as `uniform shader`
    .float("uZoom", zoom));
fframes::svgr!(<image href={layer.href()} width="1920" height="1080" />)

On Skia this could stay on the GPU: draw the subtree into an offscreen surface and bind its image snapshot as the child shader, with no CPU readback. On the CPU backend it could fall back to rasterizing the subtree. Is this something you'd want in fframes? I can help with the API, an example or tests.

Example (C)

The project is at https://github.com/mrsarac/ff-tracking (MIT): 6 s, 12 scenes, a two-pass render plus a synthesized sound track (~40 s end to end on an M3 Pro). If it's useful, I can contribute a trimmed version as examples/screen-camera, now as a two-pass example or later as the single-pass demo of the feature above. Video: https://x.com/0xsarac/status/2108515856544317489