Skip to content

armgit / Cap

Public Imported from GitHub
0

Open source Loom alternative. Beautiful, shareable screen recordings.

chore: drop the benchmark notes and parity scripts

Richie McIlroya884fb64dc6d
diff --git a/apps/desktop-gpui/docs/video-only-export-performance.md b/apps/desktop-gpui/docs/video-only-export-performance.md
deleted file mode 100644
index 6f4ac85bf..000000000
--- a/apps/desktop-gpui/docs/video-only-export-performance.md
+++ /dev/null
@@ -1,97 +0,0 @@
-# Video-only export preparation
-
-GIF and cursor-only MOV export no audio. Their GPUI preparation previously
-waited for every recording mic/system track to decode into retained PCM.
-`build_video_only()` uses the existing segment loader without audio. Generic
-`build()` and MP4 preparation keep their existing behavior. Recording metadata
-still probes audio files; this change removes full PCM decoding and retention,
-not all audio I/O.
-
-## Measurements
-
-Baseline source: `68d6c31b44db6b3c68d8da7510b159c538c06d49`.
-Measured on Apple M5 Pro, macOS 26.6.2 (25G83), Rust 1.97.1.
-Three paired repetitions per format/input, alternating pair order, separate
-processes, same release binary with baseline and video-only modes. The binary
-used the GPUI release profile: optimization level 3, thin LTO, one codegen unit.
-It called actual `ExporterBuilder` and GIF/ProRes encoders. The timer includes
-builder setup through encoder finish, excluding configuration JSON loading,
-fixture generation, validation, and GPUI dialogs/UI.
-
-Synthetic inputs: 30-second control and 1800-second stress recordings, each
-with two stereo AAC tracks at 48 kHz and a four-second trimmed timeline.
-Output is 640x360 at 15 fps, 60 decoded frames. Cursor fixtures reuse checked-in
-CLI assets with normalized movement, a shape change, and one click.
-
-| Source | Format | Median total, baseline to candidate | Median peak RSS, baseline to candidate |
-|---|---|---|---|
-| 30 s | GIF | 0.209761 to 0.197071 s | 138.98 to 122.94 MiB |
-| 30 s | Cursor MOV | 0.460177 to 0.450832 s | 88.77 to 66.45 MiB |
-| 1800 s | GIF | 0.871047 to 0.209086 s | 1458.03 to 123.23 MiB |
-| 1800 s | Cursor MOV | 1.120212 to 0.479418 s | 1397.25 to 69.45 MiB |
-
-All 12 baseline/candidate pairs passed decoded RGBA checksums, frame count,
-PTS/order/duration and stream metadata parity, with no audio stream. All six
-MOV pairs additionally passed decoded alpha parity and visible moving alpha.
-Results establish the benefit for these trimmed synthetic inputs; they do not
-predict export speedups for every recording. Short controls show modest gains.
-
-## Reproduce
-
-Requires the project's native FFmpeg setup, `ffmpeg`, `ffprobe`, and Python 3.
-Use the GPUI lockfile and workspace to retain its dependency pins. Build once,
-then time separate binary processes; exclude Cargo/build time from results.
-
-```sh
-python3 scripts/prepare-video-only-export-fixture.py /tmp/video-only-1800.cap --source-seconds 1800 --export-seconds 4 --with-cursor
-cargo +stable run --manifest-path apps/desktop-gpui/Cargo.toml --example video-only-preparation --release -- --project /tmp/video-only-1800.cap --output /tmp/video-only-baseline.gif --mode baseline --format gif
-cargo +stable run --manifest-path apps/desktop-gpui/Cargo.toml --example video-only-preparation --release -- --project /tmp/video-only-1800.cap --output /tmp/video-only-candidate.gif --mode video-only --format gif
-python3 scripts/check-video-only-export-parity.py /tmp/video-only-baseline.gif /tmp/video-only-candidate.gif
-```
-
-For MOV use `--format cursor-mov`, `.mov` output paths, and parity flags
-`--alpha --require-visible-alpha`. Repeat with `--source-seconds 30` for the
-control. Use unique output paths because the example refuses existing files.
-Capture JSON stdout and `/usr/bin/time -l` stderr for peak RSS on macOS.
-`--build-only` isolates preparation but is not a complete-export benchmark.
-The first progress timestamp measures a callback, not an encoded packet.
-
-Corrupt unused audio can now produce a video-only export after a metadata
-warning instead of failing the complete PCM/EOF wait. Display-video errors
-still fail preparation. Native GPUI UI timing and cancellation are outside
-these measurements. Separate source-loading checks passed: no-audio succeeds
-in both modes; invalid mic fails baseline and passes video-only; invalid
-display fails both modes.
-
-<details>
-<summary>Raw samples</summary>
-
-```csv
-source_seconds,format,repetition,mode,build_secs,total_secs,peak_rss_mib
-30,Gif,0,Baseline,0.056724625,0.360145084,156.609375
-30,Gif,0,VideoOnly,0.012039167,0.189455042,122.937500
-30,Gif,1,VideoOnly,0.012658167,0.199531333,122.906250
-30,Gif,1,Baseline,0.022580584,0.205054375,138.984375
-30,Gif,2,Baseline,0.020476042,0.209760750,133.125000
-30,Gif,2,VideoOnly,0.009499250,0.197070667,137.000000
-30,CursorMov,0,Baseline,0.021880417,0.460177125,88.765625
-30,CursorMov,0,VideoOnly,0.011225542,0.450831625,66.375000
-30,CursorMov,1,VideoOnly,0.012083208,0.459454000,66.593750
-30,CursorMov,1,Baseline,0.021884542,0.456129417,88.562500
-30,CursorMov,2,Baseline,0.024268625,0.472918166,88.968750
-30,CursorMov,2,VideoOnly,0.009287208,0.446681750,66.453125
-1800,Gif,0,Baseline,0.657632584,0.846303542,1459.031250
-1800,Gif,0,VideoOnly,0.012863292,0.209086459,123.468750
-1800,Gif,1,VideoOnly,0.012166709,0.196155334,117.234375
-1800,Gif,1,Baseline,0.664888292,0.871047209,1458.031250
-1800,Gif,2,Baseline,0.685738792,0.914366584,1450.703125
-1800,Gif,2,VideoOnly,0.013697875,0.216982000,123.234375
-1800,CursorMov,0,Baseline,0.668535250,1.155541958,1397.250000
-1800,CursorMov,0,VideoOnly,0.015395500,0.499813208,69.453125
-1800,CursorMov,1,VideoOnly,0.015545292,0.475296500,68.562500
-1800,CursorMov,1,Baseline,0.642078958,1.118089667,1398.000000
-1800,CursorMov,2,Baseline,0.639165958,1.120211791,1397.203125
-1800,CursorMov,2,VideoOnly,0.015095083,0.479417500,70.343750
-```
-
-</details>
diff --git a/scripts/check-video-only-export-parity.py b/scripts/check-video-only-export-parity.py
deleted file mode 100644
index 85063c61d..000000000
--- a/scripts/check-video-only-export-parity.py
+++ /dev/null
@@ -1,106 +0,0 @@
-import argparse
-import json
-import subprocess
-from pathlib import Path
-
-ALPHA_READ_BYTES = 64 * 1024
-
-
-def read_alpha_extrema(stream):
- minimum = None
- maximum = None
- while True:
- chunk = stream.read(ALPHA_READ_BYTES)
- if not chunk:
- return minimum, maximum
- chunk_minimum = min(chunk)
- chunk_maximum = max(chunk)
- minimum = chunk_minimum if minimum is None else min(minimum, chunk_minimum)
- maximum = chunk_maximum if maximum is None else max(maximum, chunk_maximum)
-
-
-def alpha_extrema(path):
- command = [
- "ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error",
- "-i", str(path), "-map", "0:v:0", "-vf", "alphaextract",
- "-pix_fmt", "gray", "-f", "rawvideo", "-",
- ]
- with subprocess.Popen(command, stdout=subprocess.PIPE) as process:
- try:
- if process.stdout is None:
- raise RuntimeError("FFmpeg alpha output pipe was not created")
- extrema = read_alpha_extrema(process.stdout)
- except BaseException:
- process.kill()
- process.wait()
- raise
- returncode = process.wait()
- if returncode:
- raise subprocess.CalledProcessError(returncode, command)
- return extrema
-
-
-def probe(path):
- raw = subprocess.check_output([
- "ffprobe", "-v", "error", "-count_frames", "-show_streams", "-show_format",
- "-of", "json", str(path),
- ])
- data = json.loads(raw)
- selected = []
- for stream in data["streams"]:
- selected.append({key: stream.get(key) for key in [
- "codec_name", "codec_type", "width", "height", "pix_fmt", "time_base",
- "duration", "nb_read_frames", "avg_frame_rate",
- ]})
- return selected
-
-
-def frames(path, alpha=False):
- command = ["ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error", "-i", str(path),
- "-map", "0:v:0"]
- if alpha:
- command += ["-vf", "alphaextract"]
- else:
- command += ["-pix_fmt", "rgba"]
- command += ["-f", "framemd5", "-"]
- raw = subprocess.check_output(command).decode()
- return [line.strip() for line in raw.splitlines() if line and not line.startswith("#")]
-
-
-def main():
- parser = argparse.ArgumentParser()
- parser.add_argument("baseline", type=Path)
- parser.add_argument("candidate", type=Path)
- parser.add_argument("--alpha", action="store_true")
- parser.add_argument("--require-visible-alpha", action="store_true")
- args = parser.parse_args()
- if args.require_visible_alpha and not args.alpha:
- parser.error("--require-visible-alpha requires --alpha")
- a, b = probe(args.baseline), probe(args.candidate)
- if a != b:
- raise SystemExit(f"Stream metadata differs: {a!r} vs {b!r}")
- if any(stream["codec_type"] == "audio" for stream in a):
- raise SystemExit("Unexpected audio stream in video-only export")
- fa, fb = frames(args.baseline), frames(args.candidate)
- if not fa or fa != fb:
- raise SystemExit("Decoded frame checksums, PTS, durations, or order differ")
- alpha_match = None
- visible_alpha = None
- if args.alpha:
- aa, ab = frames(args.baseline, True), frames(args.candidate, True)
- alpha_match = bool(aa) and aa == ab
- if not alpha_match:
- raise SystemExit("Decoded alpha differs")
- if args.require_visible_alpha:
- minimum, maximum = alpha_extrema(args.baseline)
- hashes = {line.rsplit(",", 1)[-1].strip() for line in aa}
- visible_alpha = minimum == 0 and maximum is not None and maximum > 0 and len(hashes) > 1
- if not visible_alpha:
- raise SystemExit("Cursor alpha is empty, fully opaque, or static")
- print(json.dumps({"decoded_frame_parity": True, "frames": len(fa),
- "streams": a, "alpha_parity": alpha_match,
- "visible_moving_alpha": visible_alpha}))
-
-
-if __name__ == "__main__":
- main()
diff --git a/scripts/prepare-video-only-export-fixture.py b/scripts/prepare-video-only-export-fixture.py
deleted file mode 100644
index 16b074743..000000000
--- a/scripts/prepare-video-only-export-fixture.py
+++ /dev/null
@@ -1,119 +0,0 @@
-import argparse
-import json
-import math
-import shutil
-import subprocess
-from pathlib import Path
-
-
-def add_cursor(fixture, assets_directory):
- fixture = fixture.resolve()
- marker = json.loads((fixture / "SYNTHETIC-FIXTURE.json").read_text())
- if marker.get("synthetic") is not True or marker.get("audio_only"):
- raise SystemExit("Only disposable synthetic full fixtures may be modified")
- meta_path = fixture / "recording-meta.json"
- meta = json.loads(meta_path.read_text())
- if len(meta.get("segments", [])) != 1:
- raise SystemExit("Expected one multi-segment recording entry")
- assets = json.loads((assets_directory / "cursors.json").read_text())
- for cursor_id in ("0", "2"):
- relative = assets[cursor_id]["imagePath"]
- source = assets_directory / f"cursor_{cursor_id}.png"
- destination = fixture / relative
- if destination.exists() and destination.read_bytes() != source.read_bytes():
- raise SystemExit(f"Refusing to replace different cursor asset: {destination}")
- destination.parent.mkdir(parents=True, exist_ok=True)
- shutil.copyfile(source, destination)
- duration = float(marker["export_seconds"])
- sample_count = max(2, math.ceil(duration * 60))
- moves = []
- for index in range(sample_count + 1):
- phase = index / sample_count
- moves.append({
- "active_modifiers": [], "cursor_id": "0" if phase < 0.5 else "2",
- "time_ms": phase * duration * 1000,
- "x": 0.2 + 0.6 * phase,
- "y": 0.5 + 0.2 * math.sin(phase * math.tau),
- })
- click_time_ms = duration * 0.7 * 1000
- clicks = [
- {"active_modifiers": [], "cursor_num": 1, "cursor_id": "2",
- "time_ms": click_time_ms, "down": True},
- {"active_modifiers": [], "cursor_num": 1, "cursor_id": "2",
- "time_ms": click_time_ms + 80, "down": False},
- ]
- cursor_path = fixture / "content/cursor-campaign.json"
- cursor_path.write_text(json.dumps({"moves": moves, "clicks": clicks}, indent=2) + "\n")
- meta["cursors"] = {cursor_id: assets[cursor_id] for cursor_id in ("0", "2")}
- meta["segments"][0]["cursor"] = "content/cursor-campaign.json"
- meta_path.write_text(json.dumps(meta, indent=2) + "\n")
- marker["cursor_trajectory"] = "Synthetic normalized 60Hz movement, arrow-to-hand, one click"
- marker["cursor_asset_source"] = str(assets_directory.resolve())
- (fixture / "SYNTHETIC-FIXTURE.json").write_text(json.dumps(marker, indent=2) + "\n")
- print(json.dumps({"fixture": str(fixture), "moves": len(moves), "clicks": len(clicks),
- "cursor_ids": ["0", "2"], "source": str(assets_directory.resolve())}))
-
-
-
-def main():
- parser = argparse.ArgumentParser()
- parser.add_argument("destination", type=Path)
- parser.add_argument("--source-seconds", type=int, default=600)
- parser.add_argument("--export-seconds", type=float, default=4)
- parser.add_argument("--audio-only", action="store_true")
- parser.add_argument("--with-cursor", action="store_true")
- parser.add_argument("--cursor-assets", type=Path, default=Path(__file__).resolve().parents[1] /
- "apps/cli/skill/cap-demo/assets")
- args = parser.parse_args()
- if args.audio_only and args.with_cursor:
- parser.error("--with-cursor requires video generation")
- target = args.destination.resolve()
- if target.exists():
- raise SystemExit("Refusing to replace an existing fixture directory")
- if args.source_seconds <= 0 or not 0 < args.export_seconds <= args.source_seconds:
- raise SystemExit("Invalid durations")
- content = target / "content"
- content.mkdir(parents=True)
- for name, frequency in [("mic", 440), ("system", 880)]:
- subprocess.run([
- "ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error",
- "-f", "lavfi", "-i", f"sine=frequency={frequency}:sample_rate=48000:duration={args.source_seconds}",
- "-ac", "2", "-c:a", "aac", "-b:a", "96k", str(content / f"{name}.m4a"),
- ], check=True)
- if not args.audio_only:
- subprocess.run([
- "ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error",
- "-f", "lavfi", "-i", f"color=c=0x234567:size=640x360:rate=15:duration={args.source_seconds}",
- "-c:v", "libx264", "-preset", "ultrafast", "-threads", "2",
- "-pix_fmt", "yuv420p", str(content / "display.mp4"),
- ], check=True)
- meta = {
- "pretty_name": "SYNTHETIC disposable video-only export benchmark",
- "sharing": None,
- "segments": [{
- "display": {"path": "content/display.mp4", "fps": 15},
- "mic": {"path": "content/mic.m4a"},
- "system_audio": {"path": "content/system.m4a"},
- }],
- }
- config = {
- "background": {"source": {"type": "color", "value": [30, 30, 46]}, "padding": 0},
- "camera": {"hide": True},
- "audio": {"mute": True},
- "timeline": {"segments": [{"recordingSegment": 0, "timescale": 1.0, "start": 0.0,
- "end": args.export_seconds}], "zoomSegments": []},
- }
- (target / "recording-meta.json").write_text(json.dumps(meta, indent=2) + "\n")
- (target / "project-config.json").write_text(json.dumps(config, indent=2) + "\n")
- (target / "SYNTHETIC-FIXTURE.json").write_text(json.dumps({
- "synthetic": True, "source_seconds": args.source_seconds,
- "export_seconds": args.export_seconds, "audio_only": args.audio_only,
- "purpose": "Real Cap builder/export or ProgressiveAudio workload, synthetic inputs",
- }, indent=2) + "\n")
- if args.with_cursor:
- add_cursor(target, args.cursor_assets)
- print(target)
-
-
-if __name__ == "__main__":
- main()
diff --git a/scripts/test-video-only-export-parity.py b/scripts/test-video-only-export-parity.py
deleted file mode 100644
index 6bc1e6ae2..000000000
--- a/scripts/test-video-only-export-parity.py
+++ /dev/null
@@ -1,92 +0,0 @@
-import importlib.util
-import io
-import subprocess
-import unittest
-from pathlib import Path
-from unittest.mock import MagicMock, patch
-
-SPEC = importlib.util.spec_from_file_location(
- "video_only_export_parity",
- Path(__file__).with_name("check-video-only-export-parity.py"),
-)
-CHECKER = importlib.util.module_from_spec(SPEC)
-SPEC.loader.exec_module(CHECKER)
-
-
-class VirtualChunk:
- def __init__(self, size, tracking):
- self.size = size
- self.tracking = tracking
- tracking["live"] += 1
- tracking["peak"] = max(tracking["peak"], tracking["live"])
-
- def __len__(self):
- return self.size
-
- def __iter__(self):
- return iter((0, 255))
-
- def __del__(self):
- self.tracking["live"] -= 1
-
-
-class VirtualAlphaStream:
- def __init__(self, size):
- self.remaining = size
- self.tracking = {"live": 0, "peak": 0}
- self.reads = 0
-
- def read(self, size=-1):
- if not 0 < size <= CHECKER.ALPHA_READ_BYTES:
- raise AssertionError("Alpha reads must have a bounded positive size")
- self.reads += 1
- size = min(size, self.remaining)
- self.remaining -= size
- return VirtualChunk(size, self.tracking) if size else b""
-
-
-class AlphaExtremaTests(unittest.TestCase):
- def test_long_virtual_stream_has_bounded_reads_and_retention(self):
- stream = VirtualAlphaStream(640 * 360 * 15 * 1800)
- self.assertEqual(CHECKER.read_alpha_extrema(stream), (0, 255))
- self.assertEqual(stream.remaining, 0)
- self.assertGreater(stream.reads, 90_000)
- self.assertLessEqual(stream.tracking["peak"], 2)
- self.assertEqual(stream.tracking["live"], 0)
-
- def test_empty_transparent_opaque_and_mixed_alpha(self):
- for data, expected in [
- (b"", (None, None)),
- (b"\x00" * 17, (0, 0)),
- (b"\xff" * 17, (255, 255)),
- (b"\x00\x7f\xff", (0, 255)),
- ]:
- with self.subTest(data=data):
- self.assertEqual(CHECKER.read_alpha_extrema(io.BytesIO(data)), expected)
-
- def test_ffmpeg_failure_after_valid_alpha_is_propagated(self):
- process = MagicMock()
- process.__enter__.return_value = process
- process.stdout = io.BytesIO(b"\x00\xff")
- process.wait.return_value = 7
- with patch.object(CHECKER.subprocess, "Popen", return_value=process) as spawn:
- with self.assertRaises(subprocess.CalledProcessError) as failure:
- CHECKER.alpha_extrema(Path("baseline.mov"))
- self.assertEqual(failure.exception.returncode, 7)
- self.assertEqual(spawn.call_args.kwargs, {"stdout": subprocess.PIPE})
- process.wait.assert_called_once_with()
- process.kill.assert_not_called()
-
- def test_read_failure_kills_and_reaps_ffmpeg(self):
- process = MagicMock()
- process.__enter__.return_value = process
- process.stdout.read.side_effect = OSError("Broken alpha read")
- with patch.object(CHECKER.subprocess, "Popen", return_value=process):
- with self.assertRaisesRegex(OSError, "Broken alpha read"):
- CHECKER.alpha_extrema(Path("baseline.mov"))
- process.kill.assert_called_once_with()
- process.wait.assert_called_once_with()
-
-
-if __name__ == "__main__":
- unittest.main()