Open source Loom alternative. Beautiful, shareable screen recordings.
cap-verify: engine verification suite
Oracle tests for the playback, export, recording, editing and captions engines. Fixtures are generated by code at test time. Every source video frame carries its own identity as a block code, and every audio source carries sample-accurate markers. An independent reference model computes which source frame and sample each output frame and sample must show. A case passes only when every checked frame and sample matches.
Suites
The suites are listed in scripts/verify/suites.json:
| Suite | What it checks | FAST tier | FULL tier |
|---|---|---|---|
verify-self |
marker codecs, reference model, report | linux | linux, macos |
export |
every exported frame (MP4, GIF, both decoders on macOS) | linux, macos, windows | all |
playback |
offline-driven editor playback frames | linux, macos | all |
seek |
random, keyframe and scrub seeks | linux, macos | all |
audio |
export and offline playback audio against sample-accurate markers | linux, macos, windows | all |
editing |
random edit sequences exported and checked frame by frame | linux | linux |
chaos |
hostile device data, files, projects and settings: no panic, abort, signal, Objective-C exception, hang or wrong success | macos | macos |
captions |
fake-transcriber captions: segments and rendered captions on the right frames | linux | linux |
decoder-model-ffmpeg |
FFmpeg decoder decisions on scripted media (cap-rendering decoder::model) |
linux | linux |
decoder-model-avf |
AVAssetReader decoder decisions on scripted media | macos | macos |
The upload fault suite (apps/desktop-gpui/src/upload/verify_tests.rs and
upload/queue/verify_resume_tests.rs: a mock Cap API with scripted 5xx,
resets, outages, hangs and quits, byte-identical objects and request and byte
budgets) lives in the GPUI app's own workspace. It runs inside
cargo test --bin cap-gpui, which the CI / Build GPUI Desktop job already
runs on all three OSes, so that job is its gate. The app-level caption tests
(apps/desktop-gpui/src/transcription/verify_tests.rs: a fake transcriber
behind the real extraction and caption pipeline) run in the same job. Run them
locally with
cargo test --manifest-path apps/desktop-gpui/Cargo.toml --bin cap-gpui upload::
(or transcription::).
Platform-independent logic runs on Linux only in CI. Suites that test platform code run on that platform: AVFoundation, VideoToolbox and Metal on macOS; Media Foundation, Direct3D and WASAPI on Windows. When you run the suite locally, the platform-independent suites run on every OS.
CAP_VERIFY_TIER=full (the FULL tier) uses longer fixtures and more seeds.
Chaos suite
crates/verify/tests/chaos.rs tries to crash the engines. Every probe feeds one
hostile input to a product entry point and runs in its own child process (the
test binary re-run with CAP_VERIFY_CHAOS_CHILD), so an abort, a fatal signal,
an Objective-C exception or a hang ends the child, not the run. A panic hook in
the child records every panic on any thread. Outcomes:
pass: the engine returned an error, or it succeeded and its output checks out (readable media with packets when the input had usable frames).panic,abort(fatal signal or unexpected exit),objc-exception,hang(no verdict before the probe's timeout, reproduced once more) andwrong-success(success reported for an empty or unreadable output).
Areas, one test per area (chaos_recording, chaos_files, chaos_export,
chaos_editor, chaos_fragments, chaos_fuzz):
chaos/record/: synthetic frames, audio buffers and control sequences throughOutputPipelinewith the Studio and Instant muxers (in-process and out-of-process on macOS), the microphone feed and source, the macOS camera conversion, and the Studio and Instant recording actors in camera-only mode with synthetic camera and microphone feeds, finalized like the app.chaos/files/,chaos/export/,chaos/editor/: a generated project with one damaged file, metadata or configuration field, opened like the editor's preflight, inEditorInstance(preview, playback, seeks) and exported; hostile export settings; non-finite values pushed in memory.chaos/fragments/: damaged fragmented recordings through crash recovery and Instant finalization.chaos/fuzz/: seeded fuzzing (structure-aware JSON, box-aware MP4 bytes). Each seed runs in a batch child; a crash or hang is pinned to its seed, and a failing seed is shrunk by dropping mutations while it fails the same way. The failure printsCAP_VERIFY_CHAOS_SEEDandCAP_VERIFY_CHAOS_KEEPfor its repro.
CAP_VERIFY_CHAOS_JOBS sets the children run at once per test (default a
quarter of the cores, at most 3). CAP_VERIFY_CHAOS_KEEP_SCRATCH=1 keeps
each probe's scratch directory, child log and stderr.
Running locally
scripts/verify/run.sh fast # what CI runs on this OS for a PR
scripts/verify/run.sh full # merge-queue and nightly tier
scripts/verify/run.sh fast --suites seek # one suite
scripts/verify/run.sh fast --case 'export/basic/*'
scripts/verify/run.sh fast --list # print the plan
Install cargo-nextest (cargo install cargo-nextest --locked) to run the
tests in parallel, one process per test. Without it, the script runs
cargo test per suite. The local run uses the dev profile and shares its
build with normal development. CI uses the ci-verify profile
(CAP_VERIFY_CARGO_PROFILE=ci-verify). Generated fixtures are cached in
target/verify-fixtures, or in CAP_VERIFY_FIXTURE_DIR when it is set.
Every failure prints the exact frames or samples that are wrong, the seed, and a one-line repro command, for example:
CAP_VERIFY_CASE='export/backward-hold/backward-jumps/mp460/avf' CAP_VERIFY_TIER=fast cargo test -p cap-verify --test export -- --nocapture
CAP_DECODER_MODEL_CASE=decoder-model/avf/syn/edlist/playback CAP_DECODER_MODEL_SEED_INDEX=3 cargo test -p cap-rendering --lib decoder::model::avassetreader_answers_match_the_reference_model -- --nocapture
Known failures: strict expectations
Known bugs on main are strict expected failures. They are listed in
crates/verify/expectations.toml and in
crates/rendering/src/decoder/model/expectations.toml, using the format from
crates/verify-expect:
[[xfail]]
case = "export/early-camera/full/*" # case id, * globs allowed
class = "future" # failure class that must still occur
platforms = ["macos"] # optional, absent = every platform
seeds = "0 3-7" # optional, seeded suites only
flaky = true # optional; state the measured rate
reason = "what is wrong"
link = "https://github.com/CapSoftware/Cap/pull/2595" # empty = no fix yet
- A failure that is not listed fails the run.
- A listed class or seed that no longer fails also fails the run, with "remove this xfail". When a fix lands, its expected failures must be removed in the same PR, so the fix is locked in.
- A class that cannot be measured because an earlier check failed (A/V sync when no audio marker of a source could be identified) is recorded as "not measured", never as a pass: its xfail is neither confirmed nor reported as fixed, and bless leaves it alone. The failure that prevented the measurement is judged on its own, so it still has to be listed.
To update the files after a deliberate change, run:
scripts/verify/bless.sh fast # run here and rewrite from the results
scripts/verify/bless.sh --run <run id> # rewrite from a CI run, all three OSes
Bless changes only the entries of the platforms that ran. It keeps the
reasons and links of existing entries, and gives each new failure a TODO:
reason. Replace every TODO: reason before you merge. Review the diff: a
blessed failure is a bug that the suite keeps checking.
CI
.github/workflows/engine-verify.yml:
- Plan.
scripts/verify/plan.pymaps the changed paths to suites through the root workspace dependency graph (cargo metadata). The standalone GPUI app is outside that graph, so any change underapps/desktop-gpui/conservatively selects every engine suite. This selection does not replace the app's own tests or prove coverage of its orchestration. A PR that changes only docs or the web app runs no engine suite. - Tiers. FAST runs on every PR push. FULL runs when the PR has the
verify-fulllabel, in the merge queue, and nightly. - Jobs. One job per platform builds the selected tests with the
ci-verifyprofile and runs them with nextest. Dependencies are optimised and cached; workspace crates are not optimised. A platform withshards > 1insuites.jsonuploads a nextest archive, and shard jobs run partitions of it. - No retries, except for a proven machine stall. Every test process
runs a starvation sentinel (
crates/verify/src/sentinel.rs): a high-priority thread that sleeps 2 ms and records how late it wakes, plus the process CPU time over wall time. Each case record carries that series (machine), andsentinel-<pid>.logkeeps it for tests that are killed. A wake gap of 100 ms or more proves the runner stopped scheduling the process. Only wall-clock failures are excused by it: a real-time playback case that delivered too few frames becomesinconclusive, and a test killed at its timeout, by wgpu's 6.3 s queue drain or by a lost GPU device is retryable when its sentinel log shows the gap.scripts/verify/retry.pyreruns those tests once withCAP_VERIFY_RETRY=1; a case that is inconclusive again fails asenvironment, and the job passes only if every failure was retryable and every retry passed. Content, frame identity, timestamps and sync never become inconclusive. Any other failed test is rerun once only so that the summary can label it flaky; the job still fails. - Summary. Each job summary lists the failing cases, their exact frames
or samples, and a repro command. The report records are uploaded as the
verify-report-*artifacts, whichbless.sh --runreads. - Gate. Make the
Engine verifyjob a required check.
Baseline comparison
Run python3 scripts/verify/diff.py BASE_DIR MAIN_DIR --json comparison.json with downloaded report artifacts. Comma-separated run directories combine independent runs on each side. Keep both sides on the same OS image.
A required track absent from a recording leaves its dependent classes unobserved with reason track absent. A newly observed class appears in the coverage-gained column and is still judged by main's own expectations. Coverage gain alone is not an equal or worse result.
The missing-after-hold class measures an environment-sensitive ScreenCaptureKit idle-resume event. Compare at least three independent real-capture runs per side per OS image. A distribution is worse only when every main per-run count is above the baseline range. Insufficient repetitions and overlapping ranges are reported explicitly. Retries of one capture do not add independent samples. Other classes retain the strict zero-tolerance comparison.
Finalize crash cases hold the test muxer at entry to finalization after source EOF. The parent kills it while the barrier is held and checks that the recorder never wrote done. Missing the barrier or finding done is a harness failure. The barrier exists only in the integration-test wrapper and is disabled for all ordinary cases.