Skip to content

armgit / Cap

Public Imported from GitHub
0

Open source Loom alternative. Beautiful, shareable screen recordings.

11.3 KB
Raw Blame

cap-verify: engine verification suite

Oracle tests for the playback, export, recording, editing and captions engines. Fixtures are generated by code at test time. Every source video frame carries its own identity as a block code, and every audio source carries sample-accurate markers. An independent reference model computes which source frame and sample each output frame and sample must show. A case passes only when every checked frame and sample matches.

Suites

The suites are listed in scripts/verify/suites.json:

Suite What it checks FAST tier FULL tier
verify-self marker codecs, reference model, report linux linux, macos
export every exported frame (MP4, GIF, both decoders on macOS) linux, macos, windows all
playback offline-driven editor playback frames linux, macos all
seek random, keyframe and scrub seeks linux, macos all
audio export and offline playback audio against sample-accurate markers linux, macos, windows all
editing random edit sequences exported and checked frame by frame linux linux
chaos hostile device data, files, projects and settings: no panic, abort, signal, Objective-C exception, hang or wrong success macos macos
captions fake-transcriber captions: segments and rendered captions on the right frames linux linux
decoder-model-ffmpeg FFmpeg decoder decisions on scripted media (cap-rendering decoder::model) linux linux
decoder-model-avf AVAssetReader decoder decisions on scripted media macos macos

The upload fault suite (apps/desktop-gpui/src/upload/verify_tests.rs and upload/queue/verify_resume_tests.rs: a mock Cap API with scripted 5xx, resets, outages, hangs and quits, byte-identical objects and request and byte budgets) lives in the GPUI app's own workspace. It runs inside cargo test --bin cap-gpui, which the CI / Build GPUI Desktop job already runs on all three OSes, so that job is its gate. The app-level caption tests (apps/desktop-gpui/src/transcription/verify_tests.rs: a fake transcriber behind the real extraction and caption pipeline) run in the same job. Run them locally with cargo test --manifest-path apps/desktop-gpui/Cargo.toml --bin cap-gpui upload:: (or transcription::).

Platform-independent logic runs on Linux only in CI. Suites that test platform code run on that platform: AVFoundation, VideoToolbox and Metal on macOS; Media Foundation, Direct3D and WASAPI on Windows. When you run the suite locally, the platform-independent suites run on every OS.

CAP_VERIFY_TIER=full (the FULL tier) uses longer fixtures and more seeds.

Chaos suite

crates/verify/tests/chaos.rs tries to crash the engines. Every probe feeds one hostile input to a product entry point and runs in its own child process (the test binary re-run with CAP_VERIFY_CHAOS_CHILD), so an abort, a fatal signal, an Objective-C exception or a hang ends the child, not the run. A panic hook in the child records every panic on any thread. Outcomes:

  • pass: the engine returned an error, or it succeeded and its output checks out (readable media with packets when the input had usable frames).
  • panic, abort (fatal signal or unexpected exit), objc-exception, hang (no verdict before the probe's timeout, reproduced once more) and wrong-success (success reported for an empty or unreadable output).

Areas, one test per area (chaos_recording, chaos_files, chaos_export, chaos_editor, chaos_fragments, chaos_fuzz):

  • chaos/record/: synthetic frames, audio buffers and control sequences through OutputPipeline with the Studio and Instant muxers (in-process and out-of-process on macOS), the microphone feed and source, the macOS camera conversion, and the Studio and Instant recording actors in camera-only mode with synthetic camera and microphone feeds, finalized like the app.
  • chaos/files/, chaos/export/, chaos/editor/: a generated project with one damaged file, metadata or configuration field, opened like the editor's preflight, in EditorInstance (preview, playback, seeks) and exported; hostile export settings; non-finite values pushed in memory.
  • chaos/fragments/: damaged fragmented recordings through crash recovery and Instant finalization.
  • chaos/fuzz/: seeded fuzzing (structure-aware JSON, box-aware MP4 bytes). Each seed runs in a batch child; a crash or hang is pinned to its seed, and a failing seed is shrunk by dropping mutations while it fails the same way. The failure prints CAP_VERIFY_CHAOS_SEED and CAP_VERIFY_CHAOS_KEEP for its repro.

CAP_VERIFY_CHAOS_JOBS sets the children run at once per test (default a quarter of the cores, at most 3). CAP_VERIFY_CHAOS_KEEP_SCRATCH=1 keeps each probe's scratch directory, child log and stderr.

Running locally

scripts/verify/run.sh fast                    # what CI runs on this OS for a PR
scripts/verify/run.sh full                    # merge-queue and nightly tier
scripts/verify/run.sh fast --suites seek      # one suite
scripts/verify/run.sh fast --case 'export/basic/*'
scripts/verify/run.sh fast --list             # print the plan

Install cargo-nextest (cargo install cargo-nextest --locked) to run the tests in parallel, one process per test. Without it, the script runs cargo test per suite. The local run uses the dev profile and shares its build with normal development. CI uses the ci-verify profile (CAP_VERIFY_CARGO_PROFILE=ci-verify). Generated fixtures are cached in target/verify-fixtures, or in CAP_VERIFY_FIXTURE_DIR when it is set.

Every failure prints the exact frames or samples that are wrong, the seed, and a one-line repro command, for example:

CAP_VERIFY_CASE='export/backward-hold/backward-jumps/mp460/avf' CAP_VERIFY_TIER=fast cargo test -p cap-verify --test export -- --nocapture
CAP_DECODER_MODEL_CASE=decoder-model/avf/syn/edlist/playback CAP_DECODER_MODEL_SEED_INDEX=3 cargo test -p cap-rendering --lib decoder::model::avassetreader_answers_match_the_reference_model -- --nocapture

Known failures: strict expectations

Known bugs on main are strict expected failures. They are listed in crates/verify/expectations.toml and in crates/rendering/src/decoder/model/expectations.toml, using the format from crates/verify-expect:

[[xfail]]
case = "export/early-camera/full/*"   # case id, * globs allowed
class = "future"                       # failure class that must still occur
platforms = ["macos"]                  # optional, absent = every platform
seeds = "0 3-7"                        # optional, seeded suites only
flaky = true                           # optional; state the measured rate
reason = "what is wrong"
link = "https://github.com/CapSoftware/Cap/pull/2595"   # empty = no fix yet
  • A failure that is not listed fails the run.
  • A listed class or seed that no longer fails also fails the run, with "remove this xfail". When a fix lands, its expected failures must be removed in the same PR, so the fix is locked in.
  • A class that cannot be measured because an earlier check failed (A/V sync when no audio marker of a source could be identified) is recorded as "not measured", never as a pass: its xfail is neither confirmed nor reported as fixed, and bless leaves it alone. The failure that prevented the measurement is judged on its own, so it still has to be listed.

To update the files after a deliberate change, run:

scripts/verify/bless.sh fast              # run here and rewrite from the results
scripts/verify/bless.sh --run <run id>    # rewrite from a CI run, all three OSes

Bless changes only the entries of the platforms that ran. It keeps the reasons and links of existing entries, and gives each new failure a TODO: reason. Replace every TODO: reason before you merge. Review the diff: a blessed failure is a bug that the suite keeps checking.

CI

.github/workflows/engine-verify.yml:

  • Plan. scripts/verify/plan.py maps the changed paths to suites through the root workspace dependency graph (cargo metadata). The standalone GPUI app is outside that graph, so any change under apps/desktop-gpui/ conservatively selects every engine suite. This selection does not replace the app's own tests or prove coverage of its orchestration. A PR that changes only docs or the web app runs no engine suite.
  • Tiers. FAST runs on every PR push. FULL runs when the PR has the verify-full label, in the merge queue, and nightly.
  • Jobs. One job per platform builds the selected tests with the ci-verify profile and runs them with nextest. Dependencies are optimised and cached; workspace crates are not optimised. A platform with shards > 1 in suites.json uploads a nextest archive, and shard jobs run partitions of it.
  • No retries, except for a proven machine stall. Every test process runs a starvation sentinel (crates/verify/src/sentinel.rs): a high-priority thread that sleeps 2 ms and records how late it wakes, plus the process CPU time over wall time. Each case record carries that series (machine), and sentinel-<pid>.log keeps it for tests that are killed. A wake gap of 100 ms or more proves the runner stopped scheduling the process. Only wall-clock failures are excused by it: a real-time playback case that delivered too few frames becomes inconclusive, and a test killed at its timeout, by wgpu's 6.3 s queue drain or by a lost GPU device is retryable when its sentinel log shows the gap. scripts/verify/retry.py reruns those tests once with CAP_VERIFY_RETRY=1; a case that is inconclusive again fails as environment, and the job passes only if every failure was retryable and every retry passed. Content, frame identity, timestamps and sync never become inconclusive. Any other failed test is rerun once only so that the summary can label it flaky; the job still fails.
  • Summary. Each job summary lists the failing cases, their exact frames or samples, and a repro command. The report records are uploaded as the verify-report-* artifacts, which bless.sh --run reads.
  • Gate. Make the Engine verify job a required check.

Baseline comparison

Run python3 scripts/verify/diff.py BASE_DIR MAIN_DIR --json comparison.json with downloaded report artifacts. Comma-separated run directories combine independent runs on each side. Keep both sides on the same OS image.

A required track absent from a recording leaves its dependent classes unobserved with reason track absent. A newly observed class appears in the coverage-gained column and is still judged by main's own expectations. Coverage gain alone is not an equal or worse result.

The missing-after-hold class measures an environment-sensitive ScreenCaptureKit idle-resume event. Compare at least three independent real-capture runs per side per OS image. A distribution is worse only when every main per-run count is above the baseline range. Insufficient repetitions and overlapping ranges are reported explicitly. Retries of one capture do not add independent samples. Other classes retain the strict zero-tolerance comparison.

Finalize crash cases hold the test muxer at entry to finalization after source EOF. The parent kills it while the barrier is held and checks that the recorder never wrote done. Missing the barrier or finding done is a harness failure. The barrier exists only in the integration-test wrapper and is disabled for all ordinary cases.