YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(vision): bound the macOS SCK serializer wait so queued callers cannot inherit it (#6269)
Louis Beaumontb027807740fa
diff --git a/COVERAGE.md b/COVERAGE.mdindex 3a6418a47..5d495de9e 100644--- a/COVERAGE.md+++ b/COVERAGE.md@@ -32,15 +32,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.- Mapped suites: 32- Mapped Rust files: 323-- Active test blocks: 3125+- Active test blocks: 3131- Ignored/manual test blocks: 137-- Weighted coverage points: 2570.7+- Weighted coverage points: 2575.2| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2994 | 132 | 2510.7 | 21 | 11 | 100% |-| macos | 29 | 3047 | 112 | 2521.1 | 22 | 11 | 100% |-| linux | 25 | 2670 | 105 | 2217.5 | 20 | 11 | 100% |+| windows | 29 | 3000 | 132 | 2515.2 | 21 | 11 | 100% |+| macos | 29 | 3053 | 112 | 2525.6 | 22 | 11 | 100% |+| linux | 25 | 2673 | 105 | 2219.0 | 20 | 11 | 100% |## Refreshdiff --git a/crates/screenpipe-screen/src/monitor/macos.rs b/crates/screenpipe-screen/src/monitor/macos.rsindex 6084af40e..3d4604963 100644--- a/crates/screenpipe-screen/src/monitor/macos.rs+++ b/crates/screenpipe-screen/src/monitor/macos.rs@@ -206,7 +206,17 @@ whereT: Send + 'static,F: FnOnce() -> Result<T> + Send + 'static,{- let _serial_guard = serializer.lock().await;+ // Bounding only the holder leaves the wait unbounded, so a queued caller+ // inherits the sum of every holder ahead of it. See the same repair and+ // reasoning in `run_bounded_sck_enumeration`; this is the capture-path twin+ // and sits directly under the capture loop's own bounded await.+ // See tests::capture_wait_stays_bounded_when_callers_queue.+ let Ok(_serial_guard) = tokio::time::timeout(timeout, serializer.lock()).await else {+ let secs = timeout.as_secs();+ return Err(anyhow::anyhow!(+ "{name}: capture busy; serializer held {secs}s, refusing to queue"+ ));+ };let permit = workers.try_acquire_owned().map_err(|e| match e {tokio::sync::TryAcquireError::NoPermits => anyhow::anyhow!("{name}: capture retry budget exhausted; Apple callbacks remain blocked"@@ -468,11 +478,25 @@ where{// Healthy enumeration stays single-file. After a timeout this guard is// released, allowing one genuinely fresh SCK request to recover capture.- // The holder is itself bounded by `timeout` below, so waiting here cannot- // inherit the unbounded Apple callback. Avoid racing two equal 15-second- // timeouts, which could reject the queued recovery attempt at the instant- // the first holder releases this guard.- let _serial_guard = serializer.lock().await;+ //+ // Bounding only the holder is not enough. Each holder is capped at+ // `timeout`, but an unbounded wait here lets a queued caller inherit the+ // sum of every holder ahead of it: with callbacks that are slow yet do+ // return, permits recycle, nobody fast-fails, and the Nth caller waits+ // N * timeout. That is the #3939 shape — a 250ms-bounded await in the+ // capture loop reported frozen for 73-299s, which is 5-20 holders deep at+ // the 15s production timeout. Cap the wait so a caller never blocks for+ // more than roughly twice the timeout it asked for.+ // See tests::enumeration_wait_stays_bounded_when_callers_queue.+ let _serial_guard = match tokio::time::timeout(timeout, serializer.lock()).await {+ Ok(guard) => guard,+ Err(_) => {+ return Err(MonitorListError::Other(format!(+ "ScreenCaptureKit monitor enumeration busy: another caller held the serializer for {}s; serving the caller's fallback instead of queueing",+ timeout.as_secs()+ )))+ }+ };// A timed-out spawn_blocking task cannot be cancelled because the Apple// callback is outside Rust's control. Two permits bound the damage while@@ -1342,6 +1366,149 @@ mod tests {assert!(matches!(recovered, Ok(4)));}+ /// Production #3939 freezes report 73-299s frozen in a 250ms-bounded await.+ /// `run_bounded_sck_enumeration` bounds the *holder* at `timeout`, but the+ /// wait for `serializer` is unbounded, so queued callers inherit the sum of+ /// every holder ahead of them. With callbacks that are slow but do return,+ /// permits recycle, nobody fast-fails, and the queue grows without limit.+ ///+ /// A caller must never wait for more than a small multiple of the timeout+ /// it asked for, however many callers are queued.+ #[tokio::test(flavor = "multi_thread", worker_threads = 4)]+ async fn enumeration_wait_stays_bounded_when_callers_queue() {+ const TIMEOUT: Duration = Duration::from_millis(100);+ // Longer than TIMEOUT so every holder times out, but finite so the+ // permit recycles and later callers never hit the retry budget.+ const CALLBACK: Duration = Duration::from_millis(160);+ const CALLERS: usize = 6;++ let serializer: &'static tokio::sync::Mutex<()> =+ Box::leak(Box::new(tokio::sync::Mutex::new(())));+ let workers = Arc::new(tokio::sync::Semaphore::new(2));++ let started = Instant::now();+ let mut handles = Vec::new();+ for _ in 0..CALLERS {+ let workers = workers.clone();+ handles.push(tokio::spawn(async move {+ let _ = run_bounded_sck_enumeration(serializer, workers, TIMEOUT, move || {+ std::thread::sleep(CALLBACK);+ Ok(1u8)+ })+ .await;+ started.elapsed()+ }));+ }++ let mut worst = Duration::ZERO;+ for handle in handles {+ worst = worst.max(handle.await.expect("caller task panicked"));+ }++ // Two permits let two holders overlap, so ~2x TIMEOUT is the honest+ // ceiling for a bounded design. Allow 3x for scheduling slack.+ let ceiling = TIMEOUT * 3;+ assert!(+ worst <= ceiling,+ "queued caller waited {worst:?} for a {TIMEOUT:?} bounded call \+ ({CALLERS} callers, ceiling {ceiling:?}) — the serializer wait is unbounded, \+ so waiters inherit every holder ahead of them"+ );+ }++ /// The capture-path twin of the enumeration queue defect. This serializer+ /// sits under the capture loop's own bounded await, so an unbounded wait+ /// here is what lets a 250ms-bounded loop stage report minutes frozen.+ #[tokio::test(flavor = "multi_thread", worker_threads = 4)]+ async fn capture_wait_stays_bounded_when_callers_queue() {+ const TIMEOUT: Duration = Duration::from_millis(100);+ const CALLBACK: Duration = Duration::from_millis(160);+ const CALLERS: usize = 6;++ let serializer: &'static tokio::sync::Mutex<()> =+ Box::leak(Box::new(tokio::sync::Mutex::new(())));+ let workers = Arc::new(tokio::sync::Semaphore::new(2));++ let started = Instant::now();+ let mut handles = Vec::new();+ for _ in 0..CALLERS {+ let workers = workers.clone();+ handles.push(tokio::spawn(async move {+ let _ = run_bounded_macos_capture(+ "test-capture",+ serializer,+ workers,+ TIMEOUT,+ move || {+ std::thread::sleep(CALLBACK);+ Ok(1u8)+ },+ )+ .await;+ started.elapsed()+ }));+ }++ let mut worst = Duration::ZERO;+ for handle in handles {+ worst = worst.max(handle.await.expect("caller task panicked"));+ }++ let ceiling = TIMEOUT * 3;+ assert!(+ worst <= ceiling,+ "queued capture caller waited {worst:?} for a {TIMEOUT:?} bounded call \+ ({CALLERS} callers, ceiling {ceiling:?})"+ );+ }++ /// The other regime, for contrast: when callbacks never return, both+ /// permits stay pinned and later callers fast-fail on the retry budget+ /// instead of queueing. This caps the wait at ~2x timeout, which means a+ /// permanent wedge alone cannot explain a multi-minute production freeze.+ #[tokio::test(flavor = "multi_thread", worker_threads = 4)]+ async fn enumeration_fast_fails_once_both_permits_are_pinned() {+ const TIMEOUT: Duration = Duration::from_millis(100);+ const CALLERS: usize = 6;++ let serializer: &'static tokio::sync::Mutex<()> =+ Box::leak(Box::new(tokio::sync::Mutex::new(())));+ let workers = Arc::new(tokio::sync::Semaphore::new(2));+ let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();+ let release_rx = Arc::new(std::sync::Mutex::new(release_rx));++ let started = Instant::now();+ let mut handles = Vec::new();+ for _ in 0..CALLERS {+ let workers = workers.clone();+ let release_rx = release_rx.clone();+ handles.push(tokio::spawn(async move {+ let _ = run_bounded_sck_enumeration(serializer, workers, TIMEOUT, move || {+ // Park until the test releases us; models an Apple callback+ // that never comes back.+ let _ = release_rx.lock().expect("release lock").recv();+ Ok(1u8)+ })+ .await;+ started.elapsed()+ }));+ }++ let mut worst = Duration::ZERO;+ for handle in handles {+ worst = worst.max(handle.await.expect("caller task panicked"));+ }+ for _ in 0..CALLERS {+ let _ = release_tx.send(());+ }++ assert!(+ worst <= TIMEOUT * 4,+ "with both permits pinned every later caller should fast-fail, \+ but the worst wait was {worst:?}"+ );+ }+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]async fn capture_timeout_keeps_permit_and_caps_abandoned_workers() {let serializer = tokio::sync::Mutex::new(());diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.mdindex 8bd8a5b25..8b554ce73 100644--- a/docs/coverage/CORE.md+++ b/docs/coverage/CORE.md@@ -9,10 +9,10 @@ confidence, and criticality.- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y- Mapped suites: 32- Mapped Rust files: 323-- Active test blocks: 3125+- Active test blocks: 3131- Ignored/manual test blocks: 137-- Declared test blocks: 3262-- Weighted coverage points: 2570.7+- Declared test blocks: 3268+- Weighted coverage points: 2575.2Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.Criticality weights: high=1.0, medium=0.7, low=0.4.@@ -23,19 +23,19 @@ are explicitly enabled in a runtime lane.| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2994 | 132 | 2510.7 | 21 | 11 | 100% |-| macos | 29 | 3047 | 112 | 2521.1 | 22 | 11 | 100% |-| linux | 25 | 2670 | 105 | 2217.5 | 20 | 11 | 100% |+| windows | 29 | 3000 | 132 | 2515.2 | 21 | 11 | 100% |+| macos | 29 | 3053 | 112 | 2525.6 | 22 | 11 | 100% |+| linux | 25 | 2673 | 105 | 2219.0 | 20 | 11 | 100% |## Crate Summary| Crate | Suites | Integration files | Source unit files | Active tests | Ignored tests | Weighted points | Flows || --- | --- | --- | --- | --- | --- | --- | --- |-| screenpipe-engine | 10 | 19 | 103 | 1490 | 42 | 1141.9 | 10 |+| screenpipe-engine | 10 | 19 | 103 | 1493 | 42 | 1144.9 | 10 || screenpipe-db | 5 | 52 | 14 | 447 | 16 | 424.2 | 9 || screenpipe-sqlite-coordinator | 1 | 0 | 2 | 16 | 0 | 16.0 | 2 || screenpipe-audio | 6 | 25 | 51 | 591 | 43 | 517.4 | 5 |-| screenpipe-screen | 6 | 9 | 18 | 247 | 9 | 222.9 | 4 |+| screenpipe-screen | 6 | 9 | 18 | 250 | 9 | 224.4 | 4 || screenpipe-a11y | 4 | 2 | 28 | 334 | 27 | 248.3 | 3 |## Line Coverage@@ -69,7 +69,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-search | 2 suites / 111 active / 9 ignored / 111.0 pts | 2 suites / 111 active / 9 ignored / 111.0 pts | 2 suites / 111 active / 9 ignored / 111.0 pts || engine-lifecycle | 6 suites / 208 active / 1 ignored / 183.6 pts | 6 suites / 208 active / 1 ignored / 183.6 pts | 5 suites / 202 active / 1 ignored / 181.9 pts || local-api | 2 suites / 336 active / 9 ignored / 237.0 pts | 2 suites / 336 active / 9 ignored / 237.0 pts | 2 suites / 336 active / 9 ignored / 237.0 pts |-| meeting | 6 suites / 1429 active / 19 ignored / 1168.6 pts | 6 suites / 1429 active / 19 ignored / 1168.6 pts | 4 suites / 1141 active / 15 ignored / 903.1 pts |+| meeting | 6 suites / 1432 active / 19 ignored / 1171.6 pts | 6 suites / 1432 active / 19 ignored / 1171.6 pts | 4 suites / 1141 active / 15 ignored / 903.1 pts || ocr | 4 suites / 124 active / 7 ignored / 118.3 pts | 4 suites / 128 active / 7 ignored / 123.8 pts | 3 suites / 119 active / 6 ignored / 114.8 pts || os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - || performance | 13 suites / 1405 active / 67 ignored / 1235.4 pts | 14 suites / 1503 active / 70 ignored / 1274.6 pts | 13 suites / 1405 active / 67 ignored / 1235.4 pts |@@ -82,7 +82,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| timeline | 4 suites / 974 active / 33 ignored / 789.2 pts | 4 suites / 974 active / 33 ignored / 789.2 pts | 4 suites / 974 active / 33 ignored / 789.2 pts || transcription | 5 suites / 709 active / 40 ignored / 558.9 pts | 5 suites / 709 active / 40 ignored / 558.9 pts | 5 suites / 709 active / 40 ignored / 558.9 pts || ui-events | 4 suites / 702 active / 28 ignored / 534.6 pts | 3 suites / 653 active / 5 ignored / 500.3 pts | 3 suites / 653 active / 5 ignored / 500.3 pts |-| vision-capture | 5 suites / 524 active / 32 ignored / 414.1 pts | 5 suites / 528 active / 32 ignored / 419.6 pts | 4 suites / 519 active / 31 ignored / 410.6 pts |+| vision-capture | 5 suites / 527 active / 32 ignored / 415.6 pts | 5 suites / 531 active / 32 ignored / 421.1 pts | 4 suites / 522 active / 31 ignored / 412.1 pts |## Critical Flow Matrix@@ -140,14 +140,14 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| engine-focus-os | screenpipe-engine | windows, macos | engine-lifecycle, os-integration | engine-health-lifecycle, performance-liveness | medium | conditional | unit | 3 | 6 | 0 | Platform focus-tracker parsing/helpers. These files are cfg-gated and only execute on their target OS. || engine-local-api-search-integration | screenpipe-engine | windows, macos, linux | local-api, db-search | local-api-search | high | strong | integration | 1 | 6 | 5 | Active /search route test builds an audio-disabled router, seeds captured-screen-shaped OCR data into an in-memory DB, and asserts the HTTP response and pagination. || engine-meeting-privacy-sync | screenpipe-engine | windows, macos, linux | meeting, privacy, ui-events, pipes, sync | meeting-live-notes, privacy-and-redaction, accessibility-ui-events, performance-liveness | medium | strong | unit | 30 | 463 | 3 | Unit-heavy coverage for privacy filter policy, capture exclusions, UI recorder safety, pipes/live-view/structured-output helpers, sync helpers, and CLI parsing. Meeting detection moved to the engine-meeting-watcher suite. |-| engine-meeting-watcher | screenpipe-engine | windows, macos | meeting | meeting-live-notes | high | strong | mixed | 9 | 213 | 3 | Successor of src/meeting_detector.rs and src/meeting_telemetry.rs. Audio-process and UI-scan backend state machines, candidate resolution, shared telemetry, and a scored trajectory eval. ui_scan/macos.rs and ui_scan/windows.rs are cfg-gated and only execute on their target OS; both backends are null on Linux. |+| engine-meeting-watcher | screenpipe-engine | windows, macos | meeting | meeting-live-notes | high | strong | mixed | 9 | 216 | 3 | Successor of src/meeting_detector.rs and src/meeting_telemetry.rs. Audio-process and UI-scan backend state machines, candidate resolution, shared telemetry, and a scored trajectory eval. ui_scan/macos.rs and ui_scan/windows.rs are cfg-gated and only execute on their target OS; both backends are null on Linux. || engine-retention-storage | screenpipe-engine | windows, macos, linux | storage, engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | mixed | 5 | 38 | 0 | Local retention deletion (including lean-mode heavy-text stripping), cloud archive watermarking, atomic state-file replacement, and the low-disk capture monitor. || engine-telemetry-observability | screenpipe-engine | windows, macos, linux | engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | unit | 6 | 29 | 0 | PostHog capture gating, telemetry context shaping, piggyback telemetry forwarding, crash-log helpers, resource monitoring, and the recording-coverage reliability metric. || screen-capture-ocr-contract | screenpipe-screen | windows, macos, linux | vision-capture, ocr | capture-ocr-pipeline | high | partial | unit | 3 | 14 | 0 | Cross-platform cached-OCR unit coverage for RawCaptureResult to CaptureResult metadata, browser URL, focus state, window-to-screen OCR coordinate transformation, Tesseract output parsing, and contour-based text-region detection for the meeting OCR gate. || screen-capture-windowing | screenpipe-screen | windows, macos, linux | vision-capture, timeline, performance, privacy | capture-ocr-pipeline, timeline-streaming, privacy-and-redaction, performance-liveness | high | strong | mixed | 14 | 183 | 0 | Window filtering, empty-window regressions, retry policy, URL timing, monitor cache, OCR cache, snapshots, and image comparison. || screen-custom-ocr | screenpipe-screen | windows, macos, linux | ocr | capture-ocr-pipeline | medium | conditional | manual | 1 | 0 | 2 | Custom OCR tests are ignored by default and only contribute when explicitly run. || screen-macos-ocr | screenpipe-screen | macos | ocr, vision-capture | capture-ocr-pipeline | high | strong | mixed | 2 | 9 | 1 | Apple Vision OCR source/unit coverage and fixture OCR assertions. |-| screen-monitor-platform | screenpipe-screen | windows, macos, linux | vision-capture | capture-ocr-pipeline | medium | partial | unit | 5 | 36 | 5 | Per-OS monitor enumeration (Windows, macOS, Wayland/portal on Linux) and the persistent Windows.Graphics.Capture session. Each file is cfg-gated and only executes on its target OS. |+| screen-monitor-platform | screenpipe-screen | windows, macos, linux | vision-capture | capture-ocr-pipeline | medium | partial | unit | 5 | 39 | 5 | Per-OS monitor enumeration (Windows, macOS, Wayland/portal on Linux) and the persistent Windows.Graphics.Capture session. Each file is cfg-gated and only executes on its target OS. || screen-windows-ocr | screenpipe-screen | windows | ocr, vision-capture | capture-ocr-pipeline | high | partial | integration | 2 | 5 | 1 | Windows OCR fixture coverage plus an ignored continuous-capture probe that requires a live desktop. || sqlite-coordinator-durable-quarantine | screenpipe-sqlite-coordinator | windows, macos, linux | database, engine-lifecycle | engine-health-lifecycle, performance-liveness | high | strong | unit | 2 | 16 | 0 | Process-wide single-writer gates, SQLite runtime pinning, atomic hard-fault markers, OS file identity, fail-closed malformed metadata, fresh-identity resolution, and a real cross-process restart test. |@@ -374,7 +374,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/mod.rs | source | 2 | 0 | 2 || engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/shared/telemetry.rs | source | 6 | 0 | 6 || engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/ui_scan/macos.rs | source | 0 | 2 | 2 |-| engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/ui_scan/tests.rs | source | 130 | 0 | 130 |+| engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/ui_scan/tests.rs | source | 133 | 0 | 133 || engine-meeting-watcher | screenpipe-engine | src/meeting_watcher/ui_scan/windows.rs | source | 0 | 1 | 1 || engine-config-lifecycle | screenpipe-engine | src/permission_monitor.rs | source | 1 | 0 | 1 || engine-telemetry-observability | screenpipe-engine | src/piggyback_telemetry.rs | source | 1 | 0 | 1 |@@ -459,7 +459,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| screen-capture-windowing | screenpipe-screen | src/monitor.rs | source | 5 | 0 | 5 || screen-monitor-platform | screenpipe-screen | src/monitor/linux_portal.rs | source | 5 | 0 | 5 || screen-monitor-platform | screenpipe-screen | src/monitor/linux_wayland.rs | source | 5 | 0 | 5 |-| screen-monitor-platform | screenpipe-screen | src/monitor/macos.rs | source | 16 | 2 | 18 |+| screen-monitor-platform | screenpipe-screen | src/monitor/macos.rs | source | 19 | 2 | 21 || screen-monitor-platform | screenpipe-screen | src/monitor/windows.rs | source | 2 | 1 | 3 || screen-capture-windowing | screenpipe-screen | src/ocr_cache.rs | source | 15 | 0 | 15 || screen-capture-windowing | screenpipe-screen | src/snapshot_writer.rs | source | 4 | 0 | 4 |