YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(db): tell the truth about a stalled write, and escape a stuck coordinator (#6075)
Louis Beaumont6b1c447a3dbd
diff --git a/COVERAGE.md b/COVERAGE.mdindex b4cef6069..5d439380d 100644--- a/COVERAGE.md+++ b/COVERAGE.md@@ -32,15 +32,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.- Mapped suites: 32- Mapped Rust files: 318-- Active test blocks: 3018+- Active test blocks: 3025- Ignored/manual test blocks: 137-- Weighted coverage points: 2481.3+- Weighted coverage points: 2488.0| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2887 | 132 | 2421.4 | 21 | 11 | 100% |-| macos | 29 | 2941 | 112 | 2432.5 | 22 | 11 | 100% |-| linux | 25 | 2574 | 105 | 2138.8 | 20 | 11 | 100% |+| windows | 29 | 2894 | 132 | 2428.1 | 21 | 11 | 100% |+| macos | 29 | 2948 | 112 | 2439.2 | 22 | 11 | 100% |+| linux | 25 | 2581 | 105 | 2145.6 | 20 | 11 | 100% |## Refreshdiff --git a/TESTING.md b/TESTING.mdindex fc075f5d4..28b42a95d 100644--- a/TESTING.md+++ b/TESTING.md@@ -338,6 +338,8 @@ commits: `eea0c865`, `cc09de61`, `e61501da`, `d25191d7`, `60096fb9`- [ ] **Audio chunk timestamps** — `start_time` and `end_time` are correctly set for reconciled and retranscribed audio chunks in the database.- [ ] **SCREENPIPE_DATA_DIR usage** — Set the `SCREENPIPE_DATA_DIR` environment variable. Verify the app uses this directory for all its data storage. (`d5f30db71`)- [ ] **DB pool starvation prevention** — Simulate high database load (e.g., rapid screen activity, many pipes running) and monitor logs. Verify no "database is locked" errors or signs of DB pool starvation.+- [ ] **write stalls name their own cause** — Run `screenpipe db compact` (or any long `VACUUM`) while capture is active. Logs must say "another writer held the SQLite write lock", NOT "pool timed out"; the UI-event recorder must log this at `warn`, not `error`, so a routine VACUUM does not raise a Sentry issue. Starving the write pool instead must say "no write-pool connection became available". The two must never share a message. (`SCREENPIPE-CLI-SQ`)+- [ ] **stuck write coordinator recovers** — A process-wide SQLite write permit that is never released must not wedge writes forever. Verify a sustained contention run (past `CONTENTION_PERSISTENT_AFTER_WALL`) marks the write path degraded *and* requests an engine restart, while a normal multi-minute maintenance hold does not. (`SCREENPIPE-CLI-SQ`)- [ ] **DB write coalescing queue** — verify high-frequency captures (e.g. 10 FPS) don't lock the UI or cause write errors. (`c23768f41`)- [ ] **Multi-byte window titles in suggestions** — Interact with suggestions for windows that have multi-byte (e.g., Unicode, emoji) characters in their titles. Verify no char boundary panics.- [ ] **no concurrent reconciliation issues** — Verify that concurrent reconciliation processes do not cause issues during heavy load or sync operations. (`1d436bc3`)diff --git a/crates/screenpipe-db/src/lib.rs b/crates/screenpipe-db/src/lib.rsindex 11372a95b..2c0c216ba 100644--- a/crates/screenpipe-db/src/lib.rs+++ b/crates/screenpipe-db/src/lib.rs@@ -36,5 +36,7 @@ pub use screenpipe_sqlite_coordinator::{pub use text_normalizer::{expand_search_query, sanitize_fts5_query};pub use types::*;pub use write_queue::{- request_write_pause, request_write_resume, PersistentFailureHook, SyncTable, WriteQueueHealth,+ is_retryable_write_stall, is_write_lock_contention, is_write_pool_starved, request_write_pause,+ request_write_resume, PersistentFailureHook, SyncTable, WriteQueueHealth,+ WRITE_LOCK_HELD_MESSAGE, WRITE_POOL_STARVED_MESSAGE,};diff --git a/crates/screenpipe-db/src/write_queue.rs b/crates/screenpipe-db/src/write_queue.rsindex 56c25666c..d1e8b4c98 100644--- a/crates/screenpipe-db/src/write_queue.rs+++ b/crates/screenpipe-db/src/write_queue.rs@@ -107,6 +107,51 @@ const FATAL_RUN_CLEAR_AFTER: u64 = 3;/// the app can escalate further instead of wedging silently forever.const PERSISTENT_FAILURE_REFIRE_EVERY: Duration = Duration::from_secs(600);+// ── Sustained write-lock contention ──────────────────────────────────────+//+// Contention is normal: `compact()` (full VACUUM) and the RESTART checkpoint+// legitimately hold the process-wide write coordinator for longer than a+// batch's semaphore budget, and `db/maintenance.rs` documents that as "the+// recording briefly pauses the UI warns about". So a contended batch must+// NOT restart the engine.+//+// But a *permanently* held permit — a leaked `OwnedSemaphorePermit` from a+// hung or aborted task — is indistinguishable from that, batch by batch, and+// had no escape at all: `BatchOutcome::Contention` set `degraded` and nothing+// else. Every batch then timed out forever with no reopen, no hook, no cure.+//+// These thresholds are deliberately far past any legitimate maintenance hold+// so the hook only fires on a genuine wedge.++/// Fire the persistent-failure hook after this many *consecutive* contended+/// batches. Each one costs a full `write_semaphore` timeout, so 60 is already+/// ~30 minutes of uninterrupted contention; this is the fast path for+/// write-heavy load, not the primary trigger.+const CONTENTION_PERSISTENT_AFTER: u64 = 60;+/// The real trigger: a contention run that has spanned this long with no+/// successful write in between.+///+/// Deliberately generous. `compact()` (full VACUUM) and `backup_to()`+/// (VACUUM INTO) hold the coordinator for as long as it takes to rewrite a+/// multi-GB capture database, and neither announces itself — they do not use+/// `request_write_pause`, so the drain loop cannot tell them apart from a+/// wedge. Restarting the engine mid-VACUUM would roll back the whole thing and+/// waste the work, so the budget has to sit clearly beyond any plausible+/// maintenance hold rather than merely beyond a typical one.+///+/// Thirty minutes without a single successful write is unambiguous: capture+/// writes arrive continuously while recording, so a run this long is either a+/// stuck coordinator or a maintenance operation that is itself a bug.+///+/// The better long-term fix is for the long maintenance paths to announce+/// their hold (the `WRITE_PAUSED` seam already exists for sleep), at which+/// point this can come down a lot.+const CONTENTION_PERSISTENT_AFTER_WALL: Duration = Duration::from_secs(1_800);+/// Require this many contended batches in the run before the wall-clock rule+/// can fire, so a single slow checkpoint on an otherwise idle machine cannot+/// request a restart.+const CONTENTION_PERSISTENT_MIN_BATCHES: u64 = 5;+/// Pure escalation state for a fatal run — decides when the persistent-failure/// hook fires. Extracted from the drain loop so the count/wall-clock/refire/// rules are unit-testable with injected time.@@ -145,6 +190,24 @@ impl FatalRunEscalation {}}+ /// Same state machine, but with the much longer budgets a *contention* run+ /// needs. Legitimate maintenance holds the write coordinator for a while;+ /// only a genuinely stuck coordinator should ever reach the hook.+ fn for_contention(persistent_after: u64, wall: Duration, min_batches_for_wall: u64) -> Self {+ Self {+ persistent_after,+ wall,+ min_fatals_for_wall: min_batches_for_wall,+ clear_after: FATAL_RUN_CLEAR_AFTER,+ refire_every: PERSISTENT_FAILURE_REFIRE_EVERY,+ run_started: None,+ fatal_in_run: 0,+ consecutive_fatal: 0,+ healthy_streak: 0,+ last_fire: None,+ }+ }+/// Record a healthy batch. Returns true when this batch ended the fatal run.fn on_healthy(&mut self) -> bool {self.consecutive_fatal = 0;@@ -189,6 +252,55 @@ impl FatalRunEscalation {}}+// ── What a failed write actually means ───────────────────────────────────+//+// Two completely different failures used to be reported with the same+// fabricated `sqlx::Error::PoolTimedOut`, which `send_error_to_all` then+// flattened into `Error::Protocol(String)`. Both reached callers — and+// Sentry — as the single string "encountered unexpected or invalid data:+// pool timed out while waiting for an open connection", so:+//+// * SCREENPIPE-CLI-SQ conflated "another writer holds the lock" (expected+// during VACUUM, retry) with "the pool has no connections" (a real+// starvation fault) and could not be triaged;+// * no caller could branch on the cause, because the variant was gone.+//+// These messages are the stable contract. They are matched by the predicates+// below rather than re-derived at each call site, so a wording change cannot+// silently turn a retryable stall into a reported fault.++/// A batch never obtained the process-wide SQLite write coordinator inside its+/// budget: some other writer (a sibling batch, `compact()`, a checkpoint) still+/// holds it. The write did not happen, but nothing is broken.+pub const WRITE_LOCK_HELD_MESSAGE: &str =+ "screenpipe write queue: another writer held the SQLite write lock past this batch's budget";++/// A batch owned the write coordinator — so every ordinary writer was already+/// excluded — and still could not check out a pooled connection. That is real+/// pool starvation, not contention.+pub const WRITE_POOL_STARVED_MESSAGE: &str =+ "screenpipe write queue: no write-pool connection became available inside this batch's budget";++/// True when the write failed only because another writer held the SQLite write+/// lock. Expected during maintenance; retry rather than reporting a fault.+pub fn is_write_lock_contention(error: &sqlx::Error) -> bool {+ matches!(error, sqlx::Error::Protocol(msg) if msg == WRITE_LOCK_HELD_MESSAGE)+}++/// True when the write pool itself was starved of connections.+pub fn is_write_pool_starved(error: &sqlx::Error) -> bool {+ matches!(error, sqlx::Error::Protocol(msg) if msg == WRITE_POOL_STARVED_MESSAGE)+ || matches!(error, sqlx::Error::PoolTimedOut)+}++/// True when the write is worth retrying: the database is healthy, this batch+/// just lost a race for the write path. Callers use this to keep expected+/// stalls out of their error paths (and out of Sentry) without also swallowing+/// genuine faults.+pub fn is_retryable_write_stall(error: &sqlx::Error) -> bool {+ is_write_lock_contention(error) || is_write_pool_starved(error)+}+/// Outcome of draining one batch, used by the drain loop to drive recovery.#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub(crate) enum BatchOutcome {@@ -207,9 +319,9 @@ pub(crate) enum BatchOutcome {}#[derive(Clone, Copy)]-struct BatchTimeouts {- write_semaphore: Duration,- write_pool: Duration,+pub(crate) struct BatchTimeouts {+ pub(crate) write_semaphore: Duration,+ pub(crate) write_pool: Duration,}const BATCH_TIMEOUTS: BatchTimeouts = BatchTimeouts {@@ -473,6 +585,15 @@ pub(crate) struct WriteDrainOpts {pub degraded_after: u64,/// Fire the persistent-failure hook after this many consecutive fatal batches.pub persistent_after: u64,+ /// Fire the persistent-failure hook after this many consecutive *contended*+ /// batches, or once a contention run has spanned `contention_wall` with at+ /// least `contention_min_batches` batches. Injectable so the stuck-coordinator+ /// escape can be tested without waiting out the production budget.+ pub contention_persistent_after: u64,+ pub contention_wall: Duration,+ pub contention_min_batches: u64,+ /// Per-batch acquisition budgets. Injectable for the same reason.+ pub batch_timeouts: BatchTimeouts,/// Cancelled on `DatabaseManager::close()`. The drain loop exits and closes/// its current write pool (which may be a rebuilt one only it holds) so no/// SQLite connection survives teardown.@@ -488,6 +609,10 @@ impl Default for WriteDrainOpts {reopen_every: WRITE_POOL_REOPEN_EVERY,degraded_after: DEGRADED_AFTER,persistent_after: PERSISTENT_FAILURE_AFTER,+ contention_persistent_after: CONTENTION_PERSISTENT_AFTER,+ contention_wall: CONTENTION_PERSISTENT_AFTER_WALL,+ contention_min_batches: CONTENTION_PERSISTENT_MIN_BATCHES,+ batch_timeouts: BATCH_TIMEOUTS,shutdown: tokio_util::sync::CancellationToken::new(),}}@@ -883,11 +1008,20 @@ async fn drain_loop(reopen_every,degraded_after,persistent_after,+ contention_persistent_after,+ contention_wall,+ contention_min_batches,+ batch_timeouts,shutdown,} = opts;let mut batch: Vec<PendingWrite> = Vec::with_capacity(MAX_BATCH_SIZE);// Count + wall-clock + refire escalation rules (see FatalRunEscalation).let mut escalation = FatalRunEscalation::new(persistent_after);+ let mut contention_escalation = FatalRunEscalation::for_contention(+ contention_persistent_after,+ contention_wall,+ contention_min_batches,+ );loop {// Block until at least one write arrives, then take up to MAX_BATCH_SIZE@@ -941,7 +1075,7 @@ async fn drain_loop(&mut batch,&db_path,&health,- BATCH_TIMEOUTS,+ batch_timeouts,).await;batch.clear();@@ -955,12 +1089,15 @@ async fn drain_loop();}health.record_success();+ contention_escalation.on_healthy();if escalation.on_healthy() {health.note_fatal_run_recovered();info!("write_queue: fatal run cleared");}}BatchOutcome::Contention => {+ let now = std::time::Instant::now();+ let fire_hook = contention_escalation.on_fatal(now);let consecutive_contention = health.record_contention();warn!("write_queue: SQLite write contention exceeded the retry budget ({} consecutive batch(es))",@@ -972,6 +1109,26 @@ async fn drain_loop(// SQLite writer, but this must surface immediately and recover// only after a later successful batch.health.set_degraded();++ // A held permit that never comes back is a permanent wedge:+ // every batch times out, `degraded` is already set, and there+ // was no further escape — no reopen (correctly, it cannot help)+ // and no restart request. Restarting the engine drops the+ // leaked permit with the process, so it IS the cure here, but+ // only once the run has clearly outlived legitimate maintenance+ // (see CONTENTION_PERSISTENT_AFTER_WALL).+ if fire_hook {+ health.note_persistent_signal();+ error!(+ "write_queue: SQLite write lock held by another writer for {:?} across {} batch(es) — requesting engine restart to clear the stuck coordinator",+ contention_escalation.run_elapsed(now),+ contention_escalation.fatal_in_run+ );+ let hook = on_persistent_failure.hook();+ if let Some(hook) = hook {+ hook();+ }+ }}BatchOutcome::FatalConnection => {let now = std::time::Instant::now();@@ -1071,7 +1228,7 @@ async fn drain_loop(&mut tail_batch,&db_path,&health,- BATCH_TIMEOUTS,+ batch_timeouts,).await;tail_batch.clear();@@ -1102,7 +1259,13 @@ async fn execute_batch(}Err(_) => {warn!("write_queue: semaphore acquisition timed out for batch");- send_error_to_all(batch, sqlx::Error::PoolTimedOut);+ // Say what actually happened. This is NOT a pool timeout: the pool+ // was never consulted. Reporting it as one made CLI-SQ impossible+ // to tell apart from genuine starvation below.+ send_error_to_all(+ batch,+ sqlx::Error::Protocol(WRITE_LOCK_HELD_MESSAGE.to_string()),+ );// Another serialized writer still owns admission. Rebuilding the// pool cannot release that owner, but calling this healthy clears// degradation and suppresses the sustained-contention signal.@@ -1159,7 +1322,10 @@ async fn execute_batch(};}Err(_) => {- send_error_to_all(batch, sqlx::Error::PoolTimedOut);+ send_error_to_all(+ batch,+ sqlx::Error::Protocol(WRITE_POOL_STARVED_MESSAGE.to_string()),+ );// We already own the single write coordinator, so every normal// writer is excluded. Exhausting the connection wait here means// the pool itself is starved; drive the existing reopen/restart@@ -2282,13 +2448,41 @@ async fn insert_ui_event_row(Ok(result.last_insert_rowid())}+/// Fail every caller in `batch` with `error`.+///+/// The variant is preserved for the transport-level unit variants so callers+/// can still `matches!` on them — flattening *everything* into+/// `Error::Protocol(String)` is what made every write failure look alike to+/// every caller. `Error::Database` and friends are not `Clone`, so those keep+/// the existing stringified `Protocol` form, which is all issue #7331781144+/// asked for (the original *message* must survive, unmasked).fn send_error_to_all(batch: &mut Vec<PendingWrite>, error: sqlx::Error) {let err_str = error.to_string();for pw in batch.drain(..) {- let _ = pw.respond.send(Err(sqlx::Error::Protocol(err_str.clone())));+ let cloned = match &error {+ sqlx::Error::PoolTimedOut => sqlx::Error::PoolTimedOut,+ sqlx::Error::PoolClosed => sqlx::Error::PoolClosed,+ sqlx::Error::WorkerCrashed => sqlx::Error::WorkerCrashed,+ // Already a Protocol error: re-wrapping stacks another+ // "encountered unexpected or invalid data: " prefix onto the+ // message and breaks exact-message classification.+ sqlx::Error::Protocol(msg) => sqlx::Error::Protocol(msg.clone()),+ _ => sqlx::Error::Protocol(err_str.clone()),+ };+ let _ = pw.respond.send(Err(cloned));+ }+ // Log the original error that caused the batch failure. Contention is an+ // expected stall, not a fault: logging it at error! made every VACUUM and+ // checkpoint raise a Sentry issue (SCREENPIPE-CLI-SQ). The drain loop still+ // counts it and escalates a sustained run.+ if is_write_lock_contention(&error) {+ warn!(+ "write_queue: batch stalled behind another writer: {}",+ error+ );+ } else {+ error!("write_queue: batch failed: {}", error);}- // Log the original error that caused the batch failure- error!("write_queue: batch failed: {}", error);}/// Lowercased-message check shared with `is_connection_error`. Pulled@@ -2455,6 +2649,179 @@ mod tests {FatalRunEscalation::new(PERSISTENT_FAILURE_AFTER)}+ fn contention_esc() -> FatalRunEscalation {+ FatalRunEscalation::for_contention(+ CONTENTION_PERSISTENT_AFTER,+ CONTENTION_PERSISTENT_AFTER_WALL,+ CONTENTION_PERSISTENT_MIN_BATCHES,+ )+ }++ /// A permanently held write coordinator used to be unrecoverable: the+ /// `Contention` arm set `degraded` and stopped there — no pool reopen+ /// (correctly, it cannot help) and no restart request. Every subsequent+ /// batch timed out forever. A restart drops the leaked permit with the+ /// process, so the hook must eventually fire.+ #[test]+ fn sustained_contention_eventually_requests_a_restart() {+ let mut e = contention_esc();+ let t0 = std::time::Instant::now();++ // Enough batches to satisfy the minimum, but not yet long enough.+ for i in 0..CONTENTION_PERSISTENT_MIN_BATCHES {+ assert!(+ !e.on_fatal(t0 + Duration::from_secs(i)),+ "a short contention run must not request a restart"+ );+ }++ assert!(+ e.on_fatal(t0 + CONTENTION_PERSISTENT_AFTER_WALL),+ "a contention run past the wall-clock budget must request a restart"+ );+ }++ /// `compact()` (full VACUUM) and the RESTART checkpoint hold the same+ /// process-wide coordinator by design, and `db/maintenance.rs` documents+ /// the resulting pause as expected. Escalating those into an engine+ /// restart would be worse than the stall.+ #[test]+ fn legitimate_maintenance_holds_do_not_request_a_restart() {+ let mut e = contention_esc();+ let t0 = std::time::Instant::now();++ // A ten-minute VACUUM on a multi-GB capture database: far past a batch+ // budget, still well inside ours. Restarting mid-VACUUM would roll the+ // whole thing back.+ let maintenance = Duration::from_secs(600);+ assert!(+ maintenance < CONTENTION_PERSISTENT_AFTER_WALL,+ "the escalation budget must sit beyond a realistic maintenance hold"+ );+ for i in 0..CONTENTION_PERSISTENT_MIN_BATCHES * 2 {+ let at = t0 + maintenance * (i as u32) / (CONTENTION_PERSISTENT_MIN_BATCHES * 2) as u32;+ assert!(+ !e.on_fatal(at),+ "maintenance-length contention must not request a restart"+ );+ }++ // Maintenance finishes and writes resume: the run clears.+ for _ in 0..FATAL_RUN_CLEAR_AFTER {+ e.on_healthy();+ }++ // A fresh run then starts from zero rather than inheriting the old clock.+ assert!(+ !e.on_fatal(t0 + CONTENTION_PERSISTENT_AFTER_WALL * 2),+ "a recovered run must not carry its old elapsed time into the next stall"+ );+ }++ /// The unit rules above prove the state machine. This drives the *real*+ /// drain loop against a genuinely stuck coordinator — the production+ /// failure — and proves three things end to end:+ /// 1. every queued write is failed with a classifiable contention error+ /// (not a fabricated pool timeout),+ /// 2. the queue reports itself degraded, and+ /// 3. the restart hook fires, which is the only thing that can drop a+ /// leaked permit.+ ///+ /// Before this change (3) never happened: the `Contention` arm set+ /// `degraded` and stopped, so the wedge was permanent.+ #[tokio::test]+ async fn a_stuck_write_coordinator_is_escalated_by_the_real_drain_loop() {+ use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering};++ let pool = SqlitePoolOptions::new()+ .max_connections(1)+ .connect("sqlite::memory:")+ .await+ .unwrap();+ let semaphore = Arc::new(Semaphore::new(1));++ // A leaked permit: the exact production shape of a hung or aborted+ // task that never gives the coordinator back.+ let leaked_permit = semaphore.clone().acquire_owned().await.unwrap();++ let health = WriteQueueHealth::default();+ let restart_requested = Arc::new(AtomicBool::new(false));+ let hook_flag = restart_requested.clone();++ let queue = spawn_write_drain_with(+ pool.clone(),+ semaphore.clone(),+ Arc::from("sqlite::memory:"),+ WriteDrainOpts {+ health: health.clone(),+ on_persistent_failure: persistent_failure_slot(Some(Arc::new(move || {+ hook_flag.store(true, AtomicOrdering::SeqCst);+ }))),+ // Escalate on the count rule so the test does not have to wait+ // out a wall-clock budget. The wall-clock rule itself is+ // covered by `sustained_contention_eventually_requests_a_restart`.+ contention_persistent_after: 3,+ contention_wall: Duration::from_secs(600),+ contention_min_batches: 3,+ batch_timeouts: BatchTimeouts {+ write_semaphore: Duration::from_millis(10),+ write_pool: Duration::from_millis(10),+ },+ ..Default::default()+ },+ );++ for i in 0..3 {+ let error = queue+ .submit(WriteOp::InsertAudioChunk {+ file_path: format!("/tmp/wedged-{i}.wav"),+ timestamp: None,+ })+ .await+ .expect_err("a stuck coordinator must fail the write");+ assert!(+ is_write_lock_contention(&error),+ "caller must be able to tell this is contention, got: {error}"+ );+ }++ assert!(+ health.is_degraded(),+ "a stuck coordinator must report the write path degraded"+ );++ tokio::time::timeout(Duration::from_secs(5), async {+ while !restart_requested.load(AtomicOrdering::SeqCst) {+ tokio::task::yield_now().await;+ }+ })+ .await+ .expect("a stuck write coordinator must request an engine restart");++ drop(leaked_permit);+ drop(queue);+ pool.close().await;+ }++ /// One lucky commit in the middle of a wedge must not reset the clock, or+ /// an intermittently-available coordinator defers escalation forever. This+ /// is the contention twin of `escalation_single_success_does_not_reset_the_run`.+ #[test]+ fn a_single_success_does_not_clear_a_contention_run() {+ let mut e = contention_esc();+ let t0 = std::time::Instant::now();++ for i in 0..CONTENTION_PERSISTENT_MIN_BATCHES {+ e.on_fatal(t0 + Duration::from_secs(i));+ }+ assert!(!e.on_healthy(), "one success must not clear the run");++ assert!(+ e.on_fatal(t0 + CONTENTION_PERSISTENT_AFTER_WALL),+ "the run must still escalate despite an intervening success"+ );+ }+#[test]fn escalation_count_rule_fires_at_threshold() {let mut e = esc();@@ -2866,7 +3233,116 @@ mod tests {assert_eq!(outcome, BatchOutcome::Contention);let error = response.await.unwrap().unwrap_err();- assert!(error.to_string().contains("pool timed out"));+ // The pool was never consulted here, so this must not claim a pool+ // timeout. Callers branch on this to keep an expected stall out of+ // their error paths.+ assert!(+ is_write_lock_contention(&error),+ "semaphore starvation must report lock contention, got: {error}"+ );+ assert!(+ !is_write_pool_starved(&error),+ "semaphore starvation must not masquerade as pool starvation: {error}"+ );+ assert!(is_retryable_write_stall(&error));+ }++ /// The two batch timeouts have opposite meanings — "someone else is+ /// writing, retry" versus "this pool has no connections, recover" — and+ /// both used to be reported as a fabricated `PoolTimedOut`, flattened to+ /// the same string. SCREENPIPE-CLI-SQ was therefore untriageable: one+ /// Sentry issue for two unrelated faults. They must stay distinguishable.+ #[tokio::test]+ async fn lock_contention_and_pool_starvation_are_distinguishable() {+ async fn run(starve_pool: bool) -> sqlx::Error {+ let pool = SqlitePoolOptions::new()+ .max_connections(1)+ .connect("sqlite::memory:")+ .await+ .unwrap();+ let semaphore = Arc::new(Semaphore::new(1));+ // Hold whichever resource the scenario is about.+ let _held_permit = if starve_pool {+ None+ } else {+ Some(semaphore.clone().acquire_owned().await.unwrap())+ };+ let _starved_connection = if starve_pool {+ Some(pool.acquire().await.unwrap())+ } else {+ None+ };+ let health = WriteQueueHealth::default();+ let (respond, response) = tokio::sync::oneshot::channel();+ let mut batch = vec![PendingWrite {+ op: WriteOp::InsertAudioChunk {+ file_path: "/tmp/distinguishable.wav".to_string(),+ timestamp: None,+ },+ respond,+ }];+ execute_batch(+ &pool,+ &semaphore,+ &mut batch,+ "sqlite::memory:",+ &health,+ BatchTimeouts {+ write_semaphore: Duration::from_millis(10),+ write_pool: Duration::from_millis(10),+ },+ )+ .await;+ response.await.unwrap().unwrap_err()+ }++ let contention = run(false).await;+ let starvation = run(true).await;++ assert_ne!(+ contention.to_string(),+ starvation.to_string(),+ "the two failures must not share one message"+ );+ assert!(is_write_lock_contention(&contention));+ assert!(!is_write_lock_contention(&starvation));+ assert!(is_write_pool_starved(&starvation));+ assert!(!is_write_pool_starved(&contention));+ // Both are worth retrying, so a caller that only wants to suppress+ // expected stalls can use the one predicate.+ assert!(is_retryable_write_stall(&contention));+ assert!(is_retryable_write_stall(&starvation));+ }++ /// `send_error_to_all` used to flatten every error into+ /// `Error::Protocol(String)`, so no caller could ever `matches!` on the+ /// transport variants. SCREENPIPE-CLI-SN's collateral-abort signal+ /// (`WorkerCrashed`) and the shutdown signal (`PoolClosed`) both depend on+ /// the variant surviving.+ #[test]+ fn send_error_to_all_preserves_transport_variants() {+ for original in [+ sqlx::Error::PoolClosed,+ sqlx::Error::PoolTimedOut,+ sqlx::Error::WorkerCrashed,+ ] {+ let expected = std::mem::discriminant(&original);+ let (respond, response) = tokio::sync::oneshot::channel();+ let mut batch = vec![PendingWrite {+ op: WriteOp::InsertAudioChunk {+ file_path: "/tmp/variant.wav".into(),+ timestamp: None,+ },+ respond,+ }];+ send_error_to_all(&mut batch, original);+ let received = response.blocking_recv().unwrap().unwrap_err();+ assert_eq!(+ std::mem::discriminant(&received),+ expected,+ "variant must survive the queue, got: {received}"+ );+ }}#[tokio::test]@@ -2903,7 +3379,14 @@ mod tests {assert_eq!(outcome, BatchOutcome::FatalConnection);let error = response.await.unwrap().unwrap_err();- assert!(error.to_string().contains("pool timed out"));+ assert!(+ is_write_pool_starved(&error),+ "pool starvation must be reported as such, got: {error}"+ );+ assert!(+ !is_write_lock_contention(&error),+ "pool starvation must not masquerade as lock contention: {error}"+ );}#[tokio::test]diff --git a/crates/screenpipe-engine/src/ui_recorder.rs b/crates/screenpipe-engine/src/ui_recorder.rsindex 9fd68be78..e3a34512a 100644--- a/crates/screenpipe-engine/src/ui_recorder.rs+++ b/crates/screenpipe-engine/src/ui_recorder.rs@@ -946,10 +946,16 @@ pub async fn start_ui_recording(.await;last_flush = std::time::Instant::now();- // Exponential backoff on consecutive failures+ // Exponential backoff on consecutive failures. The shift+ // is capped at 6 so the documented 30s ceiling is+ // actually reachable: `.min(5)` topped out at 16s and+ // made the `.min(30_000)` clamp dead code, so a long+ // write stall was re-attacked twice as often as+ // intended.if consecutive_failures > 0 {let backoff = Duration::from_millis(- (500 * (1u64 << consecutive_failures.min(5))).min(30_000),+ (500u64.saturating_mul(1u64 << consecutive_failures.min(6)))+ .min(30_000),);debug!("UI recorder: backing off {}ms after {} failures",@@ -1134,7 +1140,25 @@ async fn flush_batch(}Err(e) => {*consecutive_failures += 1;- if *consecutive_failures <= 3 {+ // A write that lost a race for the SQLite write path is an expected+ // stall, not a fault: the batch is retained above and the next flush+ // retries it, and `compact()`/checkpoint hold that same process-wide+ // lock by design. Raising it at error! sent every VACUUM to Sentry+ // as SCREENPIPE-CLI-SQ. A genuinely stuck coordinator is escalated+ // by the write queue itself, not by counting log lines here.+ if screenpipe_db::is_retryable_write_stall(&e) {+ if *consecutive_failures <= 3 {+ warn!(+ "UI events batch stalled behind the SQLite write path, retrying: {}",+ e+ );+ } else {+ debug!(+ "UI events batch still stalled (attempt #{}): {}",+ consecutive_failures, e+ );+ }+ } else if *consecutive_failures <= 3 {error!("Failed to insert UI events batch: {}", e);} else {// Reduce log spam during contention storms@@ -1654,6 +1678,36 @@ mod event_batch_tests {);}+ /// SCREENPIPE-CLI-SQ: a batch that lost a race for the SQLite write path+ /// is an expected stall — `compact()` and the RESTART checkpoint hold that+ /// same process-wide lock by design — so it must not be reported as a+ /// fault. What it *must* still do is retain the events for retry.+ ///+ /// This asserts the classification the log level is derived from, which is+ /// the part that decides whether a routine VACUUM raises a Sentry issue.+ #[test]+ fn write_path_contention_is_classified_as_a_retryable_stall() {+ let contention = sqlx::Error::Protocol(screenpipe_db::WRITE_LOCK_HELD_MESSAGE.to_string());+ let starvation =+ sqlx::Error::Protocol(screenpipe_db::WRITE_POOL_STARVED_MESSAGE.to_string());+ assert!(screenpipe_db::is_retryable_write_stall(&contention));+ assert!(screenpipe_db::is_retryable_write_stall(&starvation));++ // A genuine fault must keep its error path. `DROP TABLE ui_events`+ // (the scenario in the test above) produces exactly this shape, and it+ // must never be silently downgraded to a stall.+ let real_fault = sqlx::Error::Protocol(+ "error returned from database: (code: 1) no such table: ui_events".to_string(),+ );+ assert!(+ !screenpipe_db::is_retryable_write_stall(&real_fault),+ "a real schema/IO fault must not be treated as contention"+ );+ assert!(!screenpipe_db::is_retryable_write_stall(+ &sqlx::Error::RowNotFound+ ));+ }+/// The flip side: a successful flush clears the batch and resets the/// failure counter so the full-batch flush path re-engages.#[tokio::test]diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.mdindex 500921c7b..5b481c4e7 100644--- a/docs/coverage/CORE.md+++ b/docs/coverage/CORE.md@@ -9,10 +9,10 @@ confidence, and criticality.- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y- Mapped suites: 32- Mapped Rust files: 318-- Active test blocks: 3018+- Active test blocks: 3025- Ignored/manual test blocks: 137-- Declared test blocks: 3155-- Weighted coverage points: 2481.3+- Declared test blocks: 3162+- Weighted coverage points: 2488.0Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.Criticality weights: high=1.0, medium=0.7, low=0.4.@@ -23,16 +23,16 @@ are explicitly enabled in a runtime lane.| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2887 | 132 | 2421.4 | 21 | 11 | 100% |-| macos | 29 | 2941 | 112 | 2432.5 | 22 | 11 | 100% |-| linux | 25 | 2574 | 105 | 2138.8 | 20 | 11 | 100% |+| windows | 29 | 2894 | 132 | 2428.1 | 21 | 11 | 100% |+| macos | 29 | 2948 | 112 | 2439.2 | 22 | 11 | 100% |+| linux | 25 | 2581 | 105 | 2145.6 | 20 | 11 | 100% |## Crate Summary| Crate | Suites | Integration files | Source unit files | Active tests | Ignored tests | Weighted points | Flows || --- | --- | --- | --- | --- | --- | --- | --- |-| screenpipe-engine | 10 | 18 | 102 | 1438 | 42 | 1102.5 | 10 |-| screenpipe-db | 5 | 51 | 14 | 424 | 16 | 402.1 | 9 |+| screenpipe-engine | 10 | 18 | 102 | 1439 | 42 | 1103.2 | 10 |+| screenpipe-db | 5 | 51 | 14 | 430 | 16 | 408.1 | 9 || screenpipe-sqlite-coordinator | 1 | 0 | 2 | 11 | 0 | 11.0 | 2 || screenpipe-audio | 6 | 25 | 49 | 575 | 43 | 503.2 | 5 || screenpipe-screen | 6 | 9 | 18 | 237 | 9 | 215.0 | 4 |@@ -65,23 +65,23 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| audio | 7 suites / 671 active / 44 ignored / 599.2 pts | 7 suites / 671 active / 44 ignored / 599.2 pts | 6 suites / 596 active / 43 ignored / 546.7 pts || audio-device | 2 suites / 211 active / 7 ignored / 188.5 pts | 2 suites / 211 active / 7 ignored / 188.5 pts | 1 suites / 136 active / 6 ignored / 136.0 pts || configuration | 2 suites / 138 active / 3 ignored / 124.2 pts | 2 suites / 138 active / 3 ignored / 124.2 pts | 2 suites / 138 active / 3 ignored / 124.2 pts |-| database | 6 suites / 338 active / 12 ignored / 316.1 pts | 6 suites / 338 active / 12 ignored / 316.1 pts | 6 suites / 338 active / 12 ignored / 316.1 pts |+| database | 6 suites / 344 active / 12 ignored / 322.1 pts | 6 suites / 344 active / 12 ignored / 322.1 pts | 6 suites / 344 active / 12 ignored / 322.1 pts || db-search | 2 suites / 111 active / 9 ignored / 111.0 pts | 2 suites / 111 active / 9 ignored / 111.0 pts | 2 suites / 111 active / 9 ignored / 111.0 pts || engine-lifecycle | 6 suites / 203 active / 1 ignored / 178.6 pts | 6 suites / 203 active / 1 ignored / 178.6 pts | 5 suites / 197 active / 1 ignored / 176.9 pts || local-api | 2 suites / 325 active / 9 ignored / 229.3 pts | 2 suites / 325 active / 9 ignored / 229.3 pts | 2 suites / 325 active / 9 ignored / 229.3 pts |-| meeting | 6 suites / 1383 active / 19 ignored / 1126.5 pts | 6 suites / 1383 active / 19 ignored / 1126.5 pts | 4 suites / 1105 active / 15 ignored / 871.0 pts |+| meeting | 6 suites / 1384 active / 19 ignored / 1127.2 pts | 6 suites / 1384 active / 19 ignored / 1127.2 pts | 4 suites / 1106 active / 15 ignored / 871.7 pts || ocr | 4 suites / 124 active / 7 ignored / 118.3 pts | 4 suites / 128 active / 7 ignored / 123.8 pts | 3 suites / 119 active / 6 ignored / 114.8 pts || os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - |-| performance | 13 suites / 1354 active / 67 ignored / 1195.8 pts | 14 suites / 1452 active / 70 ignored / 1235.0 pts | 13 suites / 1354 active / 67 ignored / 1195.8 pts |-| pipes | 1 suites / 461 active / 3 ignored / 322.7 pts | 1 suites / 461 active / 3 ignored / 322.7 pts | 1 suites / 461 active / 3 ignored / 322.7 pts |-| privacy | 5 suites / 869 active / 36 ignored / 706.1 pts | 5 suites / 919 active / 16 ignored / 711.7 pts | 5 suites / 845 active / 14 ignored / 684.3 pts |+| performance | 13 suites / 1360 active / 67 ignored / 1201.8 pts | 14 suites / 1458 active / 70 ignored / 1241.0 pts | 13 suites / 1360 active / 67 ignored / 1201.8 pts |+| pipes | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts |+| privacy | 5 suites / 870 active / 36 ignored / 706.8 pts | 5 suites / 920 active / 16 ignored / 712.4 pts | 5 suites / 846 active / 14 ignored / 685.0 pts || real-app | - | 1 suites / 98 active / 3 ignored / 39.2 pts | - || speaker | 2 suites / 325 active / 8 ignored / 325.0 pts | 2 suites / 325 active / 8 ignored / 325.0 pts | 2 suites / 325 active / 8 ignored / 325.0 pts |-| storage | 3 suites / 464 active / 29 ignored / 375.5 pts | 3 suites / 464 active / 29 ignored / 375.5 pts | 3 suites / 464 active / 29 ignored / 375.5 pts |-| sync | 1 suites / 461 active / 3 ignored / 322.7 pts | 1 suites / 461 active / 3 ignored / 322.7 pts | 1 suites / 461 active / 3 ignored / 322.7 pts |-| timeline | 4 suites / 922 active / 33 ignored / 749.2 pts | 4 suites / 922 active / 33 ignored / 749.2 pts | 4 suites / 922 active / 33 ignored / 749.2 pts |+| storage | 3 suites / 470 active / 29 ignored / 381.5 pts | 3 suites / 470 active / 29 ignored / 381.5 pts | 3 suites / 470 active / 29 ignored / 381.5 pts |+| sync | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts |+| timeline | 4 suites / 928 active / 33 ignored / 755.2 pts | 4 suites / 928 active / 33 ignored / 755.2 pts | 4 suites / 928 active / 33 ignored / 755.2 pts || transcription | 5 suites / 683 active / 40 ignored / 538.0 pts | 5 suites / 683 active / 40 ignored / 538.0 pts | 5 suites / 683 active / 40 ignored / 538.0 pts |-| ui-events | 4 suites / 699 active / 28 ignored / 532.5 pts | 3 suites / 651 active / 5 ignored / 498.9 pts | 3 suites / 651 active / 5 ignored / 498.9 pts |+| ui-events | 4 suites / 700 active / 28 ignored / 533.2 pts | 3 suites / 652 active / 5 ignored / 499.6 pts | 3 suites / 652 active / 5 ignored / 499.6 pts || vision-capture | 5 suites / 485 active / 32 ignored / 385.9 pts | 5 suites / 489 active / 32 ignored / 391.4 pts | 4 suites / 480 active / 31 ignored / 382.4 pts |## Critical Flow Matrix@@ -132,14 +132,14 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-audio-meetings-speakers | screenpipe-db | windows, macos, linux | database, audio, meeting, speaker | audio-record-transcribe, audio-device-health, meeting-live-notes | high | strong | integration | 15 | 96 | 1 | Audio transcript dedupe, live meeting mirroring, end-generation and open-meeting invariants, liveness, and speaker reassignment coverage. || db-runtime-reliability | screenpipe-db | windows, macos, linux | database, performance | performance-liveness | high | partial | mixed | 12 | 27 | 6 | SQLite hard-fault classification, failpoint VFS injection, fresh-identity recovery verification with integrity/FK/write canaries, query cancellation, close-severs-connections regressions, multi-pool WAL parity, runtime version pinning, and WAL chaos plus memory-pressure probes. || db-search-indexing | screenpipe-db | windows, macos, linux | db-search, ocr, accessibility, performance | local-api-search, capture-ocr-pipeline, accessibility-ui-events, performance-liveness | high | strong | mixed | 13 | 105 | 4 | FTS, tokenizer, OCR snapshot search, query planning, ordering, accessibility search, and contention coverage. |-| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 18 | 169 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. |+| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 18 | 175 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. || engine-api-routes | screenpipe-engine | windows, macos, linux | local-api, timeline, meeting, transcription | local-api-search, timeline-streaming, meeting-live-notes, audio-record-transcribe | high | partial | mixed | 31 | 319 | 4 | Route/unit coverage for search, health, streaming, meetings, time/timezone, and transcription. Legacy endpoint/websocket tests require local data and remain ignored. || engine-capture-timeline | screenpipe-engine | windows, macos, linux | vision-capture, timeline, storage, performance | capture-ocr-pipeline, timeline-streaming, performance-liveness | high | partial | mixed | 23 | 257 | 26 | Covers capture trigger logic, frame/audio linking, hot cache, timeline refresh regressions, fragmented MP4 extraction, and HD-mode control. Several real-data tests are intentionally ignored by default. || engine-config-lifecycle | screenpipe-engine | windows, macos, linux | configuration, engine-lifecycle, performance | settings-to-engine-config, engine-health-lifecycle, performance-liveness | high | strong | mixed | 11 | 111 | 1 | Fast logic coverage for the config bridge, tray health debounce, sleep/power policies, and queue backpressure. || engine-db-recovery-cli | screenpipe-engine | windows, macos, linux | database, engine-lifecycle | engine-health-lifecycle, performance-liveness | high | strong | unit | 1 | 8 | 0 | Exact DB/WAL/SHM working-copy preservation, rollback on archive failure, and restart repair for crashes during the multi-file generation swap. || engine-focus-os | screenpipe-engine | windows, macos | engine-lifecycle, os-integration | engine-health-lifecycle, performance-liveness | medium | conditional | unit | 3 | 6 | 0 | Platform focus-tracker parsing/helpers. These files are cfg-gated and only execute on their target OS. || engine-local-api-search-integration | screenpipe-engine | windows, macos, linux | local-api, db-search | local-api-search | high | strong | integration | 1 | 6 | 5 | Active /search route test builds an audio-disabled router, seeds captured-screen-shaped OCR data into an in-memory DB, and asserts the HTTP response and pagination. |-| engine-meeting-privacy-sync | screenpipe-engine | windows, macos, linux | meeting, privacy, ui-events, pipes, sync | meeting-live-notes, privacy-and-redaction, accessibility-ui-events, performance-liveness | medium | strong | unit | 30 | 461 | 3 | Unit-heavy coverage for privacy filter policy, capture exclusions, UI recorder safety, pipes/live-view/structured-output helpers, sync helpers, and CLI parsing. Meeting detection moved to the engine-meeting-watcher suite. |+| engine-meeting-privacy-sync | screenpipe-engine | windows, macos, linux | meeting, privacy, ui-events, pipes, sync | meeting-live-notes, privacy-and-redaction, accessibility-ui-events, performance-liveness | medium | strong | unit | 30 | 462 | 3 | Unit-heavy coverage for privacy filter policy, capture exclusions, UI recorder safety, pipes/live-view/structured-output helpers, sync helpers, and CLI parsing. Meeting detection moved to the engine-meeting-watcher suite. || engine-meeting-watcher | screenpipe-engine | windows, macos | meeting | meeting-live-notes | high | strong | mixed | 9 | 203 | 3 | Successor of src/meeting_detector.rs and src/meeting_telemetry.rs. Audio-process and UI-scan backend state machines, candidate resolution, shared telemetry, and a scored trajectory eval. ui_scan/macos.rs and ui_scan/windows.rs are cfg-gated and only execute on their target OS; both backends are null on Linux. || engine-retention-storage | screenpipe-engine | windows, macos, linux | storage, engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | mixed | 5 | 38 | 0 | Local retention deletion (including lean-mode heavy-text stripping), cloud archive watermarking, atomic state-file replacement, and the low-disk capture monitor. || engine-telemetry-observability | screenpipe-engine | windows, macos, linux | engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | unit | 6 | 29 | 0 | PostHog capture gating, telemetry context shaping, piggyback telemetry forwarding, crash-log helpers, resource monitoring, and the recording-coverage reliability metric. |@@ -272,7 +272,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-search-indexing | screenpipe-db | src/text_normalizer.rs | source | 19 | 0 | 19 || db-search-indexing | screenpipe-db | src/text_similarity.rs | source | 20 | 0 | 20 || db-timeline-frames | screenpipe-db | src/types.rs | source | 3 | 0 | 3 |-| db-timeline-frames | screenpipe-db | src/write_queue.rs | source | 30 | 0 | 30 |+| db-timeline-frames | screenpipe-db | src/write_queue.rs | source | 36 | 0 | 36 || db-search-indexing | screenpipe-db | tests/accessibility_late_materialization_test.rs | integration | 2 | 0 | 2 || db-audio-meetings-speakers | screenpipe-db | tests/audio_duplicate_test.rs | integration | 12 | 0 | 12 || db-audio-meetings-speakers | screenpipe-db | tests/audio_search_speaker_join_test.rs | integration | 1 | 0 | 1 |@@ -420,7 +420,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| engine-meeting-privacy-sync | screenpipe-engine | src/sync_api.rs | source | 12 | 0 | 12 || engine-meeting-privacy-sync | screenpipe-engine | src/sync_provider.rs | source | 4 | 0 | 4 || engine-telemetry-observability | screenpipe-engine | src/telemetry_context.rs | source | 8 | 0 | 8 |-| engine-meeting-privacy-sync | screenpipe-engine | src/ui_recorder.rs | source | 49 | 0 | 49 |+| engine-meeting-privacy-sync | screenpipe-engine | src/ui_recorder.rs | source | 50 | 0 | 50 || engine-capture-timeline | screenpipe-engine | src/video_utils.rs | source | 1 | 0 | 1 || engine-capture-timeline | screenpipe-engine | src/vision_manager/manager.rs | source | 12 | 0 | 12 || engine-capture-timeline | screenpipe-engine | src/vision_manager/monitor_watcher.rs | source | 26 | 0 | 26 |