Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

fix(vision): split trigger-wait so #3939 freezes name the real stage (#6241)

Louis Beaumont38e3f1eff401
diff --git a/COVERAGE.md b/COVERAGE.md
index 013b5eedc..31d74f2c0 100644
--- a/COVERAGE.md
+++ b/COVERAGE.md
@@ -32,15 +32,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.
- Mapped suites: 32
- Mapped Rust files: 321
-- Active test blocks: 3098
+- Active test blocks: 3101
- Ignored/manual test blocks: 137
-- Weighted coverage points: 2550.9
+- Weighted coverage points: 2553.9
| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score |
| --- | --- | --- | --- | --- | --- | --- | --- |
-| windows | 29 | 2967 | 132 | 2490.9 | 21 | 11 | 100% |
-| macos | 29 | 3020 | 112 | 2501.3 | 22 | 11 | 100% |
-| linux | 25 | 2643 | 105 | 2197.7 | 20 | 11 | 100% |
+| windows | 29 | 2970 | 132 | 2493.9 | 21 | 11 | 100% |
+| macos | 29 | 3023 | 112 | 2504.3 | 22 | 11 | 100% |
+| linux | 25 | 2646 | 105 | 2200.7 | 20 | 11 | 100% |
## Refresh
diff --git a/apps/screenpipe-app-tauri/e2e/specs/capture-stall-stage-marker.spec.ts b/apps/screenpipe-app-tauri/e2e/specs/capture-stall-stage-marker.spec.ts
index 8a1f1f64a..b56c051d6 100644
--- a/apps/screenpipe-app-tauri/e2e/specs/capture-stall-stage-marker.spec.ts
+++ b/apps/screenpipe-app-tauri/e2e/specs/capture-stall-stage-marker.spec.ts
@@ -43,7 +43,8 @@
import { existsSync, readFileSync, readdirSync } from "node:fs";
import { homedir } from "node:os";
-import { resolve } from "node:path";
+import { dirname, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
import { E2E_SEED_FLAGS, getAppPid } from "../helpers/app-launcher.js";
import {
@@ -57,21 +58,51 @@ import { t, waitForAppReady } from "../helpers/test-utils.js";
const dataDir = resolve(homedir(), ".screenpipe", ".e2e");
const SILENT_MARKER = resolve(dataDir, "e2e-capture-loop-silent-fired");
-/** Every stage the loop can report. A frozen loop must name one of these. */
-const KNOWN_STAGES = [
- "heartbeat",
- "focus-gate",
- "warm-wait",
- "cold-wait",
- "pause-gate",
- "release-stream",
- "invalidate-streams",
- "exclusion-probe",
- "visual-probe",
- "trigger-wait",
- "capture",
- "hot-cache-push",
-];
+const METRICS_RS = resolve(
+ dirname(fileURLToPath(import.meta.url)),
+ "../../../../crates/screenpipe-screen/src/metrics.rs",
+);
+
+/**
+ * Every stage the loop can report, read from the enum that emits them.
+ *
+ * Derived rather than hand-mirrored. A copied list goes stale the moment a
+ * stage is added, and since a short-lived stage is rarely the one sampled, the
+ * stale copy fails as an intermittent flake instead of an honest error — which
+ * is how it gets written off as infrastructure. Reading the single source of
+ * truth means a new stage is either picked up automatically, or extraction
+ * fails loudly right here.
+ */
+function knownStagesFromRust(): string[] {
+ const source = readFileSync(METRICS_RS, "utf8");
+ const asStr = source.match(
+ /pub fn as_str\(self\) -> &'static str \{([\s\S]*?)\n {4}\}/,
+ );
+ if (!asStr) {
+ throw new Error(
+ `could not find CaptureLoopStage::as_str in ${METRICS_RS}. ` +
+ `Fix the extraction rather than hardcoding the stage list.`,
+ );
+ }
+
+ const stages = [...asStr[1].matchAll(/=>\s*"([a-z-]+)"/g)].map((m) => m[1]);
+
+ // Guard the extraction itself. A regex that silently matched nothing, or
+ // matched the wrong block, would leave every membership assertion below
+ // vacuously true — the spec would pass while checking nothing.
+ for (const anchor of ["unknown", "heartbeat", "capture", "trigger-wait"]) {
+ if (!stages.includes(anchor)) {
+ throw new Error(
+ `stage extraction looks wrong: expected "${anchor}" among ` +
+ `[${stages.join(", ")}]`,
+ );
+ }
+ }
+ return stages;
+}
+
+/** A frozen loop must name one of these. */
+const KNOWN_STAGES = knownStagesFromRust();
type HealthBody = {
frame_status?: string;
diff --git a/crates/screenpipe-engine/src/event_driven_capture.rs b/crates/screenpipe-engine/src/event_driven_capture.rs
index c18ed793e..b33277d36 100644
--- a/crates/screenpipe-engine/src/event_driven_capture.rs
+++ b/crates/screenpipe-engine/src/event_driven_capture.rs
@@ -1627,6 +1627,21 @@ pub(crate) async fn event_driven_capture_loop(
}
}
+ // The bounded wait above returned, so anything past this point is
+ // synchronous gate work, not the timer. Re-marking here keeps the
+ // watchdog's "frozen in <stage>" message honest: `trigger-wait`
+ // now accuses only the 250ms-bounded await (a freeze there means
+ // the task was never resumed), while `trigger-drain` accuses the
+ // synchronous gates below (a freeze there means a lock is held by
+ // another thread). Previously both shared one marker, so a
+ // multi-minute #3939 freeze reported `trigger-wait` even though a
+ // 250ms timeout cannot account for it.
+ record_loop_stage(
+ &vision_metrics,
+ &monitor_liveness,
+ screenpipe_screen::CaptureLoopStage::TriggerDrain,
+ );
+
// Drain any remaining triggers that piled up while we were
// waiting on the first one.
if !closed_now {
diff --git a/crates/screenpipe-screen/src/metrics.rs b/crates/screenpipe-screen/src/metrics.rs
index 7ed4bdf3e..8a9219626 100644
--- a/crates/screenpipe-screen/src/metrics.rs
+++ b/crates/screenpipe-screen/src/metrics.rs
@@ -87,12 +87,24 @@ pub enum CaptureLoopStage {
ExclusionProbe = 8,
/// Bounded visual-change probe screenshot.
VisualProbe = 9,
- /// Waiting for a capture trigger.
+ /// Parked on the bounded (250ms) wait for the first capture trigger.
TriggerWait = 10,
/// Running the capture itself (screenshot + a11y walk + persist).
Capture = 11,
/// Pushing the captured frame into the hot cache.
HotCachePush = 12,
+ /// Trigger wait returned; draining coalesced triggers and running the
+ /// synchronous gates (reduce, checkpoint promotion, visual-check decision)
+ /// before the exclusion probe.
+ ///
+ /// Split out of [`CaptureLoopStage::TriggerWait`] because that marker
+ /// covered both the bounded await *and* every synchronous gate after it.
+ /// A 250ms-bounded await cannot itself explain the multi-minute
+ /// gone-silent freezes, so "frozen in trigger-wait" was ambiguous exactly
+ /// where the #3939 triage needed precision: `trigger-wait` now means the
+ /// task was never resumed from a bounded timer, while `trigger-drain`
+ /// means a synchronous gate blocked (a lock held by another thread).
+ TriggerDrain = 13,
}
/// Whether the capture loop is currently taking screenshots, and if not, why.
@@ -178,6 +190,7 @@ impl CaptureLoopStage {
CaptureLoopStage::TriggerWait => "trigger-wait",
CaptureLoopStage::Capture => "capture",
CaptureLoopStage::HotCachePush => "hot-cache-push",
+ CaptureLoopStage::TriggerDrain => "trigger-drain",
}
}
@@ -198,6 +211,7 @@ impl CaptureLoopStage {
10 => CaptureLoopStage::TriggerWait,
11 => CaptureLoopStage::Capture,
12 => CaptureLoopStage::HotCachePush,
+ 13 => CaptureLoopStage::TriggerDrain,
_ => CaptureLoopStage::Unknown,
}
}
@@ -1053,4 +1067,124 @@ mod tests {
assert_eq!(s.frames_dropped_error, 1);
assert_eq!(s.frames_dropped, 3);
}
+
+ /// The wire form (discriminant + name) every stage is expected to have.
+ ///
+ /// Deliberately no wildcard arm: adding a variant to [`CaptureLoopStage`]
+ /// stops compiling *here*, which forces whoever adds it to extend
+ /// `from_u8` and `ALL_STAGES` in the same edit. `from_u8` has a `_`
+ /// fallback, so without this guard a missing arm compiles fine and the
+ /// watchdog silently reports "unknown" for the new stage — the exact
+ /// failure the stage marker exists to prevent.
+ fn expected_wire_form(stage: CaptureLoopStage) -> (u8, &'static str) {
+ match stage {
+ CaptureLoopStage::Unknown => (0, "unknown"),
+ CaptureLoopStage::Heartbeat => (1, "heartbeat"),
+ CaptureLoopStage::FocusGate => (2, "focus-gate"),
+ CaptureLoopStage::WarmWait => (3, "warm-wait"),
+ CaptureLoopStage::ColdWait => (4, "cold-wait"),
+ CaptureLoopStage::PauseGate => (5, "pause-gate"),
+ CaptureLoopStage::ReleaseStream => (6, "release-stream"),
+ CaptureLoopStage::InvalidateStreams => (7, "invalidate-streams"),
+ CaptureLoopStage::ExclusionProbe => (8, "exclusion-probe"),
+ CaptureLoopStage::VisualProbe => (9, "visual-probe"),
+ CaptureLoopStage::TriggerWait => (10, "trigger-wait"),
+ CaptureLoopStage::Capture => (11, "capture"),
+ CaptureLoopStage::HotCachePush => (12, "hot-cache-push"),
+ CaptureLoopStage::TriggerDrain => (13, "trigger-drain"),
+ }
+ }
+
+ const ALL_STAGES: &[CaptureLoopStage] = &[
+ CaptureLoopStage::Unknown,
+ CaptureLoopStage::Heartbeat,
+ CaptureLoopStage::FocusGate,
+ CaptureLoopStage::WarmWait,
+ CaptureLoopStage::ColdWait,
+ CaptureLoopStage::PauseGate,
+ CaptureLoopStage::ReleaseStream,
+ CaptureLoopStage::InvalidateStreams,
+ CaptureLoopStage::ExclusionProbe,
+ CaptureLoopStage::VisualProbe,
+ CaptureLoopStage::TriggerWait,
+ CaptureLoopStage::Capture,
+ CaptureLoopStage::HotCachePush,
+ CaptureLoopStage::TriggerDrain,
+ ];
+
+ /// Every stage must survive the `AtomicU8` round-trip the watchdog reads
+ /// back, and must keep the discriminant it was released with.
+ #[test]
+ fn every_loop_stage_round_trips_through_its_discriminant() {
+ for stage in ALL_STAGES {
+ let (discriminant, name) = expected_wire_form(*stage);
+ assert_eq!(
+ *stage as u8, discriminant,
+ "{name} changed discriminant; stored values in flight would be misread"
+ );
+ assert_eq!(
+ CaptureLoopStage::from_u8(discriminant),
+ *stage,
+ "{name} does not round-trip through discriminant {discriminant}"
+ );
+ assert_eq!(stage.as_str(), name, "{name} reports the wrong name");
+ }
+ }
+
+ /// Catches the other half of the drift: a variant added to the enum and to
+ /// `from_u8`, but never added to `ALL_STAGES`, so the loop above silently
+ /// stops covering it.
+ #[test]
+ fn all_stages_covers_every_decodable_discriminant() {
+ let highest = ALL_STAGES
+ .iter()
+ .map(|stage| *stage as u8)
+ .max()
+ .expect("ALL_STAGES is never empty");
+
+ for discriminant in 0..=highest {
+ let claimants = ALL_STAGES
+ .iter()
+ .filter(|stage| **stage as u8 == discriminant)
+ .count();
+ assert_eq!(
+ claimants, 1,
+ "discriminant {discriminant} is claimed by {claimants} stages, expected exactly 1"
+ );
+ }
+
+ // Anything past the known range must stay Unknown. If it decodes to a
+ // real stage, a variant was added to the enum and `from_u8` but left
+ // out of `ALL_STAGES`.
+ for discriminant in (u16::from(highest) + 1)..=u16::from(u8::MAX) {
+ let decoded = CaptureLoopStage::from_u8(discriminant as u8);
+ assert_eq!(
+ decoded,
+ CaptureLoopStage::Unknown,
+ "discriminant {discriminant} decodes to {} but is missing from ALL_STAGES",
+ decoded.as_str()
+ );
+ }
+ }
+
+ /// The trigger wait and the synchronous gates after it must be
+ /// distinguishable, otherwise a #3939 freeze cannot be attributed to
+ /// either the timer or a blocking lock.
+ #[test]
+ fn trigger_wait_and_trigger_drain_are_distinct_stages() {
+ assert_ne!(
+ CaptureLoopStage::TriggerWait as u8,
+ CaptureLoopStage::TriggerDrain as u8
+ );
+ assert_eq!(CaptureLoopStage::TriggerWait.as_str(), "trigger-wait");
+ assert_eq!(CaptureLoopStage::TriggerDrain.as_str(), "trigger-drain");
+
+ // The watchdog reads the stage out of the live metrics, so prove the
+ // transition is observable there and not just on the enum.
+ let m = PipelineMetrics::new();
+ m.record_loop_stage(CaptureLoopStage::TriggerWait);
+ assert_eq!(m.loop_stage().0, CaptureLoopStage::TriggerWait);
+ m.record_loop_stage(CaptureLoopStage::TriggerDrain);
+ assert_eq!(m.loop_stage().0, CaptureLoopStage::TriggerDrain);
+ }
}
diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.md
index 24ab3da44..9a97c14c1 100644
--- a/docs/coverage/CORE.md
+++ b/docs/coverage/CORE.md
@@ -9,10 +9,10 @@ confidence, and criticality.
- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y
- Mapped suites: 32
- Mapped Rust files: 321
-- Active test blocks: 3098
+- Active test blocks: 3101
- Ignored/manual test blocks: 137
-- Declared test blocks: 3235
-- Weighted coverage points: 2550.9
+- Declared test blocks: 3238
+- Weighted coverage points: 2553.9
Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.
Criticality weights: high=1.0, medium=0.7, low=0.4.
@@ -23,9 +23,9 @@ are explicitly enabled in a runtime lane.
| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score |
| --- | --- | --- | --- | --- | --- | --- | --- |
-| windows | 29 | 2967 | 132 | 2490.9 | 21 | 11 | 100% |
-| macos | 29 | 3020 | 112 | 2501.3 | 22 | 11 | 100% |
-| linux | 25 | 2643 | 105 | 2197.7 | 20 | 11 | 100% |
+| windows | 29 | 2970 | 132 | 2493.9 | 21 | 11 | 100% |
+| macos | 29 | 3023 | 112 | 2504.3 | 22 | 11 | 100% |
+| linux | 25 | 2646 | 105 | 2200.7 | 20 | 11 | 100% |
## Crate Summary
@@ -35,7 +35,7 @@ are explicitly enabled in a runtime lane.
| screenpipe-db | 5 | 52 | 14 | 447 | 16 | 424.2 | 9 |
| screenpipe-sqlite-coordinator | 1 | 0 | 2 | 16 | 0 | 16.0 | 2 |
| screenpipe-audio | 6 | 25 | 51 | 591 | 43 | 517.4 | 5 |
-| screenpipe-screen | 6 | 9 | 18 | 244 | 9 | 219.9 | 4 |
+| screenpipe-screen | 6 | 9 | 18 | 247 | 9 | 222.9 | 4 |
| screenpipe-a11y | 4 | 2 | 28 | 334 | 27 | 248.3 | 3 |
## Line Coverage
@@ -72,17 +72,17 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| meeting | 6 suites / 1428 active / 19 ignored / 1167.9 pts | 6 suites / 1428 active / 19 ignored / 1167.9 pts | 4 suites / 1140 active / 15 ignored / 902.4 pts |
| ocr | 4 suites / 124 active / 7 ignored / 118.3 pts | 4 suites / 128 active / 7 ignored / 123.8 pts | 3 suites / 119 active / 6 ignored / 114.8 pts |
| os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - |
-| performance | 13 suites / 1379 active / 67 ignored / 1216.3 pts | 14 suites / 1477 active / 70 ignored / 1255.5 pts | 13 suites / 1379 active / 67 ignored / 1216.3 pts |
+| performance | 13 suites / 1382 active / 67 ignored / 1219.3 pts | 14 suites / 1480 active / 70 ignored / 1258.5 pts | 13 suites / 1382 active / 67 ignored / 1219.3 pts |
| pipes | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts |
-| privacy | 5 suites / 874 active / 36 ignored / 710.5 pts | 5 suites / 923 active / 16 ignored / 715.4 pts | 5 suites / 849 active / 14 ignored / 688.0 pts |
+| privacy | 5 suites / 877 active / 36 ignored / 713.5 pts | 5 suites / 926 active / 16 ignored / 718.4 pts | 5 suites / 852 active / 14 ignored / 691.0 pts |
| real-app | - | 1 suites / 98 active / 3 ignored / 39.2 pts | - |
| speaker | 2 suites / 348 active / 8 ignored / 348.0 pts | 2 suites / 348 active / 8 ignored / 348.0 pts | 2 suites / 348 active / 8 ignored / 348.0 pts |
| storage | 3 suites / 476 active / 29 ignored / 385.7 pts | 3 suites / 476 active / 29 ignored / 385.7 pts | 3 suites / 476 active / 29 ignored / 385.7 pts |
| sync | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts | 1 suites / 462 active / 3 ignored / 323.4 pts |
-| timeline | 4 suites / 948 active / 33 ignored / 770.1 pts | 4 suites / 948 active / 33 ignored / 770.1 pts | 4 suites / 948 active / 33 ignored / 770.1 pts |
+| timeline | 4 suites / 951 active / 33 ignored / 773.1 pts | 4 suites / 951 active / 33 ignored / 773.1 pts | 4 suites / 951 active / 33 ignored / 773.1 pts |
| transcription | 5 suites / 709 active / 40 ignored / 558.9 pts | 5 suites / 709 active / 40 ignored / 558.9 pts | 5 suites / 709 active / 40 ignored / 558.9 pts |
| ui-events | 4 suites / 701 active / 28 ignored / 533.9 pts | 3 suites / 652 active / 5 ignored / 499.6 pts | 3 suites / 652 active / 5 ignored / 499.6 pts |
-| vision-capture | 5 suites / 498 active / 32 ignored / 395.0 pts | 5 suites / 502 active / 32 ignored / 400.5 pts | 4 suites / 493 active / 31 ignored / 391.5 pts |
+| vision-capture | 5 suites / 501 active / 32 ignored / 398.0 pts | 5 suites / 505 active / 32 ignored / 403.5 pts | 4 suites / 496 active / 31 ignored / 394.5 pts |
## Critical Flow Matrix
@@ -144,7 +144,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| engine-retention-storage | screenpipe-engine | windows, macos, linux | storage, engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | mixed | 5 | 38 | 0 | Local retention deletion (including lean-mode heavy-text stripping), cloud archive watermarking, atomic state-file replacement, and the low-disk capture monitor. |
| engine-telemetry-observability | screenpipe-engine | windows, macos, linux | engine-lifecycle, performance | engine-health-lifecycle, performance-liveness | medium | strong | unit | 6 | 29 | 0 | PostHog capture gating, telemetry context shaping, piggyback telemetry forwarding, crash-log helpers, resource monitoring, and the recording-coverage reliability metric. |
| screen-capture-ocr-contract | screenpipe-screen | windows, macos, linux | vision-capture, ocr | capture-ocr-pipeline | high | partial | unit | 3 | 14 | 0 | Cross-platform cached-OCR unit coverage for RawCaptureResult to CaptureResult metadata, browser URL, focus state, window-to-screen OCR coordinate transformation, Tesseract output parsing, and contour-based text-region detection for the meeting OCR gate. |
-| screen-capture-windowing | screenpipe-screen | windows, macos, linux | vision-capture, timeline, performance, privacy | capture-ocr-pipeline, timeline-streaming, privacy-and-redaction, performance-liveness | high | strong | mixed | 14 | 180 | 0 | Window filtering, empty-window regressions, retry policy, URL timing, monitor cache, OCR cache, snapshots, and image comparison. |
+| screen-capture-windowing | screenpipe-screen | windows, macos, linux | vision-capture, timeline, performance, privacy | capture-ocr-pipeline, timeline-streaming, privacy-and-redaction, performance-liveness | high | strong | mixed | 14 | 183 | 0 | Window filtering, empty-window regressions, retry policy, URL timing, monitor cache, OCR cache, snapshots, and image comparison. |
| screen-custom-ocr | screenpipe-screen | windows, macos, linux | ocr | capture-ocr-pipeline | medium | conditional | manual | 1 | 0 | 2 | Custom OCR tests are ignored by default and only contribute when explicitly run. |
| screen-macos-ocr | screenpipe-screen | macos | ocr, vision-capture | capture-ocr-pipeline | high | strong | mixed | 2 | 9 | 1 | Apple Vision OCR source/unit coverage and fixture OCR assertions. |
| screen-monitor-platform | screenpipe-screen | windows, macos, linux | vision-capture | capture-ocr-pipeline | medium | partial | unit | 5 | 36 | 5 | Per-OS monitor enumeration (Windows, macOS, Wayland/portal on Linux) and the persistent Windows.Graphics.Capture session. Each file is cfg-gated and only executes on its target OS. |
@@ -452,7 +452,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| screen-capture-windowing | screenpipe-screen | src/capture_screenshot_by_window.rs | source | 63 | 0 | 63 |
| screen-capture-ocr-contract | screenpipe-screen | src/core.rs | source | 1 | 0 | 1 |
| screen-capture-windowing | screenpipe-screen | src/frame_comparison.rs | source | 16 | 0 | 16 |
-| screen-capture-windowing | screenpipe-screen | src/metrics.rs | source | 13 | 0 | 13 |
+| screen-capture-windowing | screenpipe-screen | src/metrics.rs | source | 16 | 0 | 16 |
| screen-windows-ocr | screenpipe-screen | src/microsoft.rs | source | 4 | 0 | 4 |
| screen-capture-windowing | screenpipe-screen | src/monitor.rs | source | 5 | 0 | 5 |
| screen-monitor-platform | screenpipe-screen | src/monitor/linux_portal.rs | source | 5 | 0 | 5 |