YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(audio): make silent smart-recording meetings detectable (#6088)
Louis Beaumontc007cf3abfb1
diff --git a/TESTING.md b/TESTING.mdindex 7181c0e7e..c1b3d5b04 100644--- a/TESTING.md+++ b/TESTING.md@@ -117,12 +117,23 @@ commits: `28e5c247`- [ ] **piggyback takes precedence over capture mode** — flag on, capture mode "always" (continuous): joining a meeting engages the piggyback exactly as in meetings-only (Meeting Tap + resolved mic, global output and non-resolved mics suspended); on meeting end continuous capture resumes on the user's configured devices with no leftover session streams or suspensions.- [ ] **manual meeting piggyback (all mic-holders)** — flag on, start a MANUAL meeting while an app holds the mic: within ~6s (two-tick pid adoption) a Meeting Tap session starts over ALL mic-holding processes and their resolved mics are captured; a second app opening a mic mid-meeting rebuilds the tap over the widened set (one `rebuilding tap` log, NO strike, no System Audio resume/suspend churn); ending the manual meeting reverts everything. Screenpipe's own processes must never appear in the tapped set.- [ ] **enabled resolved mic displaces others (D1, latched)** — two mics enrolled, meeting app using one of them (no session stream needed): once the resolved mic delivers, the OTHER enrolled mic is suspended ONCE (single transcription stream, no re-suspend churn); a transient stall of the resolved mic (BT renegotiation) does NOT resume the others — they come back only when the app releases its mics, the resolved capture stays dead for 60s+, or the meeting ends; after such a lift, the meeting app recovering onto an enrolled mic re-latches displacement once.-- [ ] **piggyback fallback: tap build failure** — flag on, force tap failure (revoke System Audio Recording permission): capture continues on the user's configured capture — the System Audio stream via whichever backend their settings select (SCK by default, CoreAudio global tap when `experimentalCoreaudioSystemAudio` is on) plus their configured/pinned mics; a warning logs, ≤3 retries per meeting with 60s cooldown, no notification unless the fallback capture is also silent.-- [ ] **piggyback fallback is backend-neutral** — repeat the tap-build-failure test with `experimentalCoreaudioSystemAudio` ON: the fallback System Audio stream comes up as the CoreAudio global tap (log shows a Process Tap gen start, not SCK), proving the piggyback fallback re-derives the backend from user settings at start time rather than hardcoding one. Same expectation for any future backend.-- [ ] **piggyback fallback: app quits mid-meeting** — kill the meeting app: tap stream tears down within ~2s, global output resumes (unsuspended) until the watcher ends the meeting; no crash, no rebuild storm.-- [ ] **piggyback fallback: no pid (ui_scan/reattach)** — a DETECTED meeting whose sensor can't identify the process (ui_scan, post-restart reattach) behaves exactly like flag-off (stable path). Manual meetings are NOT in this bucket — they derive their pid set from the live mic-holder enumeration (see the manual-piggyback item above).+> **NO FALLBACK since #6072.** Smart recording is the *sole* capture owner for a+> confirmed meeting: `meeting_piggyback_owns_normal_capture()` is+> `piggyback_enabled && meeting_state == Some(true)`, with no term for whether the+> tap actually built. When the tap cannot build, normal capture stays suspended and+> **the meeting records nothing**. The four items below previously asserted the+> opposite; they were stale from #6072 until 2026-08-09. Verified live on 2.6.1: an+> 86s manual meeting produced 3,584 samples (~0.22s) and an empty transcript, while+> the identical audio captured normally with no meeting active.++- [ ] **tap build failure records nothing** — flag on, force tap failure (revoke System Audio Recording permission): normal capture stays suspended for the whole meeting and no audio is recorded. Confirm ≤3 retries per meeting with 60s cooldown, and that the emitted `piggyback_meeting_summary` carries `captured_no_audio: true`. This is the current contract, not the desired one — see the follow-up issue on restoring a real fallback.+- [ ] **tap build failure is backend-neutral** — repeat with `experimentalCoreaudioSystemAudio` ON: behaviour must be identical (no capture, `captured_no_audio: true`) regardless of which System Audio backend the user's settings select. If a fallback is ever restored, this is where backend re-derivation gets re-verified.+- [ ] **app quits mid-meeting** — kill the meeting app: tap stream tears down within ~2s; no crash, no rebuild storm. Normal capture does NOT resume until the watcher ends the meeting.+- [ ] **no pid (ui_scan/reattach) records nothing** — a DETECTED meeting whose sensor can't identify the process (ui_scan, post-restart reattach) keeps normal capture suspended and records nothing; outcome is `no_pid` with `captured_no_audio: true`. It does NOT behave like flag-off. Manual meetings are NOT in this bucket — they derive their pid set from the live mic-holder enumeration (see the manual-piggyback item above).+- [ ] **silent meetings are detectable** — for every case above, `piggyback_meeting_summary` must carry `captured_no_audio: true`. The `outcome` strings (`stable_fallback`, `no_pid`, `unavailable`) are all reported for meetings that captured nothing and all read as benign, so `outcome` alone cannot distinguish a silent meeting from a healthy one.+- [ ] **the gate is visible in logs** — with a meeting active and the flag on, released builds (INFO level) must log `smart recording owns capture for this meeting — not starting normal audio device: <device>`. This used to be DEBUG, i.e. invisible in shipped builds, which is why a silent meeting left no trace anywhere.- [ ] **browser pop-out / Little Arc meeting detection** — join a Google Meet call in a Chrome pop-out window or an Arc "Little Arc" window, then focus a different app: the meeting is detected within one poll of the mic being held (log shows the `UnresolvedBrowser` -> `Active` promotion); a non-meeting window whose title merely looks like a meeting code (e.g. a doc named "abc-defg-hij") with no mic held must NOT trigger detection. (`ff0337416`)-- [ ] **piggyback on unsupported OS** — flag on, macOS <14.4 / Windows <20348: one-time log "using the stable capture path", capture identical to flag-off. Windows must fail the PID-specific tap and let the normal configured/default output capture path run; the Meeting Tap stream itself must not silently downgrade to endpoint-wide loopback.+- [ ] **piggyback on unsupported OS** — flag on, macOS <14.4 / Windows <20348: one-time "using the stable capture path" log. NOTE: despite that log, capture is NOT identical to flag-off since #6072 — the meeting still keeps normal capture suspended and records nothing (`outcome: unavailable`, `captured_no_audio: true`). The Meeting Tap stream must not silently downgrade to endpoint-wide loopback.- [ ] **mid-call mic switch in the app** — change Zoom's mic mid-call: capture follows within ~6s (two confirmation ticks AFTER Zoom has the new device — the app always acquires first; old session mic stops after two missed resolver ticks, new one starts); changing the OS default while Zoom is pinned elsewhere does NOT switch capture. Switching onto a mic the sweep had displaced only clears its suspension flag — no immediate restart racing the app.- [ ] **mic switch must not flap the meeting** — switch the meeting app's mic (AirPods → phone → built-in, several times): `meeting_status_changed` stays active=true throughout — NO end/re-start pair ~20s after each switch, no Meeting Tap teardown/rebuild, no System Audio resume/suspend churn. (Root cause was device-derived session keys: macOS synthesized the audio-session id from the device set and WASAPI GUIDs are per-endpoint, so a mic switch re-keyed the session and the meeting rode the 20s ending grace into EndMeeting + instant restart.)- [ ] **piggyback never acts on silence** — during a piggybacked call, silence must never make the piggyback rebuild, restart, probe, or notify: mute in the meeting app (or leave a silent tab rendering nothing) for 5+ min → NO notification, NO Meeting Tap rebuild, NO resolved-mic restart, and the logs contain no silence-driven piggyback actions. A silent meeting device is the user's own in-meeting feedback loop — they fix it in the app and the mic/output follow tracks the switch. (The GLOBAL System Audio tap's silence watchdog — item above, `75a52603b` — is unchanged and out of scope here.) Dead-stream handling still applies: hard open failures notify once via `audio_capture_health_mic_capture_failed`, and dead capture streams retry through `MicFollow`'s backoff.diff --git a/apps/screenpipe-app-tauri/components/settings/recording-settings.tsx b/apps/screenpipe-app-tauri/components/settings/recording-settings.tsxindex 42a2bfe9a..19153237e 100644--- a/apps/screenpipe-app-tauri/components/settings/recording-settings.tsx+++ b/apps/screenpipe-app-tauri/components/settings/recording-settings.tsx@@ -3585,7 +3585,7 @@ Your screen is a pipe. Everything you see, hear, and type flows through it. Scre</Badge></h3><p className="text-xs text-muted-foreground">- during meetings, records your meeting's audio and whichever microphone you pick in the meeting app — taking precedence over your other audio settings. falls back to your configured capture automatically if unavailable.+ during meetings, records your meeting's audio and whichever microphone you pick in the meeting app — taking precedence over your other audio settings. while a meeting is running this replaces your normal capture, so if the meeting audio can't be captured that meeting may record nothing.</p>{settings.disableMeetingDetector && (<p className="text-xs text-amber-600 dark:text-amber-500">diff --git a/crates/screenpipe-audio/src/audio_manager/manager.rs b/crates/screenpipe-audio/src/audio_manager/manager.rsindex 83129607c..d80db1725 100644--- a/crates/screenpipe-audio/src/audio_manager/manager.rs+++ b/crates/screenpipe-audio/src/audio_manager/manager.rs@@ -894,8 +894,13 @@ impl AudioManager {// enter through this choke point; meeting-scoped mic/tap streams use// `start_session_device` and deliberately bypass it.if self.meeting_piggyback_owns_normal_capture().await {- debug!(- "skipping normal audio device start while meeting piggyback owns capture: {}",+ // INFO, not DEBUG. This gate suppresses the user's normal capture+ // for the whole meeting, and if the Meeting Tap never builds+ // nothing records at all. Released builds log at INFO, so at DEBUG+ // the only user-visible symptom of a silent meeting was an empty+ // transcript with no trace anywhere in the log.+ info!(+ "smart recording owns capture for this meeting — not starting normal audio device: {}",device);return Ok(());@@ -953,8 +958,8 @@ impl AudioManager {// opening a normal stream. Session streams never pass through here.if self.meeting_piggyback_owns_normal_capture().await {self.stop_device_recording(device).await?;- debug!(- "stopped newly-opened normal audio device after meeting piggyback engaged: {}",+ info!(+ "smart recording engaged mid-open — stopped normal audio device: {}",device);return Ok(());diff --git a/crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rs b/crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rsindex 6bc54313d..e30be509d 100644--- a/crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rs+++ b/crates/screenpipe-audio/src/audio_manager/meeting_piggyback.rs@@ -826,11 +826,29 @@ pub(crate) struct PiggybackMeetingSummary {pub meeting_app_bundle_id: Option<String>,pub pid_known: bool,pub manual: bool,+ /// True when the piggyback owned capture for a meeting and delivered+ /// nothing: the tap never streamed AND no meeting mic session ever+ /// started. Normal capture is suspended for the meeting duration, so this+ /// is the signal that the meeting recorded no audio at all.+ ///+ /// The `outcome` strings cannot carry this. `stable_fallback`,+ /// `no_pid` and `unavailable` are all reported for meetings that captured+ /// nothing, and all three read as benign, so a silent meeting is+ /// indistinguishable from a healthy one in the dashboards without this+ /// field. Additive on purpose — renaming an outcome would break the+ /// existing insights.+ pub captured_no_audio: bool,pub platform: &'static str,#[serde(skip_serializing_if = "Option::is_none")]pub os_version: Option<String>,}+/// Whether the piggyback delivered no audio at all for a meeting it owned.+/// Pure so the invariant is unit-testable without a sweep.+pub(crate) fn captured_no_audio(t: &MeetingTelemetry) -> bool {+ t.meeting_seen && t.tap_streaming_ticks == 0 && !t.mic_session_started+}+pub(crate) fn classify_outcome(t: &MeetingTelemetry) -> &'static str {if t.unavailable {return "unavailable";@@ -865,6 +883,7 @@ pub(crate) fn build_meeting_summary(t: &MeetingTelemetry) -> PiggybackMeetingSummeeting_app_bundle_id: t.bundle_id.clone(),pid_known: t.pid_known,manual: t.manual,+ captured_no_audio: captured_no_audio(t),platform: std::env::consts::OS,os_version: os_version_string(),}@@ -2328,6 +2347,55 @@ mod tests {assert_eq!(classify_outcome(&t), "stable_fallback");}+ /// The three outcomes reported for meetings that captured nothing all read+ /// as benign, so `outcome` alone cannot surface a silent meeting. Observed+ /// live on 2.6.1: an 86s meeting produced 3,584 samples (~0.22s) and an+ /// empty transcript while reporting `stable_fallback`.+ #[test]+ fn silent_meeting_is_flagged_under_every_benign_outcome() {+ for (label, mutate) in [+ (+ "stable_fallback",+ (|_: &mut MeetingTelemetry| {}) as fn(&mut MeetingTelemetry),+ ),+ ("no_pid", |t: &mut MeetingTelemetry| t.pid_known = false),+ ("unavailable", |t: &mut MeetingTelemetry| {+ t.unavailable = true+ }),+ ] {+ let mut t = telem();+ mutate(&mut t);+ assert_eq!(classify_outcome(&t), label, "outcome for {label}");+ assert!(+ captured_no_audio(&t),+ "{label} captured nothing but was not flagged"+ );+ assert!(build_meeting_summary(&t).captured_no_audio);+ }+ }++ #[test]+ fn a_streaming_tap_is_not_a_silent_meeting() {+ let mut t = telem();+ t.tap_streaming_ticks = 1;+ assert!(!captured_no_audio(&t));+ }++ /// A resolved meeting mic carries the audio even when the far-end tap never+ /// streams, so that is a real fallback rather than silence.+ #[test]+ fn a_resolved_meeting_mic_is_not_a_silent_meeting() {+ let mut t = telem();+ t.mic_session_started = true;+ assert!(!captured_no_audio(&t));+ }++ #[test]+ fn no_meeting_is_never_flagged_silent() {+ let t = MeetingTelemetry::default(); // meeting_seen == false+ assert!(!captured_no_audio(&t));+ }+#[test]fn summary_math_and_fields() {let mut t = telem();