YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(audio): fail over to built-in mic when pinned input (AirPods) disconnects (#4356)
Louis Beaumonte56e1a2b403e
diff --git a/TESTING.md b/TESTING.mdindex 03c525fc6..3d9f07619 100644--- a/TESTING.md+++ b/TESTING.md@@ -156,6 +156,7 @@ commits: device_monitor.rs atomic swap, tiered backoff, empty device list guard- [ ] **DB gap query after device switch** — run: `sqlite3 ~/.screenpipe/db.sqlite "SELECT t1.timestamp as gap_start, t2.timestamp as gap_end, (julianday(t2.timestamp) - julianday(t1.timestamp)) * 86400 as gap_seconds FROM audio_transcriptions t1 JOIN audio_transcriptions t2 ON t2.id = (SELECT MIN(id) FROM audio_transcriptions WHERE id > t1.id AND is_input_device = 0) WHERE t1.is_input_device = 0 AND (julianday(t2.timestamp) - julianday(t1.timestamp)) * 86400 > 60 ORDER BY t1.timestamp;"` — should return no rows if output was continuously captured.- [ ] **Bluetooth audio device hijack recovery** — Join a video call (e.g., Zoom, Proton Meet) on AirPods, end the call, join another call ~1 min later on same AirPods. Verify the second call is fully captured (audio_chunks created throughout). CoreAudio sometimes delivers zero-fill when another app has exclusive claim; the watchdog detects sustained silence (>30s of exact zeros) after initial healthy audio and rebuilds the device. Regression: `a2e89b2ae` (initial detection), fixed by `357e4dfcc` (only trip watchdog after stream is healthy). USB devices that never produce real audio should NOT trigger rebuild storms.- [ ] **Manual-mode pinned-input fallback** — In manual mode (NOT "Follow System Default"), pin only AirPods (input) as your mic. Mid-recording, turn off the AirPods. Verify: within ~20-25s the monitor engages the system default mic as a substitute and capture continues. Re-connect AirPods. Verify the substitute is torn down and capture returns to AirPods. Log markers: `grep "PINNED_FALLBACK" ~/.screenpipe/screenpipe-app.*.log` shows `pinned input '...' missing > 20s, capturing from system default '...' until it returns` then `clearing fallback '...': pinned input returned`. Edge: if you also user-disabled the default mic (privacy mode), expect "system default ... is user-disabled — no fallback engaged" log + zero capture rather than auto-fallback.+- [ ] **Pinned-input fallback when the dead device IS the system default (or there is none)** — Same as above, but make AirPods BOTH the pinned input AND the macOS default input (the common real case), then turn them off. Verify capture continues from the built-in mic within ~20-25s — NOT zero capture. The decider now fails over to any *available* input when the system default is unusable (it IS the dead pinned device — Bluetooth lingers as the registered default — or CoreAudio reports no default). Reconnect AirPods → substitute torn down. Sub-checks: (a) prefers an on-board mic over virtual/aggregate inputs (e.g. BlackHole/Aggregate present → built-in chosen); (b) still respects privacy — if the only other input is user-disabled, expect zero capture, not auto-fallback; (c) transient empty device list (SCK failure) → no fallback that cycle, no disconnect cascade; (d) sleep/wake flap < grace does not engage a spurious fallback. Regression: `ba23c8531` (ruark 6/18 "no recording either side of the call", frames=0/samples=0 — AirPods were the only input AND the default).#### meeting detection & speaker identificationdiff --git a/crates/screenpipe-audio/src/audio_manager/device_monitor.rs b/crates/screenpipe-audio/src/audio_manager/device_monitor.rsindex 2f9a34b99..867701174 100644--- a/crates/screenpipe-audio/src/audio_manager/device_monitor.rs+++ b/crates/screenpipe-audio/src/audio_manager/device_monitor.rs@@ -145,6 +145,15 @@ fn is_permanent_input_error(err: &anyhow::Error) -> bool {msg.contains("No default input device detected")}+/// Heuristic: does this input device name look like an on-board mic? Used to+/// prefer the built-in mic over a virtual/aggregate input when failing over a+/// disconnected pinned device, so we don't grab some random loopback input+/// when a real microphone is sitting right there.+fn is_builtin_input(name: &str) -> bool {+ let n = name.to_lowercase();+ n.contains("built-in") || n.contains("macbook") || n.contains("imac")+}+/// Grace window before engaging a fallback for a missing pinned input device.////// Bluetooth devices commonly flap for a few seconds during sleep/wake or app@@ -209,6 +218,12 @@ pub(crate) struct PinnedFallbackInputs<'a> {pub running: &'a HashSet<String>,pub user_disabled: &'a HashSet<String>,pub default_input: Option<&'a str>,+ /// Currently-available input device names in suffix form (e.g.+ /// `"MacBook Pro Microphone (input)"`). When the system default is unusable+ /// as a fail-over target (it IS the dead pinned device, or there is no+ /// default), the decider falls back to a present device from this set+ /// rather than capturing nothing. Empty ⇒ no substitute available.+ pub available_inputs: &'a HashSet<String>,pub missing_since: &'a HashMap<String, Instant>,pub active: Option<&'a ActivePinnedFallback>,pub grace: Duration,@@ -263,26 +278,56 @@ pub(crate) fn decide_pinned_input_fallback(inputs: PinnedFallbackInputs<'_>) ->return FallbackDecision::Idle;};- let Some(default_name) = inputs.default_input else {- return FallbackDecision::Idle;+ // Prefer the system default as the substitute — it's the user's "normal"+ // mic. It's only usable if it ISN'T the dead pinned device (Bluetooth+ // commonly lingers as the registered default while disconnected) and the+ // user hasn't explicitly disabled it (privacy).+ let default_usable = inputs+ .default_input+ .filter(|d| *d != pinned.as_str() && !inputs.user_disabled.contains(*d));++ let fallback_name = match default_usable {+ Some(d) => d.to_string(),+ // Default is unusable (none, == dead pinned, or user-disabled). Rather+ // than capture NOTHING, fail over to any present input device. This is+ // the AirPods case: AirPods are both the pinned input and the lingering+ // system default, yet a built-in mic is sitting right there — capturing+ // it beats hours of silence until the user manually reconnects.+ None => {+ let mut candidates: Vec<String> = inputs+ .available_inputs+ .iter()+ .filter(|name| name.as_str() != pinned.as_str())+ .filter(|name| !inputs.user_disabled.contains(name.as_str()))+ .filter(|name| {+ crate::core::device::parse_audio_device(name.as_str())+ .map(|d| d.device_type == DeviceType::Input)+ .unwrap_or(false)+ })+ .cloned()+ .collect();+ // Deterministic, and prefer an on-board mic over virtual inputs.+ candidates.sort_unstable();+ let chosen = candidates+ .iter()+ .find(|name| is_builtin_input(name.as_str()))+ .cloned()+ .or_else(|| candidates.into_iter().next());+ match chosen {+ Some(name) => name,+ // Genuinely nothing else to capture from (headless box, or the+ // only other input is user-disabled). Wait for the pinned input.+ None => return FallbackDecision::Idle,+ }+ }};- // No useful fallback target — the default IS the pinned device.- if default_name == pinned {- return FallbackDecision::Idle;- }-- // User explicitly disabled the default mic (e.g. for privacy). Respect that.- if inputs.user_disabled.contains(default_name) {- return FallbackDecision::Idle;- }-- // If the default is already running, adopt it without starting again.- let start_fallback = !inputs.running.contains(default_name);+ // If the chosen substitute is already running, adopt it without restarting.+ let start_fallback = !inputs.running.contains(&fallback_name);FallbackDecision::Engage {pinned,- fallback_name: default_name.to_string(),+ fallback_name,start_fallback,}}@@ -1300,12 +1345,24 @@ async fn run_pinned_input_fallback_sweep(let default_name = default_input_device().ok().map(|d| d.to_string());+ // Currently-available input devices — the universe of substitutes the+ // decider may fail over to when the system default is unusable as a target.+ let available_inputs: HashSet<String> = audio_manager+ .devices()+ .await+ .unwrap_or_default()+ .into_iter()+ .filter(|d| d.device_type == DeviceType::Input)+ .map(|d| d.to_string())+ .collect();+let decision = decide_pinned_input_fallback(PinnedFallbackInputs {use_system_default: false,pinned_inputs: &pinned_inputs,running: &running,user_disabled: &user_disabled,default_input: default_name.as_deref(),+ available_inputs: &available_inputs,missing_since,active: active.as_ref(),grace: Duration::from_secs(PINNED_INPUT_FALLBACK_GRACE_SECS),@@ -1464,6 +1521,12 @@ impl RestartCooldown {mod tests {use super::*;+ lazy_static::lazy_static! {+ /// Default for builders that don't exercise the fail-over-to-any-available+ /// path (most fallback tests only care about the system-default target).+ static ref EMPTY_AVAILABLE_INPUTS: HashSet<String> = HashSet::new();+ }+#[test]fn test_cooldown_allows_restarts_under_limit() {let mut cd = RestartCooldown::new(3, Duration::from_secs(300));@@ -1634,6 +1697,7 @@ mod tests {running,user_disabled,default_input,+ available_inputs: &EMPTY_AVAILABLE_INPUTS,missing_since,active,grace: Duration::from_secs(20),@@ -1764,10 +1828,13 @@ mod tests {}#[test]- fn fallback_skipped_when_default_equals_pinned() {- // Edge: the pinned device IS macOS's current default (likely because- // AirPods is/was the default before disconnect). Falling back to itself- // is a no-op; just wait for it to come back.+ fn fallback_skipped_when_default_equals_pinned_and_no_other_input() {+ // The pinned device IS macOS's current default (AirPods was the default+ // before disconnect) AND no other input is available (empty+ // available_inputs). Falling back to the dead device itself is a no-op+ // and there's nothing else to capture from — wait for it to return.+ // (When another input IS present, see+ // `fails_over_to_builtin_when_default_equals_dead_pinned`.)let pinned = set(&["AirPods (input)"]);let running = HashSet::new();let now = Instant::now();@@ -1790,9 +1857,11 @@ mod tests {}#[test]- fn fallback_skipped_when_no_default_available() {- // No usable system default (unusual — laptop without a built-in mic, or- // headless box). Nothing to fall back to.+ fn fallback_skipped_when_no_default_and_no_other_input() {+ // No usable system default AND no other available input (headless box,+ // or a laptop with no built-in mic). Nothing to fall back to.+ // (When a built-in mic IS present, see+ // `fails_over_to_builtin_when_no_system_default`.)let pinned = set(&["AirPods (input)"]);let running = HashSet::new();let now = Instant::now();@@ -1814,6 +1883,155 @@ mod tests {assert_eq!(decision, FallbackDecision::Idle);}+ // --- Fail over to an available input when the system default is unusable ---+ //+ // Regression coverage for the AirPods-disconnect total-audio-loss bug+ // ([email protected], 2026-06-18, mac 2.5.50): the only/pinned input was+ // AirPods, which were ALSO the macOS system default. AirPods disconnected →+ // the monitor retried the vanished device every ~2s for 24h+ and never+ // failed over to the built-in mic that was present the whole time; the call+ // that followed recorded frames=0, samples=0. The decider used to sit Idle+ // whenever the system default was unusable as a target (it WAS the dead+ // pinned device, or was None). It now falls over to any present input.++ #[test]+ fn fails_over_to_builtin_when_default_equals_dead_pinned() {+ // ruark's exact state: AirPods is pinned AND the lingering system+ // default; the built-in mic is present. Must fail over to the built-in+ // rather than capture nothing.+ let available = set(&["AirPods (input)", "MacBook Pro Microphone (input)"]);+ let pinned = set(&["AirPods (input)"]);+ let running = HashSet::new();+ let no_disabled = HashSet::new();+ let now = Instant::now();+ let missing_since: HashMap<String, Instant> = [(+ "AirPods (input)".to_string(),+ now.checked_sub(Duration::from_secs(60)).unwrap(),+ )]+ .into_iter()+ .collect();+ let mut inputs = build_inputs(+ &pinned,+ &running,+ &no_disabled,+ Some("AirPods (input)"), // system default == the dead device+ &missing_since,+ None,+ now,+ );+ inputs.available_inputs = &available;+ assert_eq!(+ decide_pinned_input_fallback(inputs),+ FallbackDecision::Engage {+ pinned: "AirPods (input)".to_string(),+ fallback_name: "MacBook Pro Microphone (input)".to_string(),+ start_fallback: true,+ }+ );+ }++ #[test]+ fn fails_over_to_builtin_when_no_system_default() {+ // CoreAudio briefly reports no default once the only/default device+ // (AirPods) vanishes. The built-in mic is still physically present.+ let available = set(&["MacBook Pro Microphone (input)"]);+ let pinned = set(&["AirPods (input)"]);+ let running = HashSet::new();+ let no_disabled = HashSet::new();+ let now = Instant::now();+ let missing_since: HashMap<String, Instant> = [(+ "AirPods (input)".to_string(),+ now.checked_sub(Duration::from_secs(60)).unwrap(),+ )]+ .into_iter()+ .collect();+ let mut inputs = build_inputs(+ &pinned,+ &running,+ &no_disabled,+ None, // no usable system default reported this instant+ &missing_since,+ None,+ now,+ );+ inputs.available_inputs = &available;+ assert_eq!(+ decide_pinned_input_fallback(inputs),+ FallbackDecision::Engage {+ pinned: "AirPods (input)".to_string(),+ fallback_name: "MacBook Pro Microphone (input)".to_string(),+ start_fallback: true,+ }+ );+ }++ #[test]+ fn fallback_prefers_builtin_over_virtual_input() {+ // Several substitutes available: pick the on-board mic, not a virtual /+ // aggregate input (e.g. BlackHole), and do so deterministically.+ let available = set(&[+ "Aggregate Device (input)",+ "BlackHole 2ch (input)",+ "MacBook Pro Microphone (input)",+ ]);+ let pinned = set(&["AirPods (input)"]);+ let running = HashSet::new();+ let no_disabled = HashSet::new();+ let now = Instant::now();+ let missing_since: HashMap<String, Instant> = [(+ "AirPods (input)".to_string(),+ now.checked_sub(Duration::from_secs(60)).unwrap(),+ )]+ .into_iter()+ .collect();+ let mut inputs = build_inputs(+ &pinned,+ &running,+ &no_disabled,+ None,+ &missing_since,+ None,+ now,+ );+ inputs.available_inputs = &available;+ assert_eq!(+ decide_pinned_input_fallback(inputs),+ FallbackDecision::Engage {+ pinned: "AirPods (input)".to_string(),+ fallback_name: "MacBook Pro Microphone (input)".to_string(),+ start_fallback: true,+ }+ );+ }++ #[test]+ fn no_failover_when_only_other_input_is_user_disabled() {+ // Default unusable (== dead pinned) and the only other present input is+ // the built-in mic the user disabled for privacy. Respect that — Idle.+ let available = set(&["AirPods (input)", "MacBook Pro Microphone (input)"]);+ let pinned = set(&["AirPods (input)"]);+ let running = HashSet::new();+ let user_disabled = set(&["MacBook Pro Microphone (input)"]);+ let now = Instant::now();+ let missing_since: HashMap<String, Instant> = [(+ "AirPods (input)".to_string(),+ now.checked_sub(Duration::from_secs(60)).unwrap(),+ )]+ .into_iter()+ .collect();+ let mut inputs = build_inputs(+ &pinned,+ &running,+ &user_disabled,+ Some("AirPods (input)"),+ &missing_since,+ None,+ now,+ );+ inputs.available_inputs = &available;+ assert_eq!(decide_pinned_input_fallback(inputs), FallbackDecision::Idle);+ }+#[test]fn fallback_skipped_in_follow_system_default_mode() {// Follow-default mode has its own swap path in the monitor — the