YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(storage): resume recording after failed background migrations (#7023)
Ezra Ellette9ddd3795e281
diff --git a/COVERAGE.md b/COVERAGE.mdindex a84e3b08e..1b79507af 100644--- a/COVERAGE.md+++ b/COVERAGE.md@@ -36,15 +36,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.- Mapped suites: 32- Mapped Rust files: 351-- Active test blocks: 3472+- Active test blocks: 3476- Ignored/manual test blocks: 142-- Weighted coverage points: 2862.3+- Weighted coverage points: 2866.3| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 3329 | 136 | 2797.2 | 21 | 11 | 100% |-| macos | 29 | 3391 | 117 | 2811.2 | 22 | 11 | 100% |-| linux | 25 | 2959 | 109 | 2456.5 | 20 | 11 | 100% |+| windows | 29 | 3333 | 136 | 2801.2 | 21 | 11 | 100% |+| macos | 29 | 3395 | 117 | 2815.2 | 22 | 11 | 100% |+| linux | 25 | 2963 | 109 | 2460.5 | 20 | 11 | 100% |## Refreshdiff --git a/apps/screenpipe-app-tauri/components/settings/storage-migration-card.tsx b/apps/screenpipe-app-tauri/components/settings/storage-migration-card.tsxindex 18d0f8ca6..e22122fe3 100644--- a/apps/screenpipe-app-tauri/components/settings/storage-migration-card.tsx+++ b/apps/screenpipe-app-tauri/components/settings/storage-migration-card.tsx@@ -101,7 +101,7 @@ export function StorageMigrationCard({ dataDirectory, onBusyChange }: {{status?.blocked_reason && <p className="text-xs text-muted-foreground">{status.blocked_reason}</p>}{status?.pending && !status.busy && (<p className="text-xs text-muted-foreground">- {status.in_place ? "Migration is unfinished. Completed progress is saved. Resume to use history and recording again." : "Migration is unfinished. Resume to continue. Your original database is still kept."}+ {status.in_place ? "Migration is unfinished. Completed progress is saved. Recording uses your saved preference; retry migration when ready." : "Migration is unfinished. Resume to continue. Your original database is still kept."}</p>)}{status?.completed && !status.using_new_storage && !status.busy && (diff --git a/apps/screenpipe-app-tauri/components/storage-migration-gate.tsx b/apps/screenpipe-app-tauri/components/storage-migration-gate.tsxindex 48cc5624b..964b528a2 100644--- a/apps/screenpipe-app-tauri/components/storage-migration-gate.tsx+++ b/apps/screenpipe-app-tauri/components/storage-migration-gate.tsx@@ -123,7 +123,7 @@ export function StorageMigrationGate({ offerMigration = false }: { offerMigratio{activity.available_bytes != null && <p>Free space: {migrationBytes(activity.available_bytes)}</p>}</div><p className="border-t border-border pt-4 text-xs text-muted-foreground">- Space is recovered as each batch is verified. If interrupted, migration resumes on the next launch.+ Space is recovered as each batch is verified. If interrupted, recording resumes with your saved preference. Migration waits for you to retry.</p></div></dialog>diff --git a/apps/screenpipe-app-tauri/components/storage-migration-prompt.tsx b/apps/screenpipe-app-tauri/components/storage-migration-prompt.tsxindex 6ed688bd2..4366b3c84 100644--- a/apps/screenpipe-app-tauri/components/storage-migration-prompt.tsx+++ b/apps/screenpipe-app-tauri/components/storage-migration-prompt.tsx@@ -14,7 +14,7 @@ export function StorageMigrationDescription() {return <><span className="mt-3 block">Screenpipe will pause recording and history access while it converts and verifies your history, then restore your recording preference.</span><span className="mt-3 block">Keep the app open. Your computer will stay awake. Progress and elapsed time will be shown; the time needed depends on your database and computer.</span>- <span className="mt-3 block">Space is recovered as each batch is verified. Your existing database becomes the smaller index. If you close the app, migration resumes on the next launch.</span>+ <span className="mt-3 block">Space is recovered as each batch is verified. Your existing database becomes the smaller index. If migration fails or is interrupted, recording resumes with your saved preference. You can retry migration later.</span></>;}@@ -116,7 +116,7 @@ export function StorageMigrationPrompt({ activity }: { activity: StorageMigratio{status?.bytes_saved != null && <p>Space saved: {migrationBytes(status.bytes_saved)}</p>}</div>}{failure && <p className="text-sm text-destructive" role="alert">{failure}</p>}- {!success && <p className="text-xs text-muted-foreground">{status?.pending && status.in_place ? "Resume migration to use history and recording again. Your completed progress is saved." : "You can also start later in Settings → Storage."}</p>}+ {!success && <p className="text-xs text-muted-foreground">{status?.pending && status.in_place ? "Your completed progress is saved. Recording uses your saved preference while migration waits for you to retry." : "You can also start later in Settings → Storage."}</p>}<AlertDialogFooter><AlertDialogCancel disabled={submitting} onClick={dismiss}>{success ? "done" : "do later"}</AlertDialogCancel>{status?.can_cancel && !success && <Button variant="outline" disabled={submitting} onClick={() => void run(true)}>use original database</Button>}diff --git a/apps/screenpipe-app-tauri/src-tauri/src/enterprise_sync.rs b/apps/screenpipe-app-tauri/src-tauri/src/enterprise_sync.rsindex b00fdd171..5e6ef84da 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/enterprise_sync.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/enterprise_sync.rs@@ -1575,6 +1575,10 @@ mod imp {);}}+ // Hidden installs may never create the migration UI.+ // This returns after scheduling native maintenance so+ // policy revocation continues to be polled while it runs.+ crate::storage_migration::maybe_start_hidden_ui_migration(app.clone()).await;}NativeAuthorizationResult::RecordingDisabled => {crate::enterprise_policy::update_recording_authorized(false);diff --git a/apps/screenpipe-app-tauri/src-tauri/src/main.rs b/apps/screenpipe-app-tauri/src-tauri/src/main.rsindex c1208389b..d233d5571 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/main.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/main.rs@@ -2177,6 +2177,10 @@ async fn main() {).await {error!("Could not finish migration startup: {error}");}+ } else if let Err(error) = crate::storage_migration::finish_recording_recovery(+ &app_for_owned,+ ).await {+ error!("Could not finish recording recovery: {error}");}is_starting_clone.store(false, std::sync::atomic::Ordering::SeqCst);diff --git a/apps/screenpipe-app-tauri/src-tauri/src/recording.rs b/apps/screenpipe-app-tauri/src-tauri/src/recording.rsindex f57a59fda..4c9508824 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/recording.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/recording.rs@@ -1061,6 +1061,9 @@ pub(crate) async fn spawn_screenpipe_inner(if let Some(resumed) = resumed {crate::storage_migration::finish_startup(&app, resumed, result).await} else {+ if result.is_ok() {+ crate::storage_migration::finish_recording_recovery(&app).await?;+ }result}}diff --git a/apps/screenpipe-app-tauri/src-tauri/src/server_core.rs b/apps/screenpipe-app-tauri/src-tauri/src/server_core.rsindex 77a2fc85d..427c191be 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/server_core.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/server_core.rs@@ -337,7 +337,6 @@ impl ServerCore {// --- Database ---let local_data_dir = config.data_dir.clone();- crate::db_relaunch::set_active_database(&local_data_dir);let data_path = local_data_dir.join("data");std::fs::create_dir_all(&data_path).map_err(|error| {let message =@@ -346,8 +345,13 @@ impl ServerCore {message})?;- screenpipe_db::storage::pause_interrupted_migration(&local_data_dir)- .map_err(|error| format!("Failed to resume storage after interruption: {error}"))?;+ screenpipe_db::storage::recover_interrupted_migration(+ &local_data_dir,+ config.db_config.clone(),+ )+ .await+ .map_err(|error| format!("Failed to resume storage after interruption: {error}"))?;+ crate::db_relaunch::set_active_database(&local_data_dir);// A crash during repair may leave the committed WAL archived separately// from the main file. Reconcile the swap before ordinary DB diagnosis.diff --git a/apps/screenpipe-app-tauri/src-tauri/src/storage_migration.rs b/apps/screenpipe-app-tauri/src-tauri/src/storage_migration.rsindex e046d3462..769a1d205 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/storage_migration.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/storage_migration.rs@@ -14,6 +14,42 @@ use tauri::{Emitter, Manager, State};// Shared by every webview and recorder restart; renewed only by a new app process.static APP_SESSION_ID: LazyLock<String> = LazyLock::new(|| uuid::Uuid::new_v4().to_string());+const MIGRATION_ERROR_FILE: &str = "storage-migration-error.txt";+const INTERRUPTED_MIGRATION: &str =+ "The previous storage migration did not finish. Automatic retries are disabled; retry explicitly.";++fn saved_migration_error(root: &Path) -> Option<String> {+ match std::fs::read_to_string(root.join(MIGRATION_ERROR_FILE)) {+ Ok(error) if !error.trim().is_empty() => Some(error),+ Ok(_) => Some(INTERRUPTED_MIGRATION.into()),+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {+ // Also protect interrupted attempts made before the durable block+ // existed. Ordinary startup can pause this journal and record.+ root.join("storage-migration.json")+ .exists()+ .then(|| INTERRUPTED_MIGRATION.into())+ }+ Err(error) => Some(format!(+ "Could not read the previous migration result: {error}. Automatic retries are disabled."+ )),+ }+}++fn save_migration_error(root: &Path, error: &str) -> Result<(), String> {+ crate::store::durable_write(&root.join(MIGRATION_ERROR_FILE), error.as_bytes())+ .map_err(|e| format!("Could not save the migration retry block: {e}"))+}++fn clear_migration_error(root: &Path) -> Result<(), String> {+ match std::fs::remove_file(root.join(MIGRATION_ERROR_FILE)) {+ Ok(()) => Ok(()),+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),+ Err(error) => Err(format!(+ "Could not clear the migration retry block: {error}"+ )),+ }+}+#[derive(Default, Clone)]struct Operation {root: Option<PathBuf>,@@ -27,6 +63,7 @@ struct Operation {bytes_saved: Option<u64>,available_bytes: Option<u64>,completed: bool,+ background: bool,}impl Operation {@@ -43,7 +80,8 @@ impl Operation {total_records: self.total_records,bytes_saved: self.bytes_saved,available_bytes: self.available_bytes,- completed: self.completed,+ // A hidden migration can remove the old recovery copy.+ completed: self.completed && !self.background,}}}@@ -106,6 +144,47 @@ pub struct StorageMigrationStatus {pub blocked_reason: Option<String>,}+fn should_start_hidden_ui_migration(+ status: &StorageMigrationStatus,+ ui_hidden: bool,+ authorized: bool,+ recorder_ready: bool,+) -> bool {+ ui_hidden+ && authorized+ && recorder_ready+ && !status.busy+ // Failures and interrupted attempts stay blocked across app restarts.+ && status.error.is_none()+ && !status.pending+ && status.blocked_reason.is_none()+ && (status.can_migrate || status.can_delete_source)+}++/// Called by the native policy watcher, including when no webview exists.+/// Completion, failure and cleanup are recorded in logs without opening UI.+pub(crate) async fn maybe_start_hidden_ui_migration(app: tauri::AppHandle) {+ if !crate::enterprise_policy::is_app_ui_hidden()+ || !crate::enterprise_policy::recording_authorized()+ {+ return;+ }+ let result = async {+ let root = selected_root(&app)?;+ start_storage_migration_inner(+ app.clone(),+ app.state::<RecordingState>(),+ root.display().to_string(),+ true,+ )+ .await+ }+ .await;+ if let Err(error) = result {+ tracing::debug!(%error, "background storage migration deferred");+ }+}+pub(crate) fn is_running(app: &tauri::AppHandle) -> bool {app.try_state::<StorageMigrationState>().is_some_and(|state| state.0.lock().unwrap_or_else(|e| e.into_inner()).busy)@@ -191,8 +270,8 @@ fn save_recording_preference(Ok(())}-/// The ordinary startup path must finish an in-place conversion before opening-/// any database consumers. The server lifecycle lock is held by the caller.+/// Restore recording preference and acknowledge a completed migration at startup.+/// Conversion retries require explicit action. The caller holds the lifecycle lock.pub(crate) async fn resume_before_startup(app: &tauri::AppHandle,recording: &RecordingState,@@ -220,10 +299,22 @@ pub(crate) async fn resume_before_startup(if is_running(app) {return Ok(None);}- let needs_conversion = pending- || migration_report(&root)- .map_err(|e| e.to_string())?- .is_none();+ if pending || saved_migration_error(&root).is_some() {+ let error = saved_migration_error(&root).unwrap_or_else(|| INTERRUPTED_MIGRATION.into());+ report_migration_failure(app, &root, &error);+ if let Err(save_error) = save_migration_error(&root, &error) {+ tracing::error!(%save_error, "failed to save migration retry block");+ }+ // ServerCore restores a writable resident schema without encoding+ // another payload batch, then resumes the saved recording preference.+ return Ok(None);+ }+ if migration_report(&root)+ .map_err(|e| e.to_string())?+ .is_none()+ {+ return Ok(None);+ }let awake = screenpipe_engine::power::KeepAwakeGuard::acquire().map_err(|e| e.to_string())?;crate::health::set_boot_phase("migrating_database",@@ -238,23 +329,6 @@ pub(crate) async fn resume_before_startup(..Default::default()};});- let result = if needs_conversion {- screenpipe_db::storage::migrate_with_progress(- &root,- Default::default(),- Default::default(),- |update| progress(app, update),- )- .await- .map(|_| ())- .map_err(|e| e.to_string())- } else {- Ok(())- };- if let Err(error) = result {- finish_operation(app, &root, &Err(error.clone()));- return Err(error);- }progress(app,MigrationProgress {@@ -444,10 +518,13 @@ pub async fn get_storage_migration_status(blocked_reason =Some("Storage migration is unavailable while vault protection is enabled.".into());}- let error = if operation.root.as_ref() == Some(&root) {- operation.error- } else {+ let error = if operation.busy {None+ } else {+ operation+ .error+ .filter(|_| operation.root.as_ref() == Some(&root))+ .or_else(|| saved_migration_error(&root))};let can_migrate = !operation.busy&& blocked_reason.is_none()@@ -502,6 +579,15 @@ pub async fn start_storage_migration(app: tauri::AppHandle,recording: State<'_, RecordingState>,root: String,+) -> Result<(), String> {+ start_storage_migration_inner(app, recording, root, false).await+}++async fn start_storage_migration_inner(+ app: tauri::AppHandle,+ recording: State<'_, RecordingState>,+ root: String,+ background: bool,) -> Result<(), String> {let root = require_selected_root(&app, &root)?;let lifecycle = recording@@ -512,15 +598,49 @@ pub async fn start_storage_migration("Screenpipe is already restarting or changing storage. Try again when it finishes."})?;let status = get_storage_migration_status(app.clone(), recording).await?;- if !status.can_migrate {+ if background {+ if let Some(error) = &status.error {+ report_migration_failure(&app, &root, error);+ return Ok(());+ }+ let recording = app.state::<RecordingState>();+ let server_ready = recording.server.try_lock().is_ok_and(|server| {+ server+ .as_ref()+ .is_some_and(|server| !server.db.pool.is_closed())+ });+ let capture_ready = !recording.capture_intended()+ || recording+ .capture+ .try_lock()+ .is_ok_and(|capture| capture.is_some());+ if !should_start_hidden_ui_migration(+ &status,+ crate::enterprise_policy::is_app_ui_hidden(),+ crate::enterprise_policy::recording_authorized(),+ server_ready && capture_ready,+ ) {+ return Ok(());+ }+ } else if !status.can_migrate {return Err(status.blocked_reason.unwrap_or_else(|| "Migration is unavailable in the current storage state.".into()));}// Independent of the recording preference, which startup reapplies during switchover.// Acquire before pausing so a failed wake lock never strands recording.- let awake = screenpipe_engine::power::KeepAwakeGuard::acquire()- .map_err(|error| format!("Could not prevent sleep. Migration has not started: {error}"))?;+ let awake = screenpipe_engine::power::KeepAwakeGuard::acquire().map_err(|error| {+ let error = format!("Could not prevent sleep. Migration has not started: {error}");+ report_migration_failure(&app, &root, &error);+ let _ = save_migration_error(&root, &error);+ error+ })?;+ // Write BEFORE pausing recording. A crash, kill, or failed error write must+ // never erase the block and let the next launch pause recording again.+ save_migration_error(&root, INTERRUPTED_MIGRATION).map_err(|error| {+ report_migration_failure(&app, &root, &error);+ error+ })?;let recording_preference = app.state::<RecordingState>().capture_intended();if saved_recording_preference(&app, &root)?.is_none() {save_recording_preference(&app, &root, Some(recording_preference))?;@@ -529,47 +649,176 @@ pub async fn start_storage_migration(*operation = Operation {root: Some(root.clone()),busy: true,- message: "pausing recording".into(),+ message: if status.can_delete_source {+ "removing the original database"+ } else {+ "pausing recording"+ }+ .into(),started_at: Some(Instant::now()),+ background,..Default::default()};});tauri::async_runtime::spawn(async move {let _lifecycle = lifecycle;let _awake = awake;- let result = async {+ let mut result = async {let recording = app.state::<RecordingState>();- crate::recording::stop_screenpipe_inner(&recording).await?;- if !status.completed || status.pending {- screenpipe_db::storage::migrate_with_progress(- &root,- Default::default(),- Default::default(),- |message| progress(&app, message),- )- .await- .map_err(|e| e.to_string())?;+ // A completed legacy migration may retain its original source.+ // Verify and clean it up without pausing capture.+ if !background || !status.can_delete_source {+ crate::recording::stop_screenpipe_inner(&recording).await?;+ if !status.completed || status.pending {+ screenpipe_db::storage::migrate_with_progress(+ &root,+ Default::default(),+ Default::default(),+ |message| progress(&app, message),+ )+ .await+ .map_err(|e| e.to_string())?;+ }+ require_selected_root(&app, &root.display().to_string())?;+ progress(+ &app,+ MigrationProgress {+ message: "resuming recording on the new storage",+ completed_records: None,+ total_records: None,+ bytes_saved: None,+ available_bytes: None,+ },+ );+ crate::recording::spawn_screenpipe_inner(&recording, app.clone()).await?;}require_selected_root(&app, &root.display().to_string())?;- progress(- &app,- MigrationProgress {- message: "resuming recording on the new storage",- completed_records: None,- total_records: None,- bytes_saved: None,- available_bytes: None,- },- );- crate::recording::spawn_screenpipe_inner(&recording, app.clone()).await?;- verify_running(&app, &root).await+ verify_running(&app, &root).await?;+ let descriptor = StorageDescriptor::read(&root)+ .map_err(|e| e.to_string())?+ .ok_or("The migrated storage is not active. Saved progress has been kept.")?;+ if background+ && crate::enterprise_policy::is_app_ui_hidden()+ && crate::enterprise_policy::recording_authorized()+ {+ let server = recording.server.lock().await;+ let server = server.as_ref().ok_or(+ "The new storage is no longer running. The original database has been kept.",+ )?;+ // The DB primitive rechecks the live generation, parity receipt,+ // source identity, WAL/journal absence and a real query under+ // the source lease before deleting only the old db.sqlite.+ if server+ .db+ .retained_migration_source_bytes()+ .map_err(|e| e.to_string())?+ .is_some()+ {+ let bytes = server+ .db+ .delete_migration_source(&descriptor.generation)+ .await+ .map_err(|e| e.to_string())?;+ tracing::info!(+ bytes,+ "background storage migration removed the verified original database"+ );+ }+ }+ clear_migration_error(&root)?;+ Ok::<_, String>(())}.await;+ if let Err(error) = &result {+ report_migration_failure(&app, &root, error);+ if let Err(save_error) = save_migration_error(&root, error) {+ // The pre-pause marker still blocks automatic retries.+ tracing::error!(%save_error, "failed to save migration error details");+ }+ if let Err(restart_error) = resume_after_failed_migration(&app, &root).await {+ let error = format!("{error}\nRecording recovery failed: {restart_error}");+ report_migration_failure(&app, &root, &error);+ let _ = save_migration_error(&root, &error);+ result = Err(error);+ }+ }finish_operation(&app, &root, &result);});Ok(())}+fn report_migration_failure(app: &tauri::AppHandle, root: &Path, error: &str) {+ let state = app.state::<StorageMigrationState>();+ let mut operation = state.0.lock().unwrap_or_else(|e| e.into_inner());+ if operation.root.as_deref() == Some(root) && operation.error.as_deref() == Some(error) {+ return;+ }+ operation.root = Some(root.to_owned());+ operation.error = Some(error.into());+ drop(operation);+ // ERROR events go through the existing Sentry tracing layer when telemetry+ // is enabled. No webview, notification, or network await blocks recovery.+ tracing::error!(%error, "storage migration failed; automatic retries disabled");+}++async fn resume_after_failed_migration(app: &tauri::AppHandle, root: &Path) -> Result<(), String> {+ require_selected_root(app, &root.display().to_string())?;+ let recording = app.state::<RecordingState>();+ let server_ready = recording+ .server+ .lock()+ .await+ .as_ref()+ .is_some_and(|server| !server.db.pool.is_closed());+ let capture_ready = !recording.capture_intended() || recording.capture.lock().await.is_some();+ if server_ready && capture_ready {+ return finish_recording_recovery(app).await;+ }+ // Preserve capture intent, including an explicit pause. Teardown also+ // clears the restart cooldown, whose deferred path requires a webview.+ // ServerCore restores interrupted storage for recording without resuming+ // archival. The durable migration error remains until an explicit retry.+ crate::recording::stop_screenpipe_inner(&recording).await?;+ crate::recording::spawn_screenpipe_inner(&recording, app.clone()).await?;+ {+ let server = recording.server.lock().await;+ let server = server.as_ref().ok_or("Recording server did not restart.")?;+ server+ .db+ .query_raw_sql("SELECT id FROM frames LIMIT 1")+ .await+ .map_err(|e| e.to_string())?;+ }+ if recording.capture_intended() && recording.capture.lock().await.is_none() {+ return Err("Recording did not resume.".into());+ }+ tracing::info!("recording restored after migration failure; migration remains blocked");+ Ok(())+}++/// Once recovery has restored the saved intent, later recording controls own it.+/// Retaining the migration preference would overwrite a subsequent user pause.+pub(crate) async fn finish_recording_recovery(app: &tauri::AppHandle) -> Result<(), String> {+ let root = selected_root(app)?;+ if saved_migration_error(&root).is_none() || saved_recording_preference(app, &root)?.is_none() {+ return Ok(());+ }+ let recording = app.state::<RecordingState>();+ let ready = recording+ .server+ .lock()+ .await+ .as_ref()+ .is_some_and(|server| {+ server.data_dir.canonicalize().ok().as_ref() == Some(&root)+ && !server.db.pool.is_closed()+ });+ if ready && (!recording.capture_intended() || recording.capture.lock().await.is_some()) {+ save_recording_preference(app, &root, None)?;+ }+ Ok(())+}+#[tauri::command]#[specta::specta]pub async fn cancel_storage_migration(@@ -635,3 +884,194 @@ pub async fn delete_original_storage_database(.await.map_err(|e| e.to_string())}++#[cfg(test)]+mod tests {+ use super::*;++ fn legacy_status() -> StorageMigrationStatus {+ StorageMigrationStatus {+ root: "/fixture".into(),+ app_session_id: "test-process".into(),+ busy: false,+ message: String::new(),+ error: None,+ pending: false,+ in_place: true,+ completed: false,+ using_new_storage: false,+ generation: None,+ source_bytes: 1024,+ migrated_bytes: None,+ bytes_saved: None,+ available_bytes: None,+ can_migrate: true,+ can_cancel: false,+ can_delete_source: false,+ blocked_reason: None,+ }+ }++ #[test]+ fn hidden_ui_migration_requires_live_authorization_and_ready_recorder() {+ let status = legacy_status();+ assert!(should_start_hidden_ui_migration(&status, true, true, true));+ for (hidden, authorized, ready) in [+ (false, true, true),+ (true, false, true),+ (true, true, false),+ ] {+ assert!(!should_start_hidden_ui_migration(+ &status, hidden, authorized, ready+ ));+ }+ }++ #[test]+ fn hidden_ui_migration_waits_for_safe_storage_and_does_not_loop_on_failure() {+ for status in [+ StorageMigrationStatus {+ busy: true,+ ..legacy_status()+ },+ StorageMigrationStatus {+ error: Some("conversion failed".into()),+ ..legacy_status()+ },+ StorageMigrationStatus {+ blocked_reason: Some("vault is protected".into()),+ ..legacy_status()+ },+ StorageMigrationStatus {+ can_migrate: false,+ ..legacy_status()+ },+ ] {+ assert!(!should_start_hidden_ui_migration(&status, true, true, true));+ }+ // Even a journal from an older app without a saved error must not+ // automatically pause recording again. Explicit retry remains available.+ let resumed = StorageMigrationStatus {+ pending: true,+ ..legacy_status()+ };+ assert!(!should_start_hidden_ui_migration(+ &resumed, true, true, true+ ));+ assert!(resumed.can_migrate);+ }++ #[test]+ fn migration_failure_and_interruption_block_automatic_retries_across_restarts() {+ let root = tempfile::tempdir().unwrap();+ let other_root = tempfile::tempdir().unwrap();+ for error in [INTERRUPTED_MIGRATION, "not enough free disk space"] {+ save_migration_error(root.path(), error).unwrap();+ // A new process has no Operation.error. Read only durable state.+ let fresh_process = Operation::default();+ assert!(fresh_process.error.is_none());+ let status = StorageMigrationStatus {+ error: saved_migration_error(root.path()),+ ..legacy_status()+ };+ assert_eq!(status.error.as_deref(), Some(error));+ assert!(!should_start_hidden_ui_migration(&status, true, true, true));+ assert!(status.can_migrate, "explicit retry remains possible");+ assert!(saved_migration_error(other_root.path()).is_none());+ }+ // Only successful completion clears the durable block.+ clear_migration_error(root.path()).unwrap();+ assert!(saved_migration_error(root.path()).is_none());+ }++ #[test]+ fn older_interrupted_attempts_and_unreadable_markers_also_block_retries() {+ let root = tempfile::tempdir().unwrap();+ std::fs::write(root.path().join("storage-migration.json"), b"{}").unwrap();+ assert_eq!(+ saved_migration_error(root.path()).as_deref(),+ Some(INTERRUPTED_MIGRATION)+ );+ std::fs::remove_file(root.path().join("storage-migration.json")).unwrap();+ std::fs::create_dir(root.path().join(MIGRATION_ERROR_FILE)).unwrap();+ assert!(saved_migration_error(root.path()).is_some());+ }++ #[tokio::test]+ async fn preflight_failure_preserves_original_writes_without_retrying_on_later_launches() {+ use screenpipe_db::{+ storage::{migrate, pause_interrupted_migration, MigrationOptions},+ DatabaseManager,+ };++ let root = tempfile::tempdir().unwrap();+ let path = root.path().join("db.sqlite");+ let db = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ db.execute_raw_sql_write("INSERT INTO frames(id,timestamp,full_text) VALUES(1,'2026-09-14T12:00:00Z','before failed migration')").await.unwrap();+ db.close().await;+ save_migration_error(root.path(), INTERRUPTED_MIGRATION).unwrap();+ let mut options = MigrationOptions::default();+ options.budget.disk_reserve_bytes = u64::MAX / 2;+ let error = migrate(root.path(), Default::default(), options)+ .await+ .unwrap_err();+ save_migration_error(root.path(), &error.to_string()).unwrap();++ for id in 2..=3 {+ // Same storage recovery used by ServerCore::start. Each simulated+ // launch restores recording's DB but never invokes migrate again.+ pause_interrupted_migration(root.path()).unwrap();+ let db = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ assert!(db.storage_descriptor().is_none());+ db.execute_raw_sql_write(&format!("INSERT INTO frames(id,timestamp,full_text) VALUES({id},'2026-09-14T12:01:00Z','recording after failed migration')")).await.unwrap();+ let rows = db+ .query_raw_sql("SELECT id FROM frames ORDER BY id")+ .await+ .unwrap();+ assert_eq!(rows.as_array().unwrap().len(), id as usize);+ let status = StorageMigrationStatus {+ error: saved_migration_error(root.path()),+ pending: root.path().join("storage-migration.json").exists(),+ ..legacy_status()+ };+ assert!(!should_start_hidden_ui_migration(&status, true, true, true));+ db.close().await;+ }+ assert!(path.is_file());+ }++ #[test]+ fn hidden_ui_cleanup_recovers_after_restart_and_stops_after_source_removal() {+ let mut status = StorageMigrationStatus {+ completed: true,+ using_new_storage: true,+ can_migrate: false,+ can_delete_source: true,+ ..legacy_status()+ };+ assert!(should_start_hidden_ui_migration(&status, true, true, true));+ status.can_delete_source = false;+ status.source_bytes = 0;+ assert!(!should_start_hidden_ui_migration(&status, true, true, true));+ }++ #[test]+ fn background_completion_does_not_offer_the_interactive_recovery_copy_prompt() {+ let mut operation = Operation {+ background: true,+ busy: true,+ ..Default::default()+ };+ // If the admin reveals the UI mid-conversion, writes remain blocked.+ assert!(operation.activity().busy);+ operation.busy = false;+ operation.completed = true;+ assert!(!operation.activity().completed);+ operation.background = false;+ assert!(operation.activity().completed);+ }+}diff --git a/crates/screenpipe-db/src/db/setup.rs b/crates/screenpipe-db/src/db/setup.rsindex 8dffd55ed..372538552 100644--- a/crates/screenpipe-db/src/db/setup.rs+++ b/crates/screenpipe-db/src/db/setup.rs@@ -559,11 +559,15 @@ impl DatabaseManager {if background && !bootstrap_storage {if let Some(storage) = &db_manager.storage {- storage.spawn_maintenance(- db_manager.pool.clone(),- db_manager.coordinated_writer(),- db_manager.close_token.clone(),- );+ // A recovered migration keeps resident recording data in SQLite.+ // Starting the sealer here would silently retry its conversion.+ if !storage.root.join("storage-migration.json").exists() {+ storage.spawn_maintenance(+ db_manager.pool.clone(),+ db_manager.coordinated_writer(),+ db_manager.close_token.clone(),+ );+ }}}diff --git a/crates/screenpipe-db/src/storage/bulk/elements/lifecycle.rs b/crates/screenpipe-db/src/storage/bulk/elements/lifecycle.rsindex cb285a233..4534f3cc9 100644--- a/crates/screenpipe-db/src/storage/bulk/elements/lifecycle.rs+++ b/crates/screenpipe-db/src/storage/bulk/elements/lifecycle.rs@@ -190,37 +190,41 @@ async fn rewrite(format!(" LIMIT {FILE_ROWS}"),)};- let sql=format!("SELECT id,_archive_generation,{} FROM elements WHERE id BETWEEN ? AND ? AND ({}) ORDER BY id{limit}",names(),TABLE.eligible);- let mut stream = sqlx::query(sqlx::AssertSqlSafe(sql))- .bind(first)- .bind(last)- .fetch(pool);- let mut rows = Vec::new();- let mut bytes = 0;+ // Release each read cursor before encoding takes the writer. Offline+ // migration has one connection; a recovered resident range can exceed one+ // file budget and must not hold that connection while waiting for itself.+ let mut after = None;let mut files = Vec::new();- while let Some(row) = stream.try_next().await? {- let row = record(&row)?;- if row.bytes() > storage.descriptor.budget.record_bytes {- return Err(storage_error("element record exceeds sealing budget"));- }- if !rows.is_empty()- && (rows.len() == FILE_ROWS- || bytes + row.bytes() > storage.descriptor.budget.file_bytes)- {- files.push(encode(storage, writer, std::mem::take(&mut rows)).await?);- bytes = 0;- // New ranges yield after one bounded batch. Existing ranges are- // replaced completely, split into bounded files as needed.- if range.is_none() {+ loop {+ let lower = after.map_or_else(|| "1".to_owned(), |id| format!("id>{id}"));+ let sql=format!("SELECT id,_archive_generation,{} FROM elements WHERE id BETWEEN ? AND ? AND {lower} AND ({}) ORDER BY id{limit}",names(),TABLE.eligible);+ let mut stream = sqlx::query(sqlx::AssertSqlSafe(sql))+ .bind(first)+ .bind(last)+ .fetch(pool);+ let mut rows = Vec::new();+ let mut bytes = 0;+ while let Some(row) = stream.try_next().await? {+ let row = record(&row)?;+ if row.bytes() > storage.descriptor.budget.record_bytes {+ return Err(storage_error("element record exceeds sealing budget"));+ }+ if !rows.is_empty()+ && (rows.len() == FILE_ROWS+ || bytes + row.bytes() > storage.descriptor.budget.file_bytes)+ {break;}+ bytes += row.bytes();+ rows.push(row);}- bytes += row.bytes();- rows.push(row);- }- drop(stream);- if !rows.is_empty() {+ drop(stream);+ let Some(last_row) = rows.last() else { break };+ after = Some(last_row.id);files.push(encode(storage, writer, rows).await?);+ if range.is_none() {+ break;+ }}let last = if range.is_some() {lastdiff --git a/crates/screenpipe-db/src/storage/in_place.rs b/crates/screenpipe-db/src/storage/in_place.rsindex 19081abe9..7ac1530b0 100644--- a/crates/screenpipe-db/src/storage/in_place.rs+++ b/crates/screenpipe-db/src/storage/in_place.rs@@ -53,6 +53,23 @@ pub(super) async fn convert(original_allocated: u64,total_records: u64,progress: &(impl Fn(MigrationProgress) + Send + Sync),+) -> Result<(), sqlx::Error> {+ run(storage, original_allocated, total_records, progress, true).await+}++/// Restore the resident schema and search indexes without encoding payloads,+/// reclaiming files, or completing the migration. Ordinary recording can then+/// use committed Parquet plus resident SQLite until an explicit migration retry.+pub(super) async fn recover_recording(storage: Arc<HybridStorage>) -> Result<(), sqlx::Error> {+ run(storage, 0, 0, &|_| {}, false).await+}++async fn run(+ storage: Arc<HybridStorage>,+ original_allocated: u64,+ total_records: u64,+ progress: &(impl Fn(MigrationProgress) + Send + Sync),+ archive: bool,) -> Result<(), sqlx::Error> {let index = storage.root.join(&storage.descriptor.index);let lease = screenpipe_sqlite_coordinator::acquire_sqlite_manager_lease(&index)@@ -86,7 +103,7 @@ pub(super) async fn convert(screenpipe_sqlite_coordinator::sqlite_write_lock(&index),);let result = async {- reserve(&storage)?;+ if archive { reserve(&storage)?; }{let permit = writer.lock().await?;let mut conn = permit.pool().acquire().await?;@@ -112,29 +129,40 @@ pub(super) async fn convert(schema::finish_step(&mut tx,"suspend-original-triggers").await?;tx.commit().await?;}+ // A paused migration records resident payloads like the original+ // SQLite store. Its disabled sealer must not impose a staging cap+ // that eventually stops recording. Explicit completion restores it.+ sqlx::query("UPDATE storage_metadata SET staging_limit=?")+ .bind(if archive { storage.descriptor.budget.staging_bytes as i64 } else { i64::MAX })+ .execute(&mut *conn).await?;if schema::converted_step(&mut conn,"conversion-complete").await? { return Ok(()); }schema::construction_checkpoint(&mut conn).await?;}// Encoding reservations left by an interrupted attempt are handled by// the existing catalog reclamation; published files are never discarded.- storage.reclaim_once(&pool, &writer).await?;- storage.reclaim_bulk(&pool, &writer).await?;- reclaim(&storage, &writer, file).await?;- report(&storage, &pool, original_allocated, total_records, progress).await?;+ if archive {+ storage.reclaim_once(&pool, &writer).await?;+ storage.reclaim_bulk(&pool, &writer).await?;+ reclaim(&storage, &writer, file).await?;+ report(&storage, &pool, original_allocated, total_records, progress).await?;+ }loop {- reserve(&storage)?;- if storage.seal_once(&pool, &writer).await? != 0 {+ if archive { reserve(&storage)?; }+ if archive && storage.seal_once(&pool, &writer).await? != 0 {reclaim(&storage, &writer, file).await?;report(&storage, &pool, original_allocated, total_records, progress).await?;continue;}let last: Option<i64> = sqlx::query_scalar("SELECT max(frame_id) FROM frame_payloads").fetch_one(&pool).await?;- let columns = super::import::columns(&pool, "frames").await?;- let columns: Vec<_> = columns.into_iter().filter(|c| !c.starts_with("payload_")).collect();- let rows = super::import::batch(&pool, "frames", &columns, last, &storage.descriptor.budget).await?;- if rows.is_empty() { break; }- let first = rows.first().unwrap().get::<i64,_>(0);- let last = rows.last().unwrap().get::<i64,_>(0);+ let range = if archive {+ let columns = super::import::columns(&pool, "frames").await?;+ let columns: Vec<_> = columns.into_iter().filter(|c| !c.starts_with("payload_")).collect();+ let rows = super::import::batch(&pool, "frames", &columns, last, &storage.descriptor.budget).await?;+ rows.first().zip(rows.last()).map(|(first,last)| (first.get::<i64,_>(0),last.get::<i64,_>(0)))+ } else {+ resident_range(&pool, "frames", last, storage.descriptor.budget.file_bytes).await?+ };+ let Some((first, last)) = range else { break };let permit = writer.lock().await?;let mut conn = permit.pool().acquire().await?;let mut tx = conn.begin().await?;@@ -147,33 +175,47 @@ pub(super) async fn convert(if source_exists {let columns = super::import::columns(&pool, "_bulk_elements_source").await?;loop {- reserve(&storage)?;- if bulk::elements::seal(&storage, &pool, &writer).await? != 0 {+ if archive { reserve(&storage)?; }+ if archive && bulk::elements::seal(&storage, &pool, &writer).await? != 0 {reclaim(&storage, &writer, file).await?;report(&storage, &pool, original_allocated, total_records, progress).await?;continue;}- let rows = super::import::batch(&pool, "_bulk_elements_source", &columns, None, &storage.descriptor.budget).await?;- if rows.is_empty() { break; }- let first = rows.first().unwrap().get::<i64,_>(0);- let last = rows.last().unwrap().get::<i64,_>(0);+ let rows = if archive {+ super::import::batch(&pool, "_bulk_elements_source", &columns, None, &storage.descriptor.budget).await?+ } else { Vec::new() };+ let range = if archive {+ rows.first().zip(rows.last()).map(|(first,last)| (first.get::<i64,_>(0),last.get::<i64,_>(0)))+ } else {+ resident_range(&pool, "_bulk_elements_source", None, storage.descriptor.budget.file_bytes).await?+ };+ let Some((first, last)) = range else { break };let permit = writer.lock().await?;let mut conn = permit.pool().acquire().await?;let mut tx = conn.begin().await?;- super::import::insert(&mut tx, "elements", &columns, &rows, true).await?;+ if archive {+ super::import::insert(&mut tx, "elements", &columns, &rows, true).await?;+ } else {+ // SQL-to-SQL batches also recover pre-existing records that+ // exceed the archive record budget, without decoding them.+ let names = columns.iter().map(|c| format!("\"{}\"", c.replace('"', "\"\""))).collect::<Vec<_>>().join(",");+ sqlx::query(sqlx::AssertSqlSafe(format!("INSERT INTO _bulk_element_rows({names},_archive_generation) SELECT {names},1 FROM _bulk_elements_source WHERE id BETWEEN ? AND ?")))+ .bind(first).bind(last).execute(&mut *tx).await?;+ bulk::elements::import_batch(&mut tx, first, last).await?;+ }sqlx::query("DELETE FROM _bulk_elements_source WHERE id BETWEEN ? AND ?").bind(first).bind(last).execute(&mut *tx).await?;tx.commit().await?;schema::construction_checkpoint(&mut conn).await?;super::faults::checkpoint("migration_batch_staged");drop(conn);drop(permit);- reclaim(&storage, &writer, file).await?;+ if archive { reclaim(&storage, &writer, file).await?; }}let permit = writer.lock().await?;sqlx::query("DROP TABLE _bulk_elements_source").execute(permit.pool()).await?;}for table in bulk::TABLES.iter().filter(|t| t.name != "elements") {- loop {+ while archive {reserve(&storage)?;let rows = storage.select_bulk(&pool, table, None).await?;if rows.is_empty() { break; }@@ -191,7 +233,7 @@ pub(super) async fn convert(reclaim(&storage, &writer, file).await?;report(&storage, &pool, original_allocated, total_records, progress).await?;}- backfill_resident_fts(&storage, &pool, &writer, table, file).await?;+ backfill_resident_fts(&storage, &pool, &writer, table, file, archive).await?;}{let permit = writer.lock().await?;@@ -200,12 +242,20 @@ pub(super) async fn convert(let mut tx=conn.begin().await?;let original: Vec<String> = sqlx::query_scalar("SELECT sql FROM _storage_conversion_triggers WHERE restore=1").fetch_all(&mut *tx).await?;for sql in original { sqlx::raw_sql(sqlx::AssertSqlSafe(sql)).execute(&mut *tx).await?; }- schema::finish_step(&mut tx,"conversion-complete").await?;+ if archive {+ schema::finish_step(&mut tx,"conversion-complete").await?;+ } else {+ // Restored application triggers must be suspended again only+ // if the user explicitly retries archival later.+ sqlx::query("DELETE FROM _storage_conversion_steps WHERE step='suspend-original-triggers'").execute(&mut *tx).await?;+ }tx.commit().await?;}- reclaim(&storage, &writer, file).await?;- progress(MigrationProgress { message: "conversion complete", completed_records: Some(total_records), total_records: Some(total_records),- bytes_saved: Some(original_allocated.saturating_sub(super::reclaim::footprint(&storage.root)?)), available_bytes: Some(fs2::available_space(&storage.root)?) });+ if archive {+ reclaim(&storage, &writer, file).await?;+ progress(MigrationProgress { message: "conversion complete", completed_records: Some(total_records), total_records: Some(total_records),+ bytes_saved: Some(original_allocated.saturating_sub(super::reclaim::footprint(&storage.root)?)), available_bytes: Some(fs2::available_space(&storage.root)?) });+ }Ok(())}.await;if let Err(ref error) = result {@@ -219,6 +269,37 @@ pub(super) async fn convert(result}+// Only identifiers chosen by the offline runner reach this helper. Recovery+// transfers rows inside SQLite instead of allocating archive-sized Rust values.+async fn resident_range(+ pool: &SqlitePool,+ table: &str,+ after: Option<i64>,+ budget_bytes: usize,+) -> Result<Option<(i64, i64)>, sqlx::Error> {+ let lower = after.map_or_else(|| "1".to_owned(), |id| format!("id>{id}"));+ let size = super::import::columns(pool, table).await?.iter().map(|name| {+ let quoted = format!("\"{}\"", name.replace('"', "\"\""));+ format!("CASE WHEN typeof({quoted}) IN ('integer','real') THEN 8 ELSE COALESCE(length(CAST({quoted} AS BLOB)),0) END")+ }).collect::<Vec<_>>().join("+");+ let rows: Vec<(i64, i64)> = sqlx::query_as(sqlx::AssertSqlSafe(format!(+ "SELECT id,{size} FROM {table} WHERE {lower} ORDER BY id LIMIT 128"+ )))+ .fetch_all(pool)+ .await?;+ let mut range = None;+ let mut bytes = 0usize;+ for (id, size) in rows {+ let size = usize::try_from(size).map_err(storage_error)?;+ if range.is_some() && bytes.saturating_add(size) > budget_bytes {+ break;+ }+ bytes = bytes.saturating_add(size);+ range = Some((range.map_or(id, |(first, _)| first), id));+ }+ Ok(range)+}+async fn reclaim(storage: &HybridStorage,writer: &SqliteWritePool,@@ -239,12 +320,17 @@ async fn backfill_resident_fts(writer: &SqliteWritePool,table: &bulk::Table,file: &mut std::fs::File,+ archive: bool,) -> Result<(), sqlx::Error> {use futures::TryStreamExt;if table.fts.is_empty() {return Ok(());}- let step = format!("resident-fts-{}", table.name);+ let step = format!(+ "{}-fts-{}",+ if archive { "resident" } else { "recovery" },+ table.name+ );let size = table.fts.iter()@@ -257,14 +343,16 @@ async fn backfill_resident_fts(.collect::<Vec<_>>().join("+");loop {- reserve(storage)?;+ if archive {+ reserve(storage)?;+ }let after: Option<i64> =sqlx::query_scalar("SELECT last_id FROM _storage_conversion_steps WHERE step=?").bind(&step).fetch_optional(pool).await?.flatten();- let sql = format!("SELECT id,{size} FROM {t} WHERE _archive_file IS NULL AND NOT ({eligible}) AND id {comparison} ? ORDER BY id LIMIT {rows}",t=table.name,eligible=table.eligible,comparison=if after.is_some(){">"}else{">="},rows=bulk::FILE_ROWS);+ let sql = format!("SELECT id,{size} FROM {t} WHERE _archive_file IS NULL AND NOT ({eligible}) AND id {comparison} ? ORDER BY id LIMIT {rows}",t=table.name,eligible=if archive { table.eligible } else { "0" },comparison=if after.is_some(){">"}else{">="},rows=bulk::FILE_ROWS);let mut stream = sqlx::query(sqlx::AssertSqlSafe(sql)).bind(after.unwrap_or(i64::MIN)).fetch(pool);@@ -272,7 +360,7 @@ async fn backfill_resident_fts(let mut bytes = 0_usize;while let Some(row) = stream.try_next().await? {let size = row.get::<i64, _>(1) as usize;- if size > storage.descriptor.budget.record_bytes {+ if archive && size > storage.descriptor.budget.record_bytes {return Err(storage_error("resident search record exceeds migration budget",));@@ -296,8 +384,12 @@ async fn backfill_resident_fts(}sqlx::query("INSERT INTO _storage_conversion_steps(step,last_id) VALUES(?,?) ON CONFLICT(step) DO UPDATE SET last_id=excluded.last_id").bind(&step).bind(last).execute(&mut *tx).await?;tx.commit().await?;- super::reclaim::free_leaves(&mut conn, file, storage.descriptor.budget.decode_bytes)- .await?;+ if archive {+ super::reclaim::free_leaves(&mut conn, file, storage.descriptor.budget.decode_bytes)+ .await?;+ } else {+ schema::construction_checkpoint(&mut conn).await?;+ }}Ok(())}diff --git a/crates/screenpipe-db/src/storage/lifecycle.rs b/crates/screenpipe-db/src/storage/lifecycle.rsindex 9d9123503..b83f59f6d 100644--- a/crates/screenpipe-db/src/storage/lifecycle.rs+++ b/crates/screenpipe-db/src/storage/lifecycle.rs@@ -155,9 +155,7 @@ fn read_journal(root: &Path) -> Result<Journal, sqlx::Error> {pub(super) fn migration_is_paused(root: &Path) -> Result<bool, sqlx::Error> {let journal = read_journal(root)?;- Ok(journal.format == 1- && journal.phase == Phase::Paused- && checked_path(root, Path::new("db.sqlite"))?.is_file())+ Ok(journal.phase == Phase::Paused && checked_path(root, Path::new("db.sqlite"))?.is_file())}pub fn migration_requires_resume(root: &Path) -> Result<bool, sqlx::Error> {@@ -167,6 +165,96 @@ pub fn migration_requires_resume(root: &Path) -> Result<bool, sqlx::Error> {Ok(read_journal(root)?.format == 2)}+/// A paused in-place migration has a complete writable resident schema, but+/// retains its journal so archival can only continue after explicit user action.+pub(super) fn migration_recording_ready(root: &Path) -> Result<bool, sqlx::Error> {+ if !root.join("storage-migration.json").exists() {+ return Ok(false);+ }+ let journal = read_journal(root)?;+ if journal.format != 2 || journal.phase != Phase::Paused {+ return Ok(false);+ }+ let active = StorageDescriptor::read(root)?;+ Ok(active.as_ref() == Some(&journal.descriptor)+ || (active.is_none()+ && root.join("db.sqlite").is_file()+ && !root.join(&journal.descriptor.index).exists()))+}++/// Restore recording after an interrupted conversion without retrying archival.+/// Committed payload files are retained byte-for-byte; remaining rows stay in+/// SQLite. Recovery and subsequent recording use the existing single writer.+pub async fn recover_interrupted_migration(+ root: &Path,+ config: DbConfig,+) -> Result<(), sqlx::Error> {+ if !root.join("storage-migration.json").exists() {+ return Ok(());+ }+ if read_journal(root)?.format == 1 {+ return pause_interrupted_migration(root);+ }+ if migration_recording_ready(root)? {+ return Ok(());+ }+ let root = root.canonicalize()?;+ let lock = std::fs::OpenOptions::new()+ .read(true)+ .write(true)+ .create(true)+ .truncate(false)+ .open(root.join(".storage.lock"))?;+ fs2::FileExt::try_lock_exclusive(&lock)+ .map_err(|_| storage_error("storage lifecycle is already owned"))?;+ let mut journal = read_journal(&root)?;+ let index = checked_path(&root, &journal.descriptor.index)?;+ if root.join("db.sqlite").is_file() {+ if index.exists() || StorageDescriptor::read(&root)?.is_some() {+ return Err(storage_error(+ "migration recovery found conflicting database generations",+ ));+ }+ // Nothing was renamed or converted. Keep the original writable, and+ // refresh its verification receipts only on a later explicit retry.+ journal.phase = Phase::Paused;+ durable_json(&root.join("storage-migration.json"), &journal)?;+ return Ok(());+ }+ if !index.is_file() {+ return Err(storage_error("migration index is missing"));+ }+ if StorageDescriptor::read(&root)?.is_some_and(|active| active != journal.descriptor) {+ return Err(storage_error(+ "active descriptor differs from migration index",+ ));+ }+ let storage = HybridStorage::new(root.clone(), journal.descriptor.clone())?;+ super::in_place::recover_recording(storage.clone()).await?;+ let db = DatabaseManager::new_with_storage(+ index.to_str().unwrap(),+ config,+ Some(storage),+ false,+ false,+ )+ .await?;+ let check = sqlx::query("SELECT id FROM frames LIMIT 1")+ .fetch_optional(&db.pool)+ .await;+ db.close().await;+ check?;+ // Journal first: interruption before descriptor publication remains closed;+ // recovery is idempotent and never starts a Parquet conversion batch.+ journal.phase = Phase::Paused;+ durable_json(&root.join("storage-migration.json"), &journal)?;+ super::faults::checkpoint("migration_recovery_ready");+ durable_json(&root.join("storage.json"), &journal.descriptor)?;+ super::faults::checkpoint("migration_recovery_activated");+ tracing::info!("storage recovered for recording; migration remains paused");+ Ok(())+}+/// Reopening the desktop app restores ordinary use of its last active storage./// An unpublished candidate is paused; resuming it takes a fresh source snapshot.pub fn pause_interrupted_migration(root: &Path) -> Result<(), sqlx::Error> {@@ -361,6 +449,49 @@ pub async fn migrate_with_progress(super::faults::checkpoint("migration_before_rename");journal};+ if journal.phase == Phase::Paused {+ // Recording may have appended, edited, or retained history since the+ // failure. The explicit retry verifies that current logical dataset,+ // keeping all already committed archive files and their generations.+ let active = StorageDescriptor::read(&root)?;+ let path = if active.is_some() {+ root.join(&journal.descriptor.index)+ } else {+ source_path.clone()+ };+ let storage = active+ .map(|d| HybridStorage::new(root.clone(), d))+ .transpose()?;+ let db = DatabaseManager::new_with_storage(+ path.to_str().unwrap(),+ config.clone(),+ storage,+ false,+ false,+ )+ .await?;+ let refreshed = async {+ let receipts = table_receipts(&db, Some(&journal.source)).await?;+ let mut searches = Vec::new();+ for (term, _) in &journal.search_receipts {+ let ids = sqlx::query_scalar(MIGRATION_SEARCH)+ .bind(term)+ .fetch_all(&db.pool)+ .await?;+ searches.push((term.clone(), ids));+ }+ Ok::<_, sqlx::Error>((receipts, searches))+ }+ .await;+ db.close().await;+ let (receipts, searches) = refreshed?;+ journal.source = receipts;+ journal.search_receipts = searches;+ journal.snapshot = source_identity(&root)?;+ journal.report = None;+ journal.phase = Phase::Building;+ durable_json(&journal_path, &journal)?;+ }if let Some(active) = StorageDescriptor::read(&root)? {if active != journal.descriptor {return Err(storage_error(diff --git a/crates/screenpipe-db/src/storage/mod.rs b/crates/screenpipe-db/src/storage/mod.rsindex be51d4f7c..3744ee780 100644--- a/crates/screenpipe-db/src/storage/mod.rs+++ b/crates/screenpipe-db/src/storage/mod.rs@@ -29,7 +29,8 @@ pub use command::run_command;pub use inventory::{artifact_bytes, inventory};pub use lifecycle::{cancel_migration, migrate, migrate_with_progress, migration_report, migration_requires_resume,- pause_interrupted_migration, MigrationOptions, MigrationProgress, MigrationReport,+ pause_interrupted_migration, recover_interrupted_migration, MigrationOptions,+ MigrationProgress, MigrationReport,};pub use maintenance::{compact, export_sqlite};pub use reader::StorageReadToken;@@ -266,7 +267,9 @@ pub fn resolve_database_path(database: &Path) -> Result<PathBuf, sqlx::Error> {.filter(|path| !path.as_os_str().is_empty()).unwrap_or_else(|| Path::new("."));if database.file_name().is_some_and(|name| name == "db.sqlite") {- if lifecycle::migration_requires_resume(root)? {+ if lifecycle::migration_requires_resume(root)?+ && !lifecycle::migration_recording_ready(root)?+ {return Err(storage_error("in-place migration pending; resume migration before opening history or recording",));@@ -431,6 +434,13 @@ impl HybridStorage {let Some(root) = path.parent().and_then(Path::parent).and_then(Path::parent) else {return Ok(None);};+ if lifecycle::migration_requires_resume(root)?+ && !lifecycle::migration_recording_ready(root)?+ {+ return Err(storage_error(+ "interrupted migration must restore recording before opening its index",+ ));+ }let Some(descriptor) = StorageDescriptor::read(root)? else {return Err(storage_error("hybrid index requires an active storage descriptor",diff --git a/crates/screenpipe-db/tests/hybrid_storage.rs b/crates/screenpipe-db/tests/hybrid_storage.rsindex e91da8509..5ded61f70 100644--- a/crates/screenpipe-db/tests/hybrid_storage.rs+++ b/crates/screenpipe-db/tests/hybrid_storage.rs@@ -337,6 +337,41 @@ async fn legacy_migration_preserves_original_until_explicit_deletion_on_live_gen"after activation");db.close().await;+ // Automatic hidden-UI cleanup must also survive the next app launch:+ // opening the legacy path follows the active generation even when the+ // original file is gone, preserving old history and subsequent writes.+ let reopened = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ assert!(!path.exists());+ let payloads = reopened+ .frame_payloads(&[7, 8], Projection::Search)+ .await+ .unwrap();+ assert_eq!(payloads[&7].text(), "migration café");+ assert_eq!(payloads[&8].text(), "after activation");+ assert_eq!(+ search(&reopened, "migration")+ .await+ .as_array()+ .unwrap()+ .len(),+ 1+ );+ seed(&reopened, 9, Some("after cleanup restart"), None).await;+ assert_eq!(+ reopened+ .frame_payloads(&[9], Projection::Search)+ .await+ .unwrap()[&9]+ .text(),+ "after cleanup restart"+ );+ assert_eq!(+ search(&reopened, "cleanup").await.as_array().unwrap().len(),+ 1+ );+ reopened.close().await;}#[tokio::test]@@ -725,10 +760,11 @@ async fn interrupted_initialization_resumes_its_recorded_generation() {#[cfg(feature = "storage-fault-injection")]#[tokio::test]-async fn interrupted_desktop_migration_blocks_recording_until_resume_completes() {- use screenpipe_db::storage::pause_interrupted_migration;+async fn interrupted_desktop_migration_restores_recording_without_conversion_retry() {+ use screenpipe_db::storage::recover_interrupted_migration;for point in [+ "migration_before_rename","migration_after_rename","migration_ready","migration_activated",@@ -748,15 +784,29 @@ async fn interrupted_desktop_migration_blocks_recording_until_resume_completes().unwrap();assert_eq!(killed.status.code(), Some(86), "{point}");- pause_interrupted_migration(root.path()).unwrap();- assert!(- DatabaseManager::new(path.to_str().unwrap(), Default::default())+ for id in 8..=9 {+ recover_interrupted_migration(root.path(), Default::default()).await- .is_err()- );- migrate(root.path(), Default::default(), Default::default())+ .unwrap();+ let db = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ assert_eq!(+ db.frame_payloads(&[7], Projection::Search).await.unwrap()[&7].text(),+ "before restart"+ );+ seed(&db, id, Some("recording with migration paused"), None).await;+ assert_eq!(+ search(&db, "recording").await.as_array().unwrap().len(),+ (id - 7) as usize+ );+ db.close().await;+ }+ assert!(root.path().join("storage-migration.json").exists());+ let report = migrate(root.path(), Default::default(), Default::default()).await.unwrap();+ assert_eq!(report.frames, 3);let db = DatabaseManager::new(path.to_str().unwrap(), Default::default()).await.unwrap();@@ -764,7 +814,14 @@ async fn interrupted_desktop_migration_blocks_recording_until_resume_completes()db.frame_payloads(&[7], Projection::Search).await.unwrap()[&7].text(),"before restart");- seed(&db, 8, Some("after restart"), None).await;+ seed(&db, 10, Some("after explicit retry"), None).await;+ assert_eq!(+ db.frame_payloads(&[8, 9, 10], Projection::Search)+ .await+ .unwrap()+ .len(),+ 3+ );assert!(db.storage_descriptor().is_some());assert!(!path.is_file(), "source reused at {point}");db.close().await;diff --git a/crates/screenpipe-db/tests/in_place_migration.rs b/crates/screenpipe-db/tests/in_place_migration.rsindex 50c403c06..83402c4b9 100644--- a/crates/screenpipe-db/tests/in_place_migration.rs+++ b/crates/screenpipe-db/tests/in_place_migration.rs@@ -428,10 +428,23 @@ async fn failed_payload_write_resumes_without_discarding_published_files() {},).await;+ assert!(result.is_err());+ // Recover while the fault is still present. Recovery must not need another+ // write to the failed archive destination before recording can resume.+ screenpipe_db::storage::recover_interrupted_migration(root.path(), Default::default())+ .await+ .unwrap();+ let db = DatabaseManager::new(+ root.path().join("db.sqlite").to_str().unwrap(),+ Default::default(),+ )+ .await+ .unwrap();+ db.execute_raw_sql_write("INSERT INTO frames(id,timestamp,full_text) VALUES(49,'2026-09-14T12:00:00Z','recording while archive writes fail')").await.unwrap();+ db.close().await;for (path, permissions) in changed.into_inner().unwrap() {std::fs::set_permissions(path, permissions).unwrap();}- assert!(result.is_err());assert!(screenpipe_db::storage::migration_requires_resume(root.path()).unwrap());let files: Vec<_> = screenpipe_db::storage::inventory(root.path()).unwrap()@@ -599,3 +612,221 @@ async fn reuses_index_and_preserves_history_and_compact_backup() {.unwrap();assert_eq!(imported.frames, 48);}++#[tokio::test]+#[cfg(feature = "storage-fault-injection")]+async fn recording_recovery_keeps_committed_payloads_and_survives_restarts() {+ use screenpipe_db::storage::{inventory, recover_interrupted_migration, PrivacyPolicy};+ for (point, hit) in [+ ("migration_schema_step", 2),+ ("migration_schema_step", 8),+ ("migration_batch_staged", 2),+ ("seal_files_synced", 1),+ ("seal_committed", 1),+ ("bulk_committed", 1),+ ("migration_blocks_reclaimed", 2),+ ] {+ let root = tempfile::tempdir().unwrap();+ fixture(root.path()).await;+ let crashed = std::process::Command::new(env!("CARGO_BIN_EXE_screenpipe-storage"))+ .arg("migrate")+ .arg(root.path())+ .env("SCREENPIPE_STORAGE_CRASH_AT", point)+ .env("SCREENPIPE_STORAGE_CRASH_HIT", hit.to_string())+ .output()+ .unwrap();+ assert_eq!(+ crashed.status.code(),+ Some(86),+ "{point}: {}",+ String::from_utf8_lossy(&crashed.stderr)+ );+ let payloads = || {+ inventory(root.path())+ .unwrap()+ .into_iter()+ .filter(|p| p.extension().is_some_and(|e| e == "parquet"))+ .map(|p| {+ let data = std::fs::read(&p).unwrap();+ (p, data)+ })+ .collect::<std::collections::BTreeMap<_, _>>()+ };+ let before = payloads();+ for id in 49..=50 {+ let started = std::time::Instant::now();+ recover_interrupted_migration(root.path(), Default::default())+ .await+ .unwrap_or_else(|e| panic!("{point}/{hit}: {e}"));+ eprintln!(+ "recording recovery {point}/{hit} launch {id}: {:?}",+ started.elapsed()+ );+ let db = DatabaseManager::new(+ root.path().join("db.sqlite").to_str().unwrap(),+ Default::default(),+ )+ .await+ .unwrap();+ db.verify_storage().await.unwrap();+ assert_eq!(+ db.frame_payloads(&[1, 48], Projection::All).await.unwrap()[&48].text(),+ "searchable migration history"+ );+ assert_eq!(+ sqlx::query_scalar::<_, i64>(+ "SELECT count(*) FROM frames_fts WHERE frames_fts MATCH 'searchable'"+ )+ .fetch_one(&db.pool)+ .await+ .unwrap(),+ 48+ );+ assert_eq!(+ sqlx::query_scalar::<_, i64>(+ "SELECT count(*) FROM elements_fts WHERE elements_fts MATCH 'searchable'"+ )+ .fetch_one(&db.pool)+ .await+ .unwrap(),+ 48+ );+ db.execute_raw_sql_write(&format!("INSERT INTO frames(id,timestamp,full_text) VALUES({id},'2026-09-14T12:00:00Z','recovered recording'); INSERT INTO elements(id,frame_id,source,role,text) VALUES({id},{id},'accessibility','AXText','recovered element')")).await.unwrap();+ assert_eq!(+ sqlx::query_scalar::<_, i64>("SELECT count(*) FROM elements")+ .fetch_one(&db.pool)+ .await+ .unwrap(),+ id+ );+ db.set_frame_privacy_policy(&PrivacyPolicy::default())+ .await+ .unwrap();+ if point == "seal_committed" && id == 49 {+ // Wait beyond the normal maintenance interval: reopening must+ // not quietly continue conversion through the background sealer.+ tokio::time::sleep(std::time::Duration::from_millis(5200)).await;+ }+ db.close().await;+ assert_eq!(+ payloads(),+ before,+ "recovery changed Parquet at {point}/{hit}"+ );+ }+ let report = migrate(root.path(), Default::default(), Default::default())+ .await+ .unwrap_or_else(|e| panic!("explicit retry {point}: {e}"));+ assert_eq!(report.frames, 50);+ let db = DatabaseManager::new(+ root.path().join("db.sqlite").to_str().unwrap(),+ Default::default(),+ )+ .await+ .unwrap();+ assert_eq!(+ db.frame_payloads(&[49, 50], Projection::Search)+ .await+ .unwrap()+ .len(),+ 2+ );+ assert_eq!(+ sqlx::query_scalar::<_, i64>(+ "SELECT count(*) FROM elements_fts WHERE elements_fts MATCH 'recovered'"+ )+ .fetch_one(&db.pool)+ .await+ .unwrap(),+ 2+ );+ db.verify_storage().await.unwrap();+ db.close().await;+ }+}++#[tokio::test]+#[cfg(feature = "storage-fault-injection")]+async fn recording_recovery_child() {+ let Ok(root) = std::env::var("SCREENPIPE_TEST_RECOVERY_ROOT") else {+ return;+ };+ screenpipe_db::storage::recover_interrupted_migration(+ std::path::Path::new(&root),+ Default::default(),+ )+ .await+ .unwrap();+}++#[tokio::test]+#[cfg(feature = "storage-fault-injection")]+async fn recording_recovery_is_durable_when_interrupted_before_or_after_activation() {+ for point in ["migration_recovery_ready", "migration_recovery_activated"] {+ let root = tempfile::tempdir().unwrap();+ fixture(root.path()).await;+ let stopped = std::process::Command::new(env!("CARGO_BIN_EXE_screenpipe-storage"))+ .arg("migrate")+ .arg(root.path())+ .env("SCREENPIPE_STORAGE_CRASH_AT", "seal_committed")+ .output()+ .unwrap();+ assert_eq!(stopped.status.code(), Some(86));+ let stopped = std::process::Command::new(std::env::current_exe().unwrap())+ .args(["--exact", "recording_recovery_child", "--nocapture"])+ .env("SCREENPIPE_TEST_RECOVERY_ROOT", root.path())+ .env("SCREENPIPE_STORAGE_CRASH_AT", point)+ .output()+ .unwrap();+ assert_eq!(+ stopped.status.code(),+ Some(86),+ "{point}: {}",+ String::from_utf8_lossy(&stopped.stderr)+ );+ screenpipe_db::storage::recover_interrupted_migration(root.path(), Default::default())+ .await+ .unwrap();+ let db = DatabaseManager::new(+ root.path().join("db.sqlite").to_str().unwrap(),+ Default::default(),+ )+ .await+ .unwrap();+ db.verify_storage().await.unwrap();+ db.execute_raw_sql_write("INSERT INTO frames(id,timestamp,full_text) VALUES(49,'2026-09-14T12:00:00Z','recording after interrupted recovery')").await.unwrap();+ db.close().await;+ }+}++#[tokio::test]+async fn oversized_legacy_record_does_not_prevent_recording_recovery() {+ let root = tempfile::tempdir().unwrap();+ let path = root.path().join("db.sqlite");+ let db = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ db.execute_raw_sql_write("INSERT INTO frames(id,timestamp,full_text,accessibility_tree_json) VALUES(1,'2026-09-14T12:00:00Z','oversized history',printf('%.*c',2097152,'x'))").await.unwrap();+ db.close().await;+ let mut options = MigrationOptions::default();+ options.budget.record_bytes = 1024 * 1024;+ assert!(migrate(root.path(), Default::default(), options)+ .await+ .is_err());+ screenpipe_db::storage::recover_interrupted_migration(root.path(), Default::default())+ .await+ .unwrap();+ let db = DatabaseManager::new(path.to_str().unwrap(), Default::default())+ .await+ .unwrap();+ assert_eq!(+ db.frame_payloads(&[1], Projection::All).await.unwrap()[&1]+ .accessibility_tree_json+ .as_ref()+ .unwrap()+ .len(),+ 2097152+ );+ db.execute_raw_sql_write("INSERT INTO frames(id,timestamp,full_text) VALUES(2,'2026-09-14T12:01:00Z','new recording')").await.unwrap();+ db.close().await;+}diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.mdindex 923f90ae5..593e9cc0d 100644--- a/docs/coverage/CORE.md+++ b/docs/coverage/CORE.md@@ -9,10 +9,10 @@ confidence, and criticality.- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y- Mapped suites: 32- Mapped Rust files: 351-- Active test blocks: 3472+- Active test blocks: 3476- Ignored/manual test blocks: 142-- Declared test blocks: 3614-- Weighted coverage points: 2862.3+- Declared test blocks: 3618+- Weighted coverage points: 2866.3Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.Criticality weights: high=1.0, medium=0.7, low=0.4.@@ -23,16 +23,16 @@ are explicitly enabled in a runtime lane.| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 3329 | 136 | 2797.2 | 21 | 11 | 100% |-| macos | 29 | 3391 | 117 | 2811.2 | 22 | 11 | 100% |-| linux | 25 | 2959 | 109 | 2456.5 | 20 | 11 | 100% |+| windows | 29 | 3333 | 136 | 2801.2 | 21 | 11 | 100% |+| macos | 29 | 3395 | 117 | 2815.2 | 22 | 11 | 100% |+| linux | 25 | 2963 | 109 | 2460.5 | 20 | 11 | 100% |## Crate Summary| Crate | Suites | Integration files | Source unit files | Active tests | Ignored tests | Weighted points | Flows || --- | --- | --- | --- | --- | --- | --- | --- || screenpipe-engine | 10 | 19 | 116 | 1706 | 42 | 1315.0 | 10 |-| screenpipe-db | 5 | 56 | 22 | 517 | 19 | 490.3 | 9 |+| screenpipe-db | 5 | 56 | 22 | 521 | 19 | 494.3 | 9 || screenpipe-sqlite-coordinator | 1 | 0 | 3 | 27 | 0 | 27.0 | 2 || screenpipe-audio | 6 | 25 | 51 | 611 | 44 | 536.5 | 5 || screenpipe-screen | 6 | 9 | 18 | 257 | 9 | 232.1 | 4 |@@ -65,21 +65,21 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| audio | 7 suites / 724 active / 45 ignored / 649.5 pts | 7 suites / 724 active / 45 ignored / 649.5 pts | 6 suites / 649 active / 44 ignored / 597.0 pts || audio-device | 2 suites / 218 active / 7 ignored / 195.5 pts | 2 suites / 218 active / 7 ignored / 195.5 pts | 1 suites / 143 active / 6 ignored / 143.0 pts || configuration | 2 suites / 143 active / 3 ignored / 129.2 pts | 2 suites / 143 active / 3 ignored / 129.2 pts | 2 suites / 143 active / 3 ignored / 129.2 pts |-| database | 6 suites / 464 active / 15 ignored / 437.3 pts | 6 suites / 464 active / 15 ignored / 437.3 pts | 6 suites / 464 active / 15 ignored / 437.3 pts |+| database | 6 suites / 468 active / 15 ignored / 441.3 pts | 6 suites / 468 active / 15 ignored / 441.3 pts | 6 suites / 468 active / 15 ignored / 441.3 pts || db-search | 2 suites / 113 active / 9 ignored / 113.0 pts | 2 suites / 113 active / 9 ignored / 113.0 pts | 2 suites / 113 active / 9 ignored / 113.0 pts || engine-lifecycle | 6 suites / 243 active / 1 ignored / 218.0 pts | 6 suites / 243 active / 1 ignored / 218.0 pts | 5 suites / 237 active / 1 ignored / 216.3 pts || local-api | 2 suites / 416 active / 9 ignored / 293.6 pts | 2 suites / 416 active / 9 ignored / 293.6 pts | 2 suites / 416 active / 9 ignored / 293.6 pts || meeting | 6 suites / 1618 active / 20 ignored / 1319.8 pts | 6 suites / 1618 active / 20 ignored / 1319.8 pts | 4 suites / 1281 active / 16 ignored / 1005.3 pts || ocr | 4 suites / 125 active / 7 ignored / 119.0 pts | 4 suites / 129 active / 7 ignored / 124.5 pts | 3 suites / 120 active / 6 ignored / 115.5 pts || os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - |-| performance | 13 suites / 1519 active / 70 ignored / 1339.5 pts | 14 suites / 1626 active / 74 ignored / 1382.3 pts | 13 suites / 1519 active / 70 ignored / 1339.5 pts |+| performance | 13 suites / 1523 active / 70 ignored / 1343.5 pts | 14 suites / 1630 active / 74 ignored / 1386.3 pts | 13 suites / 1523 active / 70 ignored / 1343.5 pts || pipes | 1 suites / 511 active / 3 ignored / 357.7 pts | 1 suites / 511 active / 3 ignored / 357.7 pts | 1 suites / 511 active / 3 ignored / 357.7 pts || privacy | 5 suites / 942 active / 36 ignored / 763.8 pts | 5 suites / 1000 active / 17 ignored / 772.3 pts | 5 suites / 920 active / 14 ignored / 742.7 pts || real-app | - | 1 suites / 107 active / 4 ignored / 42.8 pts | - || speaker | 2 suites / 362 active / 9 ignored / 362.0 pts | 2 suites / 362 active / 9 ignored / 362.0 pts | 2 suites / 362 active / 9 ignored / 362.0 pts |-| storage | 3 suites / 569 active / 32 ignored / 467.0 pts | 3 suites / 569 active / 32 ignored / 467.0 pts | 3 suites / 569 active / 32 ignored / 467.0 pts |+| storage | 3 suites / 573 active / 32 ignored / 471.0 pts | 3 suites / 573 active / 32 ignored / 471.0 pts | 3 suites / 573 active / 32 ignored / 471.0 pts || sync | 1 suites / 511 active / 3 ignored / 357.7 pts | 1 suites / 511 active / 3 ignored / 357.7 pts | 1 suites / 511 active / 3 ignored / 357.7 pts |-| timeline | 4 suites / 1129 active / 36 ignored / 916.3 pts | 4 suites / 1129 active / 36 ignored / 916.3 pts | 4 suites / 1129 active / 36 ignored / 916.3 pts |+| timeline | 4 suites / 1133 active / 36 ignored / 920.3 pts | 4 suites / 1133 active / 36 ignored / 920.3 pts | 4 suites / 1133 active / 36 ignored / 920.3 pts || transcription | 5 suites / 801 active / 41 ignored / 626.6 pts | 5 suites / 801 active / 41 ignored / 626.6 pts | 5 suites / 801 active / 41 ignored / 626.6 pts || ui-events | 4 suites / 758 active / 28 ignored / 576.2 pts | 3 suites / 709 active / 5 ignored / 541.9 pts | 3 suites / 709 active / 5 ignored / 541.9 pts || vision-capture | 5 suites / 549 active / 32 ignored / 433.8 pts | 5 suites / 553 active / 32 ignored / 439.3 pts | 4 suites / 544 active / 31 ignored / 430.3 pts |@@ -132,7 +132,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-audio-meetings-speakers | screenpipe-db | windows, macos, linux | database, audio, meeting, speaker | audio-record-transcribe, audio-device-health, meeting-live-notes | high | strong | integration | 15 | 113 | 1 | Audio transcript dedupe, live meeting mirroring, end-generation and open-meeting invariants, liveness, and speaker reassignment coverage. || db-runtime-reliability | screenpipe-db | windows, macos, linux | database, performance | performance-liveness | high | partial | mixed | 15 | 43 | 6 | SQLite hard-fault classification, failpoint VFS injection, fresh-identity recovery verification with integrity/FK/write canaries, query cancellation, close-severs-connections regressions, multi-pool WAL parity, runtime version pinning, and WAL chaos plus memory-pressure probes, plus read-only quarantine self-heal verification for transient IOERR faults. || db-search-indexing | screenpipe-db | windows, macos, linux | db-search, ocr, accessibility, performance | local-api-search, capture-ocr-pipeline, accessibility-ui-events, performance-liveness | high | strong | mixed | 13 | 105 | 4 | FTS, tokenizer, OCR snapshot search, query planning, ordering, accessibility search, and contention coverage. |-| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 28 | 229 | 6 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. |+| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 28 | 233 | 6 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. || engine-api-routes | screenpipe-engine | windows, macos, linux | local-api, timeline, meeting, transcription | local-api-search, timeline-streaming, meeting-live-notes, audio-record-transcribe | high | partial | mixed | 40 | 408 | 4 | Route/unit coverage for search, health, streaming, meetings, time/timezone, and transcription. Legacy endpoint/websocket tests require local data and remain ignored. || engine-capture-timeline | screenpipe-engine | windows, macos, linux | vision-capture, timeline, storage, performance | capture-ocr-pipeline, timeline-streaming, performance-liveness | high | partial | mixed | 25 | 301 | 26 | Covers capture trigger logic, frame/audio linking, hot cache, timeline refresh regressions, fragmented MP4 extraction, and HD-mode control. Several real-data tests are intentionally ignored by default. || engine-config-lifecycle | screenpipe-engine | windows, macos, linux | configuration, engine-lifecycle, performance | settings-to-engine-config, engine-health-lifecycle, performance-liveness | high | strong | mixed | 12 | 116 | 1 | Fast logic coverage for the config bridge, health-endpoint identity, tray health debounce, sleep/power policies, and queue backpressure. |@@ -302,7 +302,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-search-indexing | screenpipe-db | tests/fts_dots_test.rs | integration | 13 | 0 | 13 || db-accessibility-ui-events | screenpipe-db | tests/heavy_read_test.rs | integration | 0 | 2 | 2 || db-timeline-frames | screenpipe-db | tests/hybrid_storage.rs | integration | 16 | 0 | 16 |-| db-timeline-frames | screenpipe-db | tests/in_place_migration.rs | integration | 7 | 3 | 10 |+| db-timeline-frames | screenpipe-db | tests/in_place_migration.rs | integration | 11 | 3 | 14 || db-search-indexing | screenpipe-db | tests/keyword_search_accessibility_test.rs | integration | 7 | 0 | 7 || db-search-indexing | screenpipe-db | tests/keyword_search_order_test.rs | integration | 3 | 0 | 3 || db-timeline-frames | screenpipe-db | tests/live_coverage_marker_test.rs | integration | 7 | 0 | 7 |