YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
fix(db): self-heal missing speaker_embeddings table on startup (#5925)
Ezra Ellette2deb5d772c0a
diff --git a/COVERAGE.md b/COVERAGE.mdindex 30f6ce99b..c8c1411ef 100644--- a/COVERAGE.md+++ b/COVERAGE.md@@ -32,15 +32,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.- Mapped suites: 32- Mapped Rust files: 311-- Active test blocks: 2924+- Active test blocks: 2925- Ignored/manual test blocks: 134-- Weighted coverage points: 2403.3+- Weighted coverage points: 2404.3| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2796 | 129 | 2344.5 | 21 | 11 | 100% |-| macos | 29 | 2847 | 109 | 2354.4 | 22 | 11 | 100% |-| linux | 25 | 2485 | 102 | 2063.4 | 20 | 11 | 100% |+| windows | 29 | 2797 | 129 | 2345.5 | 21 | 11 | 100% |+| macos | 29 | 2848 | 109 | 2355.4 | 22 | 11 | 100% |+| linux | 25 | 2486 | 102 | 2064.4 | 20 | 11 | 100% |## Refreshdiff --git a/crates/screenpipe-db/src/db/setup.rs b/crates/screenpipe-db/src/db/setup.rsindex 3f5358aca..d80c64ae0 100644--- a/crates/screenpipe-db/src/db/setup.rs+++ b/crates/screenpipe-db/src/db/setup.rs@@ -446,6 +446,15 @@ impl DatabaseManager {// upgraded across that migration boundary may have skipped it.Self::ensure_memories_sync_columns(pool).await?;+ // Self-heal the speakers + speaker_embeddings tables. The migration+ // (20241108202826) depends on sqlite-vec being loaded for the vec_length+ // CHECK constraint. If an older engine ran that migration without+ // sqlite-vec, sqlx recorded it as applied but the CREATE TABLE failed —+ // subsequent launches skip the "already applied" migration, so the table+ // stays missing and every transcription errors with "no such table:+ // speaker_embeddings".+ Self::ensure_speaker_tables(pool).await?;+Ok(())}@@ -624,6 +633,78 @@ impl DatabaseManager {Ok(())}+ /// Self-heal the `speakers` and `speaker_embeddings` tables. The original+ /// migration (20241108202826) uses `vec_length()` in a CHECK constraint,+ /// which requires the sqlite-vec extension. If the migration was applied by+ /// an engine build where sqlite-vec wasn't loaded (or failed to load), sqlx+ /// recorded the migration version but the DDL silently failed — the tables+ /// don't exist despite the migration being "applied". Re-issuing the+ /// `CREATE TABLE IF NOT EXISTS` statements fixes this for all subsequent+ /// launches now that sqlite-vec is always registered as an auto-extension.+ async fn ensure_speaker_tables(pool: &SqlitePool) -> Result<(), sqlx::Error> {+ let speakers_exists: (i64,) = sqlx::query_as(+ "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'speakers'",+ )+ .fetch_one(pool)+ .await?;++ if speakers_exists.0 == 0 {+ tracing::info!("Self-healing missing speakers table");+ sqlx::query(+ "CREATE TABLE IF NOT EXISTS speakers (+ id INTEGER PRIMARY KEY AUTOINCREMENT,+ name TEXT,+ metadata JSON,+ hallucination BOOLEAN DEFAULT FALSE,+ centroid FLOAT[512],+ embedding_count INTEGER DEFAULT 0+ )",+ )+ .execute(pool)+ .await?;+ // This index was added after the original table migration and would+ // already be recorded as applied on a database missing the table.+ sqlx::query(+ "CREATE INDEX IF NOT EXISTS idx_speakers_hallucination_name \+ ON speakers(hallucination, name COLLATE NOCASE)",+ )+ .execute(pool)+ .await?;+ }++ let embeddings_exists: (i64,) = sqlx::query_as(+ "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'speaker_embeddings'",+ )+ .fetch_one(pool)+ .await?;++ if embeddings_exists.0 == 0 {+ tracing::info!("Self-healing missing speaker_embeddings table");+ sqlx::query(+ "CREATE TABLE IF NOT EXISTS speaker_embeddings (+ id INTEGER PRIMARY KEY AUTOINCREMENT,+ embedding FLOAT[512] NOT NULL+ check(+ typeof(embedding) == 'blob'+ and vec_length(embedding) == 512+ ),+ speaker_id INTEGER REFERENCES speakers(id)+ )",+ )+ .execute(pool)+ .await?;+ // This index was also added by a later migration.+ sqlx::query(+ "CREATE INDEX IF NOT EXISTS idx_speaker_embeddings_speaker_id \+ ON speaker_embeddings(speaker_id)",+ )+ .execute(pool)+ .await?;+ }++ Ok(())+ }+/// Acquire a connection with `BEGIN IMMEDIATE`, serialized via a single-permit semaphore.////// Writers queue in Rust memory (the semaphore) instead of each holding a pool@@ -818,8 +899,80 @@ impl DatabaseManager {#[cfg(test)]mod shutdown_tests {use super::*;+ use std::collections::BTreeSet;use std::sync::atomic::{AtomicUsize, Ordering};+ #[tokio::test]+ async fn startup_rebuilds_missing_speaker_tables_with_current_schema() {+ let dir = tempfile::tempdir().expect("temp dir");+ let db_path = dir.path().join("missing-speakers.sqlite");+ let database = DatabaseManager::new(+ db_path.to_str().expect("utf-8 temp path"),+ DbConfig::for_tier(screenpipe_config::DeviceTier::Low),+ )+ .await+ .expect("database init");++ sqlx::query("DROP TABLE speaker_embeddings")+ .execute(&database.write_pool)+ .await+ .expect("drop embeddings table");+ sqlx::query("DROP TABLE speakers")+ .execute(&database.write_pool)+ .await+ .expect("drop speakers table");++ DatabaseManager::run_migrations(&database.write_pool)+ .await+ .expect("startup self-heal");++ let columns =+ sqlx::query_scalar::<_, String>("SELECT name FROM pragma_table_info('speakers')")+ .fetch_all(&database.pool)+ .await+ .expect("inspect rebuilt speakers schema")+ .into_iter()+ .collect::<BTreeSet<_>>();+ assert_eq!(+ columns,+ [+ "centroid",+ "embedding_count",+ "hallucination",+ "id",+ "metadata",+ "name"+ ]+ .into_iter()+ .map(str::to_owned)+ .collect(),+ "self-heal must restore every column added by later migrations"+ );++ let indexes = sqlx::query_scalar::<_, String>(+ "SELECT name FROM sqlite_master WHERE type = 'index' AND name IN \+ ('idx_speaker_embeddings_speaker_id', 'idx_speakers_hallucination_name')",+ )+ .fetch_all(&database.pool)+ .await+ .expect("inspect rebuilt speaker indexes")+ .into_iter()+ .collect::<BTreeSet<_>>();+ assert_eq!(+ indexes,+ [+ "idx_speaker_embeddings_speaker_id",+ "idx_speakers_hallucination_name",+ ]+ .into_iter()+ .map(str::to_owned)+ .collect(),+ "self-heal must restore indexes from later migrations"+ );++ database.close().await;+ }+#[tokio::test]async fn file_query_pool_enforces_its_platform_write_barrier() {let dir = tempfile::tempdir().expect("temp dir");diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.mdindex c6409d4c9..ba10c0a6d 100644--- a/docs/coverage/CORE.md+++ b/docs/coverage/CORE.md@@ -9,10 +9,10 @@ confidence, and criticality.- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y- Mapped suites: 32- Mapped Rust files: 311-- Active test blocks: 2924+- Active test blocks: 2925- Ignored/manual test blocks: 134-- Declared test blocks: 3058-- Weighted coverage points: 2403.3+- Declared test blocks: 3059+- Weighted coverage points: 2404.3Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.Criticality weights: high=1.0, medium=0.7, low=0.4.@@ -23,16 +23,16 @@ are explicitly enabled in a runtime lane.| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score || --- | --- | --- | --- | --- | --- | --- | --- |-| windows | 29 | 2796 | 129 | 2344.5 | 21 | 11 | 100% |-| macos | 29 | 2847 | 109 | 2354.4 | 22 | 11 | 100% |-| linux | 25 | 2485 | 102 | 2063.4 | 20 | 11 | 100% |+| windows | 29 | 2797 | 129 | 2345.5 | 21 | 11 | 100% |+| macos | 29 | 2848 | 109 | 2355.4 | 22 | 11 | 100% |+| linux | 25 | 2486 | 102 | 2064.4 | 20 | 11 | 100% |## Crate Summary| Crate | Suites | Integration files | Source unit files | Active tests | Ignored tests | Weighted points | Flows || --- | --- | --- | --- | --- | --- | --- | --- || screenpipe-engine | 10 | 18 | 100 | 1404 | 42 | 1078.7 | 10 |-| screenpipe-db | 5 | 50 | 14 | 416 | 16 | 395.7 | 9 |+| screenpipe-db | 5 | 50 | 14 | 417 | 16 | 396.7 | 9 || screenpipe-sqlite-coordinator | 1 | 0 | 2 | 11 | 0 | 11.0 | 2 || screenpipe-audio | 6 | 23 | 47 | 529 | 40 | 458.1 | 5 || screenpipe-screen | 6 | 9 | 18 | 234 | 9 | 213.5 | 4 |@@ -65,21 +65,21 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| audio | 7 suites / 625 active / 41 ignored / 554.1 pts | 7 suites / 625 active / 41 ignored / 554.1 pts | 6 suites / 552 active / 40 ignored / 503.0 pts || audio-device | 2 suites / 198 active / 7 ignored / 176.1 pts | 2 suites / 198 active / 7 ignored / 176.1 pts | 1 suites / 125 active / 6 ignored / 125.0 pts || configuration | 2 suites / 135 active / 3 ignored / 122.8 pts | 2 suites / 135 active / 3 ignored / 122.8 pts | 2 suites / 135 active / 3 ignored / 122.8 pts |-| database | 6 suites / 334 active / 12 ignored / 313.7 pts | 6 suites / 334 active / 12 ignored / 313.7 pts | 6 suites / 334 active / 12 ignored / 313.7 pts |+| database | 6 suites / 335 active / 12 ignored / 314.7 pts | 6 suites / 335 active / 12 ignored / 314.7 pts | 6 suites / 335 active / 12 ignored / 314.7 pts || db-search | 2 suites / 107 active / 9 ignored / 107.0 pts | 2 suites / 107 active / 9 ignored / 107.0 pts | 2 suites / 107 active / 9 ignored / 107.0 pts || engine-lifecycle | 6 suites / 205 active / 1 ignored / 180.0 pts | 6 suites / 205 active / 1 ignored / 180.0 pts | 5 suites / 199 active / 1 ignored / 178.3 pts || local-api | 2 suites / 303 active / 9 ignored / 213.9 pts | 2 suites / 303 active / 9 ignored / 213.9 pts | 2 suites / 303 active / 9 ignored / 213.9 pts || meeting | 6 suites / 1323 active / 16 ignored / 1074.9 pts | 6 suites / 1323 active / 16 ignored / 1074.9 pts | 4 suites / 1047 active / 12 ignored / 820.8 pts || ocr | 4 suites / 120 active / 7 ignored / 114.3 pts | 4 suites / 124 active / 7 ignored / 119.8 pts | 3 suites / 115 active / 6 ignored / 110.8 pts || os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - |-| performance | 13 suites / 1326 active / 67 ignored / 1172.0 pts | 14 suites / 1421 active / 70 ignored / 1210.0 pts | 13 suites / 1326 active / 67 ignored / 1172.0 pts |+| performance | 13 suites / 1327 active / 67 ignored / 1173.0 pts | 14 suites / 1422 active / 70 ignored / 1211.0 pts | 13 suites / 1327 active / 67 ignored / 1173.0 pts || pipes | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts || privacy | 5 suites / 865 active / 36 ignored / 703.3 pts | 5 suites / 912 active / 16 ignored / 707.7 pts | 5 suites / 841 active / 14 ignored / 681.5 pts || real-app | - | 1 suites / 95 active / 3 ignored / 38.0 pts | - || speaker | 2 suites / 293 active / 5 ignored / 293.0 pts | 2 suites / 293 active / 5 ignored / 293.0 pts | 2 suites / 293 active / 5 ignored / 293.0 pts |-| storage | 3 suites / 455 active / 29 ignored / 368.9 pts | 3 suites / 455 active / 29 ignored / 368.9 pts | 3 suites / 455 active / 29 ignored / 368.9 pts |+| storage | 3 suites / 456 active / 29 ignored / 369.9 pts | 3 suites / 456 active / 29 ignored / 369.9 pts | 3 suites / 456 active / 29 ignored / 369.9 pts || sync | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts |-| timeline | 4 suites / 889 active / 33 ignored / 725.8 pts | 4 suites / 889 active / 33 ignored / 725.8 pts | 4 suites / 889 active / 33 ignored / 725.8 pts |+| timeline | 4 suites / 890 active / 33 ignored / 726.8 pts | 4 suites / 890 active / 33 ignored / 726.8 pts | 4 suites / 890 active / 33 ignored / 726.8 pts || transcription | 5 suites / 628 active / 37 ignored / 489.9 pts | 5 suites / 628 active / 37 ignored / 489.9 pts | 5 suites / 628 active / 37 ignored / 489.9 pts || ui-events | 4 suites / 692 active / 28 ignored / 528.3 pts | 3 suites / 644 active / 5 ignored / 494.7 pts | 3 suites / 644 active / 5 ignored / 494.7 pts || vision-capture | 5 suites / 472 active / 32 ignored / 377.4 pts | 5 suites / 476 active / 32 ignored / 382.9 pts | 4 suites / 467 active / 31 ignored / 373.9 pts |@@ -132,7 +132,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-audio-meetings-speakers | screenpipe-db | windows, macos, linux | database, audio, meeting, speaker | audio-record-transcribe, audio-device-health, meeting-live-notes | high | strong | integration | 15 | 96 | 1 | Audio transcript dedupe, live meeting mirroring, end-generation and open-meeting invariants, liveness, and speaker reassignment coverage. || db-runtime-reliability | screenpipe-db | windows, macos, linux | database, performance | performance-liveness | high | partial | mixed | 12 | 27 | 6 | SQLite hard-fault classification, failpoint VFS injection, fresh-identity recovery verification with integrity/FK/write canaries, query cancellation, close-severs-connections regressions, multi-pool WAL parity, runtime version pinning, and WAL chaos plus memory-pressure probes. || db-search-indexing | screenpipe-db | windows, macos, linux | db-search, ocr, accessibility, performance | local-api-search, capture-ocr-pipeline, accessibility-ui-events, performance-liveness | high | strong | mixed | 13 | 101 | 4 | FTS, tokenizer, OCR snapshot search, query planning, ordering, accessibility search, and contention coverage. |-| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 17 | 168 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. |+| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 17 | 169 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. || engine-api-routes | screenpipe-engine | windows, macos, linux | local-api, timeline, meeting, transcription | local-api-search, timeline-streaming, meeting-live-notes, audio-record-transcribe | high | partial | mixed | 29 | 297 | 4 | Route/unit coverage for search, health, streaming, meetings, time/timezone, and transcription. Legacy endpoint/websocket tests require local data and remain ignored. || engine-capture-timeline | screenpipe-engine | windows, macos, linux | vision-capture, timeline, storage, performance | capture-ocr-pipeline, timeline-streaming, performance-liveness | high | partial | mixed | 23 | 247 | 26 | Covers capture trigger logic, frame/audio linking, hot cache, timeline refresh regressions, fragmented MP4 extraction, and HD-mode control. Several real-data tests are intentionally ignored by default. || engine-config-lifecycle | screenpipe-engine | windows, macos, linux | configuration, engine-lifecycle, performance | settings-to-engine-config, engine-health-lifecycle, performance-liveness | high | strong | mixed | 11 | 111 | 1 | Fast logic coverage for the config bridge, tray health debounce, sleep/power policies, and queue backpressure. |@@ -259,7 +259,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su| db-accessibility-ui-events | screenpipe-db | src/db/elements.rs | source | 2 | 0 | 2 || db-timeline-frames | screenpipe-db | src/db/feedback.rs | source | 2 | 0 | 2 || db-timeline-frames | screenpipe-db | src/db/maintenance.rs | source | 4 | 1 | 5 |-| db-timeline-frames | screenpipe-db | src/db/setup.rs | source | 5 | 0 | 5 |+| db-timeline-frames | screenpipe-db | src/db/setup.rs | source | 6 | 0 | 6 || db-timeline-frames | screenpipe-db | src/db/tests.rs | source | 31 | 1 | 32 || db-timeline-frames | screenpipe-db | src/db/truncation_tests.rs | source | 1 | 0 | 1 || db-runtime-reliability | screenpipe-db | src/failpoint_vfs.rs | source | 4 | 0 | 4 |