Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

fix(db): self-heal missing speaker_embeddings table on startup (#5925)

Ezra Ellette2deb5d772c0a
diff --git a/COVERAGE.md b/COVERAGE.md
index 30f6ce99b..c8c1411ef 100644
--- a/COVERAGE.md
+++ b/COVERAGE.md
@@ -32,15 +32,15 @@ results and `cargo llvm-cov` data on top when judging release confidence.
- Mapped suites: 32
- Mapped Rust files: 311
-- Active test blocks: 2924
+- Active test blocks: 2925
- Ignored/manual test blocks: 134
-- Weighted coverage points: 2403.3
+- Weighted coverage points: 2404.3
| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score |
| --- | --- | --- | --- | --- | --- | --- | --- |
-| windows | 29 | 2796 | 129 | 2344.5 | 21 | 11 | 100% |
-| macos | 29 | 2847 | 109 | 2354.4 | 22 | 11 | 100% |
-| linux | 25 | 2485 | 102 | 2063.4 | 20 | 11 | 100% |
+| windows | 29 | 2797 | 129 | 2345.5 | 21 | 11 | 100% |
+| macos | 29 | 2848 | 109 | 2355.4 | 22 | 11 | 100% |
+| linux | 25 | 2486 | 102 | 2064.4 | 20 | 11 | 100% |
## Refresh
diff --git a/crates/screenpipe-db/src/db/setup.rs b/crates/screenpipe-db/src/db/setup.rs
index 3f5358aca..d80c64ae0 100644
--- a/crates/screenpipe-db/src/db/setup.rs
+++ b/crates/screenpipe-db/src/db/setup.rs
@@ -446,6 +446,15 @@ impl DatabaseManager {
// upgraded across that migration boundary may have skipped it.
Self::ensure_memories_sync_columns(pool).await?;
+ // Self-heal the speakers + speaker_embeddings tables. The migration
+ // (20241108202826) depends on sqlite-vec being loaded for the vec_length
+ // CHECK constraint. If an older engine ran that migration without
+ // sqlite-vec, sqlx recorded it as applied but the CREATE TABLE failed —
+ // subsequent launches skip the "already applied" migration, so the table
+ // stays missing and every transcription errors with "no such table:
+ // speaker_embeddings".
+ Self::ensure_speaker_tables(pool).await?;
+
Ok(())
}
@@ -624,6 +633,78 @@ impl DatabaseManager {
Ok(())
}
+ /// Self-heal the `speakers` and `speaker_embeddings` tables. The original
+ /// migration (20241108202826) uses `vec_length()` in a CHECK constraint,
+ /// which requires the sqlite-vec extension. If the migration was applied by
+ /// an engine build where sqlite-vec wasn't loaded (or failed to load), sqlx
+ /// recorded the migration version but the DDL silently failed — the tables
+ /// don't exist despite the migration being "applied". Re-issuing the
+ /// `CREATE TABLE IF NOT EXISTS` statements fixes this for all subsequent
+ /// launches now that sqlite-vec is always registered as an auto-extension.
+ async fn ensure_speaker_tables(pool: &SqlitePool) -> Result<(), sqlx::Error> {
+ let speakers_exists: (i64,) = sqlx::query_as(
+ "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'speakers'",
+ )
+ .fetch_one(pool)
+ .await?;
+
+ if speakers_exists.0 == 0 {
+ tracing::info!("Self-healing missing speakers table");
+ sqlx::query(
+ "CREATE TABLE IF NOT EXISTS speakers (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ name TEXT,
+ metadata JSON,
+ hallucination BOOLEAN DEFAULT FALSE,
+ centroid FLOAT[512],
+ embedding_count INTEGER DEFAULT 0
+ )",
+ )
+ .execute(pool)
+ .await?;
+ // This index was added after the original table migration and would
+ // already be recorded as applied on a database missing the table.
+ sqlx::query(
+ "CREATE INDEX IF NOT EXISTS idx_speakers_hallucination_name \
+ ON speakers(hallucination, name COLLATE NOCASE)",
+ )
+ .execute(pool)
+ .await?;
+ }
+
+ let embeddings_exists: (i64,) = sqlx::query_as(
+ "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'speaker_embeddings'",
+ )
+ .fetch_one(pool)
+ .await?;
+
+ if embeddings_exists.0 == 0 {
+ tracing::info!("Self-healing missing speaker_embeddings table");
+ sqlx::query(
+ "CREATE TABLE IF NOT EXISTS speaker_embeddings (
+ id INTEGER PRIMARY KEY AUTOINCREMENT,
+ embedding FLOAT[512] NOT NULL
+ check(
+ typeof(embedding) == 'blob'
+ and vec_length(embedding) == 512
+ ),
+ speaker_id INTEGER REFERENCES speakers(id)
+ )",
+ )
+ .execute(pool)
+ .await?;
+ // This index was also added by a later migration.
+ sqlx::query(
+ "CREATE INDEX IF NOT EXISTS idx_speaker_embeddings_speaker_id \
+ ON speaker_embeddings(speaker_id)",
+ )
+ .execute(pool)
+ .await?;
+ }
+
+ Ok(())
+ }
+
/// Acquire a connection with `BEGIN IMMEDIATE`, serialized via a single-permit semaphore.
///
/// Writers queue in Rust memory (the semaphore) instead of each holding a pool
@@ -818,8 +899,80 @@ impl DatabaseManager {
#[cfg(test)]
mod shutdown_tests {
use super::*;
+ use std::collections::BTreeSet;
use std::sync::atomic::{AtomicUsize, Ordering};
+ #[tokio::test]
+ async fn startup_rebuilds_missing_speaker_tables_with_current_schema() {
+ let dir = tempfile::tempdir().expect("temp dir");
+ let db_path = dir.path().join("missing-speakers.sqlite");
+ let database = DatabaseManager::new(
+ db_path.to_str().expect("utf-8 temp path"),
+ DbConfig::for_tier(screenpipe_config::DeviceTier::Low),
+ )
+ .await
+ .expect("database init");
+
+ sqlx::query("DROP TABLE speaker_embeddings")
+ .execute(&database.write_pool)
+ .await
+ .expect("drop embeddings table");
+ sqlx::query("DROP TABLE speakers")
+ .execute(&database.write_pool)
+ .await
+ .expect("drop speakers table");
+
+ DatabaseManager::run_migrations(&database.write_pool)
+ .await
+ .expect("startup self-heal");
+
+ let columns =
+ sqlx::query_scalar::<_, String>("SELECT name FROM pragma_table_info('speakers')")
+ .fetch_all(&database.pool)
+ .await
+ .expect("inspect rebuilt speakers schema")
+ .into_iter()
+ .collect::<BTreeSet<_>>();
+ assert_eq!(
+ columns,
+ [
+ "centroid",
+ "embedding_count",
+ "hallucination",
+ "id",
+ "metadata",
+ "name"
+ ]
+ .into_iter()
+ .map(str::to_owned)
+ .collect(),
+ "self-heal must restore every column added by later migrations"
+ );
+
+ let indexes = sqlx::query_scalar::<_, String>(
+ "SELECT name FROM sqlite_master WHERE type = 'index' AND name IN \
+ ('idx_speaker_embeddings_speaker_id', 'idx_speakers_hallucination_name')",
+ )
+ .fetch_all(&database.pool)
+ .await
+ .expect("inspect rebuilt speaker indexes")
+ .into_iter()
+ .collect::<BTreeSet<_>>();
+ assert_eq!(
+ indexes,
+ [
+ "idx_speaker_embeddings_speaker_id",
+ "idx_speakers_hallucination_name",
+ ]
+ .into_iter()
+ .map(str::to_owned)
+ .collect(),
+ "self-heal must restore indexes from later migrations"
+ );
+
+ database.close().await;
+ }
+
#[tokio::test]
async fn file_query_pool_enforces_its_platform_write_barrier() {
let dir = tempfile::tempdir().expect("temp dir");
diff --git a/docs/coverage/CORE.md b/docs/coverage/CORE.md
index c6409d4c9..ba10c0a6d 100644
--- a/docs/coverage/CORE.md
+++ b/docs/coverage/CORE.md
@@ -9,10 +9,10 @@ confidence, and criticality.
- Tracked crates: screenpipe-engine, screenpipe-db, screenpipe-sqlite-coordinator, screenpipe-audio, screenpipe-screen, screenpipe-a11y
- Mapped suites: 32
- Mapped Rust files: 311
-- Active test blocks: 2924
+- Active test blocks: 2925
- Ignored/manual test blocks: 134
-- Declared test blocks: 3058
-- Weighted coverage points: 2403.3
+- Declared test blocks: 3059
+- Weighted coverage points: 2404.3
Confidence weights: strong=1.0, partial=0.7, conditional=0.4, smoke=0.3.
Criticality weights: high=1.0, medium=0.7, low=0.4.
@@ -23,16 +23,16 @@ are explicitly enabled in a runtime lane.
| Platform | Suites | Active tests | Ignored tests | Weighted points | Layers | Flows | Critical score |
| --- | --- | --- | --- | --- | --- | --- | --- |
-| windows | 29 | 2796 | 129 | 2344.5 | 21 | 11 | 100% |
-| macos | 29 | 2847 | 109 | 2354.4 | 22 | 11 | 100% |
-| linux | 25 | 2485 | 102 | 2063.4 | 20 | 11 | 100% |
+| windows | 29 | 2797 | 129 | 2345.5 | 21 | 11 | 100% |
+| macos | 29 | 2848 | 109 | 2355.4 | 22 | 11 | 100% |
+| linux | 25 | 2486 | 102 | 2064.4 | 20 | 11 | 100% |
## Crate Summary
| Crate | Suites | Integration files | Source unit files | Active tests | Ignored tests | Weighted points | Flows |
| --- | --- | --- | --- | --- | --- | --- | --- |
| screenpipe-engine | 10 | 18 | 100 | 1404 | 42 | 1078.7 | 10 |
-| screenpipe-db | 5 | 50 | 14 | 416 | 16 | 395.7 | 9 |
+| screenpipe-db | 5 | 50 | 14 | 417 | 16 | 396.7 | 9 |
| screenpipe-sqlite-coordinator | 1 | 0 | 2 | 11 | 0 | 11.0 | 2 |
| screenpipe-audio | 6 | 23 | 47 | 529 | 40 | 458.1 | 5 |
| screenpipe-screen | 6 | 9 | 18 | 234 | 9 | 213.5 | 4 |
@@ -65,21 +65,21 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| audio | 7 suites / 625 active / 41 ignored / 554.1 pts | 7 suites / 625 active / 41 ignored / 554.1 pts | 6 suites / 552 active / 40 ignored / 503.0 pts |
| audio-device | 2 suites / 198 active / 7 ignored / 176.1 pts | 2 suites / 198 active / 7 ignored / 176.1 pts | 1 suites / 125 active / 6 ignored / 125.0 pts |
| configuration | 2 suites / 135 active / 3 ignored / 122.8 pts | 2 suites / 135 active / 3 ignored / 122.8 pts | 2 suites / 135 active / 3 ignored / 122.8 pts |
-| database | 6 suites / 334 active / 12 ignored / 313.7 pts | 6 suites / 334 active / 12 ignored / 313.7 pts | 6 suites / 334 active / 12 ignored / 313.7 pts |
+| database | 6 suites / 335 active / 12 ignored / 314.7 pts | 6 suites / 335 active / 12 ignored / 314.7 pts | 6 suites / 335 active / 12 ignored / 314.7 pts |
| db-search | 2 suites / 107 active / 9 ignored / 107.0 pts | 2 suites / 107 active / 9 ignored / 107.0 pts | 2 suites / 107 active / 9 ignored / 107.0 pts |
| engine-lifecycle | 6 suites / 205 active / 1 ignored / 180.0 pts | 6 suites / 205 active / 1 ignored / 180.0 pts | 5 suites / 199 active / 1 ignored / 178.3 pts |
| local-api | 2 suites / 303 active / 9 ignored / 213.9 pts | 2 suites / 303 active / 9 ignored / 213.9 pts | 2 suites / 303 active / 9 ignored / 213.9 pts |
| meeting | 6 suites / 1323 active / 16 ignored / 1074.9 pts | 6 suites / 1323 active / 16 ignored / 1074.9 pts | 4 suites / 1047 active / 12 ignored / 820.8 pts |
| ocr | 4 suites / 120 active / 7 ignored / 114.3 pts | 4 suites / 124 active / 7 ignored / 119.8 pts | 3 suites / 115 active / 6 ignored / 110.8 pts |
| os-integration | 1 suites / 6 active / 0 ignored / 1.7 pts | 1 suites / 6 active / 0 ignored / 1.7 pts | - |
-| performance | 13 suites / 1326 active / 67 ignored / 1172.0 pts | 14 suites / 1421 active / 70 ignored / 1210.0 pts | 13 suites / 1326 active / 67 ignored / 1172.0 pts |
+| performance | 13 suites / 1327 active / 67 ignored / 1173.0 pts | 14 suites / 1422 active / 70 ignored / 1211.0 pts | 13 suites / 1327 active / 67 ignored / 1173.0 pts |
| pipes | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts |
| privacy | 5 suites / 865 active / 36 ignored / 703.3 pts | 5 suites / 912 active / 16 ignored / 707.7 pts | 5 suites / 841 active / 14 ignored / 681.5 pts |
| real-app | - | 1 suites / 95 active / 3 ignored / 38.0 pts | - |
| speaker | 2 suites / 293 active / 5 ignored / 293.0 pts | 2 suites / 293 active / 5 ignored / 293.0 pts | 2 suites / 293 active / 5 ignored / 293.0 pts |
-| storage | 3 suites / 455 active / 29 ignored / 368.9 pts | 3 suites / 455 active / 29 ignored / 368.9 pts | 3 suites / 455 active / 29 ignored / 368.9 pts |
+| storage | 3 suites / 456 active / 29 ignored / 369.9 pts | 3 suites / 456 active / 29 ignored / 369.9 pts | 3 suites / 456 active / 29 ignored / 369.9 pts |
| sync | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts | 1 suites / 457 active / 3 ignored / 319.9 pts |
-| timeline | 4 suites / 889 active / 33 ignored / 725.8 pts | 4 suites / 889 active / 33 ignored / 725.8 pts | 4 suites / 889 active / 33 ignored / 725.8 pts |
+| timeline | 4 suites / 890 active / 33 ignored / 726.8 pts | 4 suites / 890 active / 33 ignored / 726.8 pts | 4 suites / 890 active / 33 ignored / 726.8 pts |
| transcription | 5 suites / 628 active / 37 ignored / 489.9 pts | 5 suites / 628 active / 37 ignored / 489.9 pts | 5 suites / 628 active / 37 ignored / 489.9 pts |
| ui-events | 4 suites / 692 active / 28 ignored / 528.3 pts | 3 suites / 644 active / 5 ignored / 494.7 pts | 3 suites / 644 active / 5 ignored / 494.7 pts |
| vision-capture | 5 suites / 472 active / 32 ignored / 377.4 pts | 5 suites / 476 active / 32 ignored / 382.9 pts | 4 suites / 467 active / 31 ignored / 373.9 pts |
@@ -132,7 +132,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| db-audio-meetings-speakers | screenpipe-db | windows, macos, linux | database, audio, meeting, speaker | audio-record-transcribe, audio-device-health, meeting-live-notes | high | strong | integration | 15 | 96 | 1 | Audio transcript dedupe, live meeting mirroring, end-generation and open-meeting invariants, liveness, and speaker reassignment coverage. |
| db-runtime-reliability | screenpipe-db | windows, macos, linux | database, performance | performance-liveness | high | partial | mixed | 12 | 27 | 6 | SQLite hard-fault classification, failpoint VFS injection, fresh-identity recovery verification with integrity/FK/write canaries, query cancellation, close-severs-connections regressions, multi-pool WAL parity, runtime version pinning, and WAL chaos plus memory-pressure probes. |
| db-search-indexing | screenpipe-db | windows, macos, linux | db-search, ocr, accessibility, performance | local-api-search, capture-ocr-pipeline, accessibility-ui-events, performance-liveness | high | strong | mixed | 13 | 101 | 4 | FTS, tokenizer, OCR snapshot search, query planning, ordering, accessibility search, and contention coverage. |
-| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 17 | 168 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. |
+| db-timeline-frames | screenpipe-db | windows, macos, linux | database, timeline, storage, performance | timeline-streaming, performance-liveness | high | strong | mixed | 17 | 169 | 3 | Frame/audio joins, timeline query shape, suggestions frames, write queue, DB primitives (src/db.rs split into src/db/ modules), feedback record upserts, media eviction anti-join regressions, SAF output registry, semantic storage, and timeline performance. |
| engine-api-routes | screenpipe-engine | windows, macos, linux | local-api, timeline, meeting, transcription | local-api-search, timeline-streaming, meeting-live-notes, audio-record-transcribe | high | partial | mixed | 29 | 297 | 4 | Route/unit coverage for search, health, streaming, meetings, time/timezone, and transcription. Legacy endpoint/websocket tests require local data and remain ignored. |
| engine-capture-timeline | screenpipe-engine | windows, macos, linux | vision-capture, timeline, storage, performance | capture-ocr-pipeline, timeline-streaming, performance-liveness | high | partial | mixed | 23 | 247 | 26 | Covers capture trigger logic, frame/audio linking, hot cache, timeline refresh regressions, fragmented MP4 extraction, and HD-mode control. Several real-data tests are intentionally ignored by default. |
| engine-config-lifecycle | screenpipe-engine | windows, macos, linux | configuration, engine-lifecycle, performance | settings-to-engine-config, engine-health-lifecycle, performance-liveness | high | strong | mixed | 11 | 111 | 1 | Fast logic coverage for the config bridge, tray health debounce, sleep/power policies, and queue backpressure. |
@@ -259,7 +259,7 @@ bun run coverage:core -- --llvm-cov-summary ../../docs/coverage/core-llvm-cov-su
| db-accessibility-ui-events | screenpipe-db | src/db/elements.rs | source | 2 | 0 | 2 |
| db-timeline-frames | screenpipe-db | src/db/feedback.rs | source | 2 | 0 | 2 |
| db-timeline-frames | screenpipe-db | src/db/maintenance.rs | source | 4 | 1 | 5 |
-| db-timeline-frames | screenpipe-db | src/db/setup.rs | source | 5 | 0 | 5 |
+| db-timeline-frames | screenpipe-db | src/db/setup.rs | source | 6 | 0 | 6 |
| db-timeline-frames | screenpipe-db | src/db/tests.rs | source | 31 | 1 | 32 |
| db-timeline-frames | screenpipe-db | src/db/truncation_tests.rs | source | 1 | 0 | 1 |
| db-runtime-reliability | screenpipe-db | src/failpoint_vfs.rs | source | 4 | 0 | 4 |