YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context
Rust agentsagiaiai-memoryaudio-recordingcomputer-visionhermeshermes-agentllmlocal-ailocal-firstmachine-learningmcpmultimodalopenclawprivacyrewindscreen-recordingspeech-to-textycombinator
docs(db): unix-excl does not put the WAL index in process memory (#6291)
Louis Beaumonta8e6e0f0f47c
diff --git a/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs b/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rsindex 14671e071..25d34be65 100644--- a/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs+++ b/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs@@ -5,15 +5,19 @@//! Launch-time self-heal for transient SQLite hard faults.//!//! A `SQLITE_IOERR_SHORT_READ` (extended code 522) under heavy concurrent load-//! desyncs the WAL index in memory. Every pool is torn down and the database is-//! durably quarantined, which is correct while the faulted process is still-//! running: its `-shm` mapping cannot be trusted.+//! desyncs the WAL index against the WAL it describes. Every pool is torn down+//! and the database is durably quarantined, which is correct while the faulted+//! process is still running: its live view of `-shm` cannot be trusted. (That+//! index is file-backed even under `unix-excl` — see the VFS note in+//! `screenpipe-db`'s `db/setup.rs`.)//!-//! The quarantine is *cross-launch*, though, and that part over-fires. A fresh-//! process has a fresh WAL index, so the poisoning is gone before the first-//! read. Three separate incidents (2026-07-02, 2026-08-05, 2026-08-11) all ended-//! the same way: `PRAGMA quick_check` returned `ok` on the quarantined file, and-//! the "recovery" rebuilt a database that was never damaged. In the meantime+//! The quarantine is *cross-launch*, though, and that part over-fires. Once+//! every handle is gone the index no longer outlives the fault: the first+//! connection in a fresh process rebuilds it from the WAL, so the desync is+//! gone before the first read. Three separate incidents (2026-07-02,+//! 2026-08-05, 2026-08-11) all ended the same way: `PRAGMA quick_check`+//! returned `ok` on the quarantined file, and the "recovery" rebuilt a+//! database that was never damaged. In the meantime//! recording stayed off — hours, in the 2026-08-11 case, because nothing resumes//! until a human notices the tray and runs a repair.//!diff --git a/crates/screenpipe-connect/src/remote_sync.rs b/crates/screenpipe-connect/src/remote_sync.rsindex 81a60955f..f7ee37679 100644--- a/crates/screenpipe-connect/src/remote_sync.rs+++ b/crates/screenpipe-connect/src/remote_sync.rs@@ -628,8 +628,8 @@ pub async fn sync_to_remote_with_opts(/// Open a short-lived sqlx pool against the live DB and run `VACUUM INTO`/// to produce a self-contained snapshot at `dest`. SQLite's WAL mode lets/// our connection coexist with the engine's writer pool; on macOS it joins-/// the recorder's unix-excl domain and process-local WAL index. The pool is-/// dropped before we return — important because we then SFTP the snapshot.+/// the recorder's unix-excl locking domain. The pool is dropped before we+/// return — important because we then SFTP the snapshot.async fn snapshot_db(live_db: &Path, dest: &Path) -> Result<()> {let url = format!("sqlite://{}", live_db.to_string_lossy());let mut opts = SqliteConnectOptions::from_str(&url)@@ -639,9 +639,11 @@ async fn snapshot_db(live_db: &Path, dest: &Path) -> Result<()> {{opts = opts.read_only(true);}- // The recorder uses unix-excl on macOS so its WAL index lives in process- // memory instead of an APFS-backed `-shm` mapping. Snapshot connections- // run in that same process and must join the same VFS/locking domain.+ // The recorder uses unix-excl on macOS, which takes one exclusive lock on+ // the database file and makes later OS locking in-process bookkeeping.+ // The `-shm` WAL index is still file-backed. Snapshot connections run in+ // that same process and must join the same VFS/locking domain, or they+ // would do real POSIX locking against handles that no longer lock at all.#[cfg(target_os = "macos")]{opts = opts.vfs("unix-excl");diff --git a/crates/screenpipe-db/src/db/setup.rs b/crates/screenpipe-db/src/db/setup.rsindex d80c64ae0..50944a038 100644--- a/crates/screenpipe-db/src/db/setup.rs+++ b/crates/screenpipe-db/src/db/setup.rs@@ -207,14 +207,27 @@ impl DatabaseManager {.pragma("mmap_size", config.mmap_size.to_string());// macOS delivers an uncatchable SIGBUS when APFS invalidates a page in// SQLite's file-backed WAL index after the `-shm` file was shortened.- // `unix-excl` keeps the WAL index in process memory and takes one OS- // lock for the recorder process, while SQLite still coordinates all- // SQLx connections internally. Screenpipe already requires exclusive- // ownership for live capture, recovery, and maintenance operations.+ // `unix-excl` does NOT move that index into process memory: in the+ // bundled SQLite (3.51.3) `unixOpenSharedMemory` always creates the+ // `-shm` file, and a heap wal-index requires `locking_mode=EXCLUSIVE`,+ // which this database deliberately never sets. What `unix-excl` does+ // is take one real exclusive lock on the database file the first time+ // any lock is attempted and turn every later OS locking operation into+ // in-process bookkeeping (`unixFileLock`). That is what keeps another+ // process from attaching and shortening `-shm` underneath us, while+ // every connection in this process shares the one shm node for the+ // inode. Screenpipe already requires exclusive ownership for live+ // capture, recovery, and maintenance operations.+ //+ // Do not restate this as "the WAL index lives in memory". That claim+ // was wrong, and `db.sqlite-shm` existing on disk is normal, not+ // evidence that some other opener touched the database.#[cfg(target_os = "macos")]if !is_in_memory {connect_options = connect_options.vfs("unix-excl");- info!("macOS capture database using unix-excl VFS with a process-local WAL index");+ info!(+ "macOS capture database using unix-excl VFS with a process-exclusive database lock"+ );}for (pragma, value) in screenpipe_config::WAL_SAFETY_PRAGMAS {connect_options = connect_options.pragma(pragma, value);@@ -247,18 +260,22 @@ impl DatabaseManager {// Every file-backed query connection enables query_only. Non-macOS// builds also open the file with mode=ro as an independent physical- // barrier. macOS keeps all unix-excl handles physically RW so they- // share one process-local WAL index instead of reopening `-shm`.- // In-memory test databases remain writable for fixtures that seed them- // directly.+ // barrier. macOS keeps all unix-excl handles physically RW so every+ // connection shares one shm node under the same exclusive-lock+ // regime, rather than splitting into two locking modes over the same+ // `-shm`. In-memory test databases remain writable for fixtures that+ // seed them directly.let read_connect_options = if is_in_memory {connect_options.clone()} else {let options = connect_options.clone().pragma("query_only", "ON");- // A read-only unix-excl handle falls back to POSIX file locks and a- // file-backed WAL index. Keep macOS query handles physically RW so- // every connection shares the process-local WAL index; query_only- // remains the connection-level write barrier.+ // `unixFileLock` only takes the one-shot exclusive lock when+ // UNIXFILE_EXCL is set and UNIXFILE_RDONLY is not, so opening a+ // handle `mode=ro` drops it out of the exclusive regime and back+ // onto real POSIX locks — against sibling handles that are no+ // longer doing OS-level locking at all. Keep macOS query handles+ // physically RW so every connection stays in one regime;+ // query_only remains the connection-level write barrier.#[cfg(not(target_os = "macos"))]let options = options.read_only(true);options