Skip to content

YC (S26) | Open Computer History | Continuously record your company computer work, map your workflows, help you find work worth automating, and power your agents' context

docs(db): unix-excl does not put the WAL index in process memory (#6291)

Louis Beaumonta8e6e0f0f47c
diff --git a/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs b/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs
index 14671e071..25d34be65 100644
--- a/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs
+++ b/apps/screenpipe-app-tauri/src-tauri/src/db_self_heal.rs
@@ -5,15 +5,19 @@
//! Launch-time self-heal for transient SQLite hard faults.
//!
//! A `SQLITE_IOERR_SHORT_READ` (extended code 522) under heavy concurrent load
-//! desyncs the WAL index in memory. Every pool is torn down and the database is
-//! durably quarantined, which is correct while the faulted process is still
-//! running: its `-shm` mapping cannot be trusted.
+//! desyncs the WAL index against the WAL it describes. Every pool is torn down
+//! and the database is durably quarantined, which is correct while the faulted
+//! process is still running: its live view of `-shm` cannot be trusted. (That
+//! index is file-backed even under `unix-excl` — see the VFS note in
+//! `screenpipe-db`'s `db/setup.rs`.)
//!
-//! The quarantine is *cross-launch*, though, and that part over-fires. A fresh
-//! process has a fresh WAL index, so the poisoning is gone before the first
-//! read. Three separate incidents (2026-07-02, 2026-08-05, 2026-08-11) all ended
-//! the same way: `PRAGMA quick_check` returned `ok` on the quarantined file, and
-//! the "recovery" rebuilt a database that was never damaged. In the meantime
+//! The quarantine is *cross-launch*, though, and that part over-fires. Once
+//! every handle is gone the index no longer outlives the fault: the first
+//! connection in a fresh process rebuilds it from the WAL, so the desync is
+//! gone before the first read. Three separate incidents (2026-07-02,
+//! 2026-08-05, 2026-08-11) all ended the same way: `PRAGMA quick_check`
+//! returned `ok` on the quarantined file, and the "recovery" rebuilt a
+//! database that was never damaged. In the meantime
//! recording stayed off — hours, in the 2026-08-11 case, because nothing resumes
//! until a human notices the tray and runs a repair.
//!
diff --git a/crates/screenpipe-connect/src/remote_sync.rs b/crates/screenpipe-connect/src/remote_sync.rs
index 81a60955f..f7ee37679 100644
--- a/crates/screenpipe-connect/src/remote_sync.rs
+++ b/crates/screenpipe-connect/src/remote_sync.rs
@@ -628,8 +628,8 @@ pub async fn sync_to_remote_with_opts(
/// Open a short-lived sqlx pool against the live DB and run `VACUUM INTO`
/// to produce a self-contained snapshot at `dest`. SQLite's WAL mode lets
/// our connection coexist with the engine's writer pool; on macOS it joins
-/// the recorder's unix-excl domain and process-local WAL index. The pool is
-/// dropped before we return — important because we then SFTP the snapshot.
+/// the recorder's unix-excl locking domain. The pool is dropped before we
+/// return — important because we then SFTP the snapshot.
async fn snapshot_db(live_db: &Path, dest: &Path) -> Result<()> {
let url = format!("sqlite://{}", live_db.to_string_lossy());
let mut opts = SqliteConnectOptions::from_str(&url)
@@ -639,9 +639,11 @@ async fn snapshot_db(live_db: &Path, dest: &Path) -> Result<()> {
{
opts = opts.read_only(true);
}
- // The recorder uses unix-excl on macOS so its WAL index lives in process
- // memory instead of an APFS-backed `-shm` mapping. Snapshot connections
- // run in that same process and must join the same VFS/locking domain.
+ // The recorder uses unix-excl on macOS, which takes one exclusive lock on
+ // the database file and makes later OS locking in-process bookkeeping.
+ // The `-shm` WAL index is still file-backed. Snapshot connections run in
+ // that same process and must join the same VFS/locking domain, or they
+ // would do real POSIX locking against handles that no longer lock at all.
#[cfg(target_os = "macos")]
{
opts = opts.vfs("unix-excl");
diff --git a/crates/screenpipe-db/src/db/setup.rs b/crates/screenpipe-db/src/db/setup.rs
index d80c64ae0..50944a038 100644
--- a/crates/screenpipe-db/src/db/setup.rs
+++ b/crates/screenpipe-db/src/db/setup.rs
@@ -207,14 +207,27 @@ impl DatabaseManager {
.pragma("mmap_size", config.mmap_size.to_string());
// macOS delivers an uncatchable SIGBUS when APFS invalidates a page in
// SQLite's file-backed WAL index after the `-shm` file was shortened.
- // `unix-excl` keeps the WAL index in process memory and takes one OS
- // lock for the recorder process, while SQLite still coordinates all
- // SQLx connections internally. Screenpipe already requires exclusive
- // ownership for live capture, recovery, and maintenance operations.
+ // `unix-excl` does NOT move that index into process memory: in the
+ // bundled SQLite (3.51.3) `unixOpenSharedMemory` always creates the
+ // `-shm` file, and a heap wal-index requires `locking_mode=EXCLUSIVE`,
+ // which this database deliberately never sets. What `unix-excl` does
+ // is take one real exclusive lock on the database file the first time
+ // any lock is attempted and turn every later OS locking operation into
+ // in-process bookkeeping (`unixFileLock`). That is what keeps another
+ // process from attaching and shortening `-shm` underneath us, while
+ // every connection in this process shares the one shm node for the
+ // inode. Screenpipe already requires exclusive ownership for live
+ // capture, recovery, and maintenance operations.
+ //
+ // Do not restate this as "the WAL index lives in memory". That claim
+ // was wrong, and `db.sqlite-shm` existing on disk is normal, not
+ // evidence that some other opener touched the database.
#[cfg(target_os = "macos")]
if !is_in_memory {
connect_options = connect_options.vfs("unix-excl");
- info!("macOS capture database using unix-excl VFS with a process-local WAL index");
+ info!(
+ "macOS capture database using unix-excl VFS with a process-exclusive database lock"
+ );
}
for (pragma, value) in screenpipe_config::WAL_SAFETY_PRAGMAS {
connect_options = connect_options.pragma(pragma, value);
@@ -247,18 +260,22 @@ impl DatabaseManager {
// Every file-backed query connection enables query_only. Non-macOS
// builds also open the file with mode=ro as an independent physical
- // barrier. macOS keeps all unix-excl handles physically RW so they
- // share one process-local WAL index instead of reopening `-shm`.
- // In-memory test databases remain writable for fixtures that seed them
- // directly.
+ // barrier. macOS keeps all unix-excl handles physically RW so every
+ // connection shares one shm node under the same exclusive-lock
+ // regime, rather than splitting into two locking modes over the same
+ // `-shm`. In-memory test databases remain writable for fixtures that
+ // seed them directly.
let read_connect_options = if is_in_memory {
connect_options.clone()
} else {
let options = connect_options.clone().pragma("query_only", "ON");
- // A read-only unix-excl handle falls back to POSIX file locks and a
- // file-backed WAL index. Keep macOS query handles physically RW so
- // every connection shares the process-local WAL index; query_only
- // remains the connection-level write barrier.
+ // `unixFileLock` only takes the one-shot exclusive lock when
+ // UNIXFILE_EXCL is set and UNIXFILE_RDONLY is not, so opening a
+ // handle `mode=ro` drops it out of the exclusive regime and back
+ // onto real POSIX locks — against sibling handles that are no
+ // longer doing OS-level locking at all. Keep macOS query handles
+ // physically RW so every connection stays in one regime;
+ // query_only remains the connection-level write barrier.
#[cfg(not(target_os = "macos"))]
let options = options.read_only(true);
options