Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(security): serialize SQLCipher rekeys with cross-process sidecar locks, durable epochs, and fenced engine caches (#6662)

LearningCircuit7c8299b98809
diff --git a/changelog.d/+change-password-conditional-evict.bugfix.md b/changelog.d/+change-password-conditional-evict.bugfix.md
index 22d43f7b97..6b4edd66f8 100644
--- a/changelog.d/+change-password-conditional-evict.bugfix.md
+++ b/changelog.d/+change-password-conditional-evict.bugfix.md
@@ -1,22 +1,17 @@
fix(database): preserve a concurrent new-password engine across change_password
`change_password` previously evicted the cached engine unconditionally
-after the rekey. A concurrent `open_user_database(new_password)` racing
-the rekey legitimately succeeds against the now-new_password-keyed file
-and caches a fresh engine with the new verifier -- the surviving
-current-key engine for the rest of the session. The unconditional
-eviction dropped it and forced the next caller to re-cold-open, and
-broke the invariant "the engine cached for a user is the one a hammer
-racing this rekey observed" that
-`test_stress_change_password_chain_vs_open_hammers` (the chained-rekey
-race test) pins.
-
-The eviction now uses a new `close_user_database_if_stale` helper that
-matches by verifier: it only drops the cached engine when its verifier
-still matches the OLD password being rekeyed away. A concurrent engine
-published under the NEW verifier is left alone, so the next caller sees
-the surviving engine the hammer saw. This guard NARROWS but does not
-CLOSE the stale-engine window on its own (eviction-at-close races when an
-external close empties the cache mid-rekey); the window is closed by the
-rekey-generation check at publish time in `_cache_connection` (see the
-rekey-epoch fragment).
+after the rekey, so a concurrent `open_user_database(new_password)` that
+had just cached the surviving current-key engine lost it and the next
+caller had to cold-open again. `change_password` now detaches any cached
+engine BEFORE the rekey, while holding the user's rekey sidecar lock that
+every cold open also takes, and evicts nothing afterwards: a new-password
+open racing the rekey waits for it to commit, then publishes the only
+engine (a rekey that holds the lock for more than 5 seconds makes that
+open fail as retryable instead). The `/auth/change-password` route no
+longer closes the user's database after a successful change either (its
+"defensive" close
+evicted that same engine), so neither the manager method nor the HTTP
+password change drops it. (#6565 first fixed this with a post-rekey
+`close_user_database_if_stale` helper; the durable rekey epoch in #6662
+supersedes that helper, which is removed.)
diff --git a/changelog.d/+change-password-orphan-dispose.bugfix.md b/changelog.d/+change-password-orphan-dispose.bugfix.md
index 310b2a32c7..789064db19 100644
--- a/changelog.d/+change-password-orphan-dispose.bugfix.md
+++ b/changelog.d/+change-password-orphan-dispose.bugfix.md
@@ -1,31 +1,18 @@
fix(database): dispose overwritten engine on cache publish + degrade on corrupt verifier
-Hardens the post-rekey cache contract that
-`close_user_database_if_stale` enforces, addressing the optional items
-flagged by the post-approval review of PR #6565:
+* When a cache publish overwrites a previously cached engine for the same
+ user, the overwritten engine is disposed exactly once (after a WAL
+ checkpoint, outside the global connection-cache lock) instead of
+ lingering with its SQLCipher pool and file handle until garbage
+ collection. A same-identity republish disposes nothing.
-* `_cache_connection` now disposes the previous cached engine it is
- overwriting, inside the same `_connections_lock` acquisition that
- publishes the new engine + verifier. Without this, an orphan SQLCipher
- pool and file handle lingered until GC on exactly the path the
- conditional-eviction fix made common (concurrent new-password opens
- against a rekeying database), leaking one stale connection per
- overwrite. Same-identity republishes do not dispose (the conditional
- guards against `orphan is engine` to avoid closing a pool a live user
- is mid-query on).
+* A malformed cached password verifier (anything that is not a
+ `(salt, digest)` pair of bytes) now counts as "no match" on both the
+ cache-hit path and the cold-open path, so a login falls through to a
+ key-validating open instead of failing with a `TypeError` /
+ `ValueError`. Well-formed entries are the only ones the manager
+ produces, so a corrupt entry can only come from external mutation.
-* `_verifier_matches` now degrades to no-match (returns `False`) instead
- of raising when `_password_verifiers[username]` is malformed. The
- helper is called from inside `change_password`'s `finally` block; a
- raise there would replace the actual `change_password` outcome with a
- spurious `TypeError` / `ValueError` and the operator would see a
- confusing "password change failed" instead of the underlying cause.
- The invariants that produce well-formed entries
- (`_make_verifier`, `_cache_connection`) are unit-locked, so a corrupt
- entry can only arrive through external dict mutation; failing safe is
- the right discipline.
-
-Pinned by `test_cache_connection_disposes_overwritten_engine`,
-`test_change_password_eviction_window_handles_concurrent_publish_deterministically`,
-and `test_verifier_matches_degrades_on_corrupt_entry` in
+Pinned by `test_cache_connection_disposes_overwritten_engine` and
+`test_verifier_matches_degrades_on_corrupt_entry` in
`tests/security/test_db_connection_change_password_race.py`.
diff --git a/changelog.d/+change-password-rekey-epoch.bugfix.md b/changelog.d/+change-password-rekey-epoch.bugfix.md
index 3b4985f6f7..d9a46d7c68 100644
--- a/changelog.d/+change-password-rekey-epoch.bugfix.md
+++ b/changelog.d/+change-password-rekey-epoch.bugfix.md
@@ -1,29 +1,26 @@
-fix(database): close the stale-publish window with a rekey generation check
+fix(database): close the stale-publish window around a password change
-Adds a per-user rekey epoch bumped once per successful `change_password`
-rekey. `_open_user_database_cold` snapshots the generation before its
-SQLCipher round-trip and `_cache_connection` refuses the publish when the
-generation advanced in between (fail CLOSED with dispose + `None`). This
-closes the window eviction-at-close cannot: an `open_user_database(old)`
-that verified against the still-old-keyed file and parked before publishing
-can no longer plant a stale OLD-key engine into a cache emptied mid-rekey
-by an external unconditional close (logout / idle sweep), after which
-`open_user_database(old)` would have been handed that engine without a
-SQLCipher round-trip. A concurrent `open_user_database(new)` snapshotting after
-the rekey (e.g. waited on init_lock and verified after the rekey landed)
-publishes normally, preserving the surviving-engine invariant. A new-password
-open whose cold-open straddles the rekey (snapshotted before, verified after)
-is also refused -- fail CLOSED for the now-correct password; retry succeeds
-(availability-only, fail-closed direction).
+An `open_user_database(old_password)` that verified against the
+still-old-keyed file can no longer publish its OLD-key engine after the
+rekey lands. Every cold open now holds the user's rekey sidecar lock from
+its epoch read through its publish, so a password change waits for it to
+finish (up to 10 seconds for one in the same process and 1 second for one
+in another process, after which the change fails as contended) and then
+detaches that engine before rekeying, and a cached
+engine is only handed out while its recorded durable rekey epoch is the
+committed one, so even an engine cached under a superseded epoch is
+detached instead of being returned without a SQLCipher round-trip.
+(#6565 first closed this window with an in-process rekey-generation
+counter checked at publish time; the durable, cross-process epoch in
+#6662 supersedes it and the counter is removed.)
-Also widens `_verifier_matches` to degrade on two-tuples with wrong-typed
-members (`("x", "y")`, `(b"salt", 123)`), and fixes the race tests'
-`_engine_usable` helper to probe a real table read (`sqlite_master`)
-instead of `SELECT 1`, which under SQLCipher succeeds on a mis-keyed engine
-and measured pool liveness rather than key validity.
+Also fixes the race tests' `_engine_usable` helper to probe a real table
+read (`sqlite_master`) instead of `SELECT 1`, which under SQLCipher
+succeeds on a mis-keyed engine and measured pool liveness rather than key
+validity.
Pinned by `test_rekey_epoch_refuses_stale_old_publish`,
-`test_verifier_matches_degrades_on_wrong_typed_members`,
-`test_engine_usable_rejects_mis_keyed_engine`, and
-`test_open_cold_refuses_stale_publish_and_disposes` in
+`test_open_cold_refuses_stale_publish_and_disposes`,
+`test_change_password_eviction_window_handles_concurrent_publish_deterministically`,
+and `test_engine_usable_rejects_mis_keyed_engine` in
`tests/security/test_db_connection_change_password_race.py`.
diff --git a/changelog.d/+rekey-quarantine-cleanup.security.md b/changelog.d/+rekey-quarantine-cleanup.security.md
new file mode 100644
index 0000000000..d207ae69f7
--- /dev/null
+++ b/changelog.d/+rekey-quarantine-cleanup.security.md
@@ -0,0 +1 @@
+Include quarantined encrypted backups in password-change cleanup and configured retention, while preserving current-key copies during recovery.
diff --git a/changelog.d/6662.security.md b/changelog.d/6662.security.md
new file mode 100644
index 0000000000..e88591cd27
--- /dev/null
+++ b/changelog.d/6662.security.md
@@ -0,0 +1,179 @@
+Password changes now add defense-in-depth cross-worker serialization and
+durable UUID epochs so stale SQLCipher engine caches are rejected across
+processes. This fencing does not revoke already-checked-out sessions or
+arbitrary writers that bypass the research-start gate.
+
+The database manager's SQLCipher connections now set `busy_timeout` before
+verifying the key, so a correct password is no longer counted as wrong when
+another connection holds a lock briefly; a lock held for longer than the
+busy timeout (`db_config.busy_timeout_ms`, 30 seconds by default) can still
+fail verification. `check_database_integrity` runs
+`cipher_integrity_check` only after a PASSIVE checkpoint has moved every WAL
+frame into the main database file; otherwise `quick_check` is the verdict, so
+while a reader pins the WAL, damage confined to free-list pages may go
+unreported. The integrity scans run after the user's rekey locks are
+released, so the scan itself never holds up that user's other requests, the
+shared log-writer thread (which stores logs for every user) or the
+idle-connection sweep. A close or idle-sweep disposal that runs during a
+scan still holds those locks while its checkpoint waits for the scan's read
+transaction, for at most one second. A password change committed during a
+scan can make that check report a failure. The rekey lock sidecars are refused when they are not a regular
+file (EINVAL), are already unlinked (EMLINK), or are a hard link to another
+file in the same directory, such as the user's own or another user's
+database, WAL or shm file (EMLINK, refused before it is opened, since closing
+a descriptor to a database would drop SQLite's locks on it). This checks
+current directory entries, not an inode's history: removing the other name
+before the check leaves one link and defeats this detection. The data
+directory must remain trusted. A hard-linked one
+whose other names are all outside the directory (a backup or snapshot tool
+that hard-links the data directory) is locked without changing its mode, instead of locking the user
+out. The epoch files (`.rekey.epoch`,
+`.rekey.pending`), which every probe reads while it holds the sidecar lock,
+are opened without following a symlink and without waiting on a FIFO: a
+symlink (ELOOP) or anything that is not a regular file (EINVAL) is an epoch
+read fault, and a file longer than the 36-byte token is a malformed token,
+refused without being read, so a file planted there cannot hang the probes
+or the shutdown, or fill memory.
+
+When a rekey sidecar cannot be opened or locked (for example fd exhaustion,
+EROFS or EACCES), or an epoch file cannot be read once the sidecar is open,
+the cached engine is kept and re-checked on the next successful probe, and
+the periodic connection-pool reset skips that engine for the sweep instead
+of disposing it. The metrics writer and the log writer treat the fault as
+no usable connection rather than raising it. `DatabaseManager.get_session`
+raises it instead of answering no session. Settings read without a session
+of their own (`get_settings_manager` without one) take "no session" to mean
+that the user has no database and answer from the defaults file, so during
+the fault a stored `strict` or `private_only` egress scope read as the
+default `adaptive` (so the journal-data download could go ahead), and a
+custom library storage path read as the default directory (so a library
+sync could remove the documents whose files it did not find there). Those
+reads now fail closed; a route that leaves the error unhandled answers with
+the 503 below. A signed-in request whose engine is kept this way is answered
+with a 503 and `Retry-After`, and its session and stored password are kept
+rather than logged out. So is a signed-in request with no engine cached
+whose reopen of the database fails with such a fault or another rekey
+operational error, instead of reading as a failed credential (a 401, which
+the home page turns into a logout). The fault can also show only after the
+authentication check passed, when the request opens its own database
+session (an fd-exhaustion spike comes and goes): `get_user_db_session` then
+raises `DatabaseSessionUnavailableError`, a `DatabaseSessionError` subclass,
+for any rekey operational error while it opens or claims the database, and
+the home page answers it with that 503 instead of logging the user out. A
+browser that opens the home page during the fault gets that 503 as an error
+page with a button that retries the home page, rather than a raw JSON body.
+The app's exception handler answers with the same 503 when a route leaves
+that error, or a rekey operational error itself, unhandled; a route that
+catches it answers with its own error response, and none ends the session
+over it. A failure to release the rekey sidecar lock after the code it
+protects finished is raised (as `RekeySidecarIOError`) even when the lock
+was taken inside an exception handler; it used to be added as a note to the
+exception being handled and dropped, so a probe made there reported a
+connection the release had failed for. Opening the
+database while such a fault hides a cached engine fails at once with the
+typed error (a login answers 503, not counted toward lockout), without first
+spending a full key derivation (PBKDF2) on a cold open that needs the same
+sidecar. The rekey epoch state is read
+pending-first, so a reader that does not hold the rekey sidecar lock cannot
+report an epoch that a concurrent password-change commit superseded as
+stable. In this release that covers only the classification of a
+password-change failure raised after the lock was released: cache probes,
+logins and password changes read the epoch under the lock, and session
+validation does not read it yet.
+
+These waits for a user's rekey sidecar give up after 5 seconds: the cache
+claim behind every authenticated request (and behind `is_user_connected`,
+`probe_user_connection`, `get_session` and an open of a cached database),
+which counts the per-user lock it queues on toward the same bound, every
+cold open, a login's included, and the epoch initialization at registration.
+Within one HTTP request they share that bound: an authenticated request
+claims the sidecar up to four times before its route runs (checking for an
+open database and reopening it, then the authentication check's two probes),
+and after one of a request's waits gives up, each later one tries once
+without waiting, until one of them gets the sidecar again. So a request
+waits out the bound at most once while the sidecar stays held, rather than
+once per claim (about 20 seconds before a 503 that says `Retry-After: 5`). A
+background research or indexing job that copied the request's context stops
+sharing its wait budget when that request ends, including on an exception;
+each later background claim gets its own bounded wait. A
+holder can keep the sidecar far longer: another worker's password change of
+a large database, a stopped process, a hung mount. Past the bound the wait
+fails as `RekeyLockContentionError`, one more of the faults above: a
+signed-in request is answered with the 503 and `Retry-After`, its session
+kept, whether its engine was cached or its reopen of the database is what
+gave up; a login with a 503 that is not counted toward lockout; and a
+registration fails and removes its partial database. The shared log-writer
+thread runs outside any request: it waits up to the bound for each of its
+two claims of an entry and drops the entry when one gives up, so an entry
+for that user waits at most two bounds for the sidecar rather than as long
+as the holder keeps it. The entries queued behind it, every other user's
+included, wait meanwhile, and entries that arrive while the queue is full
+are dropped.
+
+A password change proceeds while another connection holds a read
+transaction open: the pre-rekey checkpoint is PASSIVE and best-effort, and
+the post-rekey checkpoint, like the checkpoint before an engine disposal,
+waits at most one second (or the busy timeout, if lower) for that reader,
+because the user's sidecar lock is held meanwhile. When the post-rekey checkpoint cannot
+complete, the change still succeeds and the live database already rejects
+the old password, but the re-encrypted pages are only in the WAL: until a
+later checkpoint completes (the last connection to the file closing, an
+engine disposal, or automatic checkpoints after later writes, once the
+reader has finished), the main database file on its own, and any WAL frames
+written before the rekey, are still encrypted under the old password. The
+rekey's durability comes from SQLite itself (the rekey commits under
+`synchronous=FULL` and a complete checkpoint syncs the main file): the
+manager never opens its own descriptor on the live database or its WAL,
+because closing one would release every POSIX lock the process holds on
+that file behind SQLite's back. A filesystem that cannot fsync a directory
+(EINVAL/ENOTSUP) no longer fails every epoch publication, and an I/O failure
+while publishing the first epoch at login is reported as a sidecar error
+(503, not counted toward lockout) instead of a server error.
+
+SQLCipher's `PRAGMA rekey` reports success without changing the key when
+another connection holds a write transaction for longer than the busy
+timeout. The new password is always proven by reopening the database with
+it before the new epoch is committed; when it does not open and the old
+password still does, the password change now fails as retryable (503) with
+the old password still valid and no pending state left behind, instead of
+an indeterminate outcome that only a login with the old password could
+repair. If the epoch cannot be read right after a login has proved the
+password, or it then shows a pending or different committed epoch, the
+login answers 503 (not counted toward lockout) instead of reporting an
+invalid password.
+
+A password change whose new password has been proven (it decrypts the
+database) is reported as completed even when the new epoch cannot be
+committed (disk full, an I/O error); the pending epoch stays on disk, keeps
+cached connections fenced, and the next login with the new password commits
+it. When the outcome is indeterminate (the new password may be active), the
+change-password page no longer answers like a retryable failure: every
+session of the user is ended on every device, both stored copies of the
+password, the scheduler's credential and live sockets are dropped, and the
+user is told to try the new password first. The login that resolves the
+pending change deletes only backups proven to use a superseded key,
+including verification that their salt matches the live database's; a
+backup whose salt cannot be verified is kept with an unknown key.
+Cleanup reacquires the rekey lock, requires the login's recovered epoch to
+still be current, and holds that lock through backup checks and deletion,
+so it cannot delete backups made under a later password change. Which
+password is active is decided by whether it decrypts the database's schema
+page; `quick_check` and `cipher_integrity_check` findings (a database that
+was already damaged) are logged as warnings instead of leaving the pending
+state in place, which had answered every login with 503 for good. A cold
+open waits for the user's per-user open lock no longer than for the rekey
+sidecar (5 seconds), so concurrent opens during a long password change fail
+together with a retryable 503 instead of one after another on the shared
+worker threads. Registering a user creates a fresh epoch instead of reusing
+one left beside an earlier database at the same path. See
+`docs/security/rekey-recovery.md` for the files involved, the log lines and
+manual recovery.
+
+A correct password whose verification read met a database locked or busy
+past the busy timeout (or an I/O error) was reported as a wrong password:
+the login counted a lockout strike and a password change said "Current
+password is incorrect". Such a failure now raises
+`DatabaseKeyCheckUnavailableError`, a rekey operational error: a login
+answers 503 without a strike, a signed-in reopen keeps the session, and a
+password change fails as retryable. Only SQLite's wrong-key answer ("file is
+not a database") still counts as a wrong password.
diff --git a/docs/deployment/upgrading.md b/docs/deployment/upgrading.md
index 453e67887c..422a5ffe44 100644
--- a/docs/deployment/upgrading.md
+++ b/docs/deployment/upgrading.md
@@ -298,6 +298,19 @@ Backup file names start with their UTC creation time
not a pre-migration backup, and one created at a later sign-in on the new
release is a copy of the upgraded database.
+### Rekey lock files and "pending maintenance"
+
+The first sign-in of each user on this release creates
+`ldr_user_<hash>.db.rekey.lock` and `ldr_user_<hash>.db.rekey.epoch` next to
+the user's database in `encrypted_databases/`; a password change also uses
+`ldr_user_<hash>.db.rekey.pending` while it runs. Include them when you copy
+a user's database files, and keep the server user able to create and write
+files in that directory. A login answered with "Your account database is
+pending maintenance" (HTTP 503) means one of these files could not be read,
+written or locked. The
+[rekey recovery guide](../security/rekey-recovery.md) explains the log lines
+and how to recover with the server stopped.
+
## Rollback
Never start v1.10.7 against the upgraded data directory. v1.10.7 knows
diff --git a/docs/security/rekey-recovery.md b/docs/security/rekey-recovery.md
new file mode 100644
index 0000000000..49624a07b5
--- /dev/null
+++ b/docs/security/rekey-recovery.md
@@ -0,0 +1,170 @@
+# Password-change (rekey) recovery
+
+A password change re-encrypts the user's SQLCipher database under the new
+password (`PRAGMA rekey`). Local Deep Research stores no password hash: a
+login works only if the password decrypts the user's database. To make the
+change safe across crashes and across several server processes, the rekey is
+fenced by a few small files kept next to each user database. This page
+explains those files, the "pending maintenance" message, and how to recover
+by hand when the automatic recovery cannot finish.
+
+## The files
+
+User databases live in `{data_directory}/encrypted_databases/`. For a user
+whose username hashes to `{user_hash}` (the first 16 hex characters of the
+SHA-256 of the username), the files are:
+
+| File | What it is | Safe to delete with the server stopped? |
+|------|------------|-------------------------------------------|
+| `ldr_user_{user_hash}.db` | The encrypted database | **Never** |
+| `ldr_user_{user_hash}.db-wal` | SQLite write-ahead log; can hold committed pages, including the re-encrypted pages of a recent rekey | **Never** |
+| `ldr_user_{user_hash}.db-shm` | SQLite shared-memory index for the WAL | Leave it; SQLite rebuilds it |
+| `ldr_user_{user_hash}.db.salt` | Per-database key-derivation salt | **Never**: without it no password opens the database |
+| `ldr_user_{user_hash}.db.rekey.lock` | Lock file that serializes password changes, logins and cache checks across processes | Not needed. Remove it only if it is not a regular file (a symlink, FIFO or directory); it is recreated |
+| `ldr_user_{user_hash}.db.rekey.epoch` | The committed epoch: a random UUID that changes on every password change and lets every process reject a connection it cached under an earlier password | Yes |
+| `ldr_user_{user_hash}.db.rekey.pending` | Present only while a password change is in progress or its outcome has not been resolved yet | Yes |
+
+The epoch files hold only a UUID: no key material and no user data. The
+epoch is a cache token, not a credential. SQLCipher still verifies the key
+on every open, so deleting the epoch files cannot let a wrong password in
+and cannot change which password opens the database. When `.rekey.epoch` is
+missing, the next login creates a new one. With the server running,
+though, `.rekey.pending` is what keeps every process from trusting a
+connection it cached under the old password: deleting it then can let such
+a connection be used again, which is why the steps below stop the server
+first.
+
+A hard-linked `.rekey.lock` (for example from a backup or snapshot tool that
+hard-links the data directory) is used as is; its permissions are not
+changed. While another directory entry still names the same inode, a
+`.rekey.lock` that is a hard link to another file in the same directory
+(the user's own or another user's `.db`, `-wal` or `-shm` file, or anything
+else there) is refused without being opened (`EMLINK`): closing a descriptor
+to a database would drop SQLite's locks on it. Links whose other names are
+outside the directory are accepted. These checks inspect current names,
+not the inode's history: removing the other name before the check leaves
+one link and defeats this detection. The data directory must remain
+trusted; its writers can already replace or remove database files.
+
+## How a password change resolves itself
+
+1. A new `.rekey.pending` UUID is written. From now on, no process trusts a
+ connection it cached for this user.
+2. The database is re-encrypted under the new password and reopened with it.
+ The new password is proven when it decrypts the database's schema page:
+ SQLCipher authenticates every page, so one database state never opens
+ under both passwords.
+3. The pending UUID is committed to `.rekey.epoch` and `.rekey.pending` is
+ removed.
+
+If step 3 cannot finish (disk full, an I/O error) after the new password was
+proven, the change is reported as **completed**. `.rekey.pending` stays on
+disk and the next login with the new password commits it. The old password
+is refused from that point on.
+
+If the server cannot tell which password is active (it crashed or failed
+mid-rekey), the change is reported as possibly completed: every session of
+the user is ended on every device, and the user sees "Your password change
+may have completed. Try your NEW password first; if it does not work, use
+your old one." The first login with whichever password opens the database
+resolves `.rekey.pending`. A wrong password at that point is an ordinary
+failed login and counts toward the account lockout.
+
+When such a login resolves a pending change, it checks each of the user's
+backups with the now-active password (the old password is not known at that
+point). This cleanup holds the rekey lock through backup checks and deletion.
+If another password change has already advanced the database's epoch, the
+earlier login skips cleanup instead of treating newer backups as obsolete:
+
+- A backup whose schema page the active password decrypts is kept.
+- A backup in which no page at all authenticates under the active key, and
+ whose salt is the live database's own (a password change never changes
+ the salt), is under the superseded password and is deleted. A backup whose
+ `.salt` is missing or differs from the live database's is kept: it may
+ only need its salt restored.
+- A backup in which the schema page fails but other pages authenticate is
+ under the active key with a damaged first page. It is renamed to
+ `ldr_backup_<...>.db.damaged-<UTC timestamp>` (its salt to the same name
+ plus `.salt`), mode `0600`, in the same backup directory, and the log says
+ `Backup <name> is under the current key but its first page is damaged;
+ moved aside as <new name>`. Such files are excluded from restore choices,
+ daily verification and the ordinary backup list. They still participate
+ in configured age/count retention and later password-change cleanup.
+ Usable restore candidates take priority over damaged copies within the
+ count limit. A later recovery checks the quarantined copy again under its
+ active key: current-key damage is kept without another rename; a proven
+ superseded key is removed. An unknown key or missing/mismatched salt is
+ kept during recovery. Salvage a damaged copy before retention or a later
+ completed password change removes it.
+- A backup that cannot be checked (locked, an I/O error) is kept.
+
+A completed password change deletes all old backups and makes a fresh one
+directly.
+
+After an integrity finding (see below) the password change still completes,
+or the old password stays active if the re-encryption could not be applied
+(for example because of a damaged page).
+
+## "Your account database is pending maintenance"
+
+A login shows this message (HTTP 503, not counted toward the lockout) when
+the password could not be checked against the database because the rekey
+state needs attention. A signed-in request in the same situation gets a 503
+with `Retry-After` and keeps its session. The cause is logged as:
+
+```
+Login refused for <user>: rekey state requires operator attention (<ErrorType>)
+Login refused for <user>: rekey state requires operator attention (RekeySidecarIOError) (sidecar <operation> failed, errno=<n>)
+```
+
+| `<ErrorType>` | Meaning | Usually resolves itself? |
+|---------------|---------|--------------------------|
+| `RekeyLockContentionError` | Another holder kept the user's `.rekey.lock` for more than 5 seconds: a password change of a large database, another worker's cold open, a stopped process or a hung mount | Yes, when the holder finishes |
+| `RekeySidecarIOError` | `.rekey.lock` could not be opened or locked, or an epoch file could not be written (`operation` is `open`, `acquire`, `unlock`, `close` or `publish`; check the errno: `ENOSPC`, `EROFS`, `EACCES`, `EMFILE`, `ELOOP` for a symlink, `EINVAL` for a non-regular file, `EMLINK` for an already-unlinked lock file or one hard-linked to another file in the same directory) | Once the cause is fixed |
+| `RekeyEpochUnreadableError` | An epoch file exists but cannot be read (I/O error, permissions, a symlink or non-regular file planted at its name) | Once the cause is fixed |
+| `DatabaseKeyCheckUnavailableError` | The database could not be opened to check the password: SQLite reported it locked or busy past the busy timeout (`db_config.busy_timeout_ms`), an I/O error, or a file it cannot open. The password was not checked, so no lockout strike is counted | Yes, once the lock holder finishes or the I/O problem is fixed |
+| `RekeyRecoveryRequiredError` | The epoch files do not vouch for a stable state: a malformed token, `.rekey.pending` without `.rekey.epoch`, or the password opened the database but committing the pending epoch failed (often disk full) | No, if the files are malformed: see below |
+
+## Related log lines
+
+| Log line (abridged) | Meaning |
+|---------------------|---------|
+| `Rekey epoch publication failed for <user> after the new key was verified (...)` | The password change completed; the next login with the new password commits the pending epoch |
+| `Password change outcome indeterminate for <user>: revoking all sessions; the next login resolves the pending rekey` | The server could not prove which password is active |
+| `Password change failed for <user>: rekey outcome unavailable` | Nothing was changed; the old password is still valid |
+| `PRAGMA rekey did not change the key of <db> (...); the old password remains active` | Another connection held a write transaction past the busy timeout, or a damaged page stopped the re-encryption |
+| `PRAGMA quick_check reported <n> problem(s) for <db> after the active key was proven (first: ...)` / `PRAGMA cipher_integrity_check reported ...` | The active password is known, but the database file has damage that predates or survived the rekey. Run the integrity check (`/auth/integrity-check`) and consider restoring from a backup |
+| `Post-rekey WAL checkpoint for <db> did not complete ...` / `... failed: <error>` | The re-encrypted pages are still only in the `-wal` file; the main `.db` file alone opens with the old password until a later checkpoint |
+| `Pending rekey cleanup failed for <user>: <error>` | The epoch was committed but `.rekey.pending` could not be removed; the next login removes it |
+| `Purged <n> backup(s) under the superseded key for <user> after pending rekey recovery` | Old-password backups were deleted after a pending change was resolved |
+| `Backup <name> is under the current key but its first page is damaged; moved aside as <new name>` | A damaged current-key backup was renamed out of the backup set (see above) |
+
+## Manual recovery
+
+Use this only when the 503 persists after the cause (disk space,
+permissions, a stuck process) has been fixed, or when the epoch files are
+malformed.
+
+1. **Stop the server**, every worker and process that uses the data
+ directory. Check that nothing still holds the lock, for example with
+ `fuser ldr_user_{user_hash}.db.rekey.lock` or `lsof`.
+2. **Back up every file of that user first**: copy all
+ `ldr_user_{user_hash}.db*` files (database, `-wal`, `-shm`, `.salt` and
+ the `.rekey.*` files) together, outside the data directory.
+3. Fix the underlying cause: free disk space, make the files owned by the
+ account the server runs as with mode `0600`, and remove anything planted
+ at an epoch or lock file name that is not a regular file.
+4. Delete `ldr_user_{user_hash}.db.rekey.pending` and
+ `ldr_user_{user_hash}.db.rekey.epoch`. Delete both: `.rekey.pending`
+ without `.rekey.epoch` is itself a state that needs recovery. Do **not**
+ delete the `.db`, `-wal` or `.salt` files.
+5. Start the server. The user logs in; after a password change whose outcome
+ was unclear, they try the new password first and then the old one. Only
+ the active password opens the database, and the login creates a fresh
+ `.rekey.epoch`.
+6. Because the pending state was removed by hand, old-password backups are
+ not purged automatically. If the new password is the active one, delete
+ that user's backups under
+ `{data_directory}/encrypted_databases/backups/{user_hash}/` (each `.db`
+ with its `.salt`) that were made before the password change; see the
+ [database backup guide](database-backup.md).
diff --git a/src/local_deep_research/database/backup/backup_service.py b/src/local_deep_research/database/backup/backup_service.py
index 675b05b0ef..61e0467a4d 100644
--- a/src/local_deep_research/database/backup/backup_service.py
+++ b/src/local_deep_research/database/backup/backup_service.py
@@ -4,7 +4,9 @@ Uses sqlcipher_export() for safe atomic backups that preserve encryption
and work correctly with WAL mode.
"""
+import hmac
import os
+import re
import shutil
import tempfile
import threading
@@ -73,6 +75,13 @@ _backup_verifications: dict[str, dict[Path, tuple[_BackupIdentity, bool]]] = {}
_UNSAFE_BACKUP_PATH_CHARS = frozenset('"\\\0\n\r\t')
+# Only names produced by _move_damaged_backup_aside belong to the managed
+# quarantine. Salt files and unrelated operator files must not match.
+_QUARANTINED_BACKUP_NAME_RE = re.compile(
+ r"ldr_backup_.+\.db\.damaged-[0-9]{8}_[0-9]{6}_[0-9]{6}"
+)
+
+
def _validate_backup_path(path: Path) -> str:
"""Normalize host separators and validate the path used in ATTACH.
@@ -899,10 +908,20 @@ class BackupService:
# Interrupted publication can leave a salt without a database.
# Keep recent files alone; they may belong to an active export.
- for pattern in ("ldr_backup_*.db.salt", "ldr_backup_*.db.tmp.salt"):
+ for pattern in (
+ "ldr_backup_*.db.salt",
+ "ldr_backup_*.db.tmp.salt",
+ "ldr_backup_*.db.damaged-*.salt",
+ ):
for salt_file in self.backup_dir.glob(pattern):
if not is_safe_glob_result(salt_file, self.backup_dir):
continue
+ if pattern == "ldr_backup_*.db.damaged-*.salt" and not (
+ _QUARANTINED_BACKUP_NAME_RE.fullmatch(
+ salt_file.with_suffix("").name
+ )
+ ):
+ continue
try:
if (
not salt_file.with_suffix("").exists()
@@ -933,6 +952,16 @@ class BackupService:
if p.exists() and is_safe_glob_result(p, self.backup_dir)
]
+ # Damaged copies share the retention budget, but must never
+ # displace a usable restore candidate just because they are newer.
+ backups.extend(
+ sorted(
+ self._quarantined_backup_paths(),
+ key=_safe_mtime,
+ reverse=True,
+ )
+ )
+
# The backup to keep goes first, so it fills the first slot
# of the count limit even if another file has a later mtime.
if keep is not None:
@@ -1025,6 +1054,27 @@ class BackupService:
return backups
+ def _quarantined_backup_paths(self) -> list[Path]:
+ """Return contained damaged copies for key cleanup and retention.
+
+ These files remain encrypted, but are not usable restore candidates
+ and must stay out of list_backups(), get_latest_backup() and daily
+ backup verification. They still belong to the password lifecycle.
+ """
+ return [
+ path
+ for path in self.backup_dir.glob("ldr_backup_*.db.damaged-*")
+ if _QUARANTINED_BACKUP_NAME_RE.fullmatch(path.name)
+ and is_safe_glob_result(path, self.backup_dir)
+ and path.is_file()
+ ]
+
+ def _managed_backup_paths(self) -> list[Path]:
+ """Include quarantined copies without exposing them for restore."""
+ return [
+ self.backup_dir / info["path"] for info in self.list_backups()
+ ] + self._quarantined_backup_paths()
+
def purge_and_refresh(self) -> "BackupResult":
"""Delete all existing backups and create a fresh one.
@@ -1046,13 +1096,13 @@ class BackupService:
_get_backup_verifications(self.username).clear()
# Delete all existing backup files
- for info in self.list_backups():
+ for backup_path in self._managed_backup_paths():
try:
- self._remove_backup_files(self.backup_dir / info["path"])
- logger.debug(f"Purged old-key backup: {info['filename']}")
+ self._remove_backup_files(backup_path)
+ logger.debug(f"Purged old-key backup: {backup_path.name}")
except OSError:
logger.warning(
- f"Could not delete backup {info['filename']}"
+ f"Could not delete backup {backup_path.name}"
)
# Also clean up any stale .tmp files
@@ -1071,6 +1121,218 @@ class BackupService:
# Create fresh backup with current password (lock already held)
return self._create_backup_impl()
+ def _open_backup_with_current_key(self, backup_path: Path):
+ """Return a raw connection to ``backup_path`` keyed with this password."""
+ from ..sqlcipher_compat import connect_sqlcipher, get_sqlcipher_module
+
+ conn = connect_sqlcipher(get_sqlcipher_module(), str(backup_path))
+ try:
+ cursor = conn.cursor()
+ try:
+ set_sqlcipher_key(cursor, self.password, db_path=backup_path)
+ apply_sqlcipher_pragmas(cursor, creation_mode=False)
+ finally:
+ safe_close(cursor, "backup cursor")
+ except Exception:
+ safe_close(conn, "backup connection")
+ raise
+ return conn
+
+ def _classify_backup_key(self, backup_path: Path) -> str:
+ """Say whether a backup is under the current password's key.
+
+ Returns one of:
+
+ ``"current"``
+ The current password decrypts its schema page. Kept.
+ ``"superseded"``
+ SQLite answers "file is not a database" and SQLCipher's page
+ authentication fails for EVERY page of the file: no page was
+ written under this key, so the backup is under another key,
+ which for this user's own backup directory is a superseded
+ password. Deleted.
+ ``"damaged"``
+ The schema page fails but other pages authenticate under the
+ current key: a backup under the CURRENT key whose first page
+ is damaged. Kept for salvage, subject to retention and later
+ password changes like the other encrypted backup copies.
+ ``"unknown"``
+ Anything else (a lock, an I/O error, an unexpected size or
+ scan result). Kept; nothing proves it superseded.
+
+ The old password is not known where this runs (a login resolving
+ a pending change has only the password that opened the database),
+ so "under another key" is proven from the current key's side.
+ Only SQLite's fixed messages are inspected; they never carry key
+ material.
+ """
+ try:
+ conn = self._open_backup_with_current_key(backup_path)
+ except Exception:
+ return "unknown"
+ try:
+ try:
+ conn.execute("SELECT count(*) FROM sqlite_master").fetchone()
+ except Exception as error:
+ if "not a database" not in str(error):
+ return "unknown"
+ else:
+ return "current"
+ finally:
+ safe_close(conn, "backup connection")
+
+ # The verdicts below are about the KEY, and the key is derived from
+ # the password AND the backup's salt. A rekey never changes a
+ # database's salt (only creation writes it), so a backup under a
+ # superseded password shares the live database's salt. A backup
+ # whose salt differs (missing, damaged, from another database)
+ # failed for a reason restoring the salt may fix: keep it.
+ if not self._backup_salt_matches_live_database(backup_path):
+ return "unknown"
+
+ page_size = get_sqlcipher_settings()["page_size"]
+ try:
+ size = backup_path.stat().st_size
+ except OSError:
+ return "unknown"
+ if size == 0 or size % page_size:
+ return "unknown"
+ page_count = size // page_size
+ try:
+ conn = self._open_backup_with_current_key(backup_path)
+ except Exception:
+ return "unknown"
+ try:
+ rows = conn.execute("PRAGMA cipher_integrity_check").fetchall()
+ except Exception:
+ return "unknown"
+ finally:
+ safe_close(conn, "backup connection")
+ failed_pages = set()
+ prefix = "HMAC verification failed for page "
+ for row in rows:
+ text = str(row[0]) if row else ""
+ if not text.startswith(prefix):
+ return "unknown"
+ try:
+ failed_pages.add(int(text[len(prefix) :]))
+ except ValueError:
+ return "unknown"
+ if 1 not in failed_pages:
+ return "unknown"
+ if failed_pages >= set(range(1, page_count + 1)):
+ return "superseded"
+ return "damaged"
+
+ def _backup_salt_matches_live_database(self, backup_path: Path) -> bool:
+ """True when the backup derives its key from the live database's salt.
+
+ Both sides are read the way key derivation reads them
+ (``get_salt_for_database``: the legacy salt when no ``.salt`` file
+ exists, Windows CR/LF damage undone). A backup without a salt file
+ while the live database has one, a salt that cannot be read, or one
+ with different bytes does not match.
+ """
+ try:
+ live_has_salt = has_per_database_salt(self.db_path)
+ if live_has_salt and not get_salt_file_path(backup_path).exists():
+ return False
+ live_salt = get_salt_for_database(self.db_path)
+ backup_salt = get_salt_for_database(backup_path)
+ except Exception:
+ return False
+ return hmac.compare_digest(live_salt, backup_salt)
+
+ def _move_damaged_backup_aside(self, backup_path: Path) -> Optional[Path]:
+ """Rename a damaged current-key backup out of restore candidates.
+
+ The new name ends in ``.damaged-<UTC timestamp>`` (its salt keeps
+ the ``.salt`` suffix after that). Internal key cleanup and retention
+ still include this copy; ordinary listing and restore selection do
+ not. Mode 0600. Returns the new path, or None when the move failed.
+ """
+ stamp = datetime.now(UTC).strftime("%Y%m%d_%H%M%S_%f")
+ target = backup_path.with_name(f"{backup_path.name}.damaged-{stamp}")
+ salt = get_salt_file_path(backup_path)
+ try:
+ if target.exists():
+ return None
+ os.replace(backup_path, target)
+ os.chmod(target, 0o600)
+ if salt.exists() and not salt.is_symlink():
+ salt_target = target.with_name(f"{target.name}.salt")
+ os.replace(salt, salt_target)
+ os.chmod(salt_target, 0o600)
+ except OSError:
+ return None
+ return target
+
+ def purge_backups_under_superseded_key(self) -> int:
+ """Delete backups that are under a superseded password's key.
+
+ Used after a pending password change is resolved on login: if the
+ rekey took effect, every earlier backup is still encrypted under
+ the superseded (possibly compromised) password and is deleted, as
+ ``purge_and_refresh`` does after a completed change. If the old
+ key survived, every backup still opens and nothing is deleted.
+ A backup proven to be under the current key but with a damaged
+ first page is moved aside instead (see ``_classify_backup_key``);
+ one whose key cannot be determined is kept. A backup without the
+ salt its database needs is left to the regular verification,
+ which already rejects it.
+
+ Application callers must hold this database's rekey sidecar lock
+ and establish that this password still belongs to its current
+ stable epoch, through the entire purge. The per-user backup lock
+ below does not exclude a password change.
+
+ Returns:
+ The number of backups deleted.
+ """
+ deleted = 0
+ with _get_user_lock(self.username):
+ _get_backup_verifications(self.username).clear()
+ source_has_salt = has_per_database_salt(self.db_path)
+ for backup_path in self._managed_backup_paths():
+ backup_salt = get_salt_file_path(backup_path)
+ if (
+ not backup_path.is_file()
+ or backup_path.is_symlink()
+ or backup_salt.is_symlink()
+ or (source_has_salt and not backup_salt.exists())
+ ):
+ continue
+ verdict = self._classify_backup_key(backup_path)
+ if verdict == "damaged":
+ if _QUARANTINED_BACKUP_NAME_RE.fullmatch(backup_path.name):
+ continue
+ moved = self._move_damaged_backup_aside(backup_path)
+ if moved is None:
+ logger.warning(
+ f"Backup {backup_path.name} is under the current "
+ "key but damaged, and could not be moved aside"
+ )
+ else:
+ logger.warning(
+ f"Backup {backup_path.name} is under the current "
+ f"key but its first page is damaged; moved aside "
+ f"as {moved.name}"
+ )
+ continue
+ if verdict != "superseded":
+ continue
+ try:
+ self._remove_backup_files(backup_path)
+ deleted += 1
+ logger.info(
+ f"Purged backup under superseded key: {backup_path.name}"
+ )
+ except OSError:
+ logger.warning(
+ f"Could not delete backup {backup_path.name}"
+ )
+ return deleted
+
def get_latest_backup(self) -> Optional[Path]:
"""Get the path to the most recent backup.
diff --git a/src/local_deep_research/database/encrypted_db.py b/src/local_deep_research/database/encrypted_db.py
index 438ab2e106..a28fbeca37 100644
--- a/src/local_deep_research/database/encrypted_db.py
+++ b/src/local_deep_research/database/encrypted_db.py
@@ -10,8 +10,13 @@ import os
import stat
import threading
import time
+from collections.abc import Generator
+from contextlib import contextmanager
+from contextvars import ContextVar
+from dataclasses import dataclass
+from enum import StrEnum, unique
from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Dict, Never, Optional, assert_never
from loguru import logger
from sqlalchemy import create_engine, event, text
@@ -50,13 +55,37 @@ from .pool_config import (
POOL_RECYCLE_SECONDS,
POOL_SIZE,
)
+from .rekey_lock import (
+ DatabaseKeyCheckUnavailableError,
+ RekeyEpochState,
+ RekeyEpochUnreadableError,
+ RekeyIndeterminateError,
+ RekeyLockContentionError,
+ RekeyOperationalError,
+ RekeyRecoveryRequiredError,
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+ RekeyStage,
+ RekeyUnavailableError,
+ acquire_rekey_lock,
+ begin_rekey_epoch,
+ clear_pending_rekey_epoch,
+ commit_rekey_epoch,
+ publish_new_database_epoch,
+ read_or_initialize_rekey_epoch,
+ read_rekey_epoch,
+ read_rekey_epoch_state,
+)
from .sqlcipher_utils import (
set_sqlcipher_key,
set_sqlcipher_rekey,
apply_cipher_defaults_before_key,
apply_sqlcipher_pragmas,
apply_performance_pragmas,
- verify_sqlcipher_connection,
+ check_sqlcipher_key,
+ is_operational_sqlite_error,
+ get_sqlcipher_busy_timeout_ms,
+ sqlcipher_busy_timeout_pragma,
create_database_salt,
get_salt_file_path,
has_per_database_salt,
@@ -66,6 +95,78 @@ from .sqlcipher_utils import (
)
+@dataclass(frozen=True, slots=True)
+class _ConnectionCacheEntry:
+ engine: Engine
+ verifier: tuple[bytes, bytes]
+ epoch: str
+
+
+@dataclass(frozen=True, slots=True)
+class _ConnectionGeneration:
+ username: str
+ engine: Engine
+ verifier: tuple[bytes, bytes] | None
+ epoch: str | None
+
+
+@dataclass(frozen=True, slots=True)
+class _CurrentConnectionGeneration:
+ generation: _ConnectionGeneration
+
+
+@dataclass(frozen=True, slots=True)
+class _StaleConnectionGenerationDetached:
+ generation: _ConnectionGeneration
+
+
+@dataclass(frozen=True, slots=True)
+class _ConnectionGenerationChanged:
+ pass
+
+
+@dataclass(frozen=True, slots=True)
+class _ConnectionGenerationMissing:
+ pass
+
+
+type _ConnectionGenerationClaim = (
+ _CurrentConnectionGeneration
+ | _StaleConnectionGenerationDetached
+ | _ConnectionGenerationChanged
+ | _ConnectionGenerationMissing
+)
+
+
+@dataclass(frozen=True, slots=True)
+class _ConnectionGenerationUnverifiable:
+ """A generation is cached, but its epoch could not be checked.
+
+ The rekey sidecar could not be opened or locked, or the epoch read
+ failed once it was. Nothing was read, so nothing proves the generation
+ stale: it stays cached, and ``error`` is the typed fault for the
+ callers that ask for it (``raise_unverifiable``).
+ """
+
+ error: RekeyOperationalError
+
+
+@unique
+class UserConnectionStatus(StrEnum):
+ """Outcome of :meth:`DatabaseManager.probe_user_connection`."""
+
+ # A cached engine that the durable rekey epoch still vouches for.
+ CONNECTED = "connected"
+ # No usable engine: none is cached, or the cached one was stale and has
+ # been detached. Opening the database again needs the password.
+ NOT_CONNECTED = "not-connected"
+ # An engine is cached, but the rekey sidecar or epoch could not be read
+ # (fd exhaustion, EACCES, EIO, ...), or another holder kept the sidecar
+ # past the wait bound. It is kept, and the next successful probe
+ # re-checks it, so the condition is temporary, not a verdict.
+ UNAVAILABLE = "unavailable"
+
+
class DatabaseInitializationError(Exception):
"""Raised when a per-user database opens but its schema can't be initialised.
@@ -77,6 +178,170 @@ class DatabaseInitializationError(Exception):
"""
+# How long change_password retries its non-blocking sidecar flock: long
+# enough to ride out other processes' millisecond cache probes, short
+# enough that a genuine cross-process rekey owner still fails promptly.
+_REKEY_CONTENTION_RETRY_SECONDS = 1.0
+
+# Ceiling on each wait a request-path caller makes for the user's rekey
+# sidecar: a cache claim (lifecycle lock, sidecar gate and flock together),
+# a cold open and the epoch initialization at registration. A holder keeps
+# the sidecar for as long as its work runs (a rekey or cold open of a large
+# database: seconds to minutes) and a stopped process or a hung mount keeps
+# it indefinitely, so an unbounded wait parks every request for the user,
+# and the shared log-drain thread, behind it. Past the bound the caller gets
+# RekeyLockContentionError, which those paths answer as a retryable 503 with
+# Retry-After, like any other sidecar fault. Inside an HTTP request, a wait
+# that gives up also sets a flag (``sidecar_wait_budget``): each later claim
+# then tries once without waiting, until one of them gets the sidecar again.
+# The flag is not a running total of the time waited.
+_SIDECAR_WAIT_TIMEOUT_SECONDS = 5.0
+
+
+def _seconds_until(deadline: float) -> float:
+ return max(0.0, deadline - time.monotonic())
+
+
+class _SidecarWaitBudget:
+ """The state one ``sidecar_wait_budget`` scope shares among its waits."""
+
+ __slots__ = ("active", "spent")
+
+ def __init__(self) -> None:
+ self.active = True
+ # Set when one of the scope's sidecar waits ran out (another holder
+ # kept the sidecar past the bound); cleared when a later one gets it.
+ self.spent = False
+
+
+_sidecar_wait_budget: ContextVar[_SidecarWaitBudget | None] = ContextVar(
+ "ldr_sidecar_wait_budget", default=None
+)
+
+
+@contextmanager
+def sidecar_wait_budget() -> Generator[None, None, None]:
+ """Share a give-up flag among the rekey sidecar waits in this scope.
+
+ Entered for every HTTP request (``request_database_scope`` in
+ ``web/dependencies/auth.py``, from ``DatabaseMiddleware``). Each wait
+ is bounded by ``_SIDECAR_WAIT_TIMEOUT_SECONDS``, and one authenticated
+ request makes up to four of them before its route runs:
+ DatabaseMiddleware's probe and its reopen of the database, then
+ ``require_auth``'s probe and its classifying probe. Without the scope a
+ holder that keeps the sidecar would make each of them wait out the
+ bound, about 20 s before a 503 that says ``Retry-After: 5``, on the
+ shared threads that serve every user. In the scope, a wait that gives
+ up sets the ``spent`` flag, and each later claim then tries once
+ without waiting and raises (or reports) ``RekeyLockContentionError``
+ while the sidecar is still held, until one of them gets the sidecar
+ again, which clears the flag. So a request waits out the bound at most
+ once while the sidecar stays held, rather than once per claim.
+
+ The flag is not a running total of the time waited: a request whose
+ waits each get the sidecar just under the bound clears it every time
+ and can wait close to the full bound for each claim. Waits made
+ outside every scope (the log drain, for one) wait up to the bound each.
+
+ The flag is shared with every thread the request's context is copied
+ to (``run_db_sync`` and the AnyIO workers that run sync routes and
+ dependencies), so their waits set and clear the same one. When the
+ scope exits, copies retained by background research or indexing jobs
+ stop sharing it and each subsequent claim gets its own bounded wait.
+ """
+ budget = _SidecarWaitBudget()
+ token = _sidecar_wait_budget.set(budget)
+ try:
+ yield
+ finally:
+ # Resetting this ContextVar restores only the current context. A
+ # background worker keeps the same object in its copied context,
+ # so end its lifetime there too, including on request failure.
+ budget.active = False
+ _sidecar_wait_budget.reset(token)
+
+
+def _sidecar_wait_seconds() -> float:
+ """How long the current scope's next sidecar wait may last."""
+ budget = _sidecar_wait_budget.get()
+ if budget is not None and budget.active and budget.spent:
+ return 0.0
+ return _SIDECAR_WAIT_TIMEOUT_SECONDS
+
+
+def _record_sidecar_wait(*, ran_out: bool) -> None:
+ """Record that a sidecar wait ran out, or that one got the sidecar."""
+ budget = _sidecar_wait_budget.get()
+ if budget is not None and budget.active:
+ budget.spent = ran_out
+
+
+@contextmanager
+def _bounded_rekey_lock(db_path: Path) -> Generator[None, None, None]:
+ """Hold the sidecar like ``acquire_rekey_lock``, for a request-path caller.
+
+ Waits at most ``_sidecar_wait_seconds()`` and records how the wait
+ ended for the scope's later waits (``sidecar_wait_budget``). A
+ ``RekeyLockContentionError`` that the protected body raises is not this
+ wait's and is not recorded.
+ """
+ acquired = False
+ try:
+ with acquire_rekey_lock(db_path, timeout=_sidecar_wait_seconds()):
+ acquired = True
+ _record_sidecar_wait(ran_out=False)
+ yield
+ except RekeyLockContentionError:
+ if not acquired:
+ _record_sidecar_wait(ran_out=True)
+ raise
+
+
+# Ceiling on the busy-handler wait of a WAL checkpoint that runs while this
+# process holds the user's rekey sidecar gate (the post-rekey TRUNCATE and
+# the TRUNCATE before an engine disposal, which applies the same bound on
+# its few ungated call sites). Every request for that user
+# (is_user_connected, get_session, cold opens) queues on the gate, so a
+# checkpoint that waits out the full connection busy_timeout
+# (``db_config.busy_timeout_ms``, 30 s by default) behind a reader that
+# pins the WAL would stall all of them for that long. Both checkpoints are
+# best-effort -- the committed frames are durable in the WAL either way --
+# so they give up after this bound instead.
+_GATED_CHECKPOINT_BUSY_TIMEOUT_MS = 1000
+# Fixed statement template; only an ``int`` is ever formatted into it.
+_GATED_CHECKPOINT_BUSY_TIMEOUT_PRAGMA = "PRAGMA busy_timeout = {:d}"
+
+
+def _gated_checkpoint_busy_timeout_ms() -> int:
+ return min(
+ get_sqlcipher_busy_timeout_ms(), _GATED_CHECKPOINT_BUSY_TIMEOUT_MS
+ )
+
+
+def _gated_checkpoint_busy_timeout_pragma() -> str:
+ """Return the ``PRAGMA busy_timeout`` statement for a gated checkpoint.
+
+ Built like ``sqlcipher_busy_timeout_pragma``: PRAGMA takes no bind
+ parameters, so the bound is coerced with ``int()`` and formatted into a
+ fixed template whose ``{:d}`` spec refuses anything that is not an
+ integer, so the statement can only ever carry an integer literal.
+ """
+ timeout_ms = int(_gated_checkpoint_busy_timeout_ms())
+ return _GATED_CHECKPOINT_BUSY_TIMEOUT_PRAGMA.format(timeout_ms)
+
+
+class _RekeyVerificationError(Exception):
+ check: str
+
+ def __init__(self, check: str) -> None:
+ self.check = check
+ super().__init__(f"rekey verification failed: {check}")
+
+
+def _raise_rekey_unavailable(db_path: Path, stage: RekeyStage) -> Never:
+ raise RekeyUnavailableError(db_path, stage) from None
+
+
def _best_effort_chmod(path, mode: int, *, warn: bool = False) -> None:
"""Tighten permissions on ``path`` to ``mode``, never raising.
@@ -213,17 +478,18 @@ def _remove_partial_user_db_files(db_path: Path) -> None:
Called from ``create_user_database``'s failure paths. The encrypted
branch writes a per-database ``.salt`` file (via ``create_database_salt``)
- *before* the DB itself, and SQLCipher/WAL leaves ``-wal``/``-shm``
- sidecars. Removing only the ``.db`` — as these cleanup paths used to —
+ *before* the DB itself, and finalization publishes a ``.rekey.epoch``.
+ SQLCipher/WAL can also leave ``-wal``/``-shm`` sidecars. Removing only the
+ ``.db`` — as these cleanup paths used to —
orphans the ``.salt``: a retry then hits ``create_database_salt``'s
``FileExistsError`` (it refuses to overwrite to prevent data loss), so the
username stays permanently un-registerable even after the orphaned auth
row is cleaned up. Delete every artifact so the on-disk state is atomic
and the username is reusable on a subsequent retry.
- ``create_user_database`` calls this from four sites: its three
- failure-cleanup blocks (structure creation, engine build, migration — a
- graceful error mid-creation) and its salt-creation recovery path (an
+ ``create_user_database`` calls this from five sites: its four
+ failure-cleanup blocks (structure creation, engine build, migration,
+ finalization) and its salt-creation recovery path (an
orphan left by a prior hard-killed create or by pre-#4934 cleanup that
removed only the ``.db``). In every case the ``db_path.exists()`` guard at
the top of ``create_user_database`` already ran and found no *pre-existing*
@@ -234,6 +500,8 @@ def _remove_partial_user_db_files(db_path: Path) -> None:
Both the ``-wal``/``-shm`` (WAL mode, the default) and ``-journal``
(rollback-journal modes: DELETE/TRUNCATE/PERSIST) sidecars are removed so
the cleanup is correct regardless of the configured ``journal_mode``.
+ The stable ``.rekey.lock`` inode is deliberately preserved so processes
+ already waiting on it cannot split into a second lock domain.
Never raises — cleanup must not mask the original creation error.
"""
@@ -250,6 +518,8 @@ def _remove_partial_user_db_files(db_path: Path) -> None:
db_path.with_name(db_path.name + "-wal"),
db_path.with_name(db_path.name + "-shm"),
db_path.with_name(db_path.name + "-journal"),
+ db_path.with_name(f"{db_path.name}.rekey.epoch"),
+ db_path.with_name(f"{db_path.name}.rekey.pending"),
)
except Exception:
logger.warning(
@@ -339,6 +609,7 @@ class DatabaseManager:
# password already lives in memory in session_password_store, so a
# slow KDF would add cost without reducing exposure.
self._password_verifiers: Dict[str, tuple[bytes, bytes]] = {}
+ self._connection_epochs: dict[str, str] = {}
self._verifier_key = secrets.token_bytes(32)
self._connections_lock = threading.RLock()
# Per-user locks serializing the cold-open (engine build + migration)
@@ -346,24 +617,15 @@ class DatabaseManager:
# the same database file at once. Created lazily under
# _connections_lock; see open_user_database / _get_init_lock.
self._init_locks: Dict[str, threading.Lock] = {}
- # Per-user locks serializing the whole change_password sequence
- # (close -> open(old) -> PRAGMA rekey -> evict). Deliberately NOT
- # _init_locks: change_password calls open_user_database, which
- # acquires the init lock itself, so sharing that non-reentrant lock
- # would self-deadlock. Created lazily; see
- # _get_password_change_lock.
- self._password_change_locks: Dict[str, threading.Lock] = {}
- # Per-user rekey generation, bumped once per successful
- # ``change_password`` rekey. ``_cache_connection`` refuses a publish
- # whose ``expected_epoch`` predates the current generation, which
- # closes the stale-publish window eviction-at-close cannot: an
- # ``open_user_database(old)`` that verified against the old-keyed
- # file and parked before publishing must not populate the cache
- # after the rekey lands (see ``_cache_connection``). Bounded by the
- # number of distinct usernames, like ``_init_locks``; deliberately
- # retained across closes (logout / idle sweep must not reset it, or
- # a parked stale open would match again).
- self._rekey_epoch: Dict[str, int] = {}
+ self._rekey_locks: dict[str, threading.Lock] = {}
+ # Per-user gate serializing change_password CALLS against each
+ # other. Deliberately separate from _rekey_locks (the lifecycle
+ # lock ordinary opens/claims take briefly): sharing one lock made
+ # every cache-hit open a potential false "rekey already in
+ # progress" for a concurrent password change. The cross-exclusion
+ # between a rekey and opens is provided by the on-disk sidecar,
+ # which change_password holds across its whole staged sequence.
+ self._password_change_locks: dict[str, threading.Lock] = {}
self._data_dir_override: Optional[Path] = None
self._initialized_data_dirs: set[Path] = set()
@@ -489,10 +751,45 @@ class DatabaseManager:
username: The username to check
Returns:
- True if the user has an active connection
+ True if the user has an active connection. False also covers a
+ cached engine that cannot be checked right now because the rekey
+ sidecar or epoch could not be read, or the sidecar stayed held
+ past the wait bound (``_SIDECAR_WAIT_TIMEOUT_SECONDS``; once a
+ request's wait has given up, its later claims try once without
+ waiting until one gets the sidecar again:
+ ``sidecar_wait_budget``); use
+ :meth:`probe_user_connection` to tell that temporary case apart.
"""
- with self._connections_lock:
- return username in self.connections
+ with self._acquire_current_connection_generation(
+ username
+ ) as generation:
+ return generation is not None
+
+ def probe_user_connection(self, username: str) -> UserConnectionStatus:
+ """Classify ``username``'s cached connection, separating a fault.
+
+ ``is_user_connected`` answers False both when no usable engine is
+ cached and when a cached engine could not be checked because the
+ rekey sidecar could not be opened or locked, or the epoch read
+ failed (an fd-exhaustion spike, EACCES, EIO, another holder keeping
+ the sidecar past the wait bound). The engine is kept in
+ the second case and re-checked by the next probe, so it calls for a
+ retry, not a new login. Request authentication uses this to answer
+ such a fault with a retryable 503 instead of a 401.
+
+ Never raises a ``RekeyOperationalError``: a fault while releasing
+ the sidecar after the claim is reported as ``UNAVAILABLE`` too.
+ """
+ try:
+ with self._acquire_current_connection_generation(
+ username, raise_unverifiable=True
+ ) as generation:
+ connected = generation is not None
+ except RekeyOperationalError:
+ return UserConnectionStatus.UNAVAILABLE
+ if connected:
+ return UserConnectionStatus.CONNECTED
+ return UserConnectionStatus.NOT_CONNECTED
def _check_encryption_available(self) -> bool:
"""Probe SQLCipher before considering the development fallback.
@@ -672,7 +969,9 @@ class DatabaseManager:
"""Create a properly initialized SQLCipher connection.
Follows the canonical SQLCipher initialization order: set key,
- apply cipher pragmas, verify, then apply performance pragmas.
+ apply cipher pragmas, verify, then apply performance pragmas. The
+ busy timeout alone is set before verification, so a lock held
+ elsewhere is waited out instead of failing the key check.
Cipher pragmas (page size, HMAC algorithm, KDF iterations) must
be configured before the first query (verification) because that
query triggers page decryption with the active cipher settings.
@@ -702,12 +1001,19 @@ class DatabaseManager:
try:
set_sqlcipher_key(cursor, password, db_path=db_path)
apply_sqlcipher_pragmas(cursor, creation_mode=False)
-
- if not verify_sqlcipher_connection(cursor):
+ # Wait out a lock held by another connection before the
+ # verification read: with only the driver's shorter default
+ # wait, a busy read failed verification and a CORRECT password
+ # was reported as wrong (and, during pending-rekey recovery,
+ # counted toward the login lockout).
+ busy_timeout_sql = sqlcipher_busy_timeout_pragma()
+ cursor.execute(busy_timeout_sql)
+
+ if not check_sqlcipher_key(cursor):
raise ValueError("Failed to verify database key") # noqa: TRY301 — cleanup in except before re-raise
apply_performance_pragmas(cursor)
- except Exception:
+ except Exception as error:
try:
cursor.close()
except Exception: # noqa: BLE001
@@ -715,6 +1021,13 @@ class DatabaseManager:
from ..utilities.resource_utils import safe_close
safe_close(conn, "encrypted DB connection")
+ # A lock held past the busy timeout (or an I/O error) says
+ # nothing about the password: it must not become the ValueError
+ # every caller reads as a wrong password.
+ if is_operational_sqlite_error(error):
+ raise DatabaseKeyCheckUnavailableError(
+ db_path, type(error).__name__
+ ) from None
raise
cursor.close()
@@ -829,10 +1142,29 @@ class DatabaseManager:
if not self.has_encryption and db_path.exists():
_best_effort_chmod(db_path, 0o600, warn=True)
- # Store connection AFTER migrations complete. This password just proved
- # itself by creating the database; publish the engine and its verifier
- # together so the cache is never briefly trusted without one.
- self._cache_connection(username, engine, password)
+ # Store connection AFTER migrations complete. The sidecar keeps epoch
+ # initialization ordered with any cross-process cold-open or rekey.
+ # Its wait is bounded (and, once a request's wait has given up, the
+ # request's later claims try once without waiting until one gets the
+ # sidecar again: sidecar_wait_budget): a holder past the bound fails
+ # the creation below, which removes the partial database.
+ try:
+ with _bounded_rekey_lock(db_path):
+ # A fresh epoch, never one left beside an earlier database
+ # at this path (see publish_new_database_epoch).
+ epoch = publish_new_database_epoch(db_path)
+ entry = self._make_connection_cache_entry(
+ engine, password, epoch
+ )
+ replaced = self._cache_connection(username, entry)
+ self._dispose_detached_connection(username, replaced)
+ except Exception:
+ detached = self._detach_connection(username)
+ self._dispose_detached_connection(username, detached)
+ if detached is not engine:
+ self._dispose_detached_connection(username, engine)
+ _remove_partial_user_db_files(db_path)
+ raise
def _create_user_database_files(
self, username: str, password: str
@@ -1120,48 +1452,19 @@ class DatabaseManager:
salt = secrets.token_bytes(16)
return salt, self._compute_verifier_digest(salt, password)
- def _get_rekey_epoch(self, username: str) -> int:
- """Return the current rekey generation for ``username`` (0 if none).
-
- Tolerates managers built via ``__new__`` without ``__init__`` (several
- tests set only the attributes the method under test touches) by
- lazily creating the registry. Caller need not hold
- ``_connections_lock``; this takes it.
- """
- with self._connections_lock:
- epochs = getattr(self, "_rekey_epoch", None)
- if epochs is None:
- epochs = {}
- self._rekey_epoch = epochs
- return epochs.get(username, 0)
-
- def _bump_rekey_epoch(self, username: str) -> int:
- """Advance the rekey generation for ``username`` and return the new value.
-
- Called once per successful ``change_password`` rekey, under
- ``_connections_lock``, so a concurrent cold-open's
- pre-verification snapshot (see ``_open_user_database_cold``) either
- precedes the bump (stale publish refused) or follows it (fresh
- publish allowed). Same ``__new__`` tolerance as
- :meth:`_get_rekey_epoch`.
- """
- with self._connections_lock:
- epochs = getattr(self, "_rekey_epoch", None)
- if epochs is None:
- epochs = {}
- self._rekey_epoch = epochs
- epochs[username] = epochs.get(username, 0) + 1
- return epochs[username]
+ def _make_connection_cache_entry(
+ self, engine: Engine, password: str, epoch: str
+ ) -> _ConnectionCacheEntry:
+ return _ConnectionCacheEntry(
+ engine=engine,
+ verifier=self._make_verifier(password),
+ epoch=epoch,
+ )
def _cache_connection(
- self,
- username: str,
- engine: Engine,
- password: str,
- expected_epoch: int | None = None,
- ) -> bool:
- """Publish an engine and the verifier for the password that opened it,
- atomically.
+ self, username: str, entry: _ConnectionCacheEntry
+ ) -> Engine | None:
+ """Publish an engine, password verifier, and durable epoch atomically.
Called only on paths that have ALREADY proved the password by opening
(or creating) the SQLCipher database with it. Storing the engine and
@@ -1169,73 +1472,13 @@ class DatabaseManager:
connection always has a matching verifier" true by construction, so a
concurrent open can never observe a connection with a missing or stale
verifier.
-
- If a previous engine is already cached for ``username`` (the race
- shape ``close_user_database_if_stale`` is built for: a concurrent
- ``open_user_database(new_password)`` racing ``change_password`` and
- overwriting the OLD-key engine we just published), the previous
- engine reference is captured before the overwrite and disposed inside
- the same lock acquisition. Without this dispose the orphan's
- SQLCipher pool and file handle linger until GC, which on the path
- this PR makes common (concurrent new-password opens against a
- rekeying database) leaks one stale connection per overwrite. The
- lock covers both the cache swap and the dispose, so a concurrent
- ``open_user_database`` cannot observe the new engine as cached
- while its predecessor is still considered live; this matches the
- pattern used by ``close_user_database`` and
- ``close_user_database_if_stale``.
-
- When ``expected_epoch`` is not None, the publish is refused when a
- rekey landed between the caller's pre-verification snapshot and this
- cache swap (``current != expected_epoch``). That is the window
- eviction-at-close cannot close: an ``open_user_database(old)`` that
- verified against the still-old-keyed file and parked before this
- publish would otherwise plant a stale OLD-key engine with a matching
- old verifier into an emptied cache after the rekey (e.g. an external
- unconditional close evicted the rekey's own engine mid-flight, so
- both conditional closes no-op). Refusing the publish fails
- CLOSED -- the caller disposes its just-verified-but-now-stale
- engine and returns None, so a later ``open_user_database(old)``
- cold-opens against the now-new-keyed file and fails as it must.
- A concurrent ``open_user_database(new)`` that snapshots AFTER
- the rekey (e.g. it waited on init_lock and verified after the
- rekey landed) publishes normally, so the surviving-engine
- invariant is preserved. A new-password open whose cold-open
- STRADDLES the rekey (snapshotted before, SQLCipher-verified
- after) is also refused -- fail CLOSED for the now-correct
- password; retry succeeds. That is an availability-only cost in
- the fail-closed direction, surfaced on the login route as a
- single retryable failure. ``None`` (the default, used by the
- create path and by tests driving the primitive directly) keeps
- the unconditional publish.
"""
- verifier = self._make_verifier(password)
with self._connections_lock:
- if expected_epoch is not None:
- epochs = getattr(self, "_rekey_epoch", None)
- current = epochs.get(username, 0) if epochs is not None else 0
- if current != expected_epoch:
- logger.warning(
- f"Refusing stale publish for {username}: "
- f"rekey generation advanced "
- f"({expected_epoch} -> {current}) between "
- "verification and cache publish",
- )
- return False
- orphan = self.connections.get(username)
- self.connections[username] = engine
- self._password_verifiers[username] = verifier
- if orphan is not None and orphan is not engine:
- # Same swallow-and-log discipline as the unconditional close:
- # the orphan has already been evicted from the cache, so a
- # failed dispose only delays GC of its pool/handle.
- try:
- orphan.dispose()
- except Exception:
- logger.warning(
- f"Failed to dispose orphaned engine for {username}",
- )
- return True
+ replaced = self.connections.get(username)
+ self.connections[username] = entry.engine
+ self._password_verifiers[username] = entry.verifier
+ self._connection_epochs[username] = entry.epoch
+ return replaced if replaced is not entry.engine else None
def _record_password_verifier(self, username: str, password: str) -> None:
"""Arm the verifier for a connection cached by other means.
@@ -1245,34 +1488,336 @@ class DatabaseManager:
already-cached engine. Caller must not hold ``_connections_lock``.
"""
verifier = self._make_verifier(password)
+ epoch = read_rekey_epoch(self._get_user_db_path(username))
with self._connections_lock:
self._password_verifiers[username] = verifier
+ if epoch is None:
+ self._connection_epochs.pop(username, None)
+ else:
+ self._connection_epochs[username] = epoch
- def _verifier_matches(self, username: str, password: str) -> bool:
- """Lock-free verifier comparison. **Caller must hold _connections_lock.**
+ def _connection_epoch_matches(
+ self, username: str, current_epoch: str | None
+ ) -> bool:
+ """Compare a disk epoch while the caller holds ``_connections_lock``."""
+ return (
+ current_epoch is not None
+ and self._connection_epochs.get(username) == current_epoch
+ )
+
+ def _detach_connection_locked(self, username: str) -> Engine | None:
+ """Detach all cache state while the caller holds ``_connections_lock``."""
+ engine = self.connections.pop(username, None)
+ self._password_verifiers.pop(username, None)
+ self._connection_epochs.pop(username, None)
+ return engine
+
+ def _detach_connection(self, username: str) -> Engine | None:
+ with self._connections_lock:
+ return self._detach_connection_locked(username)
+
+ def _snapshot_connection_generation(
+ self, username: str
+ ) -> _ConnectionGeneration | None:
+ with self._connections_lock:
+ engine = self.connections.get(username)
+ if engine is None:
+ return None
+ return _ConnectionGeneration(
+ username,
+ engine,
+ self._password_verifiers.get(username),
+ self._connection_epochs.get(username),
+ )
+
+ def _connection_generation_matches_locked(
+ self, generation: _ConnectionGeneration
+ ) -> bool:
+ return (
+ self.connections.get(generation.username) is generation.engine
+ and self._password_verifiers.get(generation.username)
+ is generation.verifier
+ and self._connection_epochs.get(generation.username)
+ == generation.epoch
+ )
- Fails CLOSED: a missing verifier returns False so the caller falls
+ def _detach_connection_generation(
+ self, generation: _ConnectionGeneration
+ ) -> bool:
+ with self._connections_lock:
+ if not self._connection_generation_matches_locked(generation):
+ return False
+ self._detach_connection_locked(generation.username)
+ return True
+
+ def _claim_connection_generation(
+ self,
+ generation: _ConnectionGeneration,
+ state: RekeyEpochState,
+ ) -> _ConnectionGenerationClaim:
+ with self._connections_lock:
+ if self.connections.get(generation.username) is None:
+ return _ConnectionGenerationMissing()
+ if not self._connection_generation_matches_locked(generation):
+ return _ConnectionGenerationChanged()
+ if (
+ state.pending_epoch is not None
+ or state.committed_epoch is None
+ or generation.epoch != state.committed_epoch
+ ):
+ self._detach_connection_locked(generation.username)
+ return _StaleConnectionGenerationDetached(generation)
+ return _CurrentConnectionGeneration(generation)
+
+ @contextmanager
+ def _acquire_current_connection_generation(
+ self, username: str, *, raise_unverifiable: bool = False
+ ) -> Generator[_ConnectionGeneration | None, None, None]:
+ """Yield the epoch-validated cached generation, or None.
+
+ The body of the ``with`` runs while this thread holds the user's
+ lifecycle lock, the in-process sidecar gate and the sidecar flock.
+ Every other probe for the same user waits on them, including the
+ single shared log-drain thread (``is_user_connected``), which
+ persists log entries for ALL users, and the idle-connection sweep
+ skips the user meanwhile. Keep the body to cache bookkeeping
+ (snapshot the engine, build a sessionmaker, compare a verifier) and
+ do database work after the context exits;
+ ``check_database_integrity`` runs its scans that way.
+
+ Other holders of the same sidecar are long by nature: a cold open
+ (``_open_user_database_cold``, including the pre-migration backup
+ and alembic migrations) and the rekey inside ``change_password``,
+ in this process or another one, and a stopped process or a hung
+ mount holds it indefinitely. So a claim waits for its locks at most
+ ``_SIDECAR_WAIT_TIMEOUT_SECONDS`` in total, retries after a changed
+ generation included, and not at all when an earlier wait of the
+ same request ran out and none has got the sidecar since
+ (``sidecar_wait_budget``). The lifecycle lock counts against the
+ same budget, because the claim ahead of this one holds it while that
+ claim waits for the sidecar. Past the bound the sidecar is one that
+ could not be locked (``RekeyLockContentionError``).
+
+ A cached generation whose sidecar cannot be opened or locked, or
+ whose epoch read fails, is kept and yields None. With
+ ``raise_unverifiable`` the typed fault (``RekeySidecarIOError``,
+ ``RekeyLockContentionError`` or ``RekeyEpochUnreadableError``) is
+ raised instead, after every lock is released, for the callers that
+ must not treat it as a miss: ``open_user_database``,
+ ``probe_user_connection`` and ``get_session``.
+
+ Releasing the sidecar afterwards can fail too (unlock or close).
+ When the body of the ``with`` finished cleanly, that raises
+ ``RekeySidecarIOError`` for every caller, also when the probe runs
+ inside a caller's ``except`` block. When the body raised, its own
+ exception propagates and the release failure is added to it as a
+ note instead of replacing it.
+ """
+ if not username:
+ # Fail closed BEFORE touching the filesystem: an empty or None
+ # username must not even resolve a database path
+ # (test_db_connection_username_trust).
+ yield None
+ return
+ db_path = self._get_user_db_path(username)
+ deadline = time.monotonic() + _sidecar_wait_seconds()
+ while True:
+ generation = self._snapshot_connection_generation(username)
+ if generation is None:
+ yield None
+ return
+
+ lifecycle_lock = self._get_rekey_lock(username)
+ if not lifecycle_lock.acquire(timeout=_seconds_until(deadline)):
+ # Still held at the deadline, by the claim ahead of this one
+ # while it waits for the sidecar, or by a disposal under it:
+ # the sidecar could not be locked in time, the same
+ # unverifiable outcome as the sidecar fault below.
+ _record_sidecar_wait(ran_out=True)
+ fault = RekeyLockContentionError(
+ db_path.with_name(f"{db_path.name}.rekey.lock")
+ )
+ if raise_unverifiable:
+ raise fault from None
+ yield None
+ return
+ try:
+ sidecar_lock = acquire_rekey_lock(
+ db_path, timeout=_seconds_until(deadline)
+ )
+ try:
+ sidecar_lock.__enter__()
+ except (
+ OSError,
+ RekeyLockContentionError,
+ RekeySidecarIOError,
+ ) as error:
+ # The sidecar cannot be opened or locked (fd exhaustion,
+ # EROFS/EACCES/ENOSPC, a vanished data directory, or a
+ # refused planted sidecar -- the lock module normalizes
+ # all of these to RekeySidecarIOError), or another
+ # holder kept it past the deadline above
+ # (RekeyLockContentionError). Nothing can be
+ # proved about the durable epoch, so THIS request fails
+ # closed. The callers that treat None as "no usable
+ # connection" (is_user_connected, which the log sink
+ # gates on, _password_matches_cached, the metrics
+ # session's cache hit and the integrity check) see None
+ # and never have to catch the typed error. The three
+ # that must not read it as a miss get it raised below
+ # instead: open_user_database, whose cold path needs
+ # this same sidecar (so it fails before deriving a
+ # key); probe_user_connection, which reports it as
+ # UNAVAILABLE so request authentication answers a
+ # retryable 503 rather than a 401; and get_session,
+ # whose None get_settings_manager reads as "this user
+ # has no database" and answers from the defaults file,
+ # so a stored strict egress scope would read as the
+ # default. (The metrics session falls through to
+ # open_user_database on that None, so it raises the
+ # typed error too.)
+ #
+ # The cached engine is deliberately left in place, not
+ # evicted: a transient fault (an EMFILE spike) would
+ # otherwise dispose every user's healthy engine at once.
+ # Leaving it is safe because the cache never hands an
+ # engine out without a successful claim: the next probe
+ # re-takes the sidecar and re-checks the epoch, and a
+ # rekey committed meanwhile detaches it there.
+ if isinstance(error, RekeyLockContentionError):
+ _record_sidecar_wait(ran_out=True)
+ claim = _ConnectionGenerationUnverifiable(
+ error
+ if isinstance(error, RekeyOperationalError)
+ else RekeySidecarIOError(
+ db_path, RekeySidecarOperation.ACQUIRE, error.errno
+ )
+ )
+ else:
+ _record_sidecar_wait(ran_out=False)
+ # Whatever the body raises (the caller's with-block,
+ # thrown in at the yield, or the disposal) is handed to
+ # the lock as a with statement would hand it, so a
+ # failure to release the sidecar is added to it as a
+ # note instead of replacing it. Recorded here, not left
+ # for the lock to infer: after a clean body it would
+ # see whatever exception a caller is handling (a probe
+ # made inside an except block) and attach the release
+ # failure to that instead of raising it.
+ body_failure: BaseException | None = None
+ try:
+ try:
+ state = read_rekey_epoch_state(db_path)
+ except (OSError, RekeyEpochUnreadableError) as error:
+ # The sidecar opened but the epoch read itself
+ # failed (EMFILE right after the sidecar took
+ # the last fd, EIO, ...): nothing was read, so
+ # treat it like the sidecar fault above. THIS
+ # request sees "not connected" (or the typed
+ # error, for the three callers above); the cached
+ # engine is kept and the next successful probe
+ # re-checks the epoch before any hand-out.
+ claim = _ConnectionGenerationUnverifiable(
+ error
+ if isinstance(error, RekeyEpochUnreadableError)
+ else RekeyEpochUnreadableError(
+ db_path, error.errno
+ )
+ )
+ except RekeyRecoveryRequiredError:
+ # Read, and does not vouch (malformed token,
+ # pending without committed): detach below.
+ claim = self._claim_connection_generation(
+ generation, RekeyEpochState(None, None)
+ )
+ else:
+ claim = self._claim_connection_generation(
+ generation, state
+ )
+ match claim:
+ case _CurrentConnectionGeneration(
+ generation=current
+ ):
+ yield current
+ return
+ case _StaleConnectionGenerationDetached(
+ generation=stale
+ ):
+ self._dispose_detached_connection(
+ stale.username,
+ stale.engine,
+ checkpoint=False,
+ )
+ case (
+ _ConnectionGenerationChanged()
+ | _ConnectionGenerationMissing()
+ | _ConnectionGenerationUnverifiable()
+ ):
+ pass
+ case unreachable:
+ assert_never(unreachable)
+ except BaseException as failure:
+ body_failure = failure
+ raise
+ finally:
+ if body_failure is None:
+ sidecar_lock.__exit__(None, None, None)
+ else:
+ sidecar_lock.__exit__(
+ type(body_failure),
+ body_failure,
+ body_failure.__traceback__,
+ )
+ finally:
+ lifecycle_lock.release()
+
+ match claim:
+ case _ConnectionGenerationChanged():
+ continue
+ case _ConnectionGenerationUnverifiable(error=fault):
+ if raise_unverifiable:
+ raise fault from None
+ yield None
+ return
+ case (
+ _StaleConnectionGenerationDetached()
+ | _ConnectionGenerationMissing()
+ ):
+ yield None
+ return
+ case _CurrentConnectionGeneration():
+ raise AssertionError(
+ "current generation escaped its ownership scope"
+ )
+ case unreachable:
+ assert_never(unreachable)
+
+ def _verifier_entry_matches(
+ self, username: str, entry: object, password: str
+ ) -> bool:
+ """Compare ``password`` against one ``(salt, digest)`` verifier entry.
+
+ Fails CLOSED: a missing entry returns False so the caller falls
through to a real key-validating open. Returning True on a missing
- verifier would reopen exactly the hole this closes.
+ verifier would reopen exactly the hole the verifier closes.
A malformed entry (anything that is not a ``(salt, expected)`` tuple
of bytes, including a two-tuple with wrong-typed members such as
``("x", "y")`` or ``(b"salt", 123)``) also returns False rather than
- propagating. The call sites run inside ``change_password``'s
- ``finally`` block -- any raise here would replace the actual
- ``change_password`` outcome with a spurious ``ValueError`` /
- ``TypeError`` and the operator would see a confusing
- "password change failed" instead of the underlying cause. The
- invariants that produce well-formed entries (``_make_verifier`` and
- ``_cache_connection``) are unit-locked, so a corrupt entry can only
- arrive through external dict mutation; degrading to no-match is
- safer than letting that exception escape.
+ propagating. Every caller sits on a cache-hit or cold-open path that
+ holds the per-user lifecycle locks: a ``TypeError`` / ``ValueError``
+ escaping from here would surface as an unexplained login or session
+ failure instead of the ordinary fall-through to a key-validating
+ open. The invariants that produce well-formed entries
+ (``_make_verifier`` via ``_make_connection_cache_entry``, and
+ ``_record_password_verifier``) are unit-locked, so a corrupt entry
+ can only arrive through external dict mutation; degrading to
+ no-match is safer than letting that exception escape.
"""
- entry = self._password_verifiers.get(username)
if entry is None:
return False
try:
- salt, expected = entry
+ salt, expected = entry # type: ignore[misc]
if not isinstance(salt, bytes) or not isinstance(expected, bytes):
logger.warning(
f"Malformed verifier entry for {username}; treating as no-match"
@@ -1287,7 +1832,33 @@ class DatabaseManager:
)
return False
- def _cached_engine_trusted(self, username: str, password: str) -> bool:
+ def _verifier_matches(self, username: str, password: str) -> bool:
+ """Lock-free verifier comparison. **Caller must hold _connections_lock.**
+
+ Fails CLOSED on a missing or malformed verifier; see
+ :meth:`_verifier_entry_matches`.
+ """
+ return self._verifier_entry_matches(
+ username, self._password_verifiers.get(username), password
+ )
+
+ def _connection_generation_password_matches(
+ self, generation: _ConnectionGeneration, password: str
+ ) -> bool:
+ """Check ``password`` against a claimed generation's verifier.
+
+ Fails CLOSED on a missing or malformed verifier, exactly like
+ :meth:`_verifier_matches`; see :meth:`_verifier_entry_matches`.
+ """
+ if not self.has_encryption:
+ return True
+ return self._verifier_entry_matches(
+ generation.username, generation.verifier, password
+ )
+
+ def _cached_engine_trusted(
+ self, username: str, password: str, current_epoch: str | None
+ ) -> bool:
"""May the cached engine for ``username`` be returned to a caller
presenting ``password``? **Caller must hold _connections_lock.**
@@ -1298,18 +1869,22 @@ class DatabaseManager:
database_middleware). Trust the cache directly. Encrypted mode requires
a verifier match.
"""
+ if not self._connection_epoch_matches(username, current_epoch):
+ return False
if not self.has_encryption:
return True
return self._verifier_matches(username, password)
def _password_matches_cached(self, username: str, password: str) -> bool:
- """Does ``password`` match the one that opened the cached engine?
-
- Fails CLOSED (see :meth:`_verifier_matches`). Takes the lock; prefer
- :meth:`_verifier_matches` when already holding it.
- """
- with self._connections_lock:
- return self._verifier_matches(username, password)
+ """Check ``password`` against the exact current cache generation."""
+ with self._acquire_current_connection_generation(
+ username
+ ) as generation:
+ return generation is not None and (
+ self._connection_generation_password_matches(
+ generation, password
+ )
+ )
def _get_init_lock(self, username: str) -> threading.Lock:
"""Return the per-user cold-open lock, creating it on first use.
@@ -1328,27 +1903,49 @@ class DatabaseManager:
return lock
def _get_password_change_lock(self, username: str) -> threading.Lock:
- """Return the per-user rekey lock, creating it on first use.
+ """Return the per-user change_password gate, creating it on first use.
+
+ Serializes concurrent change_password CALLS for one user (the loser
+ gets the typed 409 contention error). Must stay a separate registry
+ from _rekey_locks: the lifecycle lock is taken briefly by ordinary
+ opens/claims, so gating on it here would turn every racing open
+ into a false contention.
+
+ The backing dict is reached through ``getattr`` because several
+ tests build a ``DatabaseManager`` via ``__new__`` and set only the
+ attributes the method under test touches, so it can legitimately
+ be absent.
+ """
+ with self._connections_lock:
+ locks = getattr(self, "_password_change_locks", None)
+ if locks is None:
+ locks = {}
+ self._password_change_locks = locks
+ lock = locks.get(username)
+ if lock is None:
+ lock = threading.Lock()
+ locks[username] = lock
+ return lock
- Serializes the whole of :meth:`change_password` for one user. Per-user
- (not the global ``_connections_lock``) so a rekey for one user never
- stalls another user's opens, and so the lock can be held across the
- multi-second PBKDF2 + ``PRAGMA rekey`` without freezing the manager.
+ def _get_rekey_lock(self, username: str) -> threading.Lock:
+ """Return the dedicated per-user password-rekey lifecycle lock.
- Deliberately a SEPARATE registry from ``_init_locks``:
- ``change_password`` calls ``open_user_database``, which acquires this
- user's init lock itself, so sharing that non-reentrant lock would
- self-deadlock on every password change.
+ Serializes the whole rekey sequence for one user in-process; the
+ on-disk sidecar lock (``acquire_rekey_lock``) extends the same
+ exclusion across processes. Deliberately a separate registry from
+ ``_init_locks``: change-password-era code paths acquire both, and
+ sharing one non-reentrant lock would self-deadlock.
- The backing dict is reached through ``getattr`` because several tests
- build a ``DatabaseManager`` via ``__new__`` and set only the attributes
- the method under test touches, so it can legitimately be absent.
+ The backing dict is reached through ``getattr`` because several
+ tests build a ``DatabaseManager`` via ``__new__`` and set only the
+ attributes the method under test touches, so it can legitimately
+ be absent.
"""
with self._connections_lock:
- locks = getattr(self, "_password_change_locks", None)
+ locks = getattr(self, "_rekey_locks", None)
if locks is None:
locks = {}
- self._password_change_locks = locks
+ self._rekey_locks = locks
lock = locks.get(username)
if lock is None:
lock = threading.Lock()
@@ -1371,6 +1968,18 @@ class DatabaseManager:
A returned engine in that mode is not evidence of authentication.
Callers must not use this fallback for shared accounts that rely on
LDR password verification without adding independent authentication.
+
+ When an engine is cached but its rekey sidecar cannot be opened or
+ locked, or its epoch cannot be read, the typed fault
+ (``RekeySidecarIOError`` / ``RekeyLockContentionError`` /
+ ``RekeyEpochUnreadableError``, all ``RekeyOperationalError``) is
+ raised before any key is derived, and the engine stays cached. A
+ cold open whose per-user init lock stays held past
+ ``_SIDECAR_WAIT_TIMEOUT_SECONDS`` raises ``RekeyLockContentionError``
+ before the derivation, and one whose sidecar stays held past it
+ raises it after the derivation (without waiting, in a request whose earlier
+ wait ran out: ``sidecar_wait_budget``). None stays reserved for a
+ credential or missing-database verdict.
"""
open_start = time.perf_counter()
@@ -1389,6 +1998,8 @@ class DatabaseManager:
"Invalid encryption key: password cannot be None or empty"
)
+ db_path = self._get_user_db_path(username)
+
# Check if already open.
#
# `connections` is keyed by username ALONE, so a hit proves only that
@@ -1398,34 +2009,172 @@ class DatabaseManager:
# which is the normal state whenever they are logged in anywhere.
# Verify the credential against the one that actually opened it, and
# fall through to a real (key-validating) open when it does not match.
- # Look up the engine and check its verifier under ONE lock acquisition
- # so the engine cannot be evicted between the lookup and the verify
- # (which would hand back a disposed, de-registered engine).
- with self._connections_lock:
- cached = self.connections.get(username)
- if cached is not None and self._cached_engine_trusted(
- username, password
+ # Claim the engine and check its verifier in one ownership scope so it
+ # cannot be evicted between validation and return.
+ #
+ # raise_unverifiable: when an engine is cached but the rekey sidecar
+ # cannot be opened or locked, or the epoch cannot be read, raise that
+ # typed fault here, BEFORE the key derivation below. The cold path
+ # needs the same sidecar, so it would only fail the same way after a
+ # full PBKDF2 derivation, and every caller (each request's
+ # DatabaseMiddleware, socket handshakes, background sessions) would
+ # pay one per attempt for as long as an fd-exhaustion spike lasts.
+ # The engine stays cached for the next successful probe. Only an
+ # existing user can have a cached engine, and the login route
+ # answers this fault with a 503 that an unknown username never
+ # gets, so skipping the derivation reveals nothing the status code
+ # does not already reveal.
+ with self._acquire_current_connection_generation(
+ username, raise_unverifiable=True
+ ) as generation:
+ if generation is not None and (
+ self._connection_generation_password_matches(
+ generation, password
+ )
):
- return cached
+ return generation.engine
# Serialize the cold-open (engine build + migration) per user so two
# concurrent first-opens never run alembic against the same database
# file at once -- alembic's module-level proxy and the version-row
# UPDATE are not safe under concurrent migration of one DB.
init_lock = self._get_init_lock(username)
- with init_lock:
- # Re-check: another thread may have completed the cold-open while
- # we were waiting for the per-user lock. One lock acquisition, as
- # above, so the engine cannot be evicted between lookup and verify.
- with self._connections_lock:
- cached = self.connections.get(username)
- if cached is not None and self._cached_engine_trusted(
- username, password
+ recovered_pending_rekey = False
+ with self._bounded_init_lock(init_lock, db_path):
+ # Derive the key UNCONDITIONALLY (an absent database must cost
+ # the same as an existing one -- username-enumeration timing
+ # parity, see _open_user_database_cold), then refuse symlinked
+ # and absent databases BEFORE any sidecar file is created: an
+ # unauthenticated caller probing usernames must not be able to
+ # materialize on-disk state for a user that does not exist.
+ prederived_hex_key = get_key_from_password(
+ password, db_path=db_path
+ ).hex()
+ if _db_path_is_symlinked(db_path, username):
+ return None
+ if not db_path.exists():
+ logger.error(f"No database found for user {username}")
+ return None
+
+ # The stable sidecar serializes cold opens with cross-process rekeys.
+ # Epoch repair and publication happen only while this inode is locked.
+ # The wait is bounded (once a request's wait has given up, its
+ # later claims try once without waiting until one gets the
+ # sidecar again: sidecar_wait_budget): a holder that keeps it
+ # longer (a rekey of a large database, a stopped process) makes
+ # this open raise RekeyLockContentionError, which a login
+ # answers with 503 (not counted toward lockout), a signed-in
+ # request's reopen (ensure_user_database) with 503 and the
+ # session kept, and the other callers as any sidecar fault.
+ with _bounded_rekey_lock(db_path):
+ # Re-checked under the lock (cheap now): the pre-gate above
+ # ran without any exclusion.
+ if _db_path_is_symlinked(db_path, username):
+ return None
+ if not db_path.exists():
+ logger.error(f"No database found for user {username}")
+ return None
+ rekey_state = read_rekey_epoch_state(db_path)
+ if rekey_state.pending_epoch is not None:
+ pending_engine = self._detach_connection(username)
+ if pending_engine is not None and not (
+ self._dispose_detached_connection(
+ username, pending_engine, checkpoint=False
+ )
+ ):
+ raise RekeyRecoveryRequiredError(db_path)
+ if not self._recover_pending_rekey(
+ db_path, password, rekey_state
+ ):
+ return None
+ recovered_pending_rekey = True
+ try:
+ current_epoch = read_or_initialize_rekey_epoch(db_path)
+ except OSError as error:
+ # First open after upgrade publishes the epoch file;
+ # an I/O failure there (ENOSPC, EIO, mkstemp EACCES)
+ # is operational, not a credential verdict -- type it
+ # so the login route answers 503 without counting it
+ # toward lockout instead of escaping as a raw 500.
+ raise RekeySidecarIOError(
+ db_path, RekeySidecarOperation.PUBLISH, error.errno
+ ) from None
+ detached = None
+ with self._connections_lock:
+ cached = self.connections.get(username)
+ if cached is not None and self._cached_engine_trusted(
+ username, password, current_epoch
+ ):
+ return cached
+ if (
+ cached is not None
+ and not self._connection_epoch_matches(
+ username, current_epoch
+ )
+ ):
+ detached = self._detach_connection_locked(username)
+ self._dispose_detached_connection(
+ username, detached, checkpoint=False
+ )
+
+ engine = self._open_user_database_cold(
+ username, password, open_start, prederived_hex_key
+ )
+ if engine is None:
+ return None
+
+ # Re-read the durable state the password was just proved
+ # under. No outcome other than the same stable committed
+ # epoch publishes the engine, and none of the others says
+ # anything against the credential: a read that fails
+ # (EMFILE, EIO, persistent EACCES:
+ # RekeyEpochUnreadableError), durable state that does not
+ # vouch (a malformed token, a missing committed epoch, or
+ # a pending epoch), and a different committed epoch (none
+ # of the last four can appear while this open holds the
+ # sidecar unless a writer bypassed it). All of them
+ # propagate as the RekeyOperationalError the reads above
+ # already raise on this path, so the login route answers
+ # 503 and leaves the lockout counter alone, instead of
+ # returning the ``(None, None, False)`` that a wrong
+ # password returns (401, counted toward lockout).
+ try:
+ rechecked_state = read_rekey_epoch_state(db_path)
+ except RekeyRecoveryRequiredError:
+ self._dispose_detached_connection(
+ username, engine, checkpoint=False
+ )
+ raise
+ if (
+ rechecked_state.pending_epoch is not None
+ or rechecked_state.committed_epoch is None
):
- return cached
- engine = self._open_user_database_cold(
- username, password, open_start
- )
+ self._dispose_detached_connection(
+ username, engine, checkpoint=False
+ )
+ raise RekeyRecoveryRequiredError(db_path)
+ if rechecked_state.committed_epoch != current_epoch:
+ # A different committed epoch was read: the engine was
+ # built under a superseded generation; discard it
+ # unpublished. Every epoch writer runs under the
+ # sidecar this open holds, so only a writer that
+ # bypassed it (an operator editing the epoch file, a
+ # foreign tool) gets here. The password DID open the
+ # database, so this is not a wrong password either:
+ # fail closed with the same RekeyRecoveryRequiredError
+ # (login answers 503, lockout counter untouched)
+ # rather than the (None, None, False) that reads as
+ # invalid credentials (401, counted toward lockout).
+ self._dispose_detached_connection(
+ username, engine, checkpoint=False
+ )
+ raise RekeyRecoveryRequiredError(db_path)
+
+ entry = self._make_connection_cache_entry(
+ engine, password, current_epoch
+ )
+ replaced = self._cache_connection(username, entry)
+ self._dispose_detached_connection(username, replaced)
# Phase 2 of the RAG vector-store cutover runs OUTSIDE init_lock (only
# on the path that actually did the cold-open — the early returns above
@@ -1435,9 +2184,93 @@ class DatabaseManager:
# self-deadlock if a concurrent close_user_database() evicted the
# freshly-cached connection between the cache write and that re-entry.
if engine is not None:
+ if recovered_pending_rekey:
+ self._purge_superseded_key_backups(
+ username, password, expected_epoch=current_epoch
+ )
self._run_phase2_rekey(username, password)
return engine
+ def _purge_superseded_key_backups(
+ self, username: str, password: str, *, expected_epoch: str
+ ) -> None:
+ """Delete backups left under the old key by an interrupted rekey.
+
+ Runs after an open resolved a pending password change with
+ ``password``. A change that completed normally purges old-key
+ backups in the change-password route; one whose outcome was
+ reported as indeterminate never reached that step, so this is
+ where the old-key backups go once the active key is proven.
+
+ The cold open released its locks before reaching here. Reacquire
+ the rekey sidecar and require the same stable epoch before deleting
+ anything: a later password change may already have created backups
+ under its newer key, which this password would mistake for a
+ superseded one. Keep the sidecar through classification and deletion,
+ acquiring the backup service's per-user lock inside it (the same
+ order as pre-migration backups). A fault only skips this best-effort
+ cleanup; it never fails the open.
+ """
+ try:
+ from .backup.backup_service import BackupService
+
+ db_path = self._get_user_db_path(username)
+ with _bounded_rekey_lock(db_path):
+ state = read_rekey_epoch_state(db_path)
+ if state != RekeyEpochState(expected_epoch, None):
+ return
+ backup_service = BackupService(
+ username=username, password=password
+ )
+ # A pinned manager data_dir can differ from the backup
+ # service's configured directory. The fence must vouch for
+ # the database whose backups would actually be deleted.
+ if backup_service.db_path != db_path:
+ return
+ purged = backup_service.purge_backups_under_superseded_key()
+ if purged:
+ logger.info(
+ f"Purged {purged} backup(s) under the superseded key "
+ f"for {username} after pending rekey recovery"
+ )
+ except Exception as error:
+ logger.warning(
+ f"Backup purge after pending rekey recovery failed for "
+ f"{username}: {type(error).__name__}"
+ )
+
+ @staticmethod
+ @contextmanager
+ def _bounded_init_lock(
+ init_lock: threading.Lock, db_path: Path
+ ) -> Generator[None, None, None]:
+ """Hold a user's init lock, waiting no longer than a sidecar wait.
+
+ The init lock's holder derives the key and then waits up to the
+ sidecar bound itself, so while another holder (a rekey of a large
+ database, a stopped process) keeps the sidecar, an unbounded wait
+ here made N concurrent opens fail one after another, each about one
+ derivation plus one bound later than the previous, while occupying
+ the shared worker threads that serve every user's requests. Waiting
+ at most the same bound (and recording a give-up for the request's
+ later waits, like a sidecar wait) makes them all fail within about
+ one bound instead. This is only a queue bound: the sidecar acquire
+ inside stays the authoritative exclusion against a rekey. The same
+ bound applies when the holder is a sibling cold open of this user
+ that runs longer (a first-open migration of a large database): the
+ waiting open answers with the retryable contention error instead of
+ parking its thread until the migration ends.
+ """
+ if not init_lock.acquire(timeout=_sidecar_wait_seconds()):
+ _record_sidecar_wait(ran_out=True)
+ raise RekeyLockContentionError(
+ db_path.with_name(f"{db_path.name}.rekey.lock")
+ )
+ try:
+ yield
+ finally:
+ init_lock.release()
+
def _run_phase2_rekey(self, username: str, password: str) -> None:
"""Re-key a user's legacy FAISS indices (uuid-keyed -> DocumentChunk.id
-keyed), once, at login. See vector_stores/legacy_rekey.py.
@@ -1469,13 +2302,21 @@ class DatabaseManager:
)
def _open_user_database_cold(
- self, username: str, password: str, open_start: float
+ self,
+ username: str,
+ password: str,
+ open_start: float,
+ prederived_hex_key: str | None = None,
) -> Optional[Engine]:
"""Build the engine and run migrations for a not-yet-cached user DB.
Must be called while holding this user's init lock (acquired in
``open_user_database``) so concurrent first-opens of the same user do
not migrate one database file simultaneously.
+
+ ``prederived_hex_key`` lets the caller (which derives the key
+ unconditionally for username-enumeration timing parity before its
+ cheap pre-gates) avoid paying the derivation twice.
"""
db_path = self._get_user_db_path(username)
@@ -1483,7 +2324,11 @@ class DatabaseManager:
# This ensures both existing and non-existent users take the same amount of time,
# preventing username enumeration via timing analysis.
# Pre-derive key before closures to avoid capturing plaintext password
- hex_key = get_key_from_password(password, db_path=db_path).hex()
+ hex_key = (
+ prederived_hex_key
+ if prederived_hex_key is not None
+ else get_key_from_password(password, db_path=db_path).hex()
+ )
# After the key derivation, not before: the derivation is deliberately
# unconditional so a non-existent user costs the same as an existing
@@ -1561,16 +2406,6 @@ class DatabaseManager:
# For unencrypted databases, just apply pragmas
event.listen(engine, "connect", self._apply_pragmas)
- # Snapshot the rekey generation BEFORE the SQLCipher round-trip below.
- # A rekey landing between this snapshot and the cache publish means
- # this engine verified against the pre-rekey key and must not be
- # cached (see _cache_connection's expected_epoch guard). Snapshotting
- # here -- rather than at open_user_database entry -- lets a
- # new-password open that waited on init_lock and verified AFTER the
- # rekey publish normally, while an old-password open that verified
- # BEFORE the rekey is refused.
- expected_epoch = self._get_rekey_epoch(username)
-
try:
# Test connection by running a simple query
with engine.connect() as conn:
@@ -1662,24 +2497,6 @@ class DatabaseManager:
f"Database initialisation failed for {username}: {safe_msg}"
) from None
- # Store connection AFTER migrations complete. This password just
- # proved itself against SQLCipher; publish the engine and its
- # verifier together so the cache is never briefly trusted without
- # one. The expected_epoch guard refuses a stale publish when a
- # rekey landed mid-cold-open (see above): dispose and fail CLOSED
- # so a later open retries against the current file key.
- published = self._cache_connection(
- username, engine, password, expected_epoch=expected_epoch
- )
- if not published:
- try:
- engine.dispose()
- except Exception:
- logger.warning(
- f"Failed to dispose stale engine for {username}",
- )
- return None
-
# NOTE: the phase-2 RAG index re-key does NOT run here. It re-enters
# open_user_database() (via get_user_db_session on a session-cache
# miss), and this method runs while holding the non-reentrant
@@ -1720,6 +2537,13 @@ class DatabaseManager:
f"after {elapsed_ms:.0f}ms: {safe_msg}"
)
engine.dispose()
+ if is_operational_sqlite_error(e):
+ # Locked or busy past the busy timeout, an I/O error: the
+ # password was not checked, so this is not the None that
+ # callers (the login lockout counter) read as a wrong one.
+ raise DatabaseKeyCheckUnavailableError(
+ db_path, type(e).__name__
+ ) from None
return None
def get_session(self, username: str) -> Optional[Session]:
@@ -1748,16 +2572,29 @@ class DatabaseManager:
auth-decision path: either turns this accessor into an unauthenticated
bypass. Credential-taking entry points must go through
``open_user_database`` instead.
+
+ ``None`` means only that no usable engine is cached: none is, or
+ the cached one was stale and has just been detached. A cached
+ engine whose rekey sidecar cannot be opened or locked (within the
+ wait bound), or whose epoch cannot be read, is not a miss: the typed
+ ``RekeySidecarIOError`` / ``RekeyLockContentionError`` /
+ ``RekeyEpochUnreadableError`` is raised
+ instead (the engine stays cached), as is a failure to release the
+ sidecar afterwards. ``None`` would fail open here:
+ ``get_settings_manager`` (``utilities/db_utils.py``) reads it as
+ "this user has no database" and answers every setting from the
+ defaults file, so during the fault a stored ``strict`` egress scope
+ or a custom library ``storage_path`` would read as its default. A
+ route that leaves the fault unhandled gets the app's retryable 503.
"""
- with self._connections_lock:
- if username not in self.connections:
- # Use debug level for this common scenario to reduce log noise
- logger.debug(f"No open database for user {username}")
- return None
- engine = self.connections[username]
- # Create session inside lock to prevent race with close_user_database()
- SessionLocal = sessionmaker(bind=engine)
- return SessionLocal()
+ with self._acquire_current_connection_generation(
+ username, raise_unverifiable=True
+ ) as generation:
+ if generation is not None:
+ SessionLocal = sessionmaker(bind=generation.engine)
+ return SessionLocal()
+ logger.debug(f"No open database for user {username}")
+ return None
def get_connected_usernames(self) -> set:
"""Return a snapshot of usernames with open connections."""
@@ -1765,16 +2602,30 @@ class DatabaseManager:
return set(self.connections.keys())
def _checkpoint_wal(self, engine, context: str = ""):
- """Checkpoint WAL before disposing engine to flush pending writes."""
+ """Checkpoint WAL before disposing engine to flush pending writes.
+
+ Most callers hold the user's rekey sidecar gate (fenced disposal of
+ a current generation, cache replacement during a cold open), so the
+ TRUNCATE's busy wait is bounded by
+ ``_GATED_CHECKPOINT_BUSY_TIMEOUT_MS`` (see there) for every caller;
+ the checkpoint is best-effort either way. The pooled connection's
+ own busy_timeout is restored before it goes back to the pool.
+ """
try:
with engine.connect() as conn:
- # TRUNCATE returns (busy, log_pages, checkpointed_pages).
- # busy=1 means a reader/writer held a WAL lock and the WAL
- # was NOT truncated — surface that so we can tell from logs
- # whether the helper actually shrank the file.
- result = conn.execute(
- text("PRAGMA wal_checkpoint(TRUNCATE)")
- ).fetchone()
+ gated_timeout_sql = _gated_checkpoint_busy_timeout_pragma()
+ conn.execute(text(gated_timeout_sql))
+ try:
+ # TRUNCATE returns (busy, log_pages, checkpointed_pages).
+ # busy=1 means a reader/writer held a WAL lock and the
+ # WAL was NOT truncated — surface that so we can tell
+ # from logs whether the helper actually shrank the file.
+ result = conn.execute(
+ text("PRAGMA wal_checkpoint(TRUNCATE)")
+ ).fetchone()
+ finally:
+ restore_timeout_sql = sqlcipher_busy_timeout_pragma()
+ conn.execute(text(restore_timeout_sql))
if result and result[0] == 1:
logger.debug(
f"WAL checkpoint busy {context} — WAL not truncated"
@@ -1782,134 +2633,548 @@ class DatabaseManager:
except Exception:
logger.debug(f"WAL checkpoint failed {context}", exc_info=True)
- def close_user_database(self, username: str):
- """Close a user's database connection."""
- with self._connections_lock:
- if username in self.connections:
- try:
- self._checkpoint_wal(
- self.connections[username], f"for {username}"
- )
- self.connections[username].dispose()
- except Exception:
- logger.warning(
- f"Failed to dispose engine for {username}",
- )
- del self.connections[username]
- # Drop the verifier with the connection it described, so a
- # stale entry can never outlive the engine it was recorded for.
- self._password_verifiers.pop(username, None)
- # Deliberately do NOT pop _init_locks[username] here. A
- # concurrent open_user_database may already hold a reference to
- # this user's lock (fetched via _get_init_lock) and be about to
- # enter its cold-open; dropping the entry would let a later open
- # create a *second* lock for the same user, so two cold-opens
- # could migrate one DB file at once -- the very race this lock
- # exists to prevent. The dict is bounded by the number of
- # distinct usernames (one small Lock each), so retaining it is
- # cheap; close_all_databases likewise retains it (see there).
- logger.info(f"Closed database for user {username}")
+ @staticmethod
+ def _require_full_synchronous(connection) -> None:
+ connection.execute("PRAGMA synchronous = FULL")
+ result = connection.execute("PRAGMA synchronous").fetchone()
+ if result != (2,):
+ raise _RekeyVerificationError("synchronous mode")
+
+ @staticmethod
+ def _pre_rekey_checkpoint(connection) -> None:
+ """Fold committed WAL frames into the main file; best-effort.
+
+ PASSIVE never invokes the busy handler and never waits for a
+ reader, so a reader pinning the WAL (a research thread's open read
+ transaction, another worker's request) cannot stall the rekey here
+ while the sidecar gate is held. Completeness is not required for
+ correctness: ``PRAGMA rekey`` reads every page through the pager,
+ WAL included, and the new key is proven afterwards by
+ ``_verify_rekey_candidate``. A SQLite error still propagates and is
+ an operational preparation failure.
+ """
+ connection.execute("PRAGMA wal_checkpoint(PASSIVE)").fetchone()
+
+ @staticmethod
+ def _best_effort_rekey_checkpoint(connection) -> bool:
+ """Checkpoint after the new key is proven; True only when complete.
+
+ A concurrent reader (another process's open read transaction, a
+ backup connection) legitimately keeps a TRUNCATE checkpoint from
+ completing. That is not an ambiguity about which key is active:
+ the rekey transaction already committed under synchronous=FULL,
+ so its WAL frames are durable and the next complete checkpoint
+ folds them in (see ``_verify_rekey_candidate`` for what that
+ leaves on disk meanwhile). The busy wait is bounded by
+ ``_GATED_CHECKPOINT_BUSY_TIMEOUT_MS`` because the sidecar gate is
+ held; the connection is closed right after verification, so its
+ busy_timeout is not restored.
+ """
+ gated_timeout_sql = _gated_checkpoint_busy_timeout_pragma()
+ connection.execute(gated_timeout_sql)
+ result = connection.execute(
+ "PRAGMA wal_checkpoint(TRUNCATE)"
+ ).fetchone()
+ return (
+ result is not None
+ and len(result) == 3
+ and result[0] == 0
+ and result[1] == result[2]
+ )
+
+ @staticmethod
+ def _warn_rekey_integrity(db_path: Path, check: str, rows) -> None:
+ """Log an integrity finding loudly without failing the rekey.
+
+ Only the number of findings and the first one (SQLite's own
+ message: page numbers, table and index names, never row data or
+ key material), truncated, are logged.
+ """
+ first = repr(rows[0])[:200] if rows else "no result"
+ logger.warning(
+ f"{check} reported {len(rows)} problem(s) for {db_path.name} "
+ f"after the active key was proven (first: {first}); the key "
+ "verdict stands, but the database needs an integrity review "
+ "(see docs/security/rekey-recovery.md)"
+ )
+
+ def _verify_rekey_candidate(self, connection, db_path: Path) -> None:
+ """Prove that ``connection``'s key is the database's active key.
+
+ The proof is that the key decrypts the schema page: SQLCipher
+ authenticates every page it reads (HMAC), and page 1 is written
+ under exactly one key, so a database state never opens under both
+ the old and the new key. Which key is active is decided here, and
+ only a failure of that proof raises.
+
+ The integrity scans after it (``quick_check`` and
+ ``cipher_integrity_check``) say whether the file is healthy, not
+ which key it is under. A database that was already damaged (a bad
+ page elsewhere, an inconsistent schema) fails them under whichever
+ key is active, so failing the rekey or its recovery on them left
+ the pending state in place for good and fenced the user out of a
+ database that opens. They are logged loudly instead.
+ """
+ self._require_full_synchronous(connection)
+ try:
+ schema = connection.execute(
+ "SELECT count(*) FROM sqlite_master"
+ ).fetchone()
+ except Exception:
+ raise _RekeyVerificationError("database key") from None
+ if schema is None:
+ raise _RekeyVerificationError("database key")
+ try:
+ quick_rows = list(connection.execute("PRAGMA quick_check"))
+ except Exception as error:
+ quick_rows = [(type(error).__name__,)]
+ if not quick_rows or any(row != ("ok",) for row in quick_rows):
+ self._warn_rekey_integrity(
+ db_path, "PRAGMA quick_check", quick_rows
+ )
+ # The new key is now proven against the committed, durable file, so
+ # a checkpoint that cannot complete (busy reader) must not make the
+ # outcome indeterminate -- that stranded users behind a pending
+ # epoch whose key had in fact changed.
+ #
+ # Checkpoint BEFORE the cipher integrity scan: after the rekey
+ # commit the WAL holds the pages re-encrypted under the NEW key,
+ # while the main database file still holds them under the OLD key
+ # until a checkpoint copies the frames back. cipher_integrity_check
+ # reads the main file directly (not the WAL), so a new-key scan
+ # reports phantom "HMAC verification failed" rows for every page
+ # not yet checkpointed. Pinning the canary with a real-SQLCipher
+ # duel showed the pre-checkpoint scan failing and the
+ # post-checkpoint scan passing on the identical file. The scan is
+ # therefore only meaningful (and only run) after a complete
+ # checkpoint.
+ try:
+ checkpointed = self._best_effort_rekey_checkpoint(connection)
+ except Exception as error:
+ # A checkpoint that fails (SQLITE_CORRUPT on a damaged page, an
+ # I/O error) leaves the committed WAL frames in place exactly
+ # like a busy one; the key verdict above stands.
+ logger.warning(
+ f"Post-rekey WAL checkpoint for {db_path.name} failed: "
+ f"{type(error).__name__}"
+ )
+ checkpointed = False
+ if checkpointed:
+ try:
+ cipher_rows = list(
+ connection.execute("PRAGMA cipher_integrity_check")
+ )
+ except Exception as error:
+ cipher_rows = [(type(error).__name__,)]
+ if cipher_rows:
+ self._warn_rekey_integrity(
+ db_path, "PRAGMA cipher_integrity_check", cipher_rows
+ )
+ else:
+ # Until a later checkpoint completes, the re-encrypted pages
+ # live only in the WAL and the main database file still holds
+ # every page under the OLD key (a copy of the main file alone
+ # opens with the old password). The live database (main file
+ # plus WAL) already rejects the old key. The window closes at
+ # the first complete checkpoint after the pinning reader ends:
+ # SQLite's own checkpoint when the last connection to the file
+ # closes, the TRUNCATE before any engine disposal (logout, idle
+ # cleanup, shutdown), or an automatic checkpoint after later
+ # writes.
+ logger.warning(
+ f"Post-rekey WAL checkpoint for {db_path.name} did not "
+ "complete (a concurrent reader holds the WAL); the new key "
+ "is verified, but the main database file keeps old-key "
+ "pages until a later checkpoint folds the WAL in"
+ )
+ # No raw fsync of the database or its WAL here, and nowhere else
+ # while this process may hold a connection to the file: closing ANY
+ # descriptor to it releases every POSIX advisory (fcntl) lock this
+ # process holds on that inode, including the SHARED lock SQLite's
+ # WAL mode relies on (sqlite.org/howtocorrupt.html, section 2.2).
+ # SQLite's per-inode lock bookkeeping would not notice, so another
+ # process could checkpoint and delete the WAL underneath a live
+ # connection here and its later commits would be lost. Durability
+ # comes from SQLite itself instead: the rekey transaction committed
+ # under synchronous=FULL (enforced by ``_require_full_synchronous``
+ # on the old-key connection), which fsyncs the WAL at commit, and a
+ # complete checkpoint on this synchronous=FULL connection fsyncs
+ # the main database file.
- def close_user_database_if_stale(
- self, username: str, expected_old_password: str
+ @staticmethod
+ def _classify_rekey_failure(
+ db_path: Path, stage: RekeyStage
+ ) -> RekeyUnavailableError | RekeyIndeterminateError:
+ try:
+ state = read_rekey_epoch_state(db_path)
+ except Exception:
+ state = None
+ if (
+ stage is RekeyStage.PENDING_PUBLICATION
+ and state is not None
+ and state.pending_epoch is None
+ ):
+ return RekeyUnavailableError(db_path, stage)
+ return RekeyIndeterminateError(db_path, stage, state)
+
+ @staticmethod
+ def _clear_committed_rekey_pending(
+ db_path: Path, pending_epoch: str, username: str
) -> None:
- """Close the cached engine for ``username`` only when its verifier
- still matches ``expected_old_password``.
-
- This is the change_password post-rekey eviction with a guard: a
- concurrent ``open_user_database(new_password)`` racing the rekey
- legitimately succeeds against the now-new_password-keyed file and
- publishes a fresh engine with a DIFFERENT verifier (the new
- one). Dropping that engine would force the next caller to
- re-cold-open AND would break the invariant "the engine cached for
- a user is the one a hammer racing this rekey observed"
- (``test_stress_change_password_chain_vs_open_hammers`` pins exactly
- that). Matching by verifier rather than by engine identity lets the
- concurrent valid engine stay cached and keeps the eviction scoped
- to the stale one whose key was rekeyed away.
-
- This guard NARROWS but does not CLOSE the stale-engine window on its
- own: eviction-at-close races by construction. If an external
- unconditional close (logout, the ~300s idle sweep) empties the cache
- mid-rekey, both conditional closes no-op on the emptied cache, and a
- parked ``open_user_database(old)`` that verified against the still
- old-keyed file can then publish a stale OLD-key engine with a
- matching old verifier into the empty cache -- after which
- ``open_user_database(old)`` is handed that engine without a SQLCipher
- round-trip (data reads then fail with "file is not a database"). The
- window is CLOSED by the rekey-generation check at publish time in
- :meth:`_cache_connection` (``expected_epoch``): the parked open
- snapshots the pre-rekey generation, the successful rekey bumps it via
- :meth:`_bump_rekey_epoch`, and the stale publish is refused, disposed
- and returns None. This helper remains the eviction half; the epoch is
- the publication half.
+ try:
+ clear_pending_rekey_epoch(db_path, pending_epoch)
+ except Exception as error:
+ logger.warning(
+ f"Pending rekey cleanup failed for {username}: "
+ f"{type(error).__name__}"
+ )
+
+ @staticmethod
+ def _close_rekey_connection(connection, username: str, phase: str) -> None:
+ try:
+ connection.close()
+ except Exception as error:
+ logger.warning(
+ f"Rekey {phase} connection close failed for {username}: "
+ f"{type(error).__name__}"
+ )
+
+ def _recover_pending_rekey(
+ self,
+ db_path: Path,
+ password: str,
+ state: RekeyEpochState,
+ ) -> bool:
+ """Verify the surviving SQLCipher key before finalizing pending state.
+
+ Returns False when ``password`` does not open the surviving key: an
+ ordinary wrong password, which callers must treat exactly like any
+ other failed credential (the login route counts it toward the
+ lockout). Only a password that opens the database but whose
+ verification or epoch publication fails raises
+ ``RekeyRecoveryRequiredError``.
"""
- with self._connections_lock:
- if username not in self.connections:
- return
- if not self._verifier_matches(username, expected_old_password):
- return
+ pending_epoch = state.pending_epoch
+ if pending_epoch is None:
+ return True
+ connection = None
+ try:
try:
- self._checkpoint_wal(
- self.connections[username],
- f"for {username} post-rekey",
+ connection = self._make_sqlcipher_connection(db_path, password)
+ except ValueError:
+ # Neither surviving key accepted this password; disk and
+ # pending metadata stay untouched.
+ return False
+ self._verify_rekey_candidate(connection, db_path)
+ if state.committed_epoch != pending_epoch:
+ commit_rekey_epoch(db_path, pending_epoch)
+ clear_pending_rekey_epoch(db_path, pending_epoch)
+ except Exception:
+ raise RekeyRecoveryRequiredError(db_path) from None
+ finally:
+ if connection is not None:
+ self._close_rekey_connection(
+ connection, db_path.name, "recovery"
)
- self.connections[username].dispose()
- except Exception:
- logger.warning(
- f"Failed to dispose engine for {username}",
+ return True
+
+ def _resolve_unapplied_rekey(
+ self, db_path: Path, old_password: str, pending_epoch: str
+ ) -> Never:
+ """Classify a rekey whose new key does not open the database.
+
+ SQLCipher's ``PRAGMA rekey`` ignores the result of the underlying
+ rekey: when another connection's write transaction outlasts the
+ busy timeout it answers ``ok``, raises nothing, and leaves the file
+ under the OLD key. The new-key open in ``change_password`` is what
+ catches that (a key that does not match the file fails SQLCipher's
+ page HMAC), so no epoch is ever committed for a key that did not
+ take effect.
+
+ If the old password still opens and verifies the database, the
+ rekey provably did not apply: the pending generation is resolved
+ exactly as a login with the surviving password resolves it
+ (``_recover_pending_rekey``), and the change fails as unavailable
+ (retryable; the old password stays valid and no pending state is
+ left behind). If neither password opens it, the pending generation
+ is no longer the one this call published, or the recovery itself
+ fails, the outcome stays indeterminate as before.
+ """
+ try:
+ state = read_rekey_epoch_state(db_path)
+ old_key_survived = (
+ state.pending_epoch == pending_epoch
+ and self._recover_pending_rekey(db_path, old_password, state)
+ )
+ except RekeyRecoveryRequiredError:
+ old_key_survived = False
+ if old_key_survived:
+ logger.warning(
+ f"PRAGMA rekey did not change the key of {db_path.name} "
+ "(another connection likely held a write transaction past "
+ "the busy timeout, or a damaged page stopped it); the old "
+ "password remains active"
+ )
+ raise RekeyUnavailableError(db_path, RekeyStage.PHYSICAL_REKEY)
+ raise self._classify_rekey_failure(
+ db_path, RekeyStage.CANDIDATE_VERIFICATION
+ )
+
+ def _dispose_detached_connection(
+ self,
+ username: str,
+ engine: Engine | None,
+ *,
+ checkpoint: bool = True,
+ ) -> bool:
+ """Dispose an engine outside cache locks, checkpointing when still current."""
+ if engine is None:
+ return False
+ if checkpoint:
+ self._checkpoint_wal(engine, f"for {username}")
+ try:
+ engine.dispose()
+ except Exception as error:
+ logger.warning(
+ f"Failed to dispose engine for {username}: "
+ f"{type(error).__name__}"
+ )
+ return False
+ return True
+
+ def _dispose_connection_generation(
+ self,
+ generation: _ConnectionGeneration,
+ *,
+ detach: bool,
+ ) -> bool:
+ """Dispose one cache generation with epoch and identity fencing."""
+ db_path = self._get_user_db_path(generation.username)
+ lifecycle_lock = self._get_rekey_lock(generation.username)
+ with lifecycle_lock:
+ try:
+ with acquire_rekey_lock(db_path):
+ return (
+ self._dispose_connection_generation_under_sidecar_lock(
+ generation, detach=detach
+ )
+ )
+ except (OSError, RekeyLockContentionError, RekeyOperationalError):
+ # Contention, or the sidecar cannot even be created (the
+ # user's data directory was removed out from under a still
+ # cached engine -- test teardown, operator cleanup). Either
+ # way there is nothing durable left to fence: fall back to
+ # disposing the engine directly, without a WAL checkpoint
+ # (checkpointing a vanished database is moot).
+ if not detach or not self._detach_connection_generation(
+ generation
+ ):
+ return False
+ return self._dispose_detached_connection(
+ generation.username,
+ generation.engine,
+ checkpoint=False,
)
- del self.connections[username]
- self._password_verifiers.pop(username, None)
+
+ def _dispose_connection_generation_under_sidecar_lock(
+ self,
+ generation: _ConnectionGeneration,
+ *,
+ detach: bool,
+ ) -> bool:
+ """Dispose one generation while its database sidecar lock is held."""
+ db_path = self._get_user_db_path(generation.username)
+ try:
+ state = read_rekey_epoch_state(db_path)
+ except RekeyEpochUnreadableError:
+ # The sidecar opened but the epoch read itself failed (EMFILE
+ # right after the sidecar took the last fd, EIO, persistent
+ # EACCES): nothing was read, so nothing proves this generation
+ # stale. Propagate it like a sidecar fault, exactly as the
+ # probe path keeps the engine on the same error: the periodic
+ # reset skips this generation for this sweep (engine kept,
+ # nothing disposed), and an explicit close still detaches and
+ # disposes it, without a WAL checkpoint. Substituting an empty
+ # state here would detach and dispose every signed-in user's
+ # engine on one transient read fault during the sweep.
+ raise
+ except RekeyRecoveryRequiredError:
+ # Read, and does not vouch (malformed token, pending without
+ # committed): the generation is stale, detach it below.
+ state = RekeyEpochState(None, None)
+ claim = self._claim_connection_generation(generation, state)
+ match claim:
+ case _CurrentConnectionGeneration(generation=current):
+ if detach and not self._detach_connection_generation(current):
+ return False
+ return self._dispose_detached_connection(
+ current.username,
+ current.engine,
+ checkpoint=True,
+ )
+ case _StaleConnectionGenerationDetached(generation=stale):
+ return self._dispose_detached_connection(
+ stale.username,
+ stale.engine,
+ checkpoint=False,
+ )
+ case (
+ _ConnectionGenerationChanged() | _ConnectionGenerationMissing()
+ ):
+ return False
+ case unreachable:
+ assert_never(unreachable)
+
+ def close_user_database(self, username: str):
+ """Close a user's database connection."""
+ generation = self._snapshot_connection_generation(username)
+ if generation is not None:
+ if self._dispose_connection_generation(generation, detach=True):
+ # Per-user init/rekey locks intentionally retain stable identities.
+ logger.info(f"Closed database for user {username}")
def close_all_databases(self):
"""Close all open user database connections and release file locks."""
with self._connections_lock:
- for username, engine in list(self.connections.items()):
- try:
- self._checkpoint_wal(engine, f"for {username}")
- engine.dispose()
- except Exception:
- logger.debug(f"Error disposing engine for {username}")
- self.connections.clear()
- self._password_verifiers.clear()
- # _init_locks is deliberately NOT cleared, for the same reason as
- # _password_change_locks below: a concurrent open_user_database
- # may already hold a reference to a user's cold-open lock (fetched
- # via _get_init_lock) and be about to enter it. Dropping the entry
- # would let a later open create a SECOND lock for that user, so
- # two cold-opens could migrate one DB file at once -- the exact
- # race the lock exists to stop. A lock.locked() sweep cannot close
- # that window either (the reference is taken before the lock is
- # acquired). Bounded by the number of distinct usernames, one
- # small Lock each, so retaining it is cheap.
- # _password_change_locks is deliberately NOT cleared. An entry may
- # be held by a change_password still in flight, and dropping it
- # would let the next caller build a SECOND gate for that user and
- # rekey the same file concurrently -- the exact race the lock
- # exists to stop. Bounded by the number of distinct usernames
- # whose database file exists (change_password looks the lock up
- # only after the path check), one small Lock each.
- # _rekey_epoch is deliberately NOT cleared either. It tracks
- # completed rekeys, not cached engines: a logout / idle-sweep
- # close must not reset it, or a parked stale open_user_database
- # (verified pre-rekey, publishing post-rekey) would match the
- # reset generation and repopulate the cache with an OLD-key
- # engine after the rekey -- the exact window the epoch closes.
+ detached = [
+ _ConnectionGeneration(
+ username,
+ engine,
+ self._password_verifiers.get(username),
+ self._connection_epochs.get(username),
+ )
+ for username, engine in self.connections.items()
+ ]
+ for generation in detached:
+ self._dispose_connection_generation(generation, detach=True)
+ with self._connections_lock:
+ connected = set(self.connections)
+ for username in set(self._password_verifiers) - connected:
+ self._password_verifiers.pop(username, None)
+ for username in set(self._connection_epochs) - connected:
+ self._connection_epochs.pop(username, None)
+
+ def reset_connection_pools(self) -> int:
+ """Reset cached pools safely and return the number successfully disposed."""
+ # _init_locks is deliberately NOT cleared above: a concurrent
+ # open_user_database may already hold a reference to a user's
+ # cold-open lock (fetched via _get_init_lock) and be about to
+ # enter it; dropping the entry would let a later open create a
+ # SECOND lock for that user, so two cold-opens could migrate one
+ # DB file at once -- the exact race the lock exists to stop.
+ # _rekey_locks is likewise retained wholesale -- an entry may be
+ # held by a change_password still in flight, and dropping it would
+ # let the next caller build a SECOND gate for that user and rekey
+ # the same file concurrently. Both registries are bounded by the
+ # number of distinct usernames, one small Lock each.
+ with self._connections_lock:
+ generations = [
+ _ConnectionGeneration(
+ username,
+ engine,
+ self._password_verifiers.get(username),
+ self._connection_epochs.get(username),
+ )
+ for username, engine in self.connections.items()
+ ]
+ return sum(
+ self._reset_connection_generation(generation)
+ for generation in generations
+ )
+
+ def _reset_connection_generation(
+ self, generation: _ConnectionGeneration
+ ) -> bool:
+ lifecycle_lock = self._get_rekey_lock(generation.username)
+ if not lifecycle_lock.acquire(blocking=False):
+ return False
+ try:
+ try:
+ with acquire_rekey_lock(
+ self._get_user_db_path(generation.username),
+ blocking=False,
+ ):
+ return (
+ self._dispose_connection_generation_under_sidecar_lock(
+ generation, detach=False
+ )
+ )
+ except (OSError, RekeyLockContentionError, RekeyOperationalError):
+ # Includes a vanished data directory: skip this generation
+ # this sweep; nothing durable remains to fence.
+ return False
+ finally:
+ lifecycle_lock.release()
+
+ @staticmethod
+ def _wal_fully_checkpointed(conn, username: str) -> bool:
+ """Run a PASSIVE checkpoint; True when no WAL frame is left behind.
+
+ A database not in WAL mode answers ``(0, -1, -1)`` and counts as
+ checkpointed. A checkpoint that cannot run, or leaves frames that a
+ reader still needs, returns False so the caller skips scans that
+ read only the main database file.
+ """
+ try:
+ row = conn.execute(
+ text("PRAGMA wal_checkpoint(PASSIVE)")
+ ).fetchone()
+ except Exception as error:
+ logger.warning(
+ f"WAL checkpoint before the integrity scan failed for "
+ f"{username}: {type(error).__name__}"
+ )
+ return False
+ if (
+ row is not None
+ and len(row) == 3
+ and row[0] == 0
+ and row[1] == row[2]
+ ):
+ return True
+ logger.warning(
+ f"Skipping the cipher integrity scan for {username}: WAL frames "
+ "are not yet checkpointed into the main database file"
+ )
+ return False
def check_database_integrity(self, username: str) -> bool:
"""Check database structure and encrypted page authentication.
Plain SQLite uses the full integrity check to verify that indexes
match table contents. SQLCipher retains the quick structural check
- and its independent page-HMAC verification.
+ and its independent page-HMAC verification; the latter reads the
+ main database file only, so it runs only once a PASSIVE checkpoint
+ has folded every WAL frame into that file.
+
+ Runs only against the current, epoch-validated cached engine
+ (``_acquire_current_connection_generation``); with no trusted open
+ connection it returns False rather than opening one.
+
+ Only the engine snapshot happens inside that claim; the scans run
+ after it is released. The claim holds the user's lifecycle lock,
+ in-process sidecar gate and sidecar flock, and every other probe for
+ this user (the shared log-drain thread, the idle-connection sweep,
+ each request's session lookup) waits on them, so a full-file
+ ``cipher_integrity_check`` under the claim stalled them all for the
+ length of the scan. A disposal that runs during the scan (a close,
+ the idle-connection sweep) still holds those locks while its
+ TRUNCATE checkpoint waits for the scan's read transaction, bounded
+ by ``_GATED_CHECKPOINT_BUSY_TIMEOUT_MS``. Accepted residual, the same as for any session
+ handed out by ``get_session``: a password change that commits after
+ the snapshot detaches and disposes the engine but does not stop this
+ check, whose connection keeps the old key. It can then return False
+ (an error or HMAC failures caused by the key change, not by
+ corruption); the engine itself is never handed to a caller.
"""
- with self._connections_lock:
- if username not in self.connections:
- return False
- engine = self.connections[username]
-
try:
+ with self._acquire_current_connection_generation(
+ username
+ ) as generation:
+ if generation is None:
+ return False
+ engine = generation.engine
+
with engine.connect() as conn:
# quick_check omits index/table consistency and UNIQUE
# checks. Check those too for the unencrypted fallback: the
@@ -1934,7 +3199,19 @@ class DatabaseManager:
# pragma with a closed result whose iteration raises; the
# except below turned that into "integrity check failed"
# for every healthy fallback database (#6357).
- if self.has_encryption:
+ #
+ # cipher_integrity_check reads the main database file,
+ # not the WAL. After a password change whose checkpoint
+ # a reader kept busy, the WAL holds new-key frames while
+ # the main file still holds those pages under the old
+ # key, so the scan reported every such page as an HMAC
+ # failure. Fold the WAL in first (PASSIVE: never waits
+ # on readers or writers) and scan only when every frame
+ # reached the main file; otherwise quick_check above,
+ # which reads through the WAL, is the verdict.
+ if self.has_encryption and self._wal_fully_checkpointed(
+ conn, username
+ ):
result = conn.execute(text("PRAGMA cipher_integrity_check"))
# If this returns any rows, there are HMAC failures
failures = list(result)
@@ -1973,10 +3250,15 @@ class DatabaseManager:
password-hash update is needed because passwords are never
stored. Login verification is done by attempting decryption.
"""
+ db_path = self._get_user_db_path(username)
if not self.has_encryption:
logger.warning(
"Cannot change password - SQLCipher not available (databases are unencrypted)"
)
+ # Unencrypted mode is a static deployment mode, not an
+ # operational failure: keep the documented boolean contract
+ # (tests/security/test_db_connection_unencrypted_mode.py) for
+ # the rekey path rather than mapping it to 503.
return False
# Validate BOTH keys before anything touches the database, mirroring
@@ -2001,109 +3283,208 @@ class DatabaseManager:
"Invalid encryption key: new password cannot be None or empty"
)
- db_path = self._get_user_db_path(username)
+ rekey_lock = self._get_password_change_lock(username)
+ if not rekey_lock.acquire(blocking=False):
+ lock_path = db_path.with_name(f"{db_path.name}.rekey.lock")
+ raise RekeyLockContentionError(lock_path)
- if not db_path.exists():
- return False
+ stage = RekeyStage.PREPARATION
+ pending_may_exist = False
+ committed = False
+ old_connection = None
+ new_connection = None
+ try:
+ # Same refusal as every other path that hands this pathname to
+ # SQLCipher (#6493): the rekey must not follow a planted link
+ # and write the rekeyed database outside the data directory.
+ # _db_path_is_symlinked logs the actionable evidence (path ->
+ # target) itself; nothing has been touched at this point.
+ if _db_path_is_symlinked(db_path, username):
+ return False
+ if not db_path.exists():
+ _raise_rekey_unavailable(db_path, stage)
- # Serialize the whole close -> open(old) -> rekey -> evict
- # sequence for this user. Built from public pieces, two concurrent
- # change_password calls for one user could otherwise interleave --
- # A opens with the old key, B disposes A's engine and opens with the
- # old key too, then both rekey -- leaving the file owned by whichever
- # rekey landed last while BOTH callers were told True, and the loser's
- # new password silently does not work. The lock makes at most one
- # rekey per user succeed; the loser's open(old) then fails against the
- # already-rekeyed file and it returns False. Per-user, and acquired
- # WITHOUT holding _connections_lock (open_user_database /
- # close_user_database take that themselves), so it neither blocks
- # other users nor inverts the lock order.
- with self._get_password_change_lock(username):
try:
- # Close existing connection if any
- self.close_user_database(username)
+ # Bounded wait for in-process holders on the sidecar gate
+ # (cache probes, but also cold opens and disposals, which
+ # can hold it for seconds), then a non-blocking flock
+ # retried for a short window so another process's
+ # millisecond cache probe is not reported as contention
+ # while a genuine cross-process owner still fails with
+ # the typed contention error. The dedicated
+ # change-password gate above already fast-fails
+ # concurrent rekeys in-process.
+ with acquire_rekey_lock(
+ db_path,
+ local_wait=10.0,
+ contention_retry=_REKEY_CONTENTION_RETRY_SECONDS,
+ ):
+ try:
+ state = read_rekey_epoch_state(db_path)
+ if (
+ state.pending_epoch is not None
+ and not self._recover_pending_rekey(
+ db_path, old_password, state
+ )
+ ):
+ return False
+ read_or_initialize_rekey_epoch(db_path)
+
+ cached_engine = self._detach_connection(username)
+ if cached_engine is not None and not (
+ self._dispose_detached_connection(
+ username, cached_engine, checkpoint=False
+ )
+ ):
+ _raise_rekey_unavailable(db_path, stage)
- # Open with old password
- engine = self.open_user_database(username, old_password)
- if not engine:
- return False
+ try:
+ old_connection = self._make_sqlcipher_connection(
+ db_path, old_password
+ )
+ except ValueError:
+ return False
+ self._require_full_synchronous(old_connection)
+ self._pre_rekey_checkpoint(old_connection)
+
+ stage = RekeyStage.PENDING_PUBLICATION
+ pending_may_exist = True
+ pending_epoch = begin_rekey_epoch(db_path)
+
+ stage = RekeyStage.PHYSICAL_REKEY
+ set_sqlcipher_rekey(
+ old_connection, new_password, db_path=db_path
+ )
+ old_connection.commit()
+ # Never write through old_connection after the
+ # rekey: when PRAGMA rekey did not apply (see
+ # _resolve_unapplied_rekey) its write context is
+ # already on the NEW key while the file stays under
+ # the old one, and one INSERT there leaves pages
+ # that neither key opens. It is closed unused.
+ self._close_rekey_connection(
+ old_connection, username, "old-key"
+ )
+ old_connection = None
- # Rekey the database (only works with SQLCipher)
- with engine.connect() as conn:
- # Use centralized rekey function
- set_sqlcipher_rekey(conn, new_password, db_path=db_path)
-
- # Advance the rekey generation BEFORE the eviction below. A
- # concurrent open_user_database(old) that verified against
- # the still-old-keyed file and is parked before its cache
- # publish snapshots the pre-rekey generation; this bump makes
- # that publish mismatch and be refused by _cache_connection
- # (fail CLOSED with dispose + None) instead of planting a
- # stale OLD-key engine into an emptied cache after the rekey
- # -- the window eviction-at-close cannot close when an
- # external unconditional close (logout / idle sweep) emptied
- # the cache mid-rekey so both conditional closes no-op. A
- # concurrent open_user_database(new) that verifies AFTER this
- # point snapshots the new generation and publishes normally.
- self._bump_rekey_epoch(username)
-
- # Evict the stale engine we opened above, but ONLY if it is
- # still the one in the cache. open_user_database() above
- # cold-opened with old_password and cached that engine with a
- # verifier for old_password. The rekey just invalidated that
- # engine's key (its creator closure still derives the OLD hex
- # key, so any freshly pooled connection is mis-keyed) yet the
- # cached verifier still matches old_password -- so until this
- # eviction a concurrent open_user_database(old_password)
- # would pass the verifier check and be handed the stale-key
- # engine.
- #
- # A concurrent open_user_database(new_password) racing the
- # rekey, however, legitimately succeeds (the file IS keyed
- # with new_password now) and caches a fresh engine with the
- # new verifier. That engine is the surviving current-key
- # engine; evicting it would force the next caller to
- # re-cold-open AND would break the invariant "the engine
- # cached for a user is the one a hammer racing this rekey
- # observed" (test_stress_change_password_chain_vs_open_hammers
- # pins exactly that). The per-user lock above only excludes
- # another change_password -- plain opens run concurrently
- # throughout -- so evict only our engine here and let a
- # concurrent valid engine stay. close_user_database_if_stale
- # matches by verifier (the OLD one we just opened with), not
- # by identity -- an engine a concurrent open published
- # carries a DIFFERENT verifier (new_password) and is left
- # alone. The finally-close below stays a backstop for the
- # exception path; it uses the same conditional close.
- self.close_user_database_if_stale(username, old_password)
+ stage = RekeyStage.CANDIDATE_VERIFICATION
+ try:
+ new_connection = self._make_sqlcipher_connection(
+ db_path, new_password
+ )
+ except ValueError:
+ self._resolve_unapplied_rekey(
+ db_path, old_password, pending_epoch
+ )
+ self._verify_rekey_candidate(new_connection, db_path)
+
+ stage = RekeyStage.COMMIT_PUBLICATION
+ # The new key is proven at this point: it opened
+ # and decrypted the database, and a database state
+ # never opens under both keys. The password change
+ # has therefore happened, whatever the epoch files
+ # say, and reporting it as not completed would send
+ # the user back to an old password that no longer
+ # works while skipping the caller's revocation of
+ # every old-password session and credential.
+ epoch_published = True
+ try:
+ commit_rekey_epoch(db_path, pending_epoch)
+ except Exception as error:
+ try:
+ committed_state = read_rekey_epoch_state(
+ db_path
+ )
+ except Exception:
+ committed_state = None
+ if committed_state != RekeyEpochState(
+ committed_epoch=pending_epoch,
+ pending_epoch=pending_epoch,
+ ):
+ # The pending file stays on disk (only this
+ # call removes it, after the commit), so the
+ # state remains fenced: read_rekey_epoch
+ # reports no stable epoch while it exists,
+ # which invalidates every cached engine, and
+ # every cold open runs _recover_pending_rekey
+ # first. That recovery rejects the old
+ # password (it no longer opens the file) and,
+ # for the new one, re-proves the key and
+ # publishes this same pending epoch.
+ epoch_published = False
+ logger.warning(
+ f"Rekey epoch publication failed for "
+ f"{username} after the new key was "
+ f"verified ({type(error).__name__}); "
+ "the password change is complete and "
+ "the pending generation will be "
+ "committed by the next successful open"
+ )
+
+ committed = True
+ if epoch_published:
+ self._clear_committed_rekey_pending(
+ db_path, pending_epoch, username
+ )
+ self._close_rekey_connection(
+ new_connection, username, "new-key"
+ )
+ new_connection = None
+ except (
+ RekeyIndeterminateError,
+ RekeyUnavailableError,
+ ):
+ raise
+ except RekeyRecoveryRequiredError:
+ if pending_may_exist:
+ raise self._classify_rekey_failure(
+ db_path, stage
+ ) from None
+ raise
+ except Exception:
+ if pending_may_exist:
+ raise self._classify_rekey_failure(
+ db_path, stage
+ ) from None
+ _raise_rekey_unavailable(db_path, stage)
logger.info(f"Password changed for user {username}")
return True
+ except RekeyLockContentionError:
+ raise
+ except (
+ RekeyIndeterminateError,
+ RekeyRecoveryRequiredError,
+ RekeyUnavailableError,
+ ):
+ raise
+ except Exception as error:
+ if committed:
+ logger.warning(
+ f"Post-commit rekey cleanup failed for {username}: "
+ f"{type(error).__name__}"
+ )
+ logger.info(f"Password changed for user {username}")
+ return True
- except Exception as e:
- # Both ``old_password`` and ``new_password`` are in lexical
- # scope. ``open_user_database`` / ``set_sqlcipher_rekey``
- # carry these into nested frames — a traceback rendered
- # with ``diagnose=True`` would leak them. Redact both and
- # drop the traceback chain.
- from ..security.log_sanitizer import redact_secrets
-
- safe_msg = redact_secrets(str(e), old_password, new_password)
logger.warning(
- f"Failed to change password for user: {username}: {safe_msg}"
+ f"Failed to change password for user {username} during "
+ f"{stage.value}: {type(error).__name__}"
)
- return False
+ if pending_may_exist:
+ raise self._classify_rekey_failure(db_path, stage) from None
+ _raise_rekey_unavailable(db_path, stage)
finally:
- # Backstop for the exception path: if the rekey threw and the
- # in-try conditional close did not run, drop any stale engine
- # (its verifier matches old_password, so its key was
- # rekeyed away on the success path or it never rekeyed at
- # all on the failure path; either way the file is no longer
- # keyed with old_password under the success case). Same
- # conditional as the in-try call so a concurrent
- # open_user_database(new_password) racing the success path
- # is not clobbered.
- self.close_user_database_if_stale(username, old_password)
+ if old_connection is not None:
+ self._close_rekey_connection(
+ old_connection, username, "old-key"
+ )
+ if new_connection is not None:
+ self._close_rekey_connection(
+ new_connection, username, "new-key"
+ )
+ finally:
+ rekey_lock.release()
def user_exists(self, username: str) -> bool:
"""Check if a user exists in the auth database."""
@@ -2152,6 +3533,21 @@ class DatabaseManager:
Returns:
A SQLAlchemy Session bound to the per-user QueuePool engine.
+
+ Raises:
+ RekeyOperationalError: The database is unavailable for an
+ operational reason (the rekey sidecar or an epoch file cannot
+ be read, or the durable rekey state needs recovery), raised
+ the way ``open_user_database`` raises it: before any key is
+ derived when a cached engine cannot be checked. Not a
+ credential verdict, so it is not folded into a None that
+ reads like one. ``ThreadSafeMetricsWriter.get_session``
+ reports it as its own "no session" failure;
+ ``ThreadLocalSessionManager.get_session`` lets it reach
+ ``get_user_db_session``, which raises
+ ``DatabaseSessionUnavailableError``.
+ ValueError: The database path is a symlink or missing, or
+ ``password`` does not open the database.
"""
db_path = self._get_user_db_path(username)
@@ -2164,28 +3560,32 @@ class DatabaseManager:
if not db_path.exists():
raise ValueError(f"No database found for user {username}")
- # Trust the cached engine only when the presented password matches the
- # one that opened it -- the same fail-closed rule as
+ # Trust the cached engine only when the presented password matches
+ # the one that opened it -- the same fail-closed rule as
# open_user_database. Returning a session on a bare cache hit here
- # would be a second instance of exactly the cached-connection password
- # bypass this change closes; falling through to open_user_database on a
- # mismatch re-validates the key against SQLCipher.
- with self._connections_lock:
- engine = self.connections.get(username)
- if engine is not None and not self._cached_engine_trusted(
- username, password
+ # would be a second instance of exactly the cached-connection
+ # password bypass this change closes; falling through to
+ # open_user_database on a mismatch re-validates the key against
+ # SQLCipher.
+ with self._acquire_current_connection_generation(
+ username
+ ) as generation:
+ if generation is not None and (
+ self._connection_generation_password_matches(
+ generation, password
+ )
):
- engine = None
-
+ Session = sessionmaker(bind=generation.engine)
+ return Session()
+
+ # A cached engine that the probe above could not check (a rekey
+ # sidecar or epoch-read fault) makes this raise that typed fault
+ # without deriving a key. It propagates (see Raises): each caller
+ # decides what an operational fault means for it.
+ engine = self.open_user_database(username, password)
if engine is None:
- # Cache miss (or a rejected cache hit) — open the user database.
- # This is idempotent: after the first call it just returns the
- # cached engine.
- engine = self.open_user_database(username, password)
- if engine is None:
- raise ValueError(f"Failed to open database for user {username}")
+ raise ValueError(f"Failed to open database for user {username}")
- # Use SQLAlchemy's default expire_on_commit=True.
Session = sessionmaker(bind=engine)
return Session()
diff --git a/src/local_deep_research/database/rekey_epoch.py b/src/local_deep_research/database/rekey_epoch.py
new file mode 100644
index 0000000000..91e8caeaf5
--- /dev/null
+++ b/src/local_deep_research/database/rekey_epoch.py
@@ -0,0 +1,41 @@
+"""Durable authorization identity projection for encrypted user databases.
+
+Nothing in the application imports this module yet: it is the read side
+for session and socket identity fencing, which the FastAPI layer does not
+perform yet (a planned follow-up). Its readers do not take the rekey
+sidecar lock, which is why ``read_rekey_epoch_state`` reads pending-first.
+"""
+
+from pathlib import Path
+
+from .rekey_lock import RekeyRecoveryRequiredError, read_rekey_epoch_state
+
+
+def read_authorization_epoch(db_path: Path) -> str:
+ """Project the epoch that may authorize identities without repairing state."""
+ state = read_rekey_epoch_state(db_path)
+ committed_epoch = state.committed_epoch
+ if committed_epoch is None or (
+ state.pending_epoch is not None
+ and state.pending_epoch != committed_epoch
+ ):
+ raise RekeyRecoveryRequiredError(db_path)
+ return committed_epoch
+
+
+def read_user_authorization_epoch(username: str) -> str:
+ """Resolve a user's database path and project its authorization epoch.
+
+ The path MUST be resolved through ``db_manager`` (lazy-imported to
+ keep this module's import graph shallow): the manager may carry a
+ pinned ``data_dir`` override while ``get_data_directory()`` resolves
+ dynamically from the environment, and epochs are written beside the
+ databases under the MANAGER's resolution. Reading via the env-only
+ path projected a different directory whenever the two diverged
+ (any process that pinned the override), which would fence perfectly
+ healthy sessions with ``RekeyRecoveryRequiredError``.
+ """
+ from .encrypted_db import db_manager
+
+ db_path = db_manager._get_user_db_path(username)
+ return read_authorization_epoch(db_path)
diff --git a/src/local_deep_research/database/rekey_lock.py b/src/local_deep_research/database/rekey_lock.py
new file mode 100644
index 0000000000..36be1d7668
--- /dev/null
+++ b/src/local_deep_research/database/rekey_lock.py
@@ -0,0 +1,928 @@
+"""Cross-process serialization for SQLCipher database rekeys."""
+
+from __future__ import annotations
+
+import errno
+import os
+import stat
+import tempfile
+import threading
+import time
+from collections.abc import Generator
+from contextlib import contextmanager
+from dataclasses import dataclass
+from enum import StrEnum, unique
+from pathlib import Path
+from typing import assert_never
+from uuid import UUID, uuid4
+
+
+class RekeyOperationalError(Exception):
+ """Base for explicit operational failures during password rekey."""
+
+
+@unique
+class AuthorizationLeaseMode(StrEnum):
+ """Cross-process authorization lease compatibility modes."""
+
+ SHARED = "shared"
+ EXCLUSIVE = "exclusive"
+
+
+@unique
+class RekeySidecarOperation(StrEnum):
+ """Safe lifecycle stages for sidecar I/O diagnostics."""
+
+ OPEN = "open"
+ ACQUIRE = "acquire"
+ UNLOCK = "unlock"
+ CLOSE = "close"
+ # Writing an epoch token (``.rekey.epoch`` / ``.rekey.pending``) while
+ # the sidecar is held: mkstemp, write+fsync, replace, directory sync.
+ PUBLISH = "publish"
+
+
+# Errnos this module raises itself when it refuses a planted sidecar.
+_SIDECAR_REFUSAL_DETAIL: dict[int | None, str] = {
+ errno.ELOOP: "; refusing a symlinked sidecar",
+ errno.EINVAL: "; refusing a non-regular sidecar",
+ # st_nlink == 0 (an already-unlinked inode), or a hard link to a
+ # database file or any other file in the sidecar's directory
+ errno.EMLINK: "; refusing an unlinked sidecar or one hard-linked within its directory",
+}
+
+
+class RekeySidecarIOError(RekeyOperationalError):
+ """Normalized sidecar I/O failure without raw operating-system text."""
+
+ db_path: Path
+ operation: RekeySidecarOperation
+ errno: int | None
+
+ def __init__(
+ self,
+ db_path: Path,
+ operation: RekeySidecarOperation,
+ error_number: int | None,
+ ) -> None:
+ self.db_path = db_path
+ self.operation = operation
+ self.errno = error_number
+ errno_text = "unknown" if error_number is None else str(error_number)
+ detail = _SIDECAR_REFUSAL_DETAIL.get(error_number, "")
+ super().__init__(
+ f"rekey sidecar {operation.value} failed "
+ f"(errno={errno_text}{detail})"
+ )
+
+
+@unique
+class RekeyStage(StrEnum):
+ """Closed stages used to classify password-rekey outcomes."""
+
+ PREPARATION = "preparation"
+ PENDING_PUBLICATION = "pending-publication"
+ PHYSICAL_REKEY = "physical-rekey"
+ CANDIDATE_VERIFICATION = "candidate-verification"
+ COMMIT_PUBLICATION = "commit-publication"
+
+
+class RekeyUnavailableError(RekeyOperationalError):
+ """Raised when rekey cannot proceed before its outcome becomes ambiguous."""
+
+ db_path: Path
+ stage: RekeyStage
+
+ def __init__(self, db_path: Path, stage: RekeyStage) -> None:
+ self.db_path = db_path
+ self.stage = stage
+ super().__init__(
+ f"rekey unavailable during {stage.value} for {db_path.name}"
+ )
+
+
+class RekeyIndeterminateError(RekeyOperationalError):
+ """Raised when durable state cannot prove which password remains active."""
+
+ db_path: Path
+ stage: RekeyStage
+ state: RekeyEpochState | None
+
+ def __init__(
+ self,
+ db_path: Path,
+ stage: RekeyStage,
+ state: RekeyEpochState | None,
+ ) -> None:
+ self.db_path = db_path
+ self.stage = stage
+ self.state = state
+ super().__init__(
+ f"rekey outcome indeterminate during {stage.value} for {db_path.name}"
+ )
+
+
+class RekeyLockContentionError(RekeyOperationalError):
+ """Raised when another password rekey already owns the database lock."""
+
+ lock_path: Path
+
+ def __init__(self, lock_path: Path) -> None:
+ self.lock_path = lock_path
+ super().__init__(f"rekey already in progress for {lock_path.name}")
+
+
+class DatabaseKeyCheckUnavailableError(RekeyOperationalError):
+ """The password could not be checked against the database right now.
+
+ Raised when opening the database to verify a password fails for an
+ operational reason (SQLite reports the database locked or busy past the
+ busy timeout, an I/O error, a file it cannot open), as opposed to the
+ key being wrong ("file is not a database"). It says nothing about the
+ password, so a login answers 503 without a lockout strike and a password
+ change answers with its retryable failure, never "password incorrect".
+ Only the SQLite exception type is kept, not its message.
+ """
+
+ db_path: Path
+ reason: str
+
+ def __init__(self, db_path: Path, reason: str) -> None:
+ self.db_path = db_path
+ self.reason = reason
+ super().__init__(
+ f"password check unavailable for {db_path.name} ({reason})"
+ )
+
+
+class RekeyRecoveryRequiredError(RekeyOperationalError):
+ """Raised when durable rekey metadata requires verified recovery."""
+
+ db_path: Path
+
+ def __init__(self, db_path: Path) -> None:
+ self.db_path = db_path
+ super().__init__(f"rekey recovery required for {db_path.name}")
+
+
+class RekeyEpochUnreadableError(RekeyRecoveryRequiredError):
+ """An epoch file exists (or may) but the read itself failed.
+
+ Raised for every ``OSError`` other than ``FileNotFoundError`` while
+ reading ``.rekey.epoch`` / ``.rekey.pending`` (EMFILE, ENFILE, EAGAIN,
+ EIO, EACCES, ...; EINTR never gets here, Python retries it itself per
+ PEP 475), and when the reader refuses a planted file without reading
+ it: a symlink (ELOOP) or anything that is not a regular file, such as
+ a FIFO or a directory (EINVAL). It is a ``RekeyRecoveryRequiredError``, so
+ every existing handler keeps failing closed exactly as before; the
+ subclass only lets a caller that can retry later (session validation)
+ tell "nothing could be read" apart from durable state that was read
+ and does not vouch for the identity. ``errno`` is kept for
+ diagnostics; the raw OS message is not.
+ """
+
+ errno: int | None
+
+ def __init__(self, db_path: Path, error_number: int | None) -> None:
+ super().__init__(db_path)
+ self.errno = error_number
+
+
+@dataclass(frozen=True, slots=True)
+class RekeyEpochState:
+ """Durable committed and in-flight rekey generations."""
+
+ committed_epoch: str | None
+ pending_epoch: str | None
+
+
+@dataclass(frozen=True, slots=True)
+class _SidecarLockRequest:
+ db_path: Path
+ lock_path: Path
+ mode: AuthorizationLeaseMode
+ blocking: bool
+ gate_wait: float | None = None
+ contention_retry: float = 0.0
+ # time.monotonic() value bounding the gate wait and the flock retries
+ # together; see acquire_rekey_lock's ``timeout``.
+ deadline: float | None = None
+
+
+# Poll interval while a non-blocking caller rides out a brief foreign hold.
+_CONTENTION_RETRY_INTERVAL = 0.01
+
+
+# In-process gate per sidecar file. flock semantics make two fds in one
+# process contend like unrelated processes, so a heavyweight sidecar user
+# (change_password) cannot tell "an in-process holder has the lock" from
+# "a cross-process rekey holds it" and either fails fast on both or blocks
+# on both. In-process holders are not all brief: cache probes hold it for
+# milliseconds, but a cold open holds it across migrations and engine
+# build, and a disposal across its WAL checkpoint (seconds on a large
+# database). Every in-process flock holder therefore takes this gate
+# first: heavyweight users wait on the gate (bounded) for local holders,
+# then take the flock non-blocking (with a short retry for other
+# processes' probes) so genuinely foreign holders still fail with the
+# typed contention error. Request-path callers bound the gate wait and
+# the flock retries together, by one deadline (``timeout``).
+_sidecar_gates: dict[Path, threading.Lock] = {}
+_sidecar_gates_guard = threading.Lock()
+
+
+def _get_sidecar_gate(lock_path: Path) -> threading.Lock:
+ with _sidecar_gates_guard:
+ lock = _sidecar_gates.get(lock_path)
+ if lock is None:
+ lock = threading.Lock()
+ _sidecar_gates[lock_path] = lock
+ return lock
+
+
+def _acquire_posix_lock(
+ fd: int,
+ lock_path: Path,
+ *,
+ mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,
+ blocking: bool = False,
+) -> None:
+ import fcntl
+
+ match mode:
+ case AuthorizationLeaseMode.SHARED:
+ operation = fcntl.LOCK_SH
+ case AuthorizationLeaseMode.EXCLUSIVE:
+ operation = fcntl.LOCK_EX
+ case _ as unreachable:
+ assert_never(unreachable)
+ if not blocking:
+ operation |= fcntl.LOCK_NB
+ try:
+ fcntl.flock(fd, operation)
+ except OSError as error:
+ if error.errno in {errno.EACCES, errno.EAGAIN}:
+ contention_error = RekeyLockContentionError(lock_path)
+ else:
+ raise
+ else:
+ return
+ raise contention_error
+
+
+def _release_posix_lock(fd: int) -> None:
+ import fcntl
+
+ fcntl.flock(fd, fcntl.LOCK_UN)
+
+
+def _acquire_windows_lock(
+ fd: int,
+ lock_path: Path,
+ *,
+ mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,
+ blocking: bool = False,
+) -> None:
+ import msvcrt
+
+ if os.fstat(fd).st_size == 0:
+ _ = os.lseek(fd, 0, os.SEEK_SET)
+ _ = os.write(fd, b"\0")
+ _ = os.lseek(fd, 0, os.SEEK_SET)
+ try:
+ match mode:
+ case AuthorizationLeaseMode.SHARED:
+ lock_mode = msvcrt.LK_RLCK if blocking else msvcrt.LK_NBRLCK
+ case AuthorizationLeaseMode.EXCLUSIVE:
+ lock_mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCK
+ case _ as unreachable:
+ assert_never(unreachable)
+ msvcrt.locking(fd, lock_mode, 1)
+ except OSError as error:
+ if error.errno in {
+ errno.EACCES,
+ errno.EAGAIN,
+ errno.EDEADLK,
+ } or getattr(error, "winerror", None) in {33, 36}:
+ contention_error = RekeyLockContentionError(lock_path)
+ else:
+ raise
+ else:
+ return
+ raise contention_error
+
+
+def _release_windows_lock(fd: int) -> None:
+ import msvcrt
+
+ _ = os.lseek(fd, 0, os.SEEK_SET)
+ msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
+
+
+def _authorization_lock_path(db_path: Path) -> Path:
+ return db_path.with_name(f"{db_path.name}.rekey.authorization.lock")
+
+
+def _sidecar_io_error(
+ request: _SidecarLockRequest,
+ operation: RekeySidecarOperation,
+ error: OSError,
+) -> RekeySidecarIOError:
+ return RekeySidecarIOError(request.db_path, operation, error.errno)
+
+
+def _open_sidecar(lock_request: _SidecarLockRequest) -> int:
+ # O_NOFOLLOW: a symlink planted at the sidecar name must not redirect the
+ # open (or the permission tightening below) onto an unrelated file the
+ # process can write; it fails with ELOOP and surfaces as a typed OPEN
+ # error. The mode is tightened through the fd for the same reason.
+ flags = os.O_CREAT | os.O_RDWR | getattr(os, "O_NOFOLLOW", 0)
+ # Checked BEFORE the open: closing any descriptor to the database (or
+ # its WAL/shm) releases every POSIX lock this process holds on it behind
+ # SQLite's back, so a sidecar hard-linked to one of them must never be
+ # opened at all, not even to refuse it.
+ try:
+ planted = os.lstat(lock_request.lock_path)
+ except OSError:
+ planted = None
+ if planted is not None and _is_database_file(lock_request, planted):
+ raise RekeySidecarIOError(
+ lock_request.db_path, RekeySidecarOperation.OPEN, errno.EMLINK
+ )
+ try:
+ fd = os.open(lock_request.lock_path, flags, 0o600)
+ except OSError as error:
+ open_error = _sidecar_io_error(
+ lock_request, RekeySidecarOperation.OPEN, error
+ )
+ else:
+ # O_NOFOLLOW does not catch a HARD link planted at the sidecar name:
+ # the open lands on the linked inode and fchmod would tighten that
+ # unrelated file. A hard-linked sidecar is still locked (an advisory
+ # lock changes nothing in the file, and refusing it locked the user
+ # out whenever a backup or snapshot tool hard-linked the data
+ # directory), but its mode is left alone.
+ refusal = _refuse_foreign_sidecar(lock_request, fd)
+ if refusal is not None:
+ _close_refused_sidecar(fd)
+ raise refusal
+ try:
+ hard_linked = os.fstat(fd).st_nlink > 1
+ except OSError:
+ hard_linked = True
+ fchmod = None if hard_linked else getattr(os, "fchmod", None)
+ if fchmod is not None:
+ try:
+ fchmod(fd, 0o600)
+ except OSError:
+ return fd
+ return fd
+ raise open_error
+
+
+def _is_database_file(
+ request: _SidecarLockRequest, status: os.stat_result
+) -> bool:
+ """True when ``status`` may be the inode of a database file.
+
+ A lock file whose inode is also a database, WAL or shm file (this
+ user's or ANOTHER user's in the same directory) must never be opened:
+ closing the lock's descriptor releases every POSIX lock this process
+ holds on that inode behind SQLite's back. Only an inode with more than
+ one link can be one. This user's own files are stat'ed directly; every
+ other entry of the sidecar's directory is then compared without
+ following symlinks, so any other name for the inode in that directory
+ refuses it. Links only outside the directory (a ``cp -al`` snapshot of
+ the data directory) stay accepted. A directory that cannot be listed
+ refuses too: nothing then shows the inode is not a database.
+ """
+ if status.st_nlink < 2:
+ return False
+ identity = (status.st_dev, status.st_ino)
+ db_path = request.db_path
+ for path in (
+ db_path,
+ db_path.with_name(f"{db_path.name}-wal"),
+ db_path.with_name(f"{db_path.name}-shm"),
+ ):
+ try:
+ other = os.stat(path)
+ except OSError:
+ continue
+ if (other.st_dev, other.st_ino) == identity:
+ return True
+ lock_name = request.lock_path.name
+ try:
+ with os.scandir(request.lock_path.parent) as entries:
+ for entry in entries:
+ if entry.name == lock_name:
+ continue
+ try:
+ other = entry.stat(follow_symlinks=False)
+ except OSError:
+ continue
+ if (other.st_dev, other.st_ino) == identity:
+ return True
+ except OSError:
+ return True
+ return False
+
+
+def _refuse_foreign_sidecar(
+ request: _SidecarLockRequest, fd: int
+) -> RekeySidecarIOError | None:
+ try:
+ status = os.fstat(fd)
+ except OSError as error:
+ return _sidecar_io_error(request, RekeySidecarOperation.OPEN, error)
+ if not stat.S_ISREG(status.st_mode):
+ return RekeySidecarIOError(
+ request.db_path, RekeySidecarOperation.OPEN, errno.EINVAL
+ )
+ if _is_database_file(request, status):
+ # Planted between the check before the open and the open itself:
+ # refused here as well, although the close that follows may
+ # already have dropped this process's locks on the database.
+ return RekeySidecarIOError(
+ request.db_path, RekeySidecarOperation.OPEN, errno.EMLINK
+ )
+ if status.st_nlink == 0:
+ # Already unlinked: a lock on it would exclude nobody who opens the
+ # name afterwards.
+ return RekeySidecarIOError(
+ request.db_path, RekeySidecarOperation.OPEN, errno.EMLINK
+ )
+ return None
+
+
+def _close_refused_sidecar(fd: int) -> None:
+ try:
+ os.close(fd)
+ except OSError:
+ return
+
+
+@contextmanager
+def _acquire_sidecar_lock(
+ request: _SidecarLockRequest,
+) -> Generator[None, None, None]:
+ """Hold one sidecar lock while preserving protected-body failures."""
+ gate = _get_sidecar_gate(request.lock_path)
+ if request.deadline is not None:
+ gate_timeout: float | None = max(
+ 0.0, request.deadline - time.monotonic()
+ )
+ elif request.gate_wait is not None:
+ gate_timeout = request.gate_wait
+ elif request.blocking:
+ gate_timeout = None
+ else:
+ gate_timeout = 0.0
+ if gate_timeout is None:
+ acquired = gate.acquire()
+ else:
+ acquired = gate.acquire(timeout=gate_timeout)
+ if not acquired:
+ raise RekeyLockContentionError(request.lock_path)
+ try:
+ yield from _acquire_sidecar_lock_locked(request)
+ finally:
+ gate.release()
+
+
+def _acquire_os_lock_with_retry(fd: int, request: _SidecarLockRequest) -> None:
+ """Take the OS lock, retrying a non-blocking attempt for a short window.
+
+ Every cache probe in any process (``is_user_connected``/``get_session``)
+ takes this flock for a few milliseconds. A single non-blocking attempt
+ therefore reported spurious contention whenever it collided with one of
+ those probes; ``contention_retry`` absorbs such brief holds while a
+ genuine long-lived owner (another process's rekey) still fails fast.
+ A request with a ``deadline`` retries until that deadline instead.
+ """
+ if request.deadline is not None:
+ deadline = request.deadline
+ else:
+ deadline = time.monotonic() + request.contention_retry
+ while True:
+ try:
+ if os.name == "nt":
+ _acquire_windows_lock(
+ fd,
+ request.lock_path,
+ mode=request.mode,
+ blocking=request.blocking,
+ )
+ else:
+ _acquire_posix_lock(
+ fd,
+ request.lock_path,
+ mode=request.mode,
+ blocking=request.blocking,
+ )
+ except RekeyLockContentionError:
+ if request.blocking or time.monotonic() >= deadline:
+ raise
+ time.sleep(_CONTENTION_RETRY_INTERVAL)
+ else:
+ return
+
+
+def _acquire_sidecar_lock_locked(
+ request: _SidecarLockRequest,
+) -> Generator[None, None, None]:
+ fd = _open_sidecar(request)
+ acquired = False
+ # The failure of the acquire or of the protected body, recorded as it
+ # passes through. A release failure is added to it as a note, and is
+ # raised itself only when there is none. Not sys.exception() in the
+ # finally below: after a clean body, this generator is resumed from
+ # its caller's frame, and when that caller is inside an except block
+ # sys.exception() returns the exception IT is handling, so a failed
+ # unlock or close was attached to that unrelated exception as a note
+ # and never raised.
+ active_error: BaseException | None = None
+ try:
+ acquire_error: RekeySidecarIOError | None = None
+ try:
+ _acquire_os_lock_with_retry(fd, request)
+ except OSError as error:
+ acquire_error = _sidecar_io_error(
+ request, RekeySidecarOperation.ACQUIRE, error
+ )
+ if acquire_error is not None:
+ raise acquire_error # noqa: TRY301 — recorded as active_error by the except below
+ acquired = True
+ yield
+ except BaseException as failure:
+ active_error = failure
+ raise
+ finally:
+ cleanup_errors: list[RekeySidecarIOError] = []
+ if acquired:
+ try:
+ if os.name == "nt":
+ _release_windows_lock(fd)
+ else:
+ _release_posix_lock(fd)
+ except OSError as error:
+ cleanup_errors.append(
+ _sidecar_io_error(
+ request, RekeySidecarOperation.UNLOCK, error
+ )
+ )
+ try:
+ os.close(fd)
+ except OSError as error:
+ cleanup_errors.append(
+ _sidecar_io_error(request, RekeySidecarOperation.CLOSE, error)
+ )
+
+ if active_error is not None:
+ for cleanup_error in cleanup_errors:
+ active_error.add_note(str(cleanup_error))
+ elif cleanup_errors:
+ primary_error = cleanup_errors[0]
+ for cleanup_error in cleanup_errors[1:]:
+ primary_error.add_note(str(cleanup_error))
+ raise primary_error
+
+
+@contextmanager
+def acquire_rekey_lock(
+ db_path: Path,
+ *,
+ blocking: bool = False,
+ local_wait: float | None = None,
+ contention_retry: float = 0.0,
+ timeout: float | None = None,
+) -> Generator[None, None, None]:
+ """Hold the database's persistent, owner-only rekey sidecar lock.
+
+ ``local_wait`` serves the heavyweight caller (change_password): wait up
+ to that many seconds for IN-PROCESS holders via the sidecar gate. Those
+ holders are not only brief cache probes: a cold open keeps the gate
+ through key derivation, alembic migrations and engine build, and an
+ engine disposal through its WAL checkpoint, so the wait is bounded
+ rather than assumed short. The flock is then taken non-blocking,
+ retried for at most ``contention_retry`` seconds to ride out other
+ processes' millisecond cache probes, so a genuine cross-process owner
+ still fails fast with ``RekeyLockContentionError``.
+
+ ``timeout`` bounds the whole acquisition instead, for the callers on a
+ request's path (``DatabaseManager``'s cache claim, cold open and
+ registration): at most that many seconds in total, first on the gate
+ for in-process holders, then on the flock, which is retried
+ non-blocking until the same deadline. A holder keeps the sidecar for as
+ long as its work runs (a rekey or a cold open of a large database), and
+ a stopped process or a hung mount keeps it indefinitely, so the wait
+ ends there with ``RekeyLockContentionError``. It overrides ``blocking``,
+ ``local_wait`` and ``contention_retry``.
+ """
+ request = _SidecarLockRequest(
+ db_path=db_path,
+ lock_path=db_path.with_name(f"{db_path.name}.rekey.lock"),
+ mode=AuthorizationLeaseMode.EXCLUSIVE,
+ blocking=blocking and timeout is None,
+ gate_wait=local_wait,
+ contention_retry=contention_retry,
+ deadline=None if timeout is None else time.monotonic() + timeout,
+ )
+ with _acquire_sidecar_lock(request):
+ yield
+
+
+@contextmanager
+def acquire_authorization_lease(
+ db_path: Path,
+ mode: AuthorizationLeaseMode,
+ blocking: bool = True,
+) -> Generator[None, None, None]:
+ """Hold a persistent shared or exclusive authorization sidecar lease."""
+ request = _SidecarLockRequest(
+ db_path=db_path,
+ lock_path=_authorization_lock_path(db_path),
+ mode=mode,
+ blocking=blocking,
+ )
+ with _acquire_sidecar_lock(request):
+ yield
+
+
+def _epoch_path(db_path: Path) -> Path:
+ return db_path.with_name(f"{db_path.name}.rekey.epoch")
+
+
+def _pending_epoch_path(db_path: Path) -> Path:
+ return db_path.with_name(f"{db_path.name}.rekey.pending")
+
+
+# Every token this module writes is a canonical UUID string: 36 ASCII bytes.
+_EPOCH_TOKEN_MAX_BYTES = len(str(UUID(int=0)))
+
+
+def _read_epoch_file(fd: int) -> bytes:
+ """Read at most one byte past a token, so an oversized file is detected."""
+ data = b""
+ while len(data) <= _EPOCH_TOKEN_MAX_BYTES:
+ chunk = os.read(fd, _EPOCH_TOKEN_MAX_BYTES + 1 - len(data))
+ if not chunk:
+ break
+ data += chunk
+ return data
+
+
+def _close_epoch_file(fd: int) -> None:
+ # Read-only descriptor: a failed close loses nothing that was read.
+ try:
+ os.close(fd)
+ except OSError:
+ return
+
+
+def _read_epoch_token(path: Path, db_path: Path) -> str | None:
+ # The guard _refuse_foreign_sidecar gives the lock sidecar, for these
+ # reads: they run while the caller holds the sidecar flock (every
+ # authenticated probe does), so a file planted at the token name must
+ # fail the read, never hang it or fill memory. O_NOFOLLOW refuses a
+ # symlink (ELOOP). O_NONBLOCK makes the open of a FIFO return at once
+ # instead of waiting for a writer, and the fstat below refuses every
+ # file that is not regular (EINVAL); both are "nothing was read". A
+ # regular file longer than a token is read no further than one byte
+ # past it, and does not vouch, like any other malformed token. The link
+ # count is not checked: unlike the sidecar's fchmod, a read-only open
+ # cannot change the file a hard link points at.
+ flags = (
+ os.O_RDONLY
+ | getattr(os, "O_NOFOLLOW", 0)
+ | getattr(os, "O_NONBLOCK", 0)
+ | getattr(os, "O_BINARY", 0)
+ )
+ try:
+ fd = os.open(path, flags)
+ except FileNotFoundError:
+ return None
+ except OSError as error:
+ raise RekeyEpochUnreadableError(db_path, error.errno) from None
+ try:
+ status = os.fstat(fd)
+ regular = stat.S_ISREG(status.st_mode)
+ oversized = status.st_size > _EPOCH_TOKEN_MAX_BYTES
+ raw = _read_epoch_file(fd) if regular and not oversized else b""
+ except OSError as error:
+ raise RekeyEpochUnreadableError(db_path, error.errno) from None
+ finally:
+ _close_epoch_file(fd)
+ if not regular:
+ raise RekeyEpochUnreadableError(db_path, errno.EINVAL)
+ if oversized or len(raw) > _EPOCH_TOKEN_MAX_BYTES:
+ raise RekeyRecoveryRequiredError(db_path)
+ try:
+ token = raw.decode("ascii")
+ except UnicodeError:
+ raise RekeyRecoveryRequiredError(db_path) from None
+
+ try:
+ canonical = str(UUID(token))
+ except ValueError:
+ raise RekeyRecoveryRequiredError(db_path) from None
+ if token != canonical:
+ raise RekeyRecoveryRequiredError(db_path)
+ return canonical
+
+
+def read_rekey_epoch_state(db_path: Path) -> RekeyEpochState:
+ """Read and parse the complete durable rekey state.
+
+ The application reads this while holding the user's rekey sidecar
+ lock, which excludes a concurrent publication, with one exception:
+ ``DatabaseManager._classify_rekey_failure`` when ``change_password``
+ fails after that lock has been released (the read only picks which
+ error to report). The read order matters for that reader and for the
+ ones that never take the lock: ``_record_password_verifier`` (only
+ tests call it) and the ``rekey_epoch`` authorization projection
+ (nothing in the application calls it yet). Such a reader may run while
+ a rekey is publishing, so the two files are read in the order that
+ cannot tear into a stale-but-stable answer. Writers publish ``.pending`` before
+ they change ``.epoch`` and remove it only after ``.epoch`` is
+ committed. Reading ``.epoch`` first could see the OLD committed epoch,
+ then (after a commit + clear landed in between) no pending file, and
+ report the old epoch as stable. Reading ``.pending`` first means a
+ missing pending file was missing BEFORE the epoch read, so the epoch
+ read is at least as new as that moment; a present one reports the
+ in-flight state.
+ """
+ pending_epoch = _read_epoch_token(_pending_epoch_path(db_path), db_path)
+ committed_epoch = _read_epoch_token(_epoch_path(db_path), db_path)
+ if committed_epoch is None and pending_epoch is not None:
+ raise RekeyRecoveryRequiredError(db_path)
+ return RekeyEpochState(committed_epoch, pending_epoch)
+
+
+def read_rekey_epoch(db_path: Path) -> str | None:
+ """Project only a stable committed epoch for cache validation."""
+ try:
+ state = read_rekey_epoch_state(db_path)
+ except RekeyRecoveryRequiredError:
+ return None
+ if state.pending_epoch is not None:
+ return None
+ return state.committed_epoch
+
+
+def _canonical_epoch(epoch: str, db_path: Path) -> str:
+ try:
+ canonical = str(UUID(epoch))
+ except ValueError:
+ raise RekeyRecoveryRequiredError(db_path) from None
+ if epoch != canonical:
+ raise RekeyRecoveryRequiredError(db_path)
+ return canonical
+
+
+# errnos with which a filesystem says it cannot fsync a directory at all
+# (vboxsf, some CIFS/9p mounts, WSL DrvFs). There is nothing stronger to
+# fall back to, so the rename stays as durable as that filesystem makes
+# it; any other errno (EIO, ENOSPC, EACCES, ...) is a real failure.
+_DIRECTORY_FSYNC_UNSUPPORTED: frozenset[int] = frozenset(
+ code
+ for code in (
+ errno.EINVAL,
+ getattr(errno, "ENOTSUP", None),
+ getattr(errno, "EOPNOTSUPP", None),
+ getattr(errno, "ENOSYS", None),
+ )
+ if code is not None
+)
+
+
+def _sync_parent_directory(path: Path) -> None:
+ if os.name == "nt":
+ return
+ flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
+ directory_fd = os.open(path.parent, flags)
+ try:
+ os.fsync(directory_fd)
+ except OSError as error:
+ # Unsupported, not failed: treating it as fatal made every epoch
+ # publication fail on such mounts (the first login after upgrade,
+ # every password change, every registration).
+ if error.errno not in _DIRECTORY_FSYNC_UNSUPPORTED:
+ raise
+ finally:
+ os.close(directory_fd)
+
+
+def _write_epoch_token(path: Path, epoch: str, db_path: Path) -> None:
+ canonical = _canonical_epoch(epoch, db_path)
+ # The temporary file materialized below holds only a random canonical
+ # UUID epoch token (no secrets, no user data), is created 0o600 by
+ # mkstemp inside the database's own encrypted-databases directory, and
+ # is unlinked in the finally below whether or not the replace lands.
+ fd, temporary_name = tempfile.mkstemp( # Safe: UUID epoch token only
+ dir=path.parent,
+ prefix=f".{path.name}.",
+ )
+ temporary_path = Path(temporary_name)
+ try:
+ # Same canonical UUID token, nothing else, before fsync+replace.
+ stream = os.fdopen(fd, "wb") # Safe: UUID epoch token only
+ fd = -1
+ with stream:
+ _ = stream.write(canonical.encode("ascii"))
+ stream.flush()
+ os.fsync(stream.fileno())
+
+ # nosemgrep: semgrep.rules.path-traversal-risk, reason: path is a fixed suffix on the sha256-named user database path and temporary_path is mkstemp output in its directory
+ os.replace(temporary_path, path)
+ _sync_parent_directory(path)
+ finally:
+ if fd >= 0:
+ os.close(fd)
+ # nosemgrep: semgrep.rules.path-traversal-risk, reason: temporary_path is mkstemp output in the directory of the sha256-named user database path
+ temporary_path.unlink(missing_ok=True)
+
+
+def begin_rekey_epoch(db_path: Path) -> str:
+ """Durably publish a fresh pending generation from stable state."""
+ state = read_rekey_epoch_state(db_path)
+ if state.committed_epoch is None or state.pending_epoch is not None:
+ raise RekeyRecoveryRequiredError(db_path)
+ pending_epoch = str(uuid4())
+ while pending_epoch == state.committed_epoch:
+ pending_epoch = str(uuid4())
+ _write_epoch_token(_pending_epoch_path(db_path), pending_epoch, db_path)
+ return pending_epoch
+
+
+def commit_rekey_epoch(db_path: Path, pending_epoch: str) -> None:
+ """Durably publish the verified pending generation as committed."""
+ candidate = _canonical_epoch(pending_epoch, db_path)
+ state = read_rekey_epoch_state(db_path)
+ if state.committed_epoch is None or state.pending_epoch != candidate:
+ raise RekeyRecoveryRequiredError(db_path)
+ if state.committed_epoch != candidate:
+ _write_epoch_token(_epoch_path(db_path), candidate, db_path)
+
+
+def clear_pending_rekey_epoch(db_path: Path, pending_epoch: str) -> None:
+ """Remove pending metadata only after its generation is committed."""
+ candidate = _canonical_epoch(pending_epoch, db_path)
+ state = read_rekey_epoch_state(db_path)
+ if state.pending_epoch is None and state.committed_epoch == candidate:
+ return
+ if state.committed_epoch != candidate or state.pending_epoch != candidate:
+ raise RekeyRecoveryRequiredError(db_path)
+ pending_path = _pending_epoch_path(db_path)
+ pending_path.unlink()
+ _sync_parent_directory(pending_path)
+
+
+def advance_rekey_epoch(db_path: Path) -> str:
+ """Atomically publish a fresh durable rekey epoch."""
+ state = read_rekey_epoch_state(db_path)
+ if state.pending_epoch is not None:
+ raise RekeyRecoveryRequiredError(db_path)
+ epoch = str(uuid4())
+ while epoch == state.committed_epoch:
+ epoch = str(uuid4())
+ _write_epoch_token(_epoch_path(db_path), epoch, db_path)
+ return epoch
+
+
+def publish_new_database_epoch(db_path: Path) -> str:
+ """Mint a fresh committed epoch for a database that was just created.
+
+ Creation proved no database existed at this path, so any epoch or
+ pending token left beside it (an account removed without its sidecars,
+ a restore of only the ``.db``) belongs to an earlier database. Reusing
+ that epoch would let an engine or identity cached under the earlier
+ database vouch for the new one, and a leftover pending token would
+ make the creation fail. The fresh epoch is published first and the
+ stale pending token removed after it, so a reader never sees the old
+ epoch as stable.
+ """
+ try:
+ previous = _read_epoch_token(_epoch_path(db_path), db_path)
+ except RekeyRecoveryRequiredError:
+ previous = None
+ epoch = str(uuid4())
+ while epoch == previous:
+ epoch = str(uuid4())
+ _write_epoch_token(_epoch_path(db_path), epoch, db_path)
+ pending_path = _pending_epoch_path(db_path)
+ try:
+ pending_path.unlink()
+ except FileNotFoundError:
+ return epoch
+ _sync_parent_directory(pending_path)
+ return epoch
+
+
+def read_or_initialize_rekey_epoch(db_path: Path) -> str:
+ """Read or initialize the epoch while the stable sidecar lock is held."""
+ state = read_rekey_epoch_state(db_path)
+ if state.pending_epoch is not None:
+ raise RekeyRecoveryRequiredError(db_path)
+ if state.committed_epoch is not None:
+ return state.committed_epoch
+ return advance_rekey_epoch(db_path)
diff --git a/src/local_deep_research/database/session_context.py b/src/local_deep_research/database/session_context.py
index 727385ca37..3d3a71086e 100644
--- a/src/local_deep_research/database/session_context.py
+++ b/src/local_deep_research/database/session_context.py
@@ -14,6 +14,7 @@ from sqlalchemy.orm import Session
from ..security.log_sanitizer import scrub_error
from ..utilities.thread_context import get_search_context
from .encrypted_db import db_manager
+from .rekey_lock import RekeyOperationalError
from .thread_local_session import thread_session_manager
# Placeholder password used when accessing unencrypted databases.
@@ -27,6 +28,30 @@ class DatabaseSessionError(Exception):
pass
+class DatabaseSessionUnavailableError(DatabaseSessionError):
+ """The session could not be established for an operational reason.
+
+ ``get_user_db_session`` raises this instead of the base class when
+ opening or claiming the user's database failed with a
+ ``RekeyOperationalError``: the rekey sidecar could not be opened, locked
+ or released, an epoch file could not be read or first published (an
+ fd-exhaustion spike, EACCES, EIO), or the durable rekey state needs
+ recovery first. None of these is a verdict on the credential or on the
+ session. A request handler that would end the session on a
+ ``DatabaseSessionError`` answers this one with a retryable 503 and keeps
+ the session instead (``database_unavailable_exception`` in
+ ``web/dependencies/auth.py``).
+
+ ``reason`` is the underlying fault's type name, for diagnostics. The
+ fault itself is neither chained nor kept: the frames it was raised
+ through hold the plaintext password.
+ """
+
+ def __init__(self, reason: str) -> None:
+ self.reason = reason
+ super().__init__("Database session unavailable")
+
+
def safe_rollback(session: Session, context: str = "") -> None:
"""Roll back the session, swallowing and logging any rollback failure.
@@ -177,6 +202,10 @@ def get_user_db_session(
Database session for the user
Raises:
+ DatabaseSessionUnavailableError: If the session cannot be
+ established because of a ``RekeyOperationalError`` (a rekey
+ sidecar or epoch fault, or pending rekey recovery). A subclass
+ of ``DatabaseSessionError``.
DatabaseSessionError: If session cannot be established
"""
# Import here to avoid circular imports
@@ -217,7 +246,24 @@ def get_user_db_session(
)
password = UNENCRYPTED_DB_PLACEHOLDER
- session = get_metrics_session(username, password)
+ # A RekeyOperationalError from opening or claiming the database (in
+ # open_user_database or in create_thread_safe_session_for_metrics, which
+ # both let it through) is not a credential verdict, so it gets its own
+ # subclass: request handlers answer it with a retryable 503 that keeps
+ # the session, where the base class may end it. Raised after the except
+ # block, so the new error does not even carry the fault as its
+ # suppressed __context__: the frames it came through hold the password.
+ session = None
+ unavailable_reason = None
+ try:
+ session = get_metrics_session(username, password)
+ except RekeyOperationalError as error:
+ unavailable_reason = type(error).__name__
+ if unavailable_reason is not None:
+ logger.warning(
+ f"Database session unavailable for {username}: {unavailable_reason}"
+ )
+ raise DatabaseSessionUnavailableError(unavailable_reason)
if not session:
raise DatabaseSessionError(
f"Could not establish session for {username}"
diff --git a/src/local_deep_research/database/sqlcipher_utils.py b/src/local_deep_research/database/sqlcipher_utils.py
index f08f0a8f6d..c63bfe5b9a 100644
--- a/src/local_deep_research/database/sqlcipher_utils.py
+++ b/src/local_deep_research/database/sqlcipher_utils.py
@@ -334,6 +334,13 @@ def set_sqlcipher_rekey(
Uses the same PBKDF2 key derivation as set_sqlcipher_key() to ensure
consistency when re-opening databases after password change.
+ Returning normally does NOT prove the key changed: SQLCipher's
+ ``PRAGMA rekey`` ignores the result of the underlying rekey and answers
+ ``ok`` without raising when, for example, another connection holds a
+ write transaction past the busy timeout, leaving the file under the old
+ key. Callers must prove the new key by opening the database with it
+ (``DatabaseManager.change_password`` does).
+
Args:
cursor_or_conn: SQLCipher cursor or connection object
new_password: The new password to use for encryption
@@ -403,6 +410,40 @@ DEFAULT_PAGE_SIZE = 16384 # 16KB pages for maximum performance with caching
DEFAULT_HMAC_ALGORITHM = "HMAC_SHA512"
DEFAULT_KDF_ALGORITHM = "PBKDF2_HMAC_SHA512"
+# How long a connection waits on another connection's lock before SQLITE_BUSY.
+# Applied by apply_performance_pragmas, and before key verification wherever a
+# busy verification read would otherwise be mistaken for a wrong key. This is
+# the default of the operator-tunable ``db_config.busy_timeout_ms`` setting;
+# read the effective value through ``get_sqlcipher_busy_timeout_ms()``.
+SQLCIPHER_BUSY_TIMEOUT_MS = 30000
+# Fixed statement template; only an ``int`` is ever formatted into it.
+_BUSY_TIMEOUT_PRAGMA = "PRAGMA busy_timeout = {:d}"
+
+
+def get_sqlcipher_busy_timeout_ms() -> int:
+ """Return the effective ``busy_timeout`` in milliseconds.
+
+ Honours ``db_config.busy_timeout_ms`` so the pre-verification wait and
+ the per-connection performance pragma always agree.
+ """
+ return int(
+ get_env_setting("db_config.busy_timeout_ms", SQLCIPHER_BUSY_TIMEOUT_MS)
+ )
+
+
+def sqlcipher_busy_timeout_pragma() -> str:
+ """Return the ``PRAGMA busy_timeout`` statement for the effective value.
+
+ PRAGMA takes no bind parameters, so the value is formatted into the
+ statement. It is coerced with ``int()`` here (on top of
+ ``get_sqlcipher_busy_timeout_ms``), and the template's ``{:d}`` format
+ spec refuses anything that is not an integer, so the statement can
+ only ever carry an integer literal.
+ """
+ timeout_ms = int(get_sqlcipher_busy_timeout_ms())
+ return _BUSY_TIMEOUT_PRAGMA.format(timeout_ms)
+
+
# Valid page sizes (powers of 2 within the SQLite range).
# IntegerSetting validates min/max but not that the value is a power of 2,
# so we check against this set as an additional safeguard.
@@ -697,9 +738,8 @@ def apply_performance_pragmas(cursor_or_conn: Any) -> None:
# tight that the spinner can't ride out even a single held
# transaction, and the upper bound (300 s) caps tail latency if a
# query goes pathological.
- busy_timeout_ms = int(get_env_setting("db_config.busy_timeout_ms", 30000))
- # nosemgrep: semgrep.rules.sql-execute-with-string-format, semgrep.rules.sql-string-concatenation, reason: PRAGMA cannot be bound; value coerced with int() above
- cursor_or_conn.execute(f"PRAGMA busy_timeout = {busy_timeout_ms}")
+ busy_timeout_sql = sqlcipher_busy_timeout_pragma()
+ cursor_or_conn.execute(busy_timeout_sql)
# SQLite defaults foreign_keys to OFF — without this every
# ondelete="CASCADE"/"SET NULL" declared on an FK is inert,
@@ -894,6 +934,78 @@ def retry_on_db_lock(
)
+# SQLite's answer when page 1 does not decrypt (SQLITE_NOTADB): with
+# SQLCipher, a wrong key. Older builds word it the second way.
+_WRONG_KEY_MESSAGES = (
+ "file is not a database",
+ "file is encrypted or is not a database",
+)
+_DBAPI_MODULE_PREFIXES = ("sqlcipher3", "sqlite3", "pysqlcipher3")
+
+
+def _exception_chain(error: BaseException):
+ seen: set[int] = set()
+ pending: list[BaseException | None] = [error]
+ while pending:
+ current = pending.pop()
+ if current is None or id(current) in seen:
+ continue
+ seen.add(id(current))
+ yield current
+ pending.extend(
+ (
+ getattr(current, "orig", None),
+ current.__cause__,
+ current.__context__,
+ )
+ )
+
+
+def is_wrong_key_error(error: BaseException) -> bool:
+ """True when ``error`` (or one it wraps) is SQLite's wrong-key answer."""
+ return any(
+ any(message in str(item) for message in _WRONG_KEY_MESSAGES)
+ for item in _exception_chain(error)
+ )
+
+
+def is_operational_sqlite_error(error: BaseException) -> bool:
+ """True for a SQLite failure that says nothing about the key.
+
+ A driver ``OperationalError`` (database locked or busy past the busy
+ timeout, disk I/O error, unable to open the file) anywhere in the
+ chain, unless the chain also carries the wrong-key answer. Callers use
+ it to keep such a failure from being reported as a wrong password.
+ """
+ if is_wrong_key_error(error):
+ return False
+ return any(
+ type(item).__name__ == "OperationalError"
+ and type(item).__module__.startswith(_DBAPI_MODULE_PREFIXES)
+ for item in _exception_chain(error)
+ )
+
+
+def check_sqlcipher_key(cursor: Any) -> bool:
+ """Run the key verification query; False only for a wrong key.
+
+ Unlike ``verify_sqlcipher_connection``, an operational failure
+ (``is_operational_sqlite_error``) propagates instead of reading as a
+ wrong key, so password-checking callers can answer it as retryable.
+ """
+ try:
+ cursor.execute("SELECT 1")
+ result = cursor.fetchone()
+ except Exception as error:
+ if is_operational_sqlite_error(error):
+ raise
+ logger.warning(
+ f"SQLCipher key verification failed: {type(error).__name__}"
+ )
+ return False
+ return result == (1,)
+
+
def verify_sqlcipher_connection(cursor_or_conn: Any) -> bool:
"""
Verify that the SQLCipher connection is working correctly.
@@ -1004,8 +1116,9 @@ def create_sqlcipher_connection(
# Apply performance settings
apply_performance_pragmas(cursor)
- # Verify connection works
- if not verify_sqlcipher_connection(cursor):
+ # Verify connection works (an operational failure propagates; see
+ # check_sqlcipher_key)
+ if not check_sqlcipher_key(cursor):
raise ValueError( # noqa: TRY301 — except does connection cleanup before re-raise
"Failed to establish encrypted database connection"
)
diff --git a/src/local_deep_research/database/thread_local_session.py b/src/local_deep_research/database/thread_local_session.py
index d04953313c..99c5fce7fe 100644
--- a/src/local_deep_research/database/thread_local_session.py
+++ b/src/local_deep_research/database/thread_local_session.py
@@ -55,6 +55,15 @@ class ThreadLocalSessionManager:
The session is created once per thread and reused for all subsequent calls.
This avoids the expensive SQLCipher decryption on every database access.
+
+ Returns None when the database cannot be opened with ``password``
+ (or its schema cannot be initialised). A ``RekeyOperationalError``
+ from opening the database or creating the session (the rekey
+ sidecar or an epoch file could not be read, or a pending rekey
+ needs recovery) is raised instead: it is not a credential verdict,
+ and ``get_user_db_session`` turns it into
+ ``DatabaseSessionUnavailableError`` so a request can answer a
+ retryable 503 rather than log the user out.
"""
thread_id = threading.get_ident()
@@ -121,6 +130,14 @@ class ThreadLocalSessionManager:
# credential failures and raises DatabaseInitializationError when
# the schema can't be initialised; from a worker-thread caller
# both mean "no usable session right now" so collapse them.
+ #
+ # A RekeyOperationalError is deliberately NOT collapsed into None,
+ # here or from create_thread_safe_session_for_metrics below. It is
+ # an operational fault (an fd-exhaustion spike on the rekey
+ # sidecar, an unreadable epoch, pending recovery), and a None would
+ # reach get_user_db_session as the same DatabaseSessionError a
+ # wrong password produces, which the home page answered by logging
+ # the user out. get_user_db_session catches it instead.
from .encrypted_db import DatabaseInitializationError
try:
@@ -336,6 +353,9 @@ def get_metrics_session(username: str, password: str) -> Optional[Session]:
Note: This specifically uses create_thread_safe_session_for_metrics internally
and should only be used for metrics-related database operations.
+
+ Raises ``RekeyOperationalError`` when the database is unavailable for an
+ operational reason; see ``ThreadLocalSessionManager.get_session``.
"""
return thread_session_manager.get_session(username, password)
diff --git a/src/local_deep_research/database/thread_metrics.py b/src/local_deep_research/database/thread_metrics.py
index 396715a7e1..774f2d786d 100644
--- a/src/local_deep_research/database/thread_metrics.py
+++ b/src/local_deep_research/database/thread_metrics.py
@@ -14,6 +14,7 @@ from loguru import logger
from sqlalchemy.orm import Session
from .encrypted_db import db_manager
+from .rekey_lock import RekeyOperationalError
from ..utilities.request_context import get_current_username # noqa: F401
@@ -89,9 +90,21 @@ class ThreadSafeMetricsWriter:
# Create a thread-safe session for this user
session = None
try:
- session = db_manager.create_thread_safe_session_for_metrics(
- username, password
- )
+ try:
+ session = db_manager.create_thread_safe_session_for_metrics(
+ username, password
+ )
+ except RekeyOperationalError as error:
+ # The database is unavailable for an operational reason (a
+ # rekey sidecar or epoch fault, or pending rekey recovery).
+ # This writer's callers do not know the typed error, so it
+ # becomes the same "no session" failure as below, raised
+ # outside this block so the fault is not chained onto it.
+ # Type name only: ``password`` is a live local in this frame.
+ logger.warning(
+ f"Metrics session unavailable for user {username}: "
+ f"{type(error).__name__}"
+ )
if not session:
raise ValueError( # noqa: TRY301 — except does session rollback before re-raise
f"Failed to create session for user {username}"
diff --git a/src/local_deep_research/exceptions.py b/src/local_deep_research/exceptions.py
index cc8cce4934..173ddd82f8 100644
--- a/src/local_deep_research/exceptions.py
+++ b/src/local_deep_research/exceptions.py
@@ -33,9 +33,12 @@ class NoUserDatabaseError(RuntimeError):
answering it with defaults means a worker silently reads *someone
else's* configuration instead of the user's.
3. Anything deeper failing inside ``db_manager.get_session`` — a corrupt or
- unopenable encrypted database, a keyring failure. Serving defaults here
- is the most dangerous of the three: a setting that is off by default
- silently reads as off, even though the user turned it on.
+ unopenable encrypted database, a keyring failure, or a rekey sidecar
+ or epoch fault on the cached engine (raised as the typed
+ ``RekeyOperationalError``, not answered with ``None``). Serving
+ defaults here is the most dangerous of the three: a setting that is
+ off by default silently reads as off, even though the user turned it
+ on, and a stored ``strict`` egress scope reads as the default.
Only case 1 should be swallowed. The other two must reach the caller.
"""
diff --git a/src/local_deep_research/utilities/db_utils.py b/src/local_deep_research/utilities/db_utils.py
index f9ea78be07..507c57e59d 100644
--- a/src/local_deep_research/utilities/db_utils.py
+++ b/src/local_deep_research/utilities/db_utils.py
@@ -88,6 +88,12 @@ def _get_cached_user_session(username: str, _namespace: str = "") -> Session:
call (where username is usually None) would key two different users'
requests served by the same threadpool worker to one entry, handing
user B a session opened for user A.
+
+ ``NoUserDatabaseError`` (which ``get_settings_manager`` answers with
+ the defaults file) is raised only for ``get_session``'s None: no usable
+ engine is cached. A rekey sidecar or epoch fault on a cached engine is
+ raised by ``get_session`` itself as a ``RekeyOperationalError`` and
+ passes through here untouched, so it never reads as "no database".
"""
user_session = db_manager.get_session(username)
if user_session:
@@ -197,7 +203,12 @@ def get_settings_manager(
# anonymous defaults is silently wrong rather than degraded --
# a worker would read another configuration than the user's, and
# an unopenable database would read as "every setting is at its
- # default" with nothing logged. Both now propagate.
+ # default" with nothing logged. Both now propagate. So does a
+ # rekey sidecar or epoch fault on the user's cached engine (a
+ # RekeyOperationalError from db_manager.get_session, e.g. an
+ # fd-exhaustion spike): read as defaults, a stored "strict"
+ # egress scope would let a public download through. The app
+ # answers it with a retryable 503.
db_session = None
username = "anonymous"
diff --git a/src/local_deep_research/utilities/log_utils.py b/src/local_deep_research/utilities/log_utils.py
index 0fe780baa4..0563d89dce 100644
--- a/src/local_deep_research/utilities/log_utils.py
+++ b/src/local_deep_research/utilities/log_utils.py
@@ -391,7 +391,9 @@ def _write_log_to_database(log_entry: dict) -> None:
try:
# Binds a session to the ALREADY-open engine. Returns None (never
# reopens) if the connection was closed in the small race between
- # the is_user_connected check above and this call. We can't use
+ # the is_user_connected check above and this call; a rekey sidecar
+ # fault in that window raises instead and is reported below like
+ # any other failed write (the entry is dropped). We can't use
# get_user_db_session here: it would resolve+cache a password and
# lazily reopen the DB, which is exactly the post-logout
# resurrection this fix removes. The session is explicitly closed
diff --git a/src/local_deep_research/web/auth/connection_cleanup.py b/src/local_deep_research/web/auth/connection_cleanup.py
index c5f0c9bf12..94b3cd0715 100644
--- a/src/local_deep_research/web/auth/connection_cleanup.py
+++ b/src/local_deep_research/web/auth/connection_cleanup.py
@@ -282,32 +282,23 @@ def cleanup_idle_connections(session_manager, db_manager):
now = time.monotonic()
if now - _last_dispose_time >= _DISPOSE_INTERVAL_SECONDS:
_last_dispose_time = now
- disposed = 0
- with db_manager._connections_lock:
- for username, engine in list(db_manager.connections.items()):
- try:
- db_manager._checkpoint_wal(engine, f"for {username}")
- engine.dispose()
- disposed += 1
- except Exception as exc:
- # Surface the failure. Pre-fix this was logger.debug,
- # which hid the symptom — if WAL checkpoint or pool
- # dispose repeatedly fails (disk pressure, lock
- # starvation, etc.) the WAL file silently grows on
- # disk and pooled connections leak. The 30-min
- # periodic-dispose workaround for ADR-0004's WAL/SHM
- # handle leak depends on this loop succeeding.
- #
- # Only the exception's TYPE NAME is logged, matching
- # the codebase's `_report_silent_exception` pattern
- # (utilities/log_utils.py:146-194). The exception
- # value itself can carry sensitive locals (DB paths,
- # query fragments, etc.) and our sensitive-logging
- # hook flags any `f"...{exc}"` interpolation.
- exc_type = type(exc).__name__
- logger.warning(
- f"Error disposing engine for {username}: {exc_type}"
- )
+ # Delegate to the fenced pool-reset API: it takes each user's
+ # rekey lifecycle lock and the on-disk sidecar lock before
+ # checkpointing and disposing, so this periodic sweep can never
+ # race a live password rekey (checkpointing a mid-rekey WAL, or
+ # disposing the engine a rekey is about to evict anyway). Engines
+ # whose sidecar is contended are skipped, not torn down. Individual
+ # dispose failures are logged (type name only) inside the API.
+ try:
+ disposed = db_manager.reset_connection_pools()
+ except Exception as exc:
+ # The sweep must never take the scheduler down with it. Only
+ # the exception TYPE NAME is logged — the value can carry
+ # sensitive locals (DB paths, query fragments).
+ logger.warning(
+ f"Periodic pool dispose failed: {type(exc).__name__}"
+ )
+ disposed = 0
if disposed:
logger.info(
f"Pool dispose: reset {disposed} engine(s) to release "
diff --git a/src/local_deep_research/web/dependencies/auth.py b/src/local_deep_research/web/dependencies/auth.py
index 8bbbbf559e..72649e8202 100644
--- a/src/local_deep_research/web/dependencies/auth.py
+++ b/src/local_deep_research/web/dependencies/auth.py
@@ -5,13 +5,20 @@ Replaces Flask's @login_required decorator and g.db_session / g.current_user
with explicit dependency injection.
"""
+from contextlib import contextmanager
+from contextvars import ContextVar
from typing import Annotated, Generator
from fastapi import Depends, HTTPException, Request
from loguru import logger
from sqlalchemy.orm import Session
-from ...database.encrypted_db import db_manager
+from ...database.encrypted_db import (
+ UserConnectionStatus,
+ db_manager,
+ sidecar_wait_budget,
+)
+from ...database.rekey_lock import RekeyOperationalError
from ...database.session_context import get_user_db_session
from ...database.session_passwords import session_password_store
from ...utilities.db_utils import get_settings_manager
@@ -23,6 +30,50 @@ def get_session_username(request: Request) -> str | None:
return request.session.get("username")
+# Type names of the rekey operational faults that DatabaseMiddleware's reopen
+# of the user's database met in the current request (ensure_user_database);
+# None outside a request_database_scope. A list, so the reopen that runs on a
+# worker thread (with a copy of the request's context) can append to the one
+# require_auth reads later on another.
+#
+# Request-scoped state: contexts copied from the request (research, RAG and
+# post-login threads) keep this list after the request ends, so it must only
+# be read by require_auth (through _database_connection_status), in the
+# request that wrote it, never from background work.
+_reopen_faults: ContextVar[list[str] | None] = ContextVar(
+ "ldr_request_reopen_faults", default=None
+)
+
+
+@contextmanager
+def request_database_scope() -> Generator[None, None, None]:
+ """Scope one HTTP request's database bookkeeping (``DatabaseMiddleware``).
+
+ * The request's waits for the user's rekey sidecar share a give-up flag
+ (``sidecar_wait_budget``): DatabaseMiddleware's probe and reopen,
+ ``require_auth``'s probes and the route's own database opens. A wait
+ that gives up sets it, and each later claim then tries once without
+ waiting, until one of them gets the sidecar again. The flag is not a
+ running total of the time waited.
+ * A rekey operational fault that the reopen in ``ensure_user_database``
+ met is kept for ``require_auth``, which answers the missing engine
+ with the retryable 503 instead of the 401 that ``/`` turns into a
+ logout.
+ """
+ token = _reopen_faults.set([])
+ try:
+ with sidecar_wait_budget():
+ yield
+ finally:
+ _reopen_faults.reset(token)
+
+
+def _record_reopen_fault(fault: RekeyOperationalError) -> None:
+ faults = _reopen_faults.get()
+ if faults is not None:
+ faults.append(type(fault).__name__)
+
+
def clear_session_if_unrecoverable(request: Request, username: str) -> bool:
"""Drop a stale session that has no database recovery credential.
@@ -122,7 +173,12 @@ def end_rejected_session(request: Request) -> None:
password change), every request 401s with ``username`` still in the
cookie. ``/auth/login`` sends any cookie with a username back to ``/``,
so a page that redirects that 401 to the login page without clearing the
- cookie loops between the two until the browser gives up.
+ cookie loops between the two until the browser gives up. A cached engine
+ that merely cannot be checked right now (a rekey sidecar fault), or a
+ reopen that failed with a rekey operational fault (another holder
+ keeping the sidecar past the wait bound, for one), is not such a
+ failure: ``require_auth`` answers it with a 503, which never reaches
+ this function.
Call this before redirecting such a 401 to the login page. It destroys
the server-side session, disconnects its sockets, drops this session's
@@ -147,18 +203,119 @@ def end_rejected_session(request: Request) -> None:
request.session.clear()
+# Seconds a client is asked to wait before retrying a request answered with
+# ``database_unavailable_exception``'s 503: the user's database could not be
+# checked or opened because of a rekey operational fault (a transient
+# sidecar fault such as an fd-exhaustion spike).
+DATABASE_UNAVAILABLE_RETRY_AFTER_SECONDS = 5
+
+
+def database_unavailable_exception() -> HTTPException:
+ """The retryable 503 for a rekey operational fault after authentication.
+
+ Every moment that meets such a fault answers the same way:
+
+ * ``require_auth``'s own probe cannot check the cached engine, or no
+ engine is cached because this request's reopen of the database met
+ such a fault (``UserConnectionStatus.UNAVAILABLE``);
+ * the probe succeeded, but opening the request's database session a
+ moment later failed on the same kind of fault
+ (``DatabaseSessionUnavailableError`` from ``get_user_db_session``),
+ which an fd-exhaustion spike that comes and goes produces: ``/`` and
+ ``get_db_session_dep`` raise this for it (``/`` answers a browser,
+ for this and for ``require_auth``'s own 503, with the branded error
+ page instead: same status and ``Retry-After``, and a button that
+ retries ``/``);
+ * a route left that error, or a ``RekeyOperationalError`` raised by a
+ direct ``db_manager`` call, unhandled: the app's exception handler
+ for both (``fastapi_app._register_exception_handlers``) answers this.
+ That includes a settings read through ``get_settings_manager``
+ without a session (``db_manager.get_session`` raises the fault rather
+ than letting it read as "no database", i.e. every setting at its
+ default); ``POST /metrics/api/journal-data/download``, whose own
+ catch-all would answer 500, passes it on to that handler.
+
+ None is a verdict on the session, so whoever raises this leaves the
+ cookie, the server-side session and the stored password alone. A fresh
+ instance each call: ``HTTPException`` carries mutable headers.
+ """
+ return HTTPException(
+ status_code=503,
+ detail="Your database is temporarily unavailable. "
+ "Please try again in a moment.",
+ headers={"Retry-After": str(DATABASE_UNAVAILABLE_RETRY_AFTER_SECONDS)},
+ )
+
+
+def _database_connection_status(username: str) -> UserConnectionStatus:
+ """Whether ``username``'s database can serve this request right now.
+
+ ``is_user_connected`` stays the single probe on the common path. Its
+ False is ambiguous, though: either no usable engine is cached (none, or
+ a stale one the probe just detached), or a cached engine could not be
+ checked because the rekey sidecar could not be opened or locked, or the
+ epoch read failed (an fd-exhaustion spike, EACCES, EIO). The manager
+ keeps the engine in the second case and re-checks it on the next
+ probe. So a False is followed by ``probe_user_connection``, which tells
+ the two apart; its answer is the more recent one, so it decides.
+
+ No cached engine is not a credential verdict either when this
+ request's reopen of the database (``ensure_user_database``) failed with
+ a rekey operational fault: its cold open could not take the sidecar
+ (another holder kept it past the wait bound, an fd-exhaustion spike),
+ could not read or publish the epoch, or found durable rekey state that
+ needs recovery. None of these says anything against the password, so
+ that is ``UNAVAILABLE`` too.
+ """
+ try:
+ if db_manager.is_user_connected(username):
+ return UserConnectionStatus.CONNECTED
+ except RekeyOperationalError:
+ # The probe takes the rekey sidecar (filesystem + a lock it waits
+ # for at most a few seconds). Opening, locking (a holder that keeps
+ # it past that bound included) and reading are reported as a plain
+ # False, but a failure to release the sidecar afterwards
+ # (unlock/close) raises RekeySidecarIOError, which is not an
+ # OSError. It is the same transient sidecar I/O fault, not a verdict
+ # on the session.
+ return UserConnectionStatus.UNAVAILABLE
+ connection = db_manager.probe_user_connection(username)
+ if (
+ connection is UserConnectionStatus.NOT_CONNECTED
+ and _reopen_faults.get()
+ ):
+ return UserConnectionStatus.UNAVAILABLE
+ return connection
+
+
def require_auth(request: Request) -> str:
- """Require authentication. Returns username or raises 401.
+ """Require authentication. Returns username or raises 401 (or 503).
Replaces Flask's @login_required decorator.
For API routes, returns JSON 401. For HTML routes, could redirect
(but callers handle that distinction).
+
+ A session that would otherwise be accepted gets a 503 with
+ ``Retry-After`` instead while its cached database engine cannot be
+ checked because of a rekey sidecar fault, or while no engine is cached
+ because this request's reopen failed with a rekey operational fault
+ (another holder keeping the sidecar past the wait bound, for one). The
+ session, its
+ server-side record and its stored password are all kept: a 401 there
+ would be turned into a logout by ``/`` (``end_rejected_session``)
+ after a transient fault such as an fd-exhaustion spike. A fault that
+ only shows when the route then opens its database session surfaces as
+ ``DatabaseSessionUnavailableError`` and gets the same 503
+ (``database_unavailable_exception``).
"""
username = request.session.get("username")
if not username:
raise HTTPException(status_code=401, detail="Authentication required")
- if not db_manager.is_user_connected(username):
+ connection = _database_connection_status(username)
+ if connection is not UserConnectionStatus.CONNECTED and (
+ connection is not UserConnectionStatus.UNAVAILABLE
+ ):
# Stale session with no recoverable credential: clear the cookie (and
# revoke the server-side session) so the client is sent back to login
# instead of 401-ing on every request until someone loads "/".
@@ -197,6 +354,20 @@ def require_auth(request: Request) -> str:
request.session.clear()
raise HTTPException(status_code=401, detail="Authentication required")
+ if connection is UserConnectionStatus.UNAVAILABLE:
+ # The session is valid, but its database cannot be checked: the
+ # user's engine is still cached and its rekey epoch could not be
+ # checked (the sidecar could not be opened, locked or released, or
+ # the epoch read failed), or no engine is cached and the reopen
+ # DatabaseMiddleware attempted failed with a rekey operational
+ # fault (_database_connection_status). Answer a
+ # retryable 503 and touch nothing: a 401 here reaches "/", whose
+ # end_rejected_session destroys the session and its stored
+ # password, so an fd-exhaustion spike or a held sidecar would log
+ # the user out. With an engine cached, that reopen failed fast too:
+ # open_user_database raises the same fault before deriving a key.
+ raise database_unavailable_exception()
+
return username
@@ -264,8 +435,16 @@ def get_db_session_dep(
Before using this: either make scope enter/exit a token API that asserts
``threading.get_ident()`` matches, or use ``run_db_sync`` (which keeps the
whole unit of work on one worker), as the routes actually do.
+
+ A ``DatabaseSessionUnavailableError`` (a rekey sidecar or epoch fault,
+ not a credential problem) is answered with the retryable 503 of
+ ``database_unavailable_exception`` and leaves the session alone; any
+ other ``DatabaseSessionError`` clears it and answers 401.
"""
- from ...database.session_context import DatabaseSessionError
+ from ...database.session_context import (
+ DatabaseSessionError,
+ DatabaseSessionUnavailableError,
+ )
session_id = request.session.get("session_id")
try:
@@ -275,6 +454,11 @@ def get_db_session_dep(
status_code=500, detail="Failed to get database session"
)
yield session
+ except DatabaseSessionUnavailableError:
+ # require_auth's probe passed, then opening the session (or a
+ # session the handler opened) met a transient rekey fault. Clearing
+ # the session here would log the user out over an fd spike.
+ raise database_unavailable_exception() from None
except DatabaseSessionError:
# Password not available (e.g. server restarted, session expired).
# Clear session to force re-login.
@@ -309,7 +493,14 @@ def ensure_user_database(request: Request) -> None:
``open_user_database()`` (``web/auth/database_middleware.py`` on main).
That is not cosmetic here: this runs from ``DatabaseMiddleware``, i.e.
before any route, so an escaping exception turns EVERY authenticated
- request into a 500 for as long as the fault lasts.
+ request into a 500 for as long as the fault lasts. One exception to the
+ 401: a rekey operational fault (``RekeyOperationalError``) from the
+ reopen is recorded for the request (``request_database_scope``), and
+ the auth gate answers it with a retryable 503 that keeps the session
+ (see ``require_auth``). That includes a cached engine that cannot be
+ checked, which ``open_user_database`` raises before deriving a key, and
+ a cold open that could not take the sidecar (another holder keeping it
+ past the wait bound, an fd-exhaustion spike) or read the epoch.
"""
username = request.session.get("username")
if not username:
@@ -372,7 +563,10 @@ def ensure_user_database(request: Request) -> None:
# Fast path: the connection is already open, so there is nothing left
# to do. Placed after the token block above rather than at the top of
# the function — see the comment there. In steady state (no token in
- # the session) reaching here still costs only one dict lookup.
+ # the session) reaching here costs one is_user_connected() probe,
+ # which is no longer a bare dict lookup: it takes the per-user
+ # lifecycle lock and the rekey sidecar flock and re-reads the
+ # durable rekey epoch to validate the cached engine.
if db_manager.is_user_connected(username):
return
@@ -395,6 +589,19 @@ def ensure_user_database(request: Request) -> None:
logger.warning(
f"open_user_database returned None for user {username}"
)
+ except RekeyOperationalError as exc:
+ # Not a verdict on the session: the sidecar or an epoch file could
+ # not be read, another holder kept the sidecar past the wait bound,
+ # or the durable rekey state needs recovery first. None of these
+ # says anything against the password. Recorded for require_auth,
+ # which answers it with the retryable 503 and keeps the session
+ # instead of reading the missing engine as a credential failure (a
+ # 401 that "/" turns into a logout). Type name only, as below.
+ _record_reopen_fault(exc)
+ logger.warning(
+ f"Database for user {username} is temporarily unavailable "
+ f"({type(exc).__name__})"
+ )
except Exception as exc:
# Deliberately no traceback and no exception message: the frames
# under this call hold the plaintext password in their locals, and
diff --git a/src/local_deep_research/web/fastapi_app.py b/src/local_deep_research/web/fastapi_app.py
index c6fcfc2c2f..4d716980a2 100644
--- a/src/local_deep_research/web/fastapi_app.py
+++ b/src/local_deep_research/web/fastapi_app.py
@@ -18,7 +18,7 @@ from contextlib import asynccontextmanager
from importlib import resources as importlib_resources
from pathlib import Path
-from fastapi import FastAPI, Request
+from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import (
HTMLResponse,
JSONResponse,
@@ -69,6 +69,13 @@ _BRANDED_ERROR_COPY: dict[int, tuple[str, str]] = {
"Request too large",
"The file or request you sent is larger than the allowed limit.",
),
+ # Only ``/`` renders this one (``_database_unavailable_page``): the
+ # retryable 503 of ``database_unavailable_exception``.
+ 503: (
+ "Temporarily unavailable",
+ "Your database is temporarily unavailable. "
+ "Please try again in a moment.",
+ ),
}
@@ -177,6 +184,31 @@ def _render_branded_error_page(
return None
+def _database_unavailable_page(request: Request, exc: HTTPException):
+ """The branded page for ``/``'s retryable 503, or None for the JSON one.
+
+ ``exc`` is ``database_unavailable_exception()``: the user's database
+ cannot be checked or opened right now (a rekey sidecar fault such as
+ an fd-exhaustion spike). ``/`` is where a browser lands after login,
+ and ``/auth/login`` sends every signed-in cookie back to it, so a fault
+ that lasts would otherwise leave the browser on a raw JSON body. The
+ branded page keeps the status and ``Retry-After``, and its button
+ retries ``/``. It renders without a database read and from a copy of
+ the session (see ``_render_branded_error_page``), so the session stays
+ exactly as the 503 promises. Returns None, so that the caller raises
+ ``exc`` for the usual JSON answer, for a caller that asked for JSON or
+ when the page cannot be rendered.
+ """
+ if _is_api_request(request):
+ return None
+ page = _render_branded_error_page(request, exc.status_code)
+ if page is None:
+ return None
+ for name, value in (exc.headers or {}).items():
+ page.headers[name] = value
+ return page
+
+
# ---------------------------------------------------------------------------
# Secret key (same logic as Flask app_factory)
# ---------------------------------------------------------------------------
@@ -1691,6 +1723,41 @@ def _register_exception_handlers(app: FastAPI) -> None:
)
return JSONResponse(exc.to_dict(), status_code=exc.status_code)
+ # A rekey operational fault that a route leaves unhandled: the user's
+ # database could not be checked or opened because the rekey sidecar or
+ # an epoch file could not be read (an fd-exhaustion spike, EACCES, EIO),
+ # or its durable rekey state needs recovery first. get_user_db_session
+ # raises it as DatabaseSessionUnavailableError; a db_manager call made
+ # directly raises the RekeyOperationalError itself, and so does a
+ # settings read through get_settings_manager without a session (it
+ # reaches db_manager.get_session). Neither error is a verdict on the
+ # session, so both get require_auth's retryable 503
+ # (same body and Retry-After, through the HTTPException handler above)
+ # instead of the catch-all's 500, and the session is left alone. Logged
+ # by type name only: the frames the fault was raised through hold the
+ # plaintext password, so no traceback is rendered.
+ from ..database.rekey_lock import RekeyOperationalError
+ from ..database.session_context import DatabaseSessionUnavailableError
+
+ @app.exception_handler(DatabaseSessionUnavailableError)
+ @app.exception_handler(RekeyOperationalError)
+ async def handle_database_unavailable(request: Request, exc: Exception):
+ from .dependencies.auth import database_unavailable_exception
+ from .routers.notes import _log_value_preview
+
+ # (#6938: the path is client-chosen, so it is bounded.)
+ logger.warning(
+ "Database unavailable while handling {} {} ({}); answering 503",
+ request.method,
+ _log_value_preview(request.url.path),
+ exc.reason
+ if isinstance(exc, DatabaseSessionUnavailableError)
+ else type(exc).__name__,
+ )
+ return await handle_http_exception(
+ request, database_unavailable_exception()
+ )
+
try:
from ..news.exceptions import NewsAPIException
@@ -1968,6 +2035,20 @@ class DatabaseMiddleware:
await self.app(scope, receive, send)
return
+ # One scope per request around everything below: the reopen of the
+ # user's database, require_auth's probes and the route's own
+ # database opens share a give-up flag for their waits on the user's
+ # rekey sidecar (a wait that gives up sets it, and each later claim
+ # then tries once without waiting, until one of them gets the
+ # sidecar again; it is not a running total of the time waited), and
+ # require_auth sees whether the reopen met a rekey fault (a 503
+ # that keeps the session, not a 401). See request_database_scope.
+ from .dependencies.auth import request_database_scope
+
+ with request_database_scope():
+ await self._call_http(scope, receive, send)
+
+ async def _call_http(self, scope, receive, send):
path = scope.get("path", "")
method = scope.get("method", "GET")
@@ -2767,7 +2848,10 @@ _setup_template_globals()
def _mount_all(app: FastAPI) -> None:
"""Mount all routers and Socket.IO on the app."""
- from ..database.session_context import get_user_db_session
+ from ..database.session_context import (
+ DatabaseSessionUnavailableError,
+ get_user_db_session,
+ )
from ..settings.logger import log_settings
from ..utilities.db_utils import get_settings_manager
from .dependencies.threadpool import prepare_router_for_worker_cleanup
@@ -2840,13 +2924,24 @@ def _mount_all(app: FastAPI) -> None:
# the user's saved settings: without a session-scoped password,
# get_user_db_session falls back to a sibling session's password.
# require_auth rejects that session (and revokes it) instead.
- from fastapi import HTTPException
-
- from .dependencies.auth import end_rejected_session, require_auth
+ from .dependencies.auth import (
+ database_unavailable_exception,
+ end_rejected_session,
+ require_auth,
+ )
try:
username = require_auth(request)
except HTTPException as exc:
+ if exc.status_code == 503:
+ # The retryable 503 of a database that cannot be checked
+ # right now: for a browser, the branded page that retries
+ # here; for a JSON caller, or if it cannot render, the
+ # 503 itself. The session is kept either way.
+ page = _database_unavailable_page(request, exc)
+ if page is None:
+ raise
+ return page
if exc.status_code != 401:
raise
# One require_auth 401 keeps the cookie (database not connected
@@ -2936,6 +3031,26 @@ def _mount_all(app: FastAPI) -> None:
"embeddings.require_local", False
),
}
+ except DatabaseSessionUnavailableError:
+ # require_auth's probe succeeded, but opening the session for the
+ # settings read then met a rekey sidecar or epoch fault: an
+ # fd-exhaustion spike can clear and come back between the two
+ # probes. It says nothing about this session, so answer
+ # require_auth's own retryable 503 and keep the cookie, the
+ # server-side session and the stored password, rather than the
+ # logout below (a browser gets it as the branded page that
+ # retries here). get_user_db_session already logged the fault's
+ # type name.
+ logger.warning(
+ "Root route could not open the database for user {}; "
+ "answering 503 and keeping the session",
+ username,
+ )
+ unavailable = database_unavailable_exception()
+ page = _database_unavailable_page(request, unavailable)
+ if page is None:
+ raise unavailable from None
+ return page
except SettingsValueError:
# An invalid configured setting is not the password case below.
# Clearing the session would only send the user back here after
diff --git a/src/local_deep_research/web/routers/auth.py b/src/local_deep_research/web/routers/auth.py
index f77dffd6df..461e263aa2 100644
--- a/src/local_deep_research/web/routers/auth.py
+++ b/src/local_deep_research/web/routers/auth.py
@@ -27,6 +27,12 @@ from loguru import logger
from ...database.auth_db import auth_db_session
from ...database.encrypted_db import DatabaseInitializationError, db_manager
+from ...database.rekey_lock import (
+ RekeyIndeterminateError,
+ RekeyLockContentionError,
+ RekeyOperationalError,
+ RekeySidecarIOError,
+)
from ...database.models.auth import User
from ...database.thread_local_session import thread_cleanup
from sqlalchemy.exc import IntegrityError
@@ -294,6 +300,39 @@ def login(
},
status_code=503,
)
+ except RekeyOperationalError as error:
+ # Durable rekey state is indeterminate, pending recovery, or the
+ # sidecar metadata cannot be read, or another holder (a password
+ # change of a large database, a stopped process) kept the sidecar
+ # past the open's wait bound (RekeyLockContentionError). Credentials
+ # cannot be validated in any of these cases, so fail closed WITHOUT
+ # counting toward lockout — the password may well be correct.
+ sidecar_detail = ""
+ if isinstance(error, RekeySidecarIOError):
+ sidecar_detail = (
+ f" (sidecar {error.operation.value} failed, "
+ f"errno={error.errno})"
+ )
+ logger.warning(
+ f"Login refused for {redact_and_bound_for_log(username, 50)}: "
+ f"rekey state requires operator attention "
+ f"({type(error).__name__}){sidecar_detail}"
+ )
+ flash(
+ request,
+ "Your account database is pending maintenance. "
+ "Please try again shortly or contact the administrator.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/login.html",
+ {
+ "has_encryption": db_manager.has_encryption,
+ "allow_registrations": config.get("allow_registrations", True),
+ },
+ status_code=503,
+ )
if engine is None:
lockout_mgr.record_failure(username)
@@ -1101,6 +1140,86 @@ def check_auth(request: Request):
return {"authenticated": True, "username": username}
+def _revoke_user_credentials_after_rekey(
+ request: Request, username: str
+) -> None:
+ """Revoke everything that carries or was authorised by the old password.
+
+ Shared by a completed password change and one whose outcome is
+ indeterminate (the new key may be active). Mirrors the logout
+ handler's cleanup, but for every session of the user, not just this
+ one. Each step runs on its own: one that fails is logged (without the
+ exception text, which could carry request data) and the others still
+ run, so a scheduler fault cannot leave other devices signed in.
+ """
+
+ def unregister_scheduler() -> None:
+ # Removes the scheduler's stored credential.
+ from ...scheduler.background import get_background_job_scheduler
+
+ sched = get_background_job_scheduler()
+ if sched.is_running:
+ sched.unregister_user(username)
+
+ def pop_per_user_locks() -> None:
+ # Drop per-user lock-dict entries (matches logout path).
+ from ..auth.connection_cleanup import _pop_per_user_locks
+
+ _pop_per_user_locks(username)
+
+ def evict_collection_engines() -> None:
+ # Drop the cached collection search engines (matches logout path;
+ # every session ends below).
+ from ..auth.connection_cleanup import _evict_collection_engines_cache
+
+ _evict_collection_engines_cache(username)
+
+ def destroy_sessions() -> None:
+ session_manager.destroy_all_user_sessions(username)
+
+ def clear_session_passwords() -> None:
+ from ...database.session_passwords import session_password_store
+
+ session_password_store.clear_all_for_user(username)
+
+ def clear_thread_credentials() -> None:
+ # Per-thread cached credentials hold the OLD plaintext password,
+ # which is (or may be) invalid now: a pooled worker that kept it
+ # would both retain a dead secret and fail to reopen the database.
+ # See clear_user_credentials() for why per-request cleanup cannot
+ # reach a worker thread under FastAPI.
+ from ...database.thread_local_session import clear_user_credentials
+
+ clear_user_credentials(username)
+
+ def disconnect_sockets() -> None:
+ # A socket authorised under an old session must stop receiving this
+ # user's events (#5535). Every session is destroyed, so this is the
+ # all-sessions scope, unlike logout's per-session one.
+ _disconnect_user_sockets(username)
+
+ def clear_request_session() -> None:
+ request.session.clear()
+
+ for step in (
+ unregister_scheduler,
+ pop_per_user_locks,
+ evict_collection_engines,
+ destroy_sessions,
+ clear_session_passwords,
+ clear_thread_credentials,
+ disconnect_sockets,
+ clear_request_session,
+ ):
+ try:
+ step()
+ except Exception as error:
+ logger.warning(
+ f"Credential revocation step {step.__name__} failed for "
+ f"{_log_value_preview(username)}: {type(error).__name__}"
+ )
+
+
@router.get("/change-password")
def change_password_page(
request: Request, username: Annotated[str, Depends(require_auth)]
@@ -1237,6 +1356,7 @@ def change_password(
)
# Attempt password change (the rekey) — holds only this user's gate.
+ outcome_indeterminate = False
try:
success = db_manager.change_password(
username, current_password, new_password
@@ -1272,6 +1392,80 @@ def change_password(
},
status_code=400,
)
+ except RekeyLockContentionError:
+ # Another password change (or a gate holder) owns this user's
+ # rekey lifecycle right now. Nothing was rekeyed here.
+ logger.warning(
+ f"Password change contended for {_log_value_preview(username)}: "
+ "another rekey lifecycle owns the lock"
+ )
+ flash(
+ request,
+ "A password change is already in progress for this account. "
+ "Please wait for it to finish and try again.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/change_password.html",
+ {
+ "password_requirements": PasswordValidator.get_requirements(),
+ },
+ status_code=409,
+ )
+ except RekeyIndeterminateError:
+ # The rekey may have taken effect: the database layer could not
+ # prove which key is active. Treat every old-password credential
+ # as revoked exactly as on success (all sessions on every device,
+ # both password stores, the scheduler's copy, live sockets and
+ # this request's session), because the old password may no
+ # longer be the key. The next login resolves the pending state
+ # with whichever password actually opens the database, and that
+ # recovery purges backups left under the superseded key
+ # (DatabaseManager._purge_superseded_key_backups). No backup is
+ # purged or refreshed here: neither key is proven.
+ # Handled below, after this user's research-start gate is
+ # released, like the success path.
+ outcome_indeterminate = True
+ success = False
+ except RekeyOperationalError:
+ # Unavailable (pre-ambiguity): the old key provably stayed
+ # active and nothing was rekeyed, or another sidecar fault
+ # stopped the change before it started. The user retries.
+ logger.warning(
+ f"Password change failed for {_log_value_preview(username)}: rekey outcome "
+ "unavailable"
+ )
+ flash(
+ request,
+ "Password change could not be completed safely. The database "
+ "was left fenced; please try again shortly or contact the "
+ "administrator if this repeats.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/change_password.html",
+ {
+ "password_requirements": PasswordValidator.get_requirements(),
+ },
+ status_code=503,
+ )
+
+ if outcome_indeterminate:
+ logger.warning(
+ f"Password change outcome indeterminate for "
+ f"{_log_value_preview(username)}: revoking all sessions; the "
+ "next login resolves the pending rekey"
+ )
+ _revoke_user_credentials_after_rekey(request, username)
+ flash(
+ request,
+ "Your password change may have completed. Try your NEW "
+ "password first; if it does not work, use your old one.",
+ "warning",
+ )
+ return RedirectResponse(url="/auth/login", status_code=302)
if success:
# The rekey is the ONLY step needed. The auth database stores no
@@ -1283,36 +1477,16 @@ def change_password(
# Clean up stale credentials before clearing session
# (mirrors logout handler cleanup steps 1–5).
- # 1. Unregister from scheduler (removes stale credential)
- try:
- from ...scheduler.background import (
- get_background_job_scheduler,
- )
+ # 1. No database close here. change_password() already detached and
+ # disposed the old-password engine BEFORE the rekey, under the
+ # user's rekey sidecar lock that every cold open also takes, so no
+ # old-key engine can be cached once it returns True. The only
+ # engine that can be cached by now is one a concurrent
+ # open_user_database(new_password) published after the rekey
+ # committed. That engine is valid, and closing it here would only
+ # force the next request to cold-open again.
- sched = get_background_job_scheduler()
- if sched.is_running:
- sched.unregister_user(username)
- except Exception:
- logger.warning(
- "Could not unregister user from scheduler",
- )
-
- # 2. Close database connection (disposes old-password engine)
- # change_password() already closes in its finally block, but
- # an explicit close here is defensive — harmless if redundant.
- db_manager.close_user_database(username)
-
- # 2a. Drop per-user lock-dict entries and the cached collection
- # engines (matches logout path; every session ends below).
- from ..auth.connection_cleanup import (
- _evict_collection_engines_cache,
- _pop_per_user_locks,
- )
-
- _pop_per_user_locks(username)
- _evict_collection_engines_cache(username)
-
- # 2b. Purge old backups (encrypted with old key) and create
+ # 2. Purge old backups (encrypted with old key) and create
# a fresh backup with the new key. Old-key backups are a
# security risk per NIST SP 800-57 / OWASP A02 — they remain
# decryptable with the compromised password.
@@ -1336,32 +1510,8 @@ def change_password(
f"for {_log_value_preview(username)}"
)
- # 3. Destroy ALL sessions for this user + clear password store
- session_manager.destroy_all_user_sessions(username)
-
- from ...database.session_passwords import (
- session_password_store,
- )
-
- session_password_store.clear_all_for_user(username)
-
- # Drop per-thread cached credentials too. These hold the OLD
- # plaintext password, which is now invalid — a pooled worker that
- # kept it would both retain a dead secret and fail to reopen the
- # database. See clear_user_credentials() for why per-request
- # cleanup cannot reach a worker thread under FastAPI.
- from ...database.thread_local_session import clear_user_credentials
-
- clear_user_credentials(username)
-
- # 3a. Disconnect live WebSocket connections so a socket authorised
- # under the old session stops receiving this user's events
- # (#5535). Every session is destroyed by a password change, so
- # this is the all-sessions scope, unlike logout's per-session one.
- _disconnect_user_sockets(username)
-
- # 4. Clear Flask session dict
- request.session.clear()
+ # 3. Revoke every old-password credential and session.
+ _revoke_user_credentials_after_rekey(request, username)
logger.info(f"Password changed for user {_log_value_preview(username)}")
flash(
diff --git a/src/local_deep_research/web/routers/metrics.py b/src/local_deep_research/web/routers/metrics.py
index 923c231ef8..8ef38a6554 100644
--- a/src/local_deep_research/web/routers/metrics.py
+++ b/src/local_deep_research/web/routers/metrics.py
@@ -30,6 +30,7 @@ from ...constants import (
get_available_strategies as get_central_available_strategies,
)
from ...domain_classifier import DomainClassifier, DomainClassification
+from ...database.rekey_lock import RekeyOperationalError
from ...database.session_context import get_user_db_session
from ...metrics import TokenCounter
from ...metrics.query_utils import (
@@ -2682,7 +2683,10 @@ async def api_journal_data_download(
# JournalReputationFilter._should_skip_journal_fetch_for_scope; this
# closes the manual button as the matching entry point.
# Settings read opens the user's SQLCipher session (sync) — keep it
- # off the event loop.
+ # off the event loop. A rekey sidecar or epoch fault on that read
+ # raises a RekeyOperationalError rather than reading as "no
+ # database", which would answer the default scope instead of the
+ # stored one; it is passed on below for the app's retryable 503.
scope, scope_raw = await run_db_sync(
lambda: _resolve_persisted_egress_scope(username)
)
@@ -2770,6 +2774,13 @@ async def api_journal_data_download(
# early-return "already up to date" branch (no fetch ran).
user_message = "Journal data is already up to date."
return {"success": True, "message": user_message}
+ except RekeyOperationalError:
+ # The scope could not be read (a rekey sidecar or epoch fault, such
+ # as an fd-exhaustion spike after require_auth's probe passed), so
+ # nothing was downloaded. Not a server error: the app's handler
+ # answers the same retryable 503 + Retry-After as require_auth,
+ # and logs only the fault's type name.
+ raise
except Exception:
logger.exception("Error downloading journal data")
return JSONResponse(
diff --git a/src/local_deep_research/web/services/socketio_asgi.py b/src/local_deep_research/web/services/socketio_asgi.py
index c4fd3b2d80..5f12e37a1a 100644
--- a/src/local_deep_research/web/services/socketio_asgi.py
+++ b/src/local_deep_research/web/services/socketio_asgi.py
@@ -469,8 +469,36 @@ async def connect(sid, environ, auth=None):
return False
from ...database.encrypted_db import db_manager
-
- if not db_manager.is_user_connected(username):
+ from ...database.rekey_lock import RekeyOperationalError
+ from ..dependencies.threadpool import run_db_sync
+
+ # The probe is not the microsecond dict lookup it once was: it routes
+ # through _acquire_current_connection_generation, which takes the
+ # per-user lifecycle lock and the cross-process rekey sidecar
+ # (filesystem I/O, and a wait of up to a few seconds for another
+ # holder). Run it through run_db_sync exactly like the
+ # open_user_database call below, so a rekey in progress holds up only
+ # this handshake's worker thread — not the event loop, which this
+ # single-worker deployment shares with every user's sockets and async
+ # routes.
+ try:
+ user_connected = await run_db_sync(
+ db_manager.is_user_connected, username
+ )
+ except RekeyOperationalError:
+ # The probe already reports a sidecar that cannot be opened or
+ # locked (fd exhaustion, EROFS, EACCES, ENOSPC), and an epoch read
+ # that fails, as a plain False. A failure to release the sidecar
+ # afterwards (unlock/close) raises RekeySidecarIOError instead:
+ # degrade that to "not connected" too — ensure_user_database's
+ # contract — so the handshake fails closed through the normal
+ # no-connection path instead of raising out of connect(). Either
+ # way, while an open/lock or epoch-read fault hides a cached
+ # engine, the lazy open below raises that fault before deriving a
+ # key, so the refused handshake costs no PBKDF2 derivation.
+ user_connected = False
+
+ if not user_connected:
from ...database.session_passwords import session_password_store
session_id = session_data.get("session_id")
@@ -500,8 +528,6 @@ async def connect(sid, environ, auth=None):
# state) on that worker before returning -- same shape as the
# equivalent fix at fastapi_app.py's DatabaseMiddleware
# (ensure_user_database).
- from ..dependencies.threadpool import run_db_sync
-
await run_db_sync(db_manager.open_user_database, username, password)
except Exception as e:
# Capture the type name only and log OUTSIDE the handler:
diff --git a/tests/database/backup/test_superseded_key_purge.py b/tests/database/backup/test_superseded_key_purge.py
new file mode 100644
index 0000000000..e076278285
--- /dev/null
+++ b/tests/database/backup/test_superseded_key_purge.py
@@ -0,0 +1,635 @@
+"""Which backups the pending-rekey recovery purge deletes, keeps or moves.
+
+``BackupService.purge_backups_under_superseded_key`` runs after a login
+resolved a pending password change, with only the password that opened the
+database. It must delete backups under the superseded key and nothing else:
+a backup under the current key, a backup it cannot read (a lock), and a
+backup under the current key with a damaged first page all survive. Real
+SQLCipher throughout.
+"""
+
+import multiprocessing
+import os
+import stat
+import time
+import uuid
+from unittest.mock import patch
+
+import pytest
+
+from local_deep_research.config.paths import get_encrypted_database_path
+from local_deep_research.database import encrypted_db
+from local_deep_research.database.backup.backup_service import BackupService
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ RekeyLockContentionError,
+ acquire_rekey_lock,
+ advance_rekey_epoch,
+ begin_rekey_epoch,
+ read_or_initialize_rekey_epoch,
+ read_rekey_epoch_state,
+)
+from local_deep_research.database.sqlcipher_compat import (
+ connect_sqlcipher,
+ get_sqlcipher_module,
+)
+from local_deep_research.database.sqlcipher_utils import (
+ apply_sqlcipher_pragmas,
+ get_salt_file_path,
+ set_sqlcipher_key,
+)
+
+OLD = "OldPurgeHorse1!" # noqa: S105
+NEW = "NewPurgeHorse2!" # noqa: S105
+LATEST = "LatestPurgeHorse3!" # noqa: S105
+
+
+def manager(tmp_path, monkeypatch):
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ mgr = DatabaseManager()
+ if not mgr.has_encryption:
+ pytest.skip("SQLCipher not available")
+ # The backup service resolves the database through the data directory.
+ mgr.data_dir = get_encrypted_database_path()
+ yield mgr
+ mgr.close_all_databases()
+
+
+def _user(manager):
+ username = f"purge_{uuid.uuid4().hex[:8]}"
+ assert manager.create_user_database(username, OLD) is not None
+ return username, manager._get_user_db_path(username)
+
+
+def _backup(username, password):
+ result = BackupService(username=username, password=password).create_backup(
+ force=True, prune=False
+ )
+ assert result.success, result.error
+ return result.backup_path
+
+
+def _completed_change_left_pending(manager, username, db_path):
+ """A proven password change whose epoch commit failed (pending left)."""
+ with patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=OSError(28, "No space left on device"),
+ ):
+ assert manager.change_password(username, OLD, NEW) is True
+ assert read_rekey_epoch_state(db_path).pending_epoch is not None
+ manager.close_all_databases()
+
+
+def test_resolving_login_deletes_old_key_backup_and_keeps_new_key_backup(
+ manager,
+):
+ # Given an old-key backup, then a backup made under the NEW key before
+ # the login that resolves the pending change
+ username, db_path = _user(manager)
+ old_backup = _backup(username, OLD)
+ _completed_change_left_pending(manager, username, db_path)
+ new_backup = _backup(username, NEW)
+
+ # When the user logs in with the new password
+ assert manager.open_user_database(username, NEW) is not None
+
+ # Then only the old-key backup is gone
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert not old_backup.exists()
+ assert new_backup.exists()
+ assert get_salt_file_path(new_backup).exists()
+
+
+def test_resolving_login_with_surviving_old_password_deletes_nothing(
+ manager,
+):
+ # Given backups under the old key and a pending change that never applied
+ username, db_path = _user(manager)
+ backups = [_backup(username, OLD), _backup(username, OLD)]
+ manager.close_all_databases()
+ begin_rekey_epoch(db_path)
+
+ # When the user logs in with the surviving old password
+ assert manager.open_user_database(username, OLD) is not None
+
+ # Then every backup is kept
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert all(path.exists() for path in backups)
+
+
+def test_resolving_login_keeps_a_later_password_changes_current_backup(
+ manager,
+):
+ username, db_path = _user(manager)
+ _completed_change_left_pending(manager, username, db_path)
+ original_purge = manager._purge_superseded_key_backups
+ later_backup = []
+
+ def change_again_before_purge(user, password, **kwargs):
+ # The recovery has finished and released its cold-open locks. A
+ # second password change can complete before its cleanup resumes.
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert manager.change_password(user, NEW, LATEST) is True
+ backup = _backup(user, LATEST)
+ assert (
+ BackupService(username=user, password=LATEST)._classify_backup_key(
+ backup
+ )
+ == "current"
+ )
+ later_backup.append(backup)
+ original_purge(user, password, **kwargs)
+
+ with (
+ patch.object(
+ manager,
+ "_purge_superseded_key_backups",
+ side_effect=change_again_before_purge,
+ ),
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ assert manager.open_user_database(username, NEW) is not None
+
+ # The stale recovery password would fail every page of this healthy
+ # backup and incorrectly classify it as superseded without fencing.
+ assert len(later_backup) == 1, "the second password change never ran"
+ backup = later_backup[0]
+ assert backup.exists(), "stale recovery deleted the current-key backup"
+ assert get_salt_file_path(backup).exists()
+ assert (
+ BackupService(username=username, password=LATEST)._classify_backup_key(
+ backup
+ )
+ == "current"
+ )
+ DatabaseManager._make_sqlcipher_connection(db_path, LATEST).close()
+
+
+def _try_rekey_lock_in_child(db_path, outcomes):
+ from pathlib import Path
+
+ try:
+ with acquire_rekey_lock(Path(db_path)):
+ outcomes.put("acquired")
+ except RekeyLockContentionError:
+ outcomes.put("contended")
+
+
+def test_recovery_holds_cross_process_rekey_lock_through_backup_deletion(
+ manager,
+):
+ username, db_path = _user(manager)
+ old_backup = _backup(username, OLD)
+ _completed_change_left_pending(manager, username, db_path)
+ original_remove = BackupService._remove_backup_files
+ observed = []
+
+ def remove_while_another_process_attempts_rekey(backup):
+ context = multiprocessing.get_context("spawn")
+ outcomes = context.Queue()
+ contender = context.Process(
+ target=_try_rekey_lock_in_child,
+ args=(str(db_path), outcomes),
+ )
+ contender.start()
+ try:
+ observed.append(outcomes.get(timeout=20))
+ contender.join(timeout=10)
+ assert contender.exitcode == 0
+ finally:
+ if contender.is_alive():
+ contender.terminate()
+ contender.join(timeout=5)
+ outcomes.close()
+ outcomes.join_thread()
+ original_remove(backup)
+
+ with (
+ patch.object(
+ BackupService,
+ "_remove_backup_files",
+ side_effect=remove_while_another_process_attempts_rekey,
+ ),
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ assert manager.open_user_database(username, NEW) is not None
+
+ # Check outside the purge's best-effort handler: an assertion raised
+ # inside deletion is caught there and must not make this test pass.
+ assert observed == ["contended"]
+ assert not old_backup.exists(), "the guarded purge never deleted anything"
+
+
+ "state", ["different", "missing", "pending", "malformed"]
+)
+def test_recovery_skips_purge_without_its_stable_epoch(manager, state):
+ db_path = manager._get_user_db_path("stale_recovery")
+ with acquire_rekey_lock(db_path):
+ epoch = read_or_initialize_rekey_epoch(db_path)
+ if state == "different":
+ advance_rekey_epoch(db_path)
+ elif state == "missing":
+ db_path.with_name(f"{db_path.name}.rekey.epoch").unlink()
+ elif state == "pending":
+ begin_rekey_epoch(db_path)
+ else:
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text(
+ "not-an-epoch", encoding="ascii"
+ )
+
+ with patch.object(
+ BackupService, "purge_backups_under_superseded_key"
+ ) as purge:
+ manager._purge_superseded_key_backups(
+ "stale_recovery", NEW, expected_epoch=epoch
+ )
+ purge.assert_not_called()
+
+
+def test_recovery_does_not_purge_backups_of_a_different_database(manager):
+ # A manager's pinned directory need not be the directory BackupService
+ # resolves from configuration. Its epoch cannot authorize that purge.
+ manager.data_dir = manager.data_dir / "pinned"
+ username = "pinned_recovery"
+ db_path = manager._get_user_db_path(username)
+ with acquire_rekey_lock(db_path):
+ epoch = read_or_initialize_rekey_epoch(db_path)
+ assert BackupService(username=username, password=NEW).db_path != db_path
+
+ with patch.object(
+ BackupService, "purge_backups_under_superseded_key"
+ ) as purge:
+ manager._purge_superseded_key_backups(
+ username, NEW, expected_epoch=epoch
+ )
+ purge.assert_not_called()
+
+
+def test_purge_keeps_a_backup_it_cannot_read(manager):
+ # Given an old-key backup that another connection holds locked
+ username, db_path = _user(manager)
+ old_backup = _backup(username, OLD)
+ holder = connect_sqlcipher(
+ get_sqlcipher_module(), str(old_backup), isolation_level=None
+ )
+ try:
+ cursor = holder.cursor()
+ set_sqlcipher_key(cursor, OLD, db_path=old_backup)
+ apply_sqlcipher_pragmas(cursor, creation_mode=False)
+ cursor.execute("PRAGMA locking_mode = EXCLUSIVE")
+ cursor.execute("BEGIN EXCLUSIVE")
+ cursor.execute("SELECT count(*) FROM sqlite_master").fetchone()
+
+ # When the purge runs under the new password
+ deleted = BackupService(
+ username=username, password=NEW
+ ).purge_backups_under_superseded_key()
+ finally:
+ holder.close()
+
+ # Then the unreadable backup is kept: nothing proved it superseded
+ assert deleted == 0
+ assert old_backup.exists()
+
+
+def test_purge_moves_a_damaged_current_key_backup_aside(manager):
+ # Given a backup under the current key whose first page is damaged
+ username, db_path = _user(manager)
+ backup = _backup(username, OLD)
+ with backup.open("r+b") as handle:
+ handle.seek(200)
+ byte = handle.read(1)
+ handle.seek(200)
+ handle.write(bytes([byte[0] ^ 0xFF]))
+ service = BackupService(username=username, password=OLD)
+
+ # When the purge runs under that same (current) password
+ deleted = service.purge_backups_under_superseded_key()
+
+ # Then it is not deleted but moved out of the backup set, owner-only
+ assert deleted == 0
+ assert not backup.exists()
+ moved = list(backup.parent.glob(f"{backup.name}.damaged-*"))
+ moved_db = [path for path in moved if not path.name.endswith(".salt")]
+ assert len(moved_db) == 1
+ assert stat.S_IMODE(os.stat(moved_db[0]).st_mode) == 0o600
+ assert moved_db[0].with_name(f"{moved_db[0].name}.salt").exists()
+ assert service.list_backups() == []
+ # And a later purge leaves it alone
+ assert service.purge_backups_under_superseded_key() == 0
+ assert moved_db[0].exists()
+
+
+def _rekey_live_database(db_path, old, new):
+ from local_deep_research.database.sqlcipher_utils import set_sqlcipher_rekey
+
+ conn = DatabaseManager._make_sqlcipher_connection(db_path, old)
+ try:
+ set_sqlcipher_rekey(conn, new, db_path=db_path)
+ conn.commit()
+ finally:
+ conn.close()
+
+
+def test_current_key_backup_with_missing_salt_is_kept(manager):
+ # Given a backup under the current key whose salt file is gone, so the
+ # key it derives (legacy salt) fails every page
+ username, _db_path = _user(manager)
+ backup = _backup(username, OLD)
+ get_salt_file_path(backup).unlink()
+ service = BackupService(username=username, password=OLD)
+
+ # When it is classified and the purge runs
+ assert service._classify_backup_key(backup) == "unknown"
+ assert service.purge_backups_under_superseded_key() == 0
+
+ # Then it is kept: restoring the salt would make it readable
+ assert backup.exists()
+
+
+def test_current_key_backup_with_damaged_salt_is_kept(manager):
+ # Given a backup under the current key with a same-size damaged salt
+ username, _db_path = _user(manager)
+ backup = _backup(username, OLD)
+ salt_path = get_salt_file_path(backup)
+ salt = bytearray(salt_path.read_bytes())
+ salt[0] ^= 0x01
+ salt_path.write_bytes(bytes(salt))
+ service = BackupService(username=username, password=OLD)
+
+ # When it is classified and the purge runs
+ assert service._classify_backup_key(backup) == "unknown"
+ assert service.purge_backups_under_superseded_key() == 0
+
+ # Then it is kept
+ assert backup.exists()
+ assert salt_path.exists()
+
+
+def test_old_password_backup_sharing_the_live_salt_is_still_deleted(manager):
+ # Given an old-password backup (same salt as the live database, which
+ # a rekey never changes) and the live database rekeyed to NEW
+ username, db_path = _user(manager)
+ backup = _backup(username, OLD)
+ manager.close_all_databases()
+ _rekey_live_database(db_path, OLD, NEW)
+ service = BackupService(username=username, password=NEW)
+
+ # When the purge runs under the new password
+ assert service._classify_backup_key(backup) == "superseded"
+ assert service.purge_backups_under_superseded_key() == 1
+
+ # Then the old-password backup is gone
+ assert not backup.exists()
+
+
+def test_legacy_salt_database_old_password_backup_is_deleted(manager):
+ # Given a legacy database (no .salt: the built-in shared salt) and an
+ # old-password backup of it, which has no .salt either; both derive
+ # their key from the same legacy salt, so a key failure is the password
+ from local_deep_research.database.sqlcipher_utils import (
+ create_sqlcipher_connection,
+ )
+ from local_deep_research.config.paths import get_user_database_filename
+
+ username = f"legacy_{uuid.uuid4().hex[:8]}"
+ db_path = get_encrypted_database_path() / get_user_database_filename(
+ username
+ )
+ conn = create_sqlcipher_connection(db_path, OLD, creation_mode=True)
+ try:
+ conn.execute("CREATE TABLE notes (body TEXT)")
+ conn.execute("INSERT INTO notes VALUES ('x')")
+ conn.commit()
+ finally:
+ conn.close()
+ assert not get_salt_file_path(db_path).exists()
+ backup = _backup(username, OLD)
+ assert not get_salt_file_path(backup).exists()
+ _rekey_live_database(db_path, OLD, NEW)
+ service = BackupService(username=username, password=NEW)
+
+ # When the purge runs under the new password
+ assert service._classify_backup_key(backup) == "superseded"
+ assert service.purge_backups_under_superseded_key() == 1
+
+ # Then it is deleted like any old-password backup
+ assert not backup.exists()
+
+
+def _quarantine_during_recovery(manager):
+ username, db_path = _user(manager)
+ backup = _backup(username, OLD)
+ with backup.open("r+b") as handle:
+ handle.seek(200)
+ byte = handle.read(1)
+ handle.seek(200)
+ handle.write(bytes([byte[0] ^ 0xFF]))
+ manager.close_all_databases()
+ with acquire_rekey_lock(db_path):
+ begin_rekey_epoch(db_path)
+ assert manager.open_user_database(username, OLD) is not None
+ quarantines = [
+ path
+ for path in backup.parent.glob(f"{backup.name}.damaged-*")
+ if not path.name.endswith(".salt")
+ ]
+ assert len(quarantines) == 1
+ quarantined = quarantines[0]
+ assert (
+ BackupService(username=username, password=OLD)._classify_backup_key(
+ quarantined
+ )
+ == "damaged"
+ )
+ assert get_salt_file_path(quarantined).exists()
+ return username, db_path, quarantined
+
+
+def test_completed_password_change_purges_quarantined_old_key_copy(manager):
+ username, db_path, quarantined = _quarantine_during_recovery(manager)
+ assert manager.change_password(username, OLD, NEW) is True
+ refreshed = BackupService(
+ username=username, password=NEW
+ ).purge_and_refresh()
+
+ assert refreshed.success, refreshed.error
+ assert not quarantined.exists()
+ assert not get_salt_file_path(quarantined).exists()
+ assert (
+ BackupService(username=username, password=NEW)._classify_backup_key(
+ refreshed.backup_path
+ )
+ == "current"
+ )
+ connection = DatabaseManager._make_sqlcipher_connection(db_path, NEW)
+ connection.close()
+
+
+def test_later_pending_recovery_purges_quarantined_old_key_copy(manager):
+ username, db_path, quarantined = _quarantine_during_recovery(manager)
+ _completed_change_left_pending(manager, username, db_path)
+
+ assert manager.open_user_database(username, NEW) is not None
+
+ assert not quarantined.exists()
+ assert not get_salt_file_path(quarantined).exists()
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+
+
+def test_recovery_keeps_current_key_quarantine_without_renaming_it(manager):
+ username, db_path, quarantined = _quarantine_during_recovery(manager)
+ before = quarantined.read_bytes()
+ manager.close_all_databases()
+ with acquire_rekey_lock(db_path):
+ begin_rekey_epoch(db_path)
+
+ assert manager.open_user_database(username, OLD) is not None
+
+ assert quarantined.read_bytes() == before
+ assert get_salt_file_path(quarantined).exists()
+ assert len(list(quarantined.parent.glob("*.damaged-*"))) == 2
+ assert BackupService(username=username, password=OLD).list_backups() == []
+ assert (
+ BackupService(username=username, password=OLD).get_latest_backup()
+ is None
+ )
+
+
[email protected]("salt_state", ["missing", "different"])
+def test_recovery_keeps_quarantined_copy_without_a_matching_salt(
+ manager, salt_state
+):
+ username, db_path, quarantined = _quarantine_during_recovery(manager)
+ salt_path = get_salt_file_path(quarantined)
+ if salt_state == "missing":
+ salt_path.unlink()
+ else:
+ salt = bytearray(salt_path.read_bytes())
+ salt[0] ^= 0x01
+ salt_path.write_bytes(salt)
+ _completed_change_left_pending(manager, username, db_path)
+
+ assert manager.open_user_database(username, NEW) is not None
+
+ assert quarantined.exists()
+ if salt_state == "different":
+ assert salt_path.exists()
+
+
+def _retention_service(tmp_path, max_backups=1, max_age_days=7):
+ service = BackupService(
+ username="quarantine_retention",
+ password=OLD,
+ max_backups=max_backups,
+ max_age_days=max_age_days,
+ )
+ service.backup_dir = tmp_path / "backups"
+ service.backup_dir.mkdir()
+ service.db_path = tmp_path / "unused.db"
+ return service
+
+
+def _quarantined_pair(service, name="fixture", age_days=0):
+ path = service.backup_dir / (
+ f"ldr_backup_{name}.db.damaged-20261018_170000_123456"
+ )
+ path.write_bytes(b"\x01" * 32)
+ get_salt_file_path(path).write_bytes(b"\x02" * 16)
+ modified = time.time() - age_days * 86400
+ os.utime(path, (modified, modified))
+ return path
+
+
+def test_retention_removes_an_aged_quarantine_and_its_salt(tmp_path):
+ service = _retention_service(tmp_path, max_backups=5)
+ quarantined = _quarantined_pair(service, age_days=8)
+
+ assert service._cleanup_old_backups() == 1
+
+ assert not quarantined.exists()
+ assert not get_salt_file_path(quarantined).exists()
+
+
+def test_retention_cleans_only_recognized_stale_quarantine_orphan_salts(
+ tmp_path,
+):
+ service = _retention_service(tmp_path)
+ quarantined = _quarantined_pair(service)
+ quarantined.unlink()
+ orphan = get_salt_file_path(quarantined)
+ unrelated = service.backup_dir / (
+ "ldr_backup_operator.db.damaged-not-a-stamp.salt"
+ )
+ unrelated.write_bytes(b"\x02" * 16)
+ modified = time.time() - 2 * 3600
+ os.utime(orphan, (modified, modified))
+ os.utime(unrelated, (modified, modified))
+
+ assert service._cleanup_old_backups() == 0
+
+ assert not orphan.exists()
+ assert unrelated.exists()
+
+
[email protected]("max_backups", [1, 2])
+def test_quarantine_count_retention_preserves_usable_restore_candidate(
+ tmp_path, max_backups
+):
+ service = _retention_service(
+ tmp_path, max_backups=max_backups, max_age_days=0
+ )
+ usable = service.backup_dir / "ldr_backup_usable.db"
+ usable.write_bytes(b"\x03" * 32)
+ modified = time.time() - 3 * 86400
+ os.utime(usable, (modified, modified))
+ old_quarantine = _quarantined_pair(service, "older", age_days=2)
+ new_quarantine = _quarantined_pair(service, "newer", age_days=1)
+
+ assert service._cleanup_old_backups() == 3 - max_backups
+
+ assert usable.exists(), "damaged copies displaced the usable backup"
+ assert service.get_latest_backup() == usable
+ assert not old_quarantine.exists()
+ assert not get_salt_file_path(old_quarantine).exists()
+ assert new_quarantine.exists() is (max_backups == 2)
+ assert get_salt_file_path(new_quarantine).exists() is (max_backups == 2)
+
+
[email protected]("operation", ["normal", "recovery", "retention"])
+def test_quarantine_cleanup_does_not_follow_links_or_match_unrelated_files(
+ tmp_path, operation
+):
+ service = _retention_service(tmp_path, max_age_days=1)
+ outside = tmp_path / "outside.db"
+ outside.write_bytes(b"\x04" * 32)
+ linked = service.backup_dir / (
+ "ldr_backup_link.db.damaged-20261018_170000_123456"
+ )
+ linked.symlink_to(outside)
+ directory = service.backup_dir / (
+ "ldr_backup_directory.db.damaged-20261018_170000_123456"
+ )
+ directory.mkdir()
+ unrelated = (
+ service.backup_dir / "ldr_backup_operator.db.damaged-not-a-stamp"
+ )
+ unrelated.write_bytes(b"\x05" * 32)
+ before = outside.read_bytes()
+ if operation == "normal":
+ with patch.object(service, "_create_backup_impl"):
+ service.purge_and_refresh()
+ elif operation == "recovery":
+ assert service.purge_backups_under_superseded_key() == 0
+ else:
+ modified = time.time() - 2 * 86400
+ os.utime(unrelated, (modified, modified))
+ assert service._cleanup_old_backups() == 0
+
+ assert outside.read_bytes() == before
+ assert linked.is_symlink()
+ assert directory.is_dir()
+ assert unrelated.exists()
diff --git a/tests/database/test_create_user_database_serialized.py b/tests/database/test_create_user_database_serialized.py
index 8960f10864..258c8de99b 100644
--- a/tests/database/test_create_user_database_serialized.py
+++ b/tests/database/test_create_user_database_serialized.py
@@ -98,3 +98,34 @@ def test_concurrent_creates_for_one_user_leave_a_usable_database(manager):
assert db_path.exists()
manager.close_all_databases()
assert manager.open_user_database(username, PASSWORD) is not None
+
+
[email protected]("leftover_pending", [False, True])
+def test_creation_mints_a_fresh_epoch_over_leftover_sidecars(
+ manager, leftover_pending
+):
+ # Given epoch sidecars left beside a database path that no longer has
+ # a database (an account removed without them, a partial restore)
+ from local_deep_research.database.rekey_lock import read_rekey_epoch_state
+
+ username = "recreated_user"
+ db_path = manager._get_user_db_path(username)
+ db_path.parent.mkdir(parents=True, exist_ok=True)
+ stale_epoch = "00000000-0000-0000-0000-000000000001"
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text(
+ stale_epoch, encoding="ascii"
+ )
+ if leftover_pending:
+ db_path.with_name(f"{db_path.name}.rekey.pending").write_text(
+ "00000000-0000-0000-0000-000000000002", encoding="ascii"
+ )
+
+ # When the account is registered again
+ assert manager.create_user_database(username, PASSWORD) is not None
+
+ # Then it gets a fresh stable epoch, not the earlier database's
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch is not None
+ assert state.committed_epoch != stale_epoch
+ assert manager._connection_epochs[username] == state.committed_epoch
diff --git a/tests/database/test_database_manager_extended.py b/tests/database/test_database_manager_extended.py
index 5af49e035a..ff1d13982b 100644
--- a/tests/database/test_database_manager_extended.py
+++ b/tests/database/test_database_manager_extended.py
@@ -9,6 +9,8 @@ import pytest
from unittest.mock import patch, MagicMock
from pathlib import Path
+from local_deep_research.database.rekey_lock import RekeyEpochState
+
class TestDatabaseEncryption:
"""Tests for database encryption functionality"""
@@ -220,12 +222,24 @@ class TestDatabaseOperations:
manager = DatabaseManager()
mock_engine = MagicMock()
- manager.connections["testuser"] = mock_engine
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
# This will try to create a real session, mock the sessionmaker
- with patch(
- "local_deep_research.database.encrypted_db.sessionmaker"
- ) as mock_sm:
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.sessionmaker"
+ ) as mock_sm,
+ ):
mock_session = MagicMock()
mock_sm.return_value = MagicMock(return_value=mock_session)
@@ -320,12 +334,25 @@ class TestDatabaseIntegrity:
MagicMock(
fetchone=MagicMock(return_value=("ok",))
), # quick_check
+ MagicMock(
+ fetchone=MagicMock(return_value=(0, 0, 0))
+ ), # wal_checkpoint(PASSIVE) - nothing left behind
iter([]), # cipher_integrity_check - no failures
]
- manager.connections["testuser"] = mock_engine
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is True
@@ -353,11 +380,27 @@ class TestDatabaseIntegrity:
# Mock failed integrity check
mock_conn.execute.return_value.fetchone.return_value = ("corrupt",)
- manager.connections["testuser"] = mock_engine
+ # A trusted, epoch-validated entry: a bare ``connections[...]``
+ # plant has no epoch behind it, is detached on the first claim, and
+ # returns False before quick_check ever runs.
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is False
+ assert mock_conn.execute.called, (
+ "the check never reached quick_check -- measuring the wrong branch"
+ )
class TestPasswordChange:
@@ -376,9 +419,10 @@ class TestPasswordChange:
with patch.dict("os.environ", {"LDR_ALLOW_UNENCRYPTED": "true"}):
manager = DatabaseManager()
- result = manager.change_password("user", "old", "new")
-
- assert result is False
+ # Documented boolean contract for the static unencrypted
+ # fallback mode (tests/security/test_db_connection_
+ # unencrypted_mode.py pins the same).
+ assert manager.change_password("user", "old", "new") is False
class TestUserExists:
diff --git a/tests/database/test_encrypted_db_coverage.py b/tests/database/test_encrypted_db_coverage.py
index baa8b94587..ae081da279 100644
--- a/tests/database/test_encrypted_db_coverage.py
+++ b/tests/database/test_encrypted_db_coverage.py
@@ -13,11 +13,17 @@ Covers:
import threading
from contextlib import contextmanager
+from pathlib import Path
+from tempfile import TemporaryDirectory
+from typing import Final
from unittest.mock import MagicMock, patch
import pytest # noqa: F401
+from local_deep_research.database.rekey_lock import RekeyEpochState
+
MODULE = "local_deep_research.database.encrypted_db"
+FIXTURE_EPOCH: Final = "00000000-0000-0000-0000-000000000000"
# ---------------------------------------------------------------------------
@@ -29,6 +35,7 @@ MODULE = "local_deep_research.database.encrypted_db"
def _unencrypted_manager():
"""Yield a DatabaseManager that has encryption disabled (SQLCipher not available)."""
with (
+ TemporaryDirectory() as temp_dir,
patch(
f"{MODULE}.get_data_directory",
return_value=MagicMock(
@@ -39,6 +46,11 @@ def _unencrypted_manager():
),
),
patch(f"{MODULE}.get_env_setting", return_value=True),
+ patch(f"{MODULE}.read_rekey_epoch", return_value=FIXTURE_EPOCH),
+ patch(
+ f"{MODULE}.read_rekey_epoch_state",
+ return_value=RekeyEpochState(FIXTURE_EPOCH, None),
+ ),
patch(
f"{MODULE}.get_sqlcipher_module",
side_effect=ImportError("no sqlcipher"),
@@ -52,12 +64,16 @@ def _unencrypted_manager():
# __init__ is bypassed here; mirror its per-user init-lock dict so the
# close_*/open paths that reference mgr._init_locks don't AttributeError.
mgr._init_locks = {}
+ mgr._rekey_locks = {}
# Likewise mirror the cached-connection password-verifier state so the
# close_*/open paths that read/clear it don't AttributeError.
import secrets
mgr._password_verifiers = {}
+ mgr._connection_epochs = {}
mgr._verifier_key = secrets.token_bytes(32)
+ mgr._data_dir_override = Path(temp_dir)
+ mgr._initialized_data_dirs = set()
mgr.has_encryption = False
import os
@@ -68,6 +84,11 @@ def _unencrypted_manager():
yield mgr
+def _cache_engine(mgr, username, engine):
+ entry = mgr._make_connection_cache_entry(engine, "", FIXTURE_EPOCH)
+ mgr._cache_connection(username, entry)
+
+
# ---------------------------------------------------------------------------
# _is_valid_encryption_key
# ---------------------------------------------------------------------------
@@ -107,7 +128,7 @@ class TestConnectionState:
def test_is_user_connected_true_when_present(self):
with _unencrypted_manager() as mgr:
- mgr.connections["alice"] = MagicMock()
+ _cache_engine(mgr, "alice", MagicMock())
assert mgr.is_user_connected("alice") is True
def test_get_connected_usernames_empty(self):
@@ -116,8 +137,8 @@ class TestConnectionState:
def test_get_connected_usernames_snapshot(self):
with _unencrypted_manager() as mgr:
- mgr.connections["alice"] = MagicMock()
- mgr.connections["bob"] = MagicMock()
+ _cache_engine(mgr, "alice", MagicMock())
+ _cache_engine(mgr, "bob", MagicMock())
names = mgr.get_connected_usernames()
assert names == {"alice", "bob"}
@@ -137,8 +158,8 @@ class TestGetMemoryUsage:
def test_with_connections(self):
with _unencrypted_manager() as mgr:
- mgr.connections["user1"] = MagicMock()
- mgr.connections["user2"] = MagicMock()
+ _cache_engine(mgr, "user1", MagicMock())
+ _cache_engine(mgr, "user2", MagicMock())
stats = mgr.get_memory_usage()
assert stats["active_connections"] == 2
assert stats["estimated_memory_mb"] == pytest.approx(2 * 3.5)
@@ -173,7 +194,7 @@ class TestCloseUserDatabase:
def test_closes_and_removes_connection(self):
with _unencrypted_manager() as mgr:
mock_engine = MagicMock()
- mgr.connections["alice"] = mock_engine
+ _cache_engine(mgr, "alice", mock_engine)
mgr.close_user_database("alice")
mock_engine.dispose.assert_called_once()
assert "alice" not in mgr.connections
@@ -186,7 +207,7 @@ class TestCloseUserDatabase:
with _unencrypted_manager() as mgr:
mock_engine = MagicMock()
mock_engine.dispose.side_effect = RuntimeError("dispose failed")
- mgr.connections["alice"] = mock_engine
+ _cache_engine(mgr, "alice", mock_engine)
mgr.close_user_database("alice")
assert "alice" not in mgr.connections
@@ -200,8 +221,8 @@ class TestCloseAllDatabases:
def test_disposes_all_engines(self):
with _unencrypted_manager() as mgr:
e1, e2 = MagicMock(), MagicMock()
- mgr.connections["a"] = e1
- mgr.connections["b"] = e2
+ _cache_engine(mgr, "a", e1)
+ _cache_engine(mgr, "b", e2)
mgr.close_all_databases()
e1.dispose.assert_called_once()
e2.dispose.assert_called_once()
diff --git a/tests/database/test_encrypted_db_manager.py b/tests/database/test_encrypted_db_manager.py
index bee9a56465..a10b4659c3 100644
--- a/tests/database/test_encrypted_db_manager.py
+++ b/tests/database/test_encrypted_db_manager.py
@@ -19,6 +19,12 @@ from unittest.mock import MagicMock, patch
import pytest
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochState,
+ RekeyStage,
+ RekeyUnavailableError,
+)
+
class TestDatabaseCreation:
"""Tests for create_user_database functionality."""
@@ -297,14 +303,19 @@ class TestDatabaseOpening:
manager = DatabaseManager()
mock_engine = MagicMock()
- manager.connections["testuser"] = mock_engine
- # A cached engine is now trusted only when a verifier proves which
- # password opened it (see the cached-connection password bypass
- # fix). Record the matching verifier so this exercises the intended
- # cache-reuse path rather than the fail-closed fallback.
- manager._record_password_verifier("testuser", "password")
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.open_user_database("testuser", "password")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.open_user_database("testuser", "password")
assert result is mock_engine
@@ -499,7 +510,6 @@ class TestPasswordChange:
def test_change_password_no_encryption_returns_false(
self, mock_data_dir, tmp_path
):
- """Test password change returns False when encryption not available."""
from local_deep_research.database.encrypted_db import DatabaseManager
mock_data_dir.return_value = tmp_path
@@ -510,17 +520,19 @@ class TestPasswordChange:
with patch.dict("os.environ", {"LDR_ALLOW_UNENCRYPTED": "true"}):
manager = DatabaseManager()
- result = manager.change_password(
+ # Documented boolean contract: unencrypted mode is a static
+ # deployment mode, not an operational failure (see
+ # tests/security/test_db_connection_unencrypted_mode.py).
+ changed = manager.change_password(
"testuser", "oldpass", "newpass"
)
- assert result is False
+ assert changed is False
@patch("local_deep_research.database.encrypted_db.get_data_directory")
- def test_change_password_nonexistent_db_returns_false(
+ def test_change_password_nonexistent_db_raises_unavailable(
self, mock_data_dir, tmp_path
):
- """Test password change returns False for nonexistent database."""
from local_deep_research.database.encrypted_db import DatabaseManager
mock_data_dir.return_value = tmp_path
@@ -534,11 +546,10 @@ class TestPasswordChange:
with patch.object(
manager, "_get_user_db_path", return_value=nonexistent
):
- result = manager.change_password(
- "testuser", "oldpass", "newpass"
- )
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("testuser", "oldpass", "newpass")
- assert result is False
+ assert raised.value.stage is RekeyStage.PREPARATION
@patch("local_deep_research.database.encrypted_db.get_data_directory")
def test_change_password_wrong_old_password_returns_false(
@@ -561,13 +572,22 @@ class TestPasswordChange:
with patch.object(
manager, "_get_user_db_path", return_value=db_file
):
- # open_user_database fails with wrong password
with patch.object(
- manager, "open_user_database", return_value=None
- ):
- result = manager.change_password(
- "testuser", "wrongpass", "newpass"
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=ValueError("Failed to verify database key"),
+ ) as make_connection:
+ with patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey"
+ ) as rekey:
+ result = manager.change_password(
+ "testuser", "wrongpass", "newpass"
+ )
+
+ make_connection.assert_called_once_with(
+ db_file, "wrongpass"
)
+ rekey.assert_not_called()
assert result is False
@@ -673,12 +693,25 @@ class TestDatabaseIntegrity:
MagicMock(
fetchone=MagicMock(return_value=("ok",))
), # quick_check
+ MagicMock(
+ fetchone=MagicMock(return_value=(0, 4, 4))
+ ), # wal_checkpoint(PASSIVE) - every frame checkpointed
iter([]), # cipher_integrity_check - no failures
]
- manager.connections["testuser"] = mock_engine
+ epoch = "00000000-0000-0000-0000-000000000000"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is True
@@ -706,11 +739,27 @@ class TestDatabaseIntegrity:
# Mock failed quick_check
mock_conn.execute.return_value.fetchone.return_value = ("corrupt",)
- manager.connections["testuser"] = mock_engine
+ # A trusted, epoch-validated entry: a bare ``connections[...]``
+ # plant has no epoch behind it, is detached on the first claim, and
+ # returns False before quick_check ever runs.
+ epoch = "00000000-0000-0000-0000-000000000000"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is False
+ assert mock_conn.execute.called, (
+ "the check never reached quick_check -- measuring the wrong branch"
+ )
@patch("local_deep_research.database.encrypted_db.get_data_directory")
def test_integrity_check_cipher_failure(self, mock_data_dir, tmp_path):
@@ -738,16 +787,99 @@ class TestDatabaseIntegrity:
MagicMock(
fetchone=MagicMock(return_value=("ok",))
), # quick_check
+ MagicMock(
+ fetchone=MagicMock(return_value=(0, 4, 4))
+ ), # wal_checkpoint(PASSIVE) - every frame checkpointed
iter(
[("HMAC failure",)]
), # cipher_integrity_check - has failures
]
- manager.connections["testuser"] = mock_engine
+ epoch = "00000000-0000-0000-0000-000000000000"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is False
+ assert mock_conn.execute.call_count == 3
+
+ @pytest.mark.parametrize(
+ "checkpoint_row",
+ [
+ pytest.param((0, 9, 3), id="frames-left-behind"),
+ pytest.param((1, -1, -1), id="checkpoint-busy"),
+ ],
+ )
+ @patch("local_deep_research.database.encrypted_db.get_data_directory")
+ def test_integrity_check_skips_cipher_scan_while_wal_not_checkpointed(
+ self, mock_data_dir, tmp_path, checkpoint_row
+ ):
+ """cipher_integrity_check reads only the main file; with WAL frames
+ still un-checkpointed (a reader pins them) or the checkpoint busy
+ (``(1, -1, -1)``: a TRUNCATE checkpoint waits on a reader) it is
+ skipped, not run against pages the WAL has superseded."""
+ from local_deep_research.database.encrypted_db import DatabaseManager
+
+ mock_data_dir.return_value = tmp_path
+
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+
+ mock_engine = MagicMock()
+ mock_conn = MagicMock()
+ mock_engine.connect.return_value.__enter__ = MagicMock(
+ return_value=mock_conn
+ )
+ mock_engine.connect.return_value.__exit__ = MagicMock(
+ return_value=False
+ )
+ # The checkpoint result also iterates to an HMAC failure row, so
+ # a cipher scan that runs in the checkpoint's place (the gate
+ # removed) reports a failure instead of passing on an empty
+ # MagicMock iteration. Any further statement fails loudly.
+ checkpoint_result = MagicMock(
+ fetchone=MagicMock(return_value=checkpoint_row)
+ )
+ checkpoint_result.__iter__.return_value = iter([("HMAC failure",)])
+ mock_conn.execute.side_effect = [
+ MagicMock(
+ fetchone=MagicMock(return_value=("ok",))
+ ), # quick_check
+ checkpoint_result, # wal_checkpoint(PASSIVE)
+ AssertionError(
+ "cipher_integrity_check ran without a full checkpoint"
+ ),
+ ]
+
+ epoch = "00000000-0000-0000-0000-000000000000"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
+
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
+
+ assert result is True
+ assert [
+ str(call.args[0]) for call in mock_conn.execute.call_args_list
+ ] == ["PRAGMA quick_check", "PRAGMA wal_checkpoint(PASSIVE)"]
@patch("local_deep_research.database.encrypted_db.get_data_directory")
def test_integrity_check_exception_returns_false(
@@ -766,11 +898,25 @@ class TestDatabaseIntegrity:
mock_engine = MagicMock()
mock_engine.connect.side_effect = Exception("Connection failed")
- manager.connections["testuser"] = mock_engine
+ # A trusted, epoch-validated entry (see the quick_check test above).
+ epoch = "00000000-0000-0000-0000-000000000000"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- result = manager.check_database_integrity("testuser")
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ result = manager.check_database_integrity("testuser")
assert result is False
+ assert mock_engine.connect.called, (
+ "the check never reached the engine -- measuring the wrong branch"
+ )
class TestMemoryUsage:
@@ -879,13 +1025,22 @@ class TestThreadSafeSessionForMetrics:
manager = DatabaseManager()
mock_engine = MagicMock()
- manager.connections["testuser"] = mock_engine
- # The cached engine is trusted on a metrics-session cache hit only
- # when the presented password matches the recorded verifier.
- manager._record_password_verifier("testuser", "password")
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- with patch.object(
- manager, "_get_user_db_path", return_value=db_file
+ with (
+ patch.object(
+ manager, "_get_user_db_path", return_value=db_file
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ),
):
session = manager.create_thread_safe_session_for_metrics(
"testuser", "password"
@@ -935,15 +1090,21 @@ class TestThreadSafeSessionForMetrics:
)
conn.commit()
- # Inject the engine so create_thread_safe_session_for_metrics
- # finds it on cache hit, with a verifier matching the password the
- # writer threads present ("unused") so the cache hit is trusted.
- manager.connections["testuser"] = engine
- manager._record_password_verifier("testuser", "unused")
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(engine, "unused", epoch),
+ )
try:
- with patch.object(
- manager, "_get_user_db_path", return_value=db_file
+ with (
+ patch.object(
+ manager, "_get_user_db_path", return_value=db_file
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ),
):
num_threads = 10
writes_per_thread = 10
@@ -1036,11 +1197,23 @@ class TestSessionManagement:
manager = DatabaseManager()
mock_engine = MagicMock()
- manager.connections["testuser"] = mock_engine
+ epoch = "test-epoch"
+ manager._cache_connection(
+ "testuser",
+ manager._make_connection_cache_entry(
+ mock_engine, "password", epoch
+ ),
+ )
- with patch(
- "local_deep_research.database.encrypted_db.sessionmaker"
- ) as mock_sm:
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.sessionmaker"
+ ) as mock_sm,
+ ):
mock_session = MagicMock()
mock_sm.return_value = MagicMock(return_value=mock_session)
@@ -1573,8 +1746,9 @@ class TestRemovePartialUserDbFiles:
def test_removes_db_salt_and_all_sidecars(self, tmp_path):
"""Every artifact a half-created DB can leave — the .db, its .salt,
- and the -wal/-shm/-journal sidecars — must be removed, so a failed
- create leaves nothing that blocks a retry.
+ rekey epoch and pending files, and journal sidecars — must be
+ removed, while the stable rekey lock inode survives so concurrent
+ processes keep one lock domain.
Guards each unlink line independently of journal_mode: a typo in any
one of them leaves its file behind and fails this test (the
@@ -1589,21 +1763,26 @@ class TestRemovePartialUserDbFiles:
)
db_path = tmp_path / "ldr_user_deadbeef.db"
+ lock_path = db_path.with_name(f"{db_path.name}.rekey.lock")
artifacts = [
db_path,
get_salt_file_path(db_path),
db_path.with_name(db_path.name + "-wal"),
db_path.with_name(db_path.name + "-shm"),
db_path.with_name(db_path.name + "-journal"),
+ db_path.with_name(f"{db_path.name}.rekey.epoch"),
+ db_path.with_name(f"{db_path.name}.rekey.pending"),
]
for path in artifacts:
path.write_bytes(b"x")
assert path.exists()
+ lock_path.write_bytes(b"stable-lock")
_remove_partial_user_db_files(db_path)
for path in artifacts:
assert not path.exists(), f"{path.name} was not removed"
+ assert lock_path.read_bytes() == b"stable-lock"
def test_missing_files_are_a_noop_not_an_error(self, tmp_path):
"""Never raises when artifacts are already absent (missing_ok)."""
diff --git a/tests/database/test_encrypted_db_manager_contracts.py b/tests/database/test_encrypted_db_manager_contracts.py
index 817d81515e..d59333ff4e 100644
--- a/tests/database/test_encrypted_db_manager_contracts.py
+++ b/tests/database/test_encrypted_db_manager_contracts.py
@@ -59,9 +59,17 @@ def _manager(root_getter, *, has_encryption=True):
yield DatabaseManager()
-def _register(mgr, username, engine, password="pw"):
- """Publish ``engine`` for ``username`` exactly as a verified open would."""
- mgr._cache_connection(username, engine, password)
+def _register(mgr, username, engine, password="pw", epoch="epoch"):
+ """Publish ``engine`` for ``username`` exactly as a verified open would.
+
+ Epoch-fenced cache: the engine, its verifier, and the durable epoch
+ publish together. Tests that exercise the claim paths mock
+ ``read_rekey_epoch_state`` to ``RekeyEpochState(epoch, None)`` via
+ ``_trust_epoch``; tests of pure teardown semantics do not need it.
+ """
+ mgr._cache_connection(
+ username, mgr._make_connection_cache_entry(engine, password, epoch)
+ )
@contextlib.contextmanager
@@ -119,6 +127,25 @@ class _FakeEngine:
# ===========================================================================
+def _trust_epoch(epoch="epoch"):
+ """Pretend the durable epoch state matches the fabricated cache.
+
+ Production caches are only published by paths that initialize the
+ durable epoch files first, so a cached engine always has matching
+ on-disk state. Contract tests that fabricate a cache with no files
+ use this to stand in for that durable state; the real-file fencing
+ behaviour is covered by tests/database/test_rekey_cache_fencing.py.
+ """
+ from local_deep_research.database.rekey_lock import RekeyEpochState
+
+ with patch(
+ "local_deep_research.database.encrypted_db.read_rekey_epoch_state",
+ return_value=RekeyEpochState(epoch, None),
+ ):
+ yield
+
+
def test_checked_out_session_outlives_close_user_database(tmp_path):
"""A Session handed out before the close keeps a LIVE connection after it.
@@ -137,12 +164,14 @@ def test_checked_out_session_outlives_close_user_database(tmp_path):
with _manager(lambda: tmp_path / "root") as mgr:
_register(mgr, "alice", engine)
- session = mgr.get_session("alice")
+ with _trust_epoch():
+ session = mgr.get_session("alice")
assert session is not None
assert session.execute(text("SELECT 1")).scalar() == 1
raw = session.connection().connection.dbapi_connection
- mgr.close_user_database("alice")
+ with _trust_epoch():
+ mgr.close_user_database("alice")
# Manager-visible state says the user is gone...
assert mgr.is_user_connected("alice") is False
@@ -200,7 +229,8 @@ def test_manager_holds_no_session_registry_to_reclaim(tmp_path):
with _manager(lambda: tmp_path / "root") as mgr:
_register(mgr, "alice", engine)
- sessions = [mgr.get_session("alice") for _ in range(2)]
+ with _trust_epoch():
+ sessions = [mgr.get_session("alice") for _ in range(2)]
for session in sessions:
session.execute(text("SELECT 1"))
@@ -211,7 +241,8 @@ def test_manager_holds_no_session_registry_to_reclaim(tmp_path):
assert usage["active_connections"] == 1
assert usage["active_sessions"] == 0
- mgr.close_user_database("alice")
+ with _trust_epoch():
+ mgr.close_user_database("alice")
# Closing the user reclaims neither Session.
for session in sessions:
@@ -222,43 +253,59 @@ def test_manager_holds_no_session_registry_to_reclaim(tmp_path):
def test_close_user_database_holds_global_lock_across_dispose(tmp_path):
- """``dispose()`` runs while holding the process-wide ``_connections_lock``.
-
- ``close_user_database`` does its WAL checkpoint and its dispose inside
- ``with self._connections_lock``. That lock also guards every other user's
- ``get_session`` / ``is_user_connected`` / cached-open fast path, so a slow
- dispose (SQLCipher + WAL close, or a pool waiting on ``pool_timeout=10``)
- stalls unrelated users, not just the one being closed.
+ """``dispose()`` excludes THIS user's rekey lifecycle, not other users.
+
+ Contract updated with the epoch-fenced teardown: the WAL checkpoint and
+ dispose run under the per-user rekey lifecycle lock (and the on-disk
+ sidecar), so a concurrent change_password for the SAME user cannot tear
+ the dispose -- while ``_connections_lock`` is held only for the cache
+ mutation, so a slow dispose no longer stalls unrelated users'
+ ``get_session`` / ``is_user_connected`` / cached-open fast path.
"""
entered = threading.Event()
release = threading.Event()
engine = _FakeEngine(dispose_gate=(entered, release))
probe_done = threading.Event()
+ rekey_gate_probe = {}
with _manager(lambda: tmp_path / "root") as mgr:
_register(mgr, "alice", engine)
_register(mgr, "bob", _FakeEngine())
+ def probe_rekey_gate():
+ # From the OUTSIDE: while alice's dispose is mid-flight, her
+ # rekey lifecycle lock must be held (a change_password for
+ # alice cannot start), but bob's must be free. Only release
+ # what was actually acquired.
+ acquired = mgr._get_rekey_lock("alice").acquire(blocking=False)
+ rekey_gate_probe["alice_blocked"] = not acquired
+ if acquired:
+ mgr._get_rekey_lock("alice").release()
+ probe_done.set()
+
closer = threading.Thread(
target=mgr.close_user_database, args=("alice",)
)
- probe = threading.Thread(
- target=lambda: (mgr.is_user_connected("bob"), probe_done.set())
- )
+ probe = threading.Thread(target=probe_rekey_gate)
closer.start()
try:
assert entered.wait(10.0), "dispose never ran"
probe.start()
- assert not probe_done.wait(0.3), (
- "is_user_connected('bob') completed while another user's "
- "dispose held _connections_lock -- the lock is no longer "
- "held across dispose (good news; update this contract)"
+ assert probe_done.wait(10.0), "probe never ran"
+ assert rekey_gate_probe["alice_blocked"], (
+ "alice's rekey lifecycle lock was free while her dispose "
+ "was mid-flight -- a concurrent change_password could tear "
+ "the teardown"
)
+ assert mgr._get_rekey_lock("bob").acquire(blocking=False), (
+ "bob's rekey lifecycle lock was held during alice's dispose "
+ "-- the exclusion is no longer per-user"
+ )
+ mgr._get_rekey_lock("bob").release()
finally:
release.set()
closer.join(10.0)
probe.join(10.0)
- assert probe_done.is_set()
assert engine.dispose_calls == 1
@@ -268,15 +315,14 @@ def test_close_user_database_holds_global_lock_across_dispose(tmp_path):
def test_dispose_failure_is_swallowed_without_any_diagnostic(tmp_path):
- """A failed dispose loses the exception AND the engine that failed.
-
- ``close_user_database``'s except block logs a bare
- ``f"Failed to dispose engine for {username}"`` -- no exception type, no
- message, no ``exc_info``. The ``del self.connections[username]`` then runs
- unconditionally (it sits outside the try), so the engine whose pool was
- never released is dropped from the registry and becomes unreachable:
- nothing can retry the dispose, and the log gives an operator nothing to
- diagnose the resulting FD leak with.
+ """A failed dispose is swallowed, de-registered, and named by type only.
+
+ Contract updated with the epoch-fenced teardown: the engine is still
+ de-registered even when its dispose raises (the pool is replaced on the
+ next open, and the dispose failure is surfaced at WARNING with exactly
+ the exception TYPE NAME -- enough for an operator to chase the FD leak,
+ never enough to leak a credential-bearing message or traceback, per the
+ repo's sensitive-logging convention).
"""
engine = _FakeEngine(dispose_error=RuntimeError("SENTINEL_DISPOSE_DETAIL"))
@@ -301,11 +347,11 @@ def test_dispose_failure_is_swallowed_without_any_diagnostic(tmp_path):
assert len(dispose_logs) == 1
entry = dispose_logs[0]
assert entry["level"] == "WARNING"
- # Everything an operator would need to diagnose the leak is absent:
+ # The type name is present; the detail and traceback are not:
assert entry["exception"] is False, "a traceback is attached after all"
assert "SENTINEL_DISPOSE_DETAIL" not in entry["message"]
- assert "RuntimeError" not in entry["message"]
- # ...and it is not recovered anywhere else either.
+ assert "RuntimeError" in entry["message"]
+ # ...and the sensitive detail is not recovered anywhere else either.
assert not [r for r in records if "SENTINEL_DISPOSE_DETAIL" in r["message"]]
@@ -321,7 +367,7 @@ def test_checkpoint_wal_failure_is_invisible_at_default_log_level(tmp_path):
with _manager(lambda: tmp_path / "root") as mgr:
_register(mgr, "alice", engine)
- with _captured_logs() as records:
+ with _captured_logs() as records, _trust_epoch():
mgr.close_user_database("alice")
checkpoint_logs = [
diff --git a/tests/database/test_locked_database_is_not_a_wrong_password.py b/tests/database/test_locked_database_is_not_a_wrong_password.py
new file mode 100644
index 0000000000..2de46951c9
--- /dev/null
+++ b/tests/database/test_locked_database_is_not_a_wrong_password.py
@@ -0,0 +1,96 @@
+"""A database locked past the busy timeout must not read as a wrong password.
+
+``verify_sqlcipher_connection`` answers False for every exception, so a
+correct password whose verification read met SQLITE_BUSY/locked used to come
+back as the ValueError (raw connection) or None (cold open) that every caller
+reads as "wrong password": the login counted a lockout strike and the change
+password page said "Current password is incorrect". Real SQLCipher; another
+connection holds the database exclusively locked.
+"""
+
+import contextlib
+import uuid
+from pathlib import Path
+
+import pytest
+
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ DatabaseKeyCheckUnavailableError,
+ RekeyOperationalError,
+ RekeyUnavailableError,
+ read_rekey_epoch_state,
+)
+
+PASSWORD = "LockedHorse1!" # noqa: S105
+
+
+def manager(tmp_path, monkeypatch):
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ # Keep the busy wait short so a held lock fails fast.
+ monkeypatch.setenv("LDR_DB_CONFIG_BUSY_TIMEOUT_MS", "200")
+ mgr = DatabaseManager()
+ mgr.data_dir = tmp_path
+ if not mgr.has_encryption:
+ pytest.skip("SQLCipher not available")
+ yield mgr
+ mgr.close_all_databases()
+
+
+def _user(manager):
+ username = f"locked_{uuid.uuid4().hex[:8]}"
+ assert manager.create_user_database(username, PASSWORD) is not None
+ manager.close_all_databases()
+ return username, manager._get_user_db_path(username)
+
+
+def _exclusively_locked(db_path: Path):
+ holder = DatabaseManager._make_sqlcipher_connection(
+ db_path, PASSWORD, isolation_level=None
+ )
+ try:
+ holder.execute("PRAGMA locking_mode = EXCLUSIVE")
+ holder.execute("BEGIN EXCLUSIVE")
+ holder.execute("CREATE TABLE IF NOT EXISTS lock_probe (x)")
+ yield
+ finally:
+ holder.close()
+
+
+def test_raw_connection_reports_a_lock_not_a_wrong_key(manager):
+ _username, db_path = _user(manager)
+ with _exclusively_locked(db_path):
+ with pytest.raises(DatabaseKeyCheckUnavailableError) as raised:
+ DatabaseManager._make_sqlcipher_connection(db_path, PASSWORD)
+ assert isinstance(raised.value, RekeyOperationalError)
+ assert not isinstance(raised.value, ValueError)
+ assert PASSWORD not in str(raised.value)
+ # Once the lock is gone the same password opens it.
+ DatabaseManager._make_sqlcipher_connection(db_path, PASSWORD).close()
+
+
+def test_wrong_key_is_still_a_value_error(manager):
+ _username, db_path = _user(manager)
+ with pytest.raises(ValueError):
+ DatabaseManager._make_sqlcipher_connection(db_path, "WrongHorse9!")
+
+
+def test_cold_open_reports_a_lock_not_a_wrong_password(manager):
+ username, db_path = _user(manager)
+ with _exclusively_locked(db_path):
+ with pytest.raises(DatabaseKeyCheckUnavailableError):
+ manager.open_user_database(username, PASSWORD)
+ assert manager.open_user_database(username, "WrongHorse9!") is None
+ assert manager.open_user_database(username, PASSWORD) is not None
+
+
+def test_change_password_reports_a_lock_as_retryable(manager):
+ username, db_path = _user(manager)
+ with _exclusively_locked(db_path):
+ with pytest.raises(RekeyUnavailableError):
+ manager.change_password(username, PASSWORD, "NewLockedHorse2!")
+ # Nothing changed and nothing is left pending.
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert manager.open_user_database(username, PASSWORD) is not None
diff --git a/tests/database/test_migration_chain_integrity.py b/tests/database/test_migration_chain_integrity.py
index d1a4ef4c28..f1c02bad15 100644
--- a/tests/database/test_migration_chain_integrity.py
+++ b/tests/database/test_migration_chain_integrity.py
@@ -195,6 +195,9 @@ KNOWN_CONNECTION_SITES = {
"database/backup/backup_service.py::BackupService._verify_backup": (
"exempt:backup-verify"
),
+ "database/backup/backup_service.py::BackupService._open_backup_with_current_key": (
+ "exempt:backup-verify"
+ ),
"database/encrypted_db.py::DatabaseManager._check_encryption_available": (
"user-db"
),
@@ -1344,6 +1347,7 @@ def test_sqlcipher_wrapper_has_only_classified_callers():
if any(kind == "connect_sqlcipher" for kind, _call in site_calls)
}
expected_callers = {
+ "database/backup/backup_service.py::BackupService._open_backup_with_current_key",
"database/backup/backup_service.py::BackupService._verify_backup",
"database/encrypted_db.py::DatabaseManager._check_encryption_available",
"database/encrypted_db.py::DatabaseManager._make_sqlcipher_connection",
@@ -1504,6 +1508,71 @@ def test_the_backup_verify_exemption_is_still_justified():
# ---------------------------------------------------------------------------
+def test_backup_key_probe_connection_never_leaves_its_classifier():
+ """``_open_backup_with_current_key`` opens a backup, not a user database.
+
+ It returns its raw connection, so the backup-verify exemption holds only
+ while its sole caller is ``_classify_backup_key``, which closes every
+ connection it gets and never returns or stores one. FK enforcement is
+ irrelevant to a connection that never leaves that classifier.
+ """
+ path = DATABASE_PKG / "backup" / "backup_service.py"
+ tree = ast.parse(path.read_text(encoding="utf-8"))
+ callers = {
+ func.name
+ for func in ast.walk(tree)
+ if isinstance(func, (ast.FunctionDef, ast.AsyncFunctionDef))
+ for node in ast.walk(func)
+ if isinstance(node, ast.Attribute)
+ and node.attr == "_open_backup_with_current_key"
+ }
+ assert callers == {"_classify_backup_key"}, (
+ "the backup key probe gained a caller; it hands back a raw backup "
+ f"connection, so classify that caller first: {sorted(callers)}"
+ )
+ classifier = next(
+ func
+ for func in ast.walk(tree)
+ if isinstance(func, ast.FunctionDef)
+ and func.name == "_classify_backup_key"
+ )
+ leaks = [
+ ast.unparse(node)
+ for node in ast.walk(classifier)
+ if (
+ isinstance(node, ast.Return)
+ and node.value is not None
+ and "conn" in ast.unparse(node.value)
+ )
+ or (
+ isinstance(node, ast.Assign)
+ and any(
+ isinstance(target, ast.Attribute) for target in node.targets
+ )
+ and "conn" in ast.unparse(node.value)
+ )
+ ]
+ assert not leaks, (
+ "_classify_backup_key now returns or stores its backup connection, "
+ f"so the exemption stops holding: {leaks}"
+ )
+ closes = [
+ node
+ for node in ast.walk(classifier)
+ if isinstance(node, ast.Call)
+ and ast.unparse(node.func) in {"safe_close", "conn.close"}
+ ]
+ opens = [
+ node
+ for node in ast.walk(classifier)
+ if isinstance(node, ast.Call)
+ and ast.unparse(node.func).endswith("_open_backup_with_current_key")
+ ]
+ assert len(closes) >= len(opens) > 0, (
+ "_classify_backup_key no longer closes every backup connection it opens"
+ )
+
+
def test_no_migration_opens_a_database_of_its_own():
"""Every user database on this project is (or may be) SQLCipher.
diff --git a/tests/database/test_rekey_cache_fencing.py b/tests/database/test_rekey_cache_fencing.py
new file mode 100644
index 0000000000..ba26260653
--- /dev/null
+++ b/tests/database/test_rekey_cache_fencing.py
@@ -0,0 +1,703 @@
+from __future__ import annotations
+
+import threading
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import contextmanager
+from pathlib import Path
+from typing import Literal, assert_never
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochState,
+ acquire_rekey_lock,
+)
+
+
+def _new_manager(data_dir: Path) -> DatabaseManager:
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = data_dir
+ return manager
+
+
+def manager(tmp_path: Path) -> DatabaseManager:
+ return _new_manager(tmp_path)
+
+
+def _existing_db(manager: DatabaseManager, username: str) -> Path:
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ return db_path
+
+
+def _initialize_epoch(db_path: Path) -> str:
+ with acquire_rekey_lock(db_path):
+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+
+type _AccessorName = Literal[
+ "is_user_connected",
+ "_password_matches_cached",
+ "open_user_database",
+ "get_session",
+ "check_database_integrity",
+ "create_thread_safe_session_for_metrics",
+]
+
+
+def _invoke_cache_accessor(
+ manager: DatabaseManager,
+ accessor: _AccessorName,
+ password: str,
+) -> MagicMock | bool | None:
+ match accessor:
+ case "is_user_connected":
+ return manager.is_user_connected("alice")
+ case "_password_matches_cached":
+ return manager._password_matches_cached("alice", password)
+ case "open_user_database":
+ return manager.open_user_database("alice", password)
+ case "get_session":
+ return manager.get_session("alice")
+ case "check_database_integrity":
+ return manager.check_database_integrity("alice")
+ case "create_thread_safe_session_for_metrics":
+ return manager.create_thread_safe_session_for_metrics(
+ "alice", password
+ )
+ case unreachable:
+ assert_never(unreachable)
+
+
+def _configure_integrity_engine(engine: MagicMock) -> None:
+ connection = engine.connect.return_value.__enter__.return_value
+ quick_check = MagicMock()
+ quick_check.fetchone.return_value = ("ok",)
+ wal_checkpoint = MagicMock()
+ wal_checkpoint.fetchone.return_value = (0, 0, 0)
+ cipher_check = MagicMock()
+ cipher_check.__iter__.return_value = iter(())
+ connection.execute.side_effect = [quick_check, wal_checkpoint, cipher_check]
+
+
+ "accessor",
+ [
+ "is_user_connected",
+ "_password_matches_cached",
+ "open_user_database",
+ "get_session",
+ "check_database_integrity",
+ "create_thread_safe_session_for_metrics",
+ ],
+)
+def test_cache_accessor_retries_e1_snapshot_after_e2_publication(
+ manager: DatabaseManager,
+ accessor: _AccessorName,
+) -> None:
+ # Given every accessor has snapshotted E1 before acquiring its sidecar
+ db_path = _existing_db(manager, "alice")
+ e1_epoch = _initialize_epoch(db_path)
+ e1_engine = MagicMock()
+ e2_engine = MagicMock()
+ _configure_integrity_engine(e2_engine)
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(
+ e1_engine, "old-password", e1_epoch
+ ),
+ )
+ snapshot_ready = threading.Event()
+ continue_claim = threading.Event()
+ real_acquire = acquire_rekey_lock
+ first_acquire = True
+ bound_engines: list[MagicMock] = []
+ expected_session = MagicMock()
+
+ @contextmanager
+ def pause_first_sidecar(path: Path, **kwargs: object):
+ nonlocal first_acquire
+ if first_acquire:
+ first_acquire = False
+ snapshot_ready.set()
+ assert continue_claim.wait(timeout=5)
+ with real_acquire(path, **kwargs):
+ yield
+
+ def make_session_factory(*, bind):
+ bound_engines.append(bind)
+ return MagicMock(return_value=expected_session)
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=pause_first_sidecar,
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.sessionmaker",
+ side_effect=make_session_factory,
+ ),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ accessor_result = executor.submit(
+ _invoke_cache_accessor, manager, accessor, "new-password"
+ )
+ assert snapshot_ready.wait(timeout=5)
+
+ # When E2 becomes durable and replaces all three local cache identities
+ with real_acquire(db_path):
+ e2_epoch = rekey_lock_module.advance_rekey_epoch(db_path)
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(
+ e2_engine, "new-password", e2_epoch
+ ),
+ )
+ continue_claim.set()
+ result = accessor_result.result(timeout=5)
+
+ # Then the accessor retries E2 without detaching or using either wrong owner
+ assert manager.connections["alice"] is e2_engine
+ e1_engine.dispose.assert_not_called()
+ e2_engine.dispose.assert_not_called()
+ e1_engine.connect.assert_not_called()
+ match accessor:
+ case "is_user_connected" | "_password_matches_cached":
+ assert result is True
+ case "open_user_database":
+ assert result is e2_engine
+ case "get_session" | "create_thread_safe_session_for_metrics":
+ assert result is expected_session
+ assert bound_engines == [e2_engine]
+ case "check_database_integrity":
+ assert result is True
+ e2_engine.connect.assert_called_once_with()
+ case unreachable:
+ assert_never(unreachable)
+
+
+def test_e1_snapshot_detaches_only_e1_after_durable_e2(
+ manager: DatabaseManager,
+) -> None:
+ # Given an accessor paused after snapshotting the only local E1 generation
+ db_path = _existing_db(manager, "alice")
+ e1_epoch = _initialize_epoch(db_path)
+ e1_engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(e1_engine, "password", e1_epoch),
+ )
+ snapshot_ready = threading.Event()
+ continue_claim = threading.Event()
+ real_acquire = acquire_rekey_lock
+
+ @contextmanager
+ def pause_sidecar(path: Path, **kwargs: object):
+ snapshot_ready.set()
+ assert continue_claim.wait(timeout=5)
+ with real_acquire(path, **kwargs):
+ yield
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=pause_sidecar,
+ ),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ connected = executor.submit(manager.is_user_connected, "alice")
+ assert snapshot_ready.wait(timeout=5)
+
+ # When only the durable generation advances to E2
+ with real_acquire(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ continue_claim.set()
+ result = connected.result(timeout=5)
+
+ # Then the exact E1 snapshot is detached and disposed once without checkpoint
+ assert result is False
+ e1_engine.dispose.assert_called_once_with()
+ e1_engine.connect.assert_not_called()
+ assert "alice" not in manager.connections
+
+
+def test_verifier_identity_change_retries_exact_generation(
+ manager: DatabaseManager,
+) -> None:
+ # Given a password check paused with an old verifier identity snapshot
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(engine, "old-password", epoch),
+ )
+ snapshot_ready = threading.Event()
+ continue_claim = threading.Event()
+ real_acquire = acquire_rekey_lock
+
+ @contextmanager
+ def pause_first_sidecar(path: Path, **kwargs: object):
+ snapshot_ready.set()
+ assert continue_claim.wait(timeout=5)
+ with real_acquire(path, **kwargs):
+ yield
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=pause_first_sidecar,
+ ),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ password_matches = executor.submit(
+ manager._password_matches_cached, "alice", "new-password"
+ )
+ assert snapshot_ready.wait(timeout=5)
+
+ # When only the verifier object is replaced before the exact claim
+ replacement_verifier = manager._make_verifier("new-password")
+ with manager._connections_lock:
+ manager._password_verifiers["alice"] = replacement_verifier
+ continue_claim.set()
+ result = password_matches.result(timeout=5)
+
+ # Then identity change retries and validates only the replacement verifier
+ assert result is True
+ assert manager._password_verifiers["alice"] is replacement_verifier
+ engine.dispose.assert_not_called()
+
+
+ "accessor",
+ [
+ "is_user_connected",
+ "_password_matches_cached",
+ "open_user_database",
+ "get_session",
+ "check_database_integrity",
+ "create_thread_safe_session_for_metrics",
+ ],
+)
+def test_pending_generation_fences_snapshotted_cache(
+ manager: DatabaseManager,
+ accessor: _AccessorName,
+) -> None:
+ # Given every accessor is paused after snapshotting a stable generation
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ replacement_engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(engine, "password", epoch),
+ )
+ snapshot_ready = threading.Event()
+ continue_claim = threading.Event()
+ real_acquire = acquire_rekey_lock
+ first_acquire = True
+ expected_session = MagicMock()
+ bound_engines: list[MagicMock] = []
+
+ @contextmanager
+ def pause_first_sidecar(path: Path, **kwargs: object):
+ nonlocal first_acquire
+ if first_acquire:
+ first_acquire = False
+ snapshot_ready.set()
+ assert continue_claim.wait(timeout=5)
+ with real_acquire(path, **kwargs):
+ yield
+
+ def recover_pending(
+ path: Path, _password: str, state: RekeyEpochState
+ ) -> bool:
+ pending_epoch = state.pending_epoch
+ assert pending_epoch is not None
+ rekey_lock_module.commit_rekey_epoch(path, pending_epoch)
+ rekey_lock_module.clear_pending_rekey_epoch(path, pending_epoch)
+ return True
+
+ def make_session_factory(*, bind):
+ bound_engines.append(bind)
+ return MagicMock(return_value=expected_session)
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=pause_first_sidecar,
+ ),
+ patch.object(
+ manager, "_recover_pending_rekey", side_effect=recover_pending
+ ) as recover,
+ patch.object(
+ manager,
+ "_open_user_database_cold",
+ return_value=replacement_engine,
+ ),
+ patch.object(manager, "_run_phase2_rekey"),
+ patch(
+ "local_deep_research.database.encrypted_db.sessionmaker",
+ side_effect=make_session_factory,
+ ),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ accessor_result = executor.submit(
+ _invoke_cache_accessor, manager, accessor, "new-password"
+ )
+ assert snapshot_ready.wait(timeout=5)
+
+ # When a pending generation becomes durable before the exact claim
+ with real_acquire(db_path):
+ rekey_lock_module.begin_rekey_epoch(db_path)
+ continue_claim.set()
+ result = accessor_result.result(timeout=5)
+
+ # Then no accessor uses the fenced engine and credentialed paths recover cold
+ engine.dispose.assert_called_once_with()
+ engine.connect.assert_not_called()
+ match accessor:
+ case "is_user_connected" | "_password_matches_cached":
+ assert result is False
+ recover.assert_not_called()
+ assert "alice" not in manager.connections
+ case "get_session":
+ assert result is None
+ recover.assert_not_called()
+ assert "alice" not in manager.connections
+ case "check_database_integrity":
+ assert result is False
+ recover.assert_not_called()
+ assert "alice" not in manager.connections
+ case "open_user_database":
+ assert result is replacement_engine
+ recover.assert_called_once()
+ assert manager.connections["alice"] is replacement_engine
+ case "create_thread_safe_session_for_metrics":
+ assert result is expected_session
+ assert bound_engines == [replacement_engine]
+ recover.assert_called_once()
+ assert manager.connections["alice"] is replacement_engine
+ case unreachable:
+ assert_never(unreachable)
+
+
+def test_cache_accessor_waits_for_live_sidecar_before_claim(
+ manager: DatabaseManager,
+) -> None:
+ # Given a current cache generation and another owner holding its sidecar
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(engine, "password", epoch),
+ )
+ sidecar_attempted = threading.Event()
+ sidecar_released = threading.Event()
+ real_acquire = acquire_rekey_lock
+
+ @contextmanager
+ def observe_sidecar_attempt(path: Path, **kwargs: object):
+ sidecar_attempted.set()
+ with real_acquire(path, **kwargs):
+ yield
+
+ real_claim = manager._claim_connection_generation
+
+ def claim_after_release(generation, state):
+ assert sidecar_released.is_set()
+ return real_claim(generation, state)
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=observe_sidecar_attempt,
+ ),
+ patch.object(
+ manager,
+ "_claim_connection_generation",
+ side_effect=claim_after_release,
+ ) as claim,
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ with real_acquire(db_path):
+ connected = executor.submit(manager.is_user_connected, "alice")
+ assert sidecar_attempted.wait(timeout=5)
+ sidecar_released.set()
+ assert connected.result(timeout=5) is True
+
+ assert sidecar_released.is_set()
+ claim.assert_called_once()
+
+
+def test_two_managers_evict_cross_worker_stale_cache(tmp_path: Path) -> None:
+ # Given two worker-local managers sharing one database directory
+ first = _new_manager(tmp_path)
+ second = _new_manager(tmp_path)
+ db_path = _existing_db(first, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ entry = first._make_connection_cache_entry(
+ stale_engine, "old-password", original_epoch
+ )
+ first._cache_connection("alice", entry)
+
+ with acquire_rekey_lock(second._get_user_db_path("alice")):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ with patch.object(first, "_checkpoint_wal"):
+ # When the first worker next consults its process-local cache
+ connected = first.is_user_connected("alice")
+
+ # Then the durable epoch fences and disposes that stale engine
+ assert connected is False
+ assert rekey_lock_module.read_rekey_epoch(db_path) != original_epoch
+ stale_engine.dispose.assert_called_once_with()
+ assert "alice" not in first.connections
+ assert "alice" not in first._password_verifiers
+ assert "alice" not in first._connection_epochs
+
+
+def test_stale_eviction_does_not_checkpoint_while_rekey_is_pending(
+ tmp_path: Path,
+) -> None:
+ # Given one worker observes durable pending metadata from another worker
+ stale_manager = _new_manager(tmp_path)
+ db_path = _existing_db(stale_manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ stale_manager._cache_connection(
+ "alice",
+ stale_manager._make_connection_cache_entry(
+ stale_engine, "old-password", original_epoch
+ ),
+ )
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.begin_rekey_epoch(db_path)
+
+ with (
+ patch.object(
+ stale_manager,
+ "_checkpoint_wal",
+ wraps=stale_manager._checkpoint_wal,
+ ) as checkpoint,
+ ):
+ # When the stale worker consults its cache during the rekey window
+ connected = stale_manager.is_user_connected("alice")
+
+ # Then eviction only disposes: checkpointing would open the old-key engine
+ assert connected is False
+ checkpoint.assert_not_called()
+ stale_engine.connect.assert_not_called()
+ stale_engine.dispose.assert_called_once_with()
+
+
+ "case",
+ [
+ ("is_user_connected", ("alice",), False),
+ ("get_session", ("alice",), None),
+ ("check_database_integrity", ("alice",), False),
+ ("_password_matches_cached", ("alice", "old-password"), False),
+ ],
+)
+def test_passwordless_cache_accessors_fence_stale_epoch(
+ manager: DatabaseManager,
+ case: tuple[str, tuple[str, ...], bool | None],
+) -> None:
+ # Given a cache entry published under an older durable epoch
+ method_name, arguments, expected = case
+ db_path = _existing_db(manager, "alice")
+ old_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ entry = manager._make_connection_cache_entry(
+ stale_engine, "old-password", old_epoch
+ )
+ manager._cache_connection("alice", entry)
+ with acquire_rekey_lock(db_path):
+ current_epoch = rekey_lock_module.advance_rekey_epoch(db_path)
+
+ with manager._connections_lock:
+ assert not manager._cached_engine_trusted(
+ "alice", "old-password", current_epoch
+ )
+
+ with patch.object(manager, "_checkpoint_wal"):
+ # When a passwordless or verifier-only accessor consults the cache
+ result = getattr(manager, method_name)(*arguments)
+
+ # Then it fails closed and evicts all parts of the stale cache entry
+ assert result == expected
+ stale_engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+ assert "alice" not in manager._password_verifiers
+ assert "alice" not in manager._connection_epochs
+
+
+def test_metrics_session_fences_stale_epoch_before_reopen(
+ manager: DatabaseManager,
+) -> None:
+ # Given a stale metrics cache entry and a replacement engine
+ db_path = _existing_db(manager, "alice")
+ old_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ replacement_engine = MagicMock()
+ entry = manager._make_connection_cache_entry(
+ stale_engine, "password", old_epoch
+ )
+ manager._cache_connection("alice", entry)
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ expected_session = MagicMock()
+
+ with (
+ patch.object(manager, "_checkpoint_wal"),
+ patch.object(
+ manager, "open_user_database", return_value=replacement_engine
+ ) as reopen,
+ patch(
+ "local_deep_research.database.encrypted_db.sessionmaker",
+ return_value=lambda: expected_session,
+ ),
+ ):
+ # When a metrics worker requests a session
+ session = manager.create_thread_safe_session_for_metrics(
+ "alice", "password"
+ )
+
+ # Then it evicts the stale engine and reopens through the fenced path
+ assert session is expected_session
+ stale_engine.dispose.assert_called_once_with()
+ reopen.assert_called_once_with("alice", "password")
+
+
+def test_open_evicts_stale_cache_before_cold_publication(
+ manager: DatabaseManager,
+) -> None:
+ # Given a cached engine whose published epoch is no longer current
+ db_path = _existing_db(manager, "alice")
+ stale_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ replacement_engine = MagicMock()
+ entry = manager._make_connection_cache_entry(
+ stale_engine, "password", stale_epoch
+ )
+ manager._cache_connection("alice", entry)
+ with acquire_rekey_lock(db_path):
+ current_epoch = rekey_lock_module.advance_rekey_epoch(db_path)
+
+ with (
+ patch.object(manager, "_checkpoint_wal"),
+ patch.object(
+ manager,
+ "_open_user_database_cold",
+ return_value=replacement_engine,
+ ) as cold_open,
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ # When the worker opens with the valid password
+ opened = manager.open_user_database("alice", "password")
+
+ # Then both cache checks reject the stale generation before publication
+ assert opened is replacement_engine
+ cold_open.assert_called_once()
+ stale_engine.dispose.assert_called_once_with()
+ assert manager.connections["alice"] is replacement_engine
+ assert manager._connection_epochs["alice"] == current_epoch
+
+
+def test_cold_open_waits_for_epoch_advance_and_rekey(
+ tmp_path: Path,
+) -> None:
+ # Given cold open reaches a sidecar owned by a generation publisher
+ opener_manager = _new_manager(tmp_path)
+ db_path = _existing_db(opener_manager, "alice")
+ _initialize_epoch(db_path)
+ replacement_engine = MagicMock()
+ sidecar_attempted = threading.Event()
+ sidecar_released = threading.Event()
+ cold_started = threading.Event()
+ real_acquire = acquire_rekey_lock
+
+ @contextmanager
+ def observe_sidecar_attempt(path: Path, **kwargs: object):
+ sidecar_attempted.set()
+ with real_acquire(path, **kwargs):
+ yield
+
+ def controlled_cold_open(
+ _username: str,
+ _password: str,
+ _open_start: float,
+ prederived_hex_key: str | None = None,
+ ) -> MagicMock:
+ assert sidecar_released.is_set()
+ cold_started.set()
+ return replacement_engine
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=observe_sidecar_attempt,
+ ),
+ patch.object(
+ opener_manager,
+ "_open_user_database_cold",
+ side_effect=controlled_cold_open,
+ ),
+ patch.object(opener_manager, "_run_phase2_rekey"),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ with real_acquire(db_path):
+ cold_open = executor.submit(
+ opener_manager.open_user_database, "alice", "new-password"
+ )
+ assert sidecar_attempted.wait(timeout=5)
+ current_epoch = rekey_lock_module.advance_rekey_epoch(db_path)
+ sidecar_released.set()
+
+ # Then physical cold-open begins only after durable publication and release
+ assert cold_open.result(timeout=5) is replacement_engine
+ assert cold_started.is_set()
+ assert opener_manager._connection_epochs["alice"] == current_epoch
+
+
+def test_close_user_database_disposes_outside_connections_lock(
+ manager: DatabaseManager,
+) -> None:
+ # Given a current-generation cached engine and a non-reentrant cache lock
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(engine, "password", epoch),
+ )
+ cache_lock = threading.Lock()
+ manager._connections_lock = cache_lock
+
+ def assert_cache_lock_released(*_args) -> None:
+ assert cache_lock.acquire(blocking=False)
+ cache_lock.release()
+
+ engine.dispose.side_effect = assert_cache_lock_released
+
+ with patch.object(
+ manager, "_checkpoint_wal", side_effect=assert_cache_lock_released
+ ) as checkpoint:
+ # When the user database is closed
+ manager.close_user_database("alice")
+
+ # Then detachment is atomic but checkpoint and dispose run after unlocking
+ checkpoint.assert_called_once_with(engine, "for alice")
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+ assert "alice" not in manager._password_verifiers
+ assert "alice" not in manager._connection_epochs
diff --git a/tests/database/test_rekey_epoch.py b/tests/database/test_rekey_epoch.py
new file mode 100644
index 0000000000..04af9dcb3c
--- /dev/null
+++ b/tests/database/test_rekey_epoch.py
@@ -0,0 +1,215 @@
+from __future__ import annotations
+
+import errno
+import os
+from pathlib import Path
+from uuid import UUID
+
+import pytest
+
+from local_deep_research.config.paths import get_user_database_filename
+from local_deep_research.database.rekey_epoch import (
+ read_authorization_epoch,
+ read_user_authorization_epoch,
+)
+from local_deep_research.database.rekey_lock import (
+ RekeyRecoveryRequiredError,
+ acquire_rekey_lock,
+ read_rekey_epoch,
+)
+
+
+_EPOCH_ONE = str(UUID(int=1))
+_EPOCH_TWO = str(UUID(int=2))
+
+
+def _write_epoch_state(
+ db_path: Path,
+ *,
+ committed: str | None,
+ pending: str | None,
+) -> None:
+ if committed is not None:
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text(
+ committed, encoding="ascii"
+ )
+ if pending is not None:
+ db_path.with_name(f"{db_path.name}.rekey.pending").write_text(
+ pending, encoding="ascii"
+ )
+
+
+def test_stable_committed_epoch_authorizes(tmp_path: Path) -> None:
+ # Given a stable committed identity
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+
+ # When authorization projects the durable identity
+ epoch = read_authorization_epoch(db_path)
+
+ # Then the committed identity authorizes sessions
+ assert epoch == _EPOCH_ONE
+
+
+def test_matching_committed_and_pending_epoch_authorizes(
+ tmp_path: Path,
+) -> None:
+ # Given verified E2 remains marked pending only for cleanup
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_TWO, pending=_EPOCH_TWO)
+
+ # When authorization projects the durable identity
+ epoch = read_authorization_epoch(db_path)
+
+ # Then verified committed E2 authorizes while cache projection stays closed
+ assert epoch == _EPOCH_TWO
+ assert read_rekey_epoch(db_path) is None
+
+
+def test_old_committed_with_candidate_pending_requires_recovery(
+ tmp_path: Path,
+) -> None:
+ # Given physical-key outcome may disagree with E1+pending-E2 metadata
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=_EPOCH_TWO)
+
+ # When authorization projects the ambiguous identity
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_authorization_epoch(db_path)
+
+ # Then no metadata is repaired or discarded
+ assert read_rekey_epoch(db_path) is None
+
+
+def test_missing_committed_epoch_requires_recovery(tmp_path: Path) -> None:
+ # Given authorization metadata has not been initialized
+ db_path = tmp_path / "alice.db"
+
+ # When authorization projects the missing identity
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_authorization_epoch(db_path)
+
+
+def test_malformed_epoch_requires_recovery_without_repair(
+ tmp_path: Path,
+) -> None:
+ # Given committed metadata is malformed
+ db_path = tmp_path / "alice.db"
+ epoch_path = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ epoch_path.write_text("not-a-canonical-epoch", encoding="ascii")
+
+ # When authorization projects the malformed identity
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_authorization_epoch(db_path)
+
+ # Then the malformed evidence is preserved for recovery
+ assert epoch_path.read_text(encoding="ascii") == "not-a-canonical-epoch"
+
+
+def test_pending_without_committed_epoch_requires_recovery(
+ tmp_path: Path,
+) -> None:
+ # Given pending metadata contradicts the missing committed identity
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=None, pending=_EPOCH_TWO)
+
+ # When authorization projects the contradictory identity
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_authorization_epoch(db_path)
+
+
+def test_unreadable_epoch_requires_recovery(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a stable committed identity whose files cannot be read: the
+ # epoch reader's own open fails (EIO), so the identity it would
+ # otherwise authorize is never seen
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ real_open = os.open
+
+ def unreadable(path, flags, *args, **kwargs):
+ if os.fspath(path).endswith((".rekey.epoch", ".rekey.pending")):
+ raise OSError(errno.EIO, "unreadable")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(os, "open", unreadable)
+
+ # When authorization projects the unreadable identity
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_authorization_epoch(db_path)
+
+
+def test_authorization_projection_does_not_acquire_sidecar(
+ tmp_path: Path,
+) -> None:
+ # Given another operation already holds the stable sidecar
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+
+ # When authorization reads while that sidecar remains held
+ with acquire_rekey_lock(db_path):
+ epoch = read_authorization_epoch(db_path)
+
+ # Then projection succeeds without lock recursion or contention
+ assert epoch == _EPOCH_ONE
+
+
+def test_user_authorization_epoch_resolves_hashed_database_path(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given Alice's metadata exists beneath the configured data root
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ database_dir = tmp_path / "encrypted_databases"
+ database_dir.mkdir()
+ db_path = database_dir / get_user_database_filename("alice")
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+
+ # When authorization resolves Alice rather than importing db_manager
+ epoch = read_user_authorization_epoch("alice")
+
+ # Then it projects the identity from Alice's hashed database path
+ assert epoch == _EPOCH_ONE
+
+
+def test_projection_cannot_tear_into_the_superseded_epoch(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a rekey in flight: committed E1, pending E2
+ from local_deep_research.database import rekey_lock as rekey_lock_module
+
+ db_path = tmp_path / "alice.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=_EPOCH_TWO)
+ real_read = rekey_lock_module._read_epoch_token
+ writer_ran = False
+
+ def read_then_let_the_writer_finish(path: Path, db: Path) -> str | None:
+ # The lock-free reader is descheduled right after its FIRST file
+ # read, and the rekey commits E2 and clears pending in between.
+ nonlocal writer_ran
+ token = real_read(path, db)
+ if not writer_ran:
+ writer_ran = True
+ monkeypatch.setattr(
+ rekey_lock_module, "_read_epoch_token", real_read
+ )
+ rekey_lock_module.commit_rekey_epoch(db_path, _EPOCH_TWO)
+ rekey_lock_module.clear_pending_rekey_epoch(db_path, _EPOCH_TWO)
+ monkeypatch.setattr(
+ rekey_lock_module,
+ "_read_epoch_token",
+ read_then_let_the_writer_finish,
+ )
+ return token
+
+ monkeypatch.setattr(
+ rekey_lock_module, "_read_epoch_token", read_then_let_the_writer_finish
+ )
+
+ # When authorization projects across that commit
+ epoch = read_authorization_epoch(db_path)
+
+ # Then it never reports the superseded E1 as a stable identity (reading
+ # .epoch first saw E1, then no pending file, and authorized E1)
+ assert writer_ran
+ assert epoch == _EPOCH_TWO
diff --git a/tests/database/test_rekey_integrity_recovery.py b/tests/database/test_rekey_integrity_recovery.py
new file mode 100644
index 0000000000..ea4dadfd4f
--- /dev/null
+++ b/tests/database/test_rekey_integrity_recovery.py
@@ -0,0 +1,173 @@
+"""Integrity scan failures must not fence a user out of a database that opens.
+
+The active key is decided by which password decrypts the schema page: a
+database state never opens under both keys. ``quick_check`` and
+``cipher_integrity_check`` findings describe file health, not the key, so a
+database that was already damaged must still finish (or resolve) a password
+change and let its user log in with the active password, with the finding
+logged loudly. Real SQLCipher throughout.
+"""
+
+import uuid
+from pathlib import Path
+
+import pytest
+
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ RekeyUnavailableError,
+ begin_rekey_epoch,
+ read_rekey_epoch_state,
+)
+
+OLD = "OldIntegrityHorse1!" # noqa: S105
+NEW = "NewIntegrityHorse2!" # noqa: S105
+
+
+def manager(tmp_path, monkeypatch):
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ mgr = DatabaseManager()
+ mgr.data_dir = tmp_path
+ if not mgr.has_encryption:
+ pytest.skip("SQLCipher not available")
+ yield mgr
+ mgr.close_all_databases()
+
+
+def _user_db(manager: DatabaseManager) -> tuple[str, Path]:
+ username = f"integrity_{uuid.uuid4().hex[:8]}"
+ assert manager.create_user_database(username, OLD) is not None
+ manager.close_all_databases()
+ return username, manager._get_user_db_path(username)
+
+
+def _share_a_root_page(db_path: Path, password: str) -> None:
+ """Repro A: two tables point at one root page; every page decrypts."""
+ conn = DatabaseManager._make_sqlcipher_connection(
+ db_path, password, isolation_level=None
+ )
+ try:
+ rows = conn.execute(
+ "SELECT name, rootpage FROM sqlite_master "
+ "WHERE type = 'table' AND rootpage > 2 ORDER BY rootpage"
+ ).fetchall()
+ conn.execute("PRAGMA writable_schema = ON")
+ conn.execute(
+ "UPDATE sqlite_master SET rootpage = ? WHERE name = ?",
+ (rows[0][1], rows[1][0]),
+ )
+ conn.execute("PRAGMA writable_schema = OFF")
+ conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+ finally:
+ conn.close()
+ fresh = DatabaseManager._make_sqlcipher_connection(db_path, password)
+ try:
+ quick = fresh.execute("PRAGMA quick_check").fetchall()
+ finally:
+ fresh.close()
+ assert quick != [("ok",)], "the repro must fail quick_check"
+
+
+def _damage_one_page(db_path: Path, password: str) -> None:
+ """Repro B: one page (not page 1) fails its HMAC."""
+ conn = DatabaseManager._make_sqlcipher_connection(
+ db_path, password, isolation_level=None
+ )
+ try:
+ page_size = int(conn.execute("PRAGMA page_size").fetchone()[0])
+ conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+ finally:
+ conn.close()
+ offset = page_size * 10 + 100
+ with db_path.open("r+b") as handle:
+ handle.seek(offset)
+ byte = handle.read(1)
+ handle.seek(offset)
+ handle.write(bytes([byte[0] ^ 0xFF]))
+
+
+def _opens_with(db_path: Path, password: str) -> bool:
+ try:
+ DatabaseManager._make_sqlcipher_connection(db_path, password).close()
+ except ValueError:
+ return False
+ return True
+
+
+def test_shared_root_page_still_completes_the_password_change(manager):
+ # Given a database whose schema fails quick_check while every page decrypts
+ username, db_path = _user_db(manager)
+ _share_a_root_page(db_path, OLD)
+
+ # When the user changes the password
+ assert manager.change_password(username, OLD, NEW) is True
+
+ # Then the change completes and only the new password logs in
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert manager.open_user_database(username, OLD) is None
+ assert manager.open_user_database(username, NEW) is not None
+
+
+def test_shared_root_page_pending_state_resolves_on_login(manager):
+ # Given a quick_check-failing database left with a pending generation
+ # (what an earlier failed verification left behind) after the rekey
+ # applied
+ username, db_path = _user_db(manager)
+ _share_a_root_page(db_path, OLD)
+ conn = DatabaseManager._make_sqlcipher_connection(db_path, OLD)
+ try:
+ from local_deep_research.database.sqlcipher_utils import (
+ set_sqlcipher_rekey,
+ )
+
+ begin_rekey_epoch(db_path)
+ set_sqlcipher_rekey(conn, NEW, db_path=db_path)
+ conn.commit()
+ finally:
+ conn.close()
+ assert _opens_with(db_path, NEW) and not _opens_with(db_path, OLD)
+ pending = read_rekey_epoch_state(db_path).pending_epoch
+ assert pending is not None
+
+ # When the user logs in with the active (new) password
+ assert manager.open_user_database(username, OLD) is None
+ assert manager.open_user_database(username, NEW) is not None
+
+ # Then the pending generation is committed and cleared
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch == pending
+
+
+def test_damaged_page_keeps_the_old_key_and_leaves_no_pending_state(manager):
+ # Given a database with one page that fails its HMAC
+ username, db_path = _user_db(manager)
+ _damage_one_page(db_path, OLD)
+
+ # When the user tries to change the password, the rekey cannot apply
+ with pytest.raises(RekeyUnavailableError):
+ manager.change_password(username, OLD, NEW)
+
+ # Then the old key is provably kept, nothing is left pending, and the
+ # user still logs in with the old password
+ assert read_rekey_epoch_state(db_path).pending_epoch is None
+ assert manager.open_user_database(username, NEW) is None
+ assert manager.open_user_database(username, OLD) is not None
+
+
+def test_damaged_page_pending_state_resolves_on_login(manager):
+ # Given a damaged database left with a pending generation whose rekey
+ # never applied
+ username, db_path = _user_db(manager)
+ _damage_one_page(db_path, OLD)
+ pending = begin_rekey_epoch(db_path)
+
+ # When the user logs in with the surviving (old) password
+ assert manager.open_user_database(username, NEW) is None
+ assert manager.open_user_database(username, OLD) is not None
+
+ # Then the pending generation is resolved instead of fencing forever
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch == pending
diff --git a/tests/database/test_rekey_lock.py b/tests/database/test_rekey_lock.py
new file mode 100644
index 0000000000..6d771da538
--- /dev/null
+++ b/tests/database/test_rekey_lock.py
@@ -0,0 +1,1548 @@
+from __future__ import annotations
+
+import errno
+import multiprocessing
+import os
+import stat
+import sys
+import threading
+import time
+import tracemalloc
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+from multiprocessing.synchronize import Event
+from pathlib import Path
+from types import ModuleType
+from typing import ClassVar, final
+from uuid import UUID
+
+import pytest
+
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochState,
+ RekeyLockContentionError,
+ RekeyRecoveryRequiredError,
+ _acquire_windows_lock,
+ _release_windows_lock,
+ acquire_rekey_lock,
+ begin_rekey_epoch,
+ clear_pending_rekey_epoch,
+ commit_rekey_epoch,
+ read_rekey_epoch_state,
+)
+
+
+def _hold_rekey_lock(
+ db_path: str,
+ acquired: Event,
+ release: Event,
+) -> None:
+ with acquire_rekey_lock(Path(db_path)):
+ acquired.set()
+ if not release.wait(timeout=10):
+ raise RuntimeError("parent did not release rekey-lock worker")
+
+
+def _hold_authorization_lease(
+ db_path: str,
+ mode: str,
+ acquired: Event,
+ release: Event,
+) -> None:
+ lease_mode = rekey_lock_module.AuthorizationLeaseMode(mode)
+ with rekey_lock_module.acquire_authorization_lease(
+ Path(db_path), lease_mode
+ ):
+ acquired.set()
+ if not release.wait(timeout=10):
+ raise _LockWorkerTimeoutError
+
+
+def _assert_sidecar_io_error(
+ error: rekey_lock_module.RekeySidecarIOError,
+ db_path: Path,
+ operation: rekey_lock_module.RekeySidecarOperation,
+) -> None:
+ assert error.db_path == db_path
+ assert error.operation is operation
+ assert error.errno == errno.EIO
+ assert error.__context__ is None
+ assert str(db_path) not in str(error)
+ assert "sensitive raw OS text" not in str(error)
+
+
+class _ProtectedBodyError(Exception):
+ pass
+
+
+class _LockWorkerTimeoutError(Exception):
+ pass
+
+
+def test_rekey_lock_contends_across_processes(tmp_path: Path) -> None:
+ # Given a separate process holding this database's sidecar lock
+ db_path = tmp_path / "user.db"
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_rekey_lock,
+ args=(str(db_path), acquired, release),
+ )
+ process.start()
+
+ try:
+ assert acquired.wait(timeout=10), "child did not acquire the rekey lock"
+
+ # When this process attempts the same nonblocking lock
+ with pytest.raises(RekeyLockContentionError):
+ with acquire_rekey_lock(db_path):
+ pytest.fail("contended rekey lock was acquired")
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then only the child acquired it and exited cleanly
+ assert process.exitcode == 0
+
+
+def _os_lock_attempt_name() -> str:
+ return "_acquire_windows_lock" if os.name == "nt" else "_acquire_posix_lock"
+
+
+def test_contention_retry_rides_out_a_holder_that_releases_in_the_window(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given another process holding the sidecar lock, which it releases only
+ # once this process has already been refused at least once
+ db_path = tmp_path / "user.db"
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_rekey_lock,
+ args=(str(db_path), acquired, release),
+ )
+ process.start()
+ attempt_name = _os_lock_attempt_name()
+ real_attempt = getattr(rekey_lock_module, attempt_name)
+ refusals = 0
+
+ def attempt_then_release_holder(*args: object, **kwargs: object) -> None:
+ nonlocal refusals
+ try:
+ real_attempt(*args, **kwargs)
+ except RekeyLockContentionError:
+ refusals += 1
+ release.set()
+ raise
+
+ monkeypatch.setattr(
+ rekey_lock_module, attempt_name, attempt_then_release_holder
+ )
+
+ try:
+ assert acquired.wait(timeout=10), "child did not acquire the rekey lock"
+
+ # When this process takes the lock with a generous retry window
+ with acquire_rekey_lock(db_path, contention_retry=30.0):
+ held_here = True
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then it was refused while the holder lived and acquired once released
+ assert held_here
+ assert refusals >= 1
+ assert process.exitcode == 0
+
+
+def test_contention_retry_gives_up_on_a_holder_beyond_the_window(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given another process holding the sidecar lock for longer than the window
+ db_path = tmp_path / "user.db"
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_rekey_lock,
+ args=(str(db_path), acquired, release),
+ )
+ process.start()
+ attempt_name = _os_lock_attempt_name()
+ real_attempt = getattr(rekey_lock_module, attempt_name)
+ attempts = 0
+
+ def counting_attempt(*args: object, **kwargs: object) -> None:
+ nonlocal attempts
+ attempts += 1
+ real_attempt(*args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module, attempt_name, counting_attempt)
+ window = 0.5
+
+ try:
+ assert acquired.wait(timeout=10), "child did not acquire the rekey lock"
+
+ # When this process retries the nonblocking lock for the window
+ started = time.monotonic()
+ with pytest.raises(RekeyLockContentionError):
+ with acquire_rekey_lock(db_path, contention_retry=window):
+ pytest.fail("lock held by another process was acquired")
+ elapsed = time.monotonic() - started
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then it kept retrying for the whole window and then failed as contention
+ assert elapsed >= window
+ assert elapsed < window + 5
+ assert attempts > 1
+ assert process.exitcode == 0
+
+
+@contextmanager
+def _flock_held_by_another_descriptor(db_path: Path) -> Iterator[None]:
+ # A second open file description of the sidecar: flock makes it contend
+ # with this process's own descriptor exactly as another process's
+ # would, and it never takes this process's gate, like a foreign holder.
+ import fcntl
+
+ fd = os.open(
+ db_path.with_name(f"{db_path.name}.rekey.lock"),
+ os.O_RDWR | os.O_CREAT,
+ 0o600,
+ )
+ try:
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ yield
+ finally:
+ os.close(fd)
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_timeout_gives_up_on_a_flock_holder_after_the_bound(
+ tmp_path: Path,
+) -> None:
+ # Given a holder that does not release on its own (another process's
+ # rekey of a large database, a stopped process, a hung mount)
+ db_path = tmp_path / "user.db"
+ bound = 0.3
+ with _flock_held_by_another_descriptor(db_path):
+ # When a request-path caller takes the sidecar with a timeout
+ started = time.monotonic()
+ with pytest.raises(RekeyLockContentionError):
+ with acquire_rekey_lock(db_path, timeout=bound):
+ pytest.fail("a flock held elsewhere was acquired")
+ elapsed = time.monotonic() - started
+
+ # Then it retried for the whole bound and gave up as typed contention
+ assert bound <= elapsed < bound + 5
+ # and the same bounded acquisition succeeds once the holder is gone
+ with acquire_rekey_lock(db_path, timeout=bound):
+ pass
+
+
+def test_timeout_bounds_the_wait_for_an_in_process_holder(
+ tmp_path: Path,
+) -> None:
+ # Given another thread of this process holding the sidecar, so the
+ # in-process gate as well, until the test lets it go
+ db_path = tmp_path / "user.db"
+ bound = 0.3
+ acquired = threading.Event()
+ release = threading.Event()
+
+ def hold() -> None:
+ with acquire_rekey_lock(db_path):
+ acquired.set()
+ release.wait(timeout=30)
+
+ holder = threading.Thread(target=hold, daemon=True)
+ holder.start()
+ try:
+ assert acquired.wait(timeout=10), "holder never took the sidecar"
+ # When a request-path caller takes the sidecar with a timeout
+ started = time.monotonic()
+ with pytest.raises(RekeyLockContentionError):
+ with acquire_rekey_lock(db_path, timeout=bound):
+ pytest.fail("the gate held by another thread was acquired")
+ elapsed = time.monotonic() - started
+ finally:
+ release.set()
+ holder.join(timeout=10)
+
+ # Then the gate wait was bounded the same way
+ assert not holder.is_alive()
+ assert bound <= elapsed < bound + 5
+
+
+def test_authorization_shared_leases_coexist_across_processes(
+ tmp_path: Path,
+) -> None:
+ # Given another process holds a shared authorization lease
+ db_path = tmp_path / "user.db"
+ shared = rekey_lock_module.AuthorizationLeaseMode.SHARED
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_authorization_lease,
+ args=(str(db_path), shared.value, acquired, release),
+ )
+ parent_acquired = False
+ process.start()
+
+ try:
+ assert acquired.wait(timeout=10), "child did not acquire shared lease"
+
+ # When this process requests another nonblocking shared lease
+ with rekey_lock_module.acquire_authorization_lease(
+ db_path, shared, blocking=False
+ ):
+ parent_acquired = True
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then both shared leases coexist and the child exits cleanly
+ assert parent_acquired is True
+ assert process.exitcode == 0
+
+
+def test_authorization_exclusive_contends_with_shared_holder(
+ tmp_path: Path,
+) -> None:
+ # Given another process holds a shared authorization lease
+ db_path = tmp_path / "user.db"
+ modes = rekey_lock_module.AuthorizationLeaseMode
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_authorization_lease,
+ args=(str(db_path), modes.SHARED.value, acquired, release),
+ )
+ process.start()
+
+ try:
+ assert acquired.wait(timeout=10), "child did not acquire shared lease"
+
+ # When this process requests a nonblocking exclusive lease
+ with pytest.raises(RekeyLockContentionError):
+ with rekey_lock_module.acquire_authorization_lease(
+ db_path, modes.EXCLUSIVE, blocking=False
+ ):
+ pytest.fail("exclusive lease bypassed shared holder")
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then contention is distinct and the shared holder exits cleanly
+ assert process.exitcode == 0
+
+
+def test_authorization_shared_contends_with_exclusive_holder(
+ tmp_path: Path,
+) -> None:
+ # Given another process holds an exclusive authorization lease
+ db_path = tmp_path / "user.db"
+ modes = rekey_lock_module.AuthorizationLeaseMode
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ process = context.Process(
+ target=_hold_authorization_lease,
+ args=(str(db_path), modes.EXCLUSIVE.value, acquired, release),
+ )
+ process.start()
+
+ try:
+ assert acquired.wait(timeout=10), (
+ "child did not acquire exclusive lease"
+ )
+
+ # When this process requests a nonblocking shared lease
+ with pytest.raises(RekeyLockContentionError):
+ with rekey_lock_module.acquire_authorization_lease(
+ db_path, modes.SHARED, blocking=False
+ ):
+ pytest.fail("shared lease bypassed exclusive holder")
+ finally:
+ release.set()
+ process.join(timeout=10)
+ if process.is_alive():
+ process.terminate()
+ process.join(timeout=5)
+
+ # Then contention is distinct and the exclusive holder exits cleanly
+ assert process.exitcode == 0
+
+
+def test_authorization_sidecar_is_persistent_and_owner_only(
+ tmp_path: Path,
+) -> None:
+ # Given a database path without an authorization sidecar
+ db_path = tmp_path / "user.db"
+ lock_path = tmp_path / "user.db.rekey.authorization.lock"
+ shared = rekey_lock_module.AuthorizationLeaseMode.SHARED
+
+ # When the authorization lease is acquired and released twice
+ with rekey_lock_module.acquire_authorization_lease(db_path, shared):
+ assert lock_path.exists()
+ original_inode = lock_path.stat().st_ino
+ with rekey_lock_module.acquire_authorization_lease(db_path, shared):
+ pass
+
+ # Then the same owner-only sidecar remains available
+ assert lock_path.exists()
+ assert lock_path.stat().st_ino == original_inode
+ if os.name != "nt":
+ assert stat.S_IMODE(lock_path.stat().st_mode) == 0o600
+
+
+def test_rekey_lock_sidecar_is_persistent_and_owner_only(
+ tmp_path: Path,
+) -> None:
+ # Given a database path without a sidecar
+ db_path = tmp_path / "user.db"
+ lock_path = tmp_path / "user.db.rekey.lock"
+
+ # When the rekey lock is acquired and released
+ with acquire_rekey_lock(db_path):
+ assert lock_path.exists()
+ original_inode = lock_path.stat().st_ino
+
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+
+ # Then the owner-only sidecar remains for future lock acquisitions
+ assert lock_path.exists()
+ assert lock_path.stat().st_ino == original_inode
+ if os.name != "nt":
+ assert stat.S_IMODE(lock_path.stat().st_mode) == 0o600
+
+
+def test_rekey_lock_chmod_failure_is_best_effort(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a filesystem that rejects permission changes
+ db_path = tmp_path / "user.db"
+
+ def fail_chmod(_path_or_fd: str | Path | int, _mode: int) -> None:
+ raise OSError(errno.EPERM, "chmod unsupported")
+
+ monkeypatch.setattr(os, "chmod", fail_chmod)
+ # The sidecar mode is tightened through its fd (never by path).
+ monkeypatch.setattr(os, "fchmod", fail_chmod, raising=False)
+
+ # When the stable lock is acquired, then chmod cannot abort locking
+ with acquire_rekey_lock(db_path):
+ assert db_path.with_name("user.db.rekey.lock").exists()
+
+
+ not hasattr(os, "O_NOFOLLOW"), reason="O_NOFOLLOW unavailable"
+)
+def test_symlinked_rekey_lock_sidecar_is_refused(tmp_path: Path) -> None:
+ # Given a symlink planted at the sidecar name, pointing at another file
+ db_path = tmp_path / "user.db"
+ target = tmp_path / "unrelated.txt"
+ target.write_text("unrelated\n", encoding="utf-8")
+ os.chmod(target, 0o644)
+ db_path.with_name("user.db.rekey.lock").symlink_to(target)
+
+ # When the stable lock is acquired, then the link is refused, typed
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("sidecar lock acquired through a symlink")
+
+ # And neither the open nor the permission tightening reached the target
+ assert (
+ raised.value.operation is rekey_lock_module.RekeySidecarOperation.OPEN
+ )
+ assert raised.value.errno == errno.ELOOP
+ assert "symlink" in str(raised.value)
+ assert stat.S_IMODE(target.stat().st_mode) == 0o644
+ assert target.read_text(encoding="utf-8") == "unrelated\n"
+
+
+ os.name == "nt" or not hasattr(os, "link"),
+ reason="POSIX hard links required",
+)
+ "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]
+)
+def test_hard_linked_sidecar_is_locked_without_touching_its_mode(
+ tmp_path: Path, lock_suffix: str
+) -> None:
+ # Given a hard link at the sidecar name whose other name is outside the
+ # sidecar's directory (a backup or snapshot tool that hard-links the
+ # data directory into another one)
+ data_dir = tmp_path / "encrypted_databases"
+ data_dir.mkdir()
+ snapshot_dir = tmp_path / "snapshot"
+ snapshot_dir.mkdir()
+ db_path = data_dir / "user.db"
+ target = snapshot_dir / "unrelated.txt"
+ target.write_text("unrelated\n", encoding="utf-8")
+ os.chmod(target, 0o644)
+ os.link(target, db_path.with_name(f"user.db{lock_suffix}"))
+
+ # When the sidecar is opened for the rekey lock or authorization lease
+ if lock_suffix == ".rekey.lock":
+ acquire = acquire_rekey_lock(db_path)
+ else:
+ acquire = rekey_lock_module.acquire_authorization_lease(
+ db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED
+ )
+ entered = False
+ with acquire:
+ entered = True
+
+ # Then the lock is taken (no lockout), and the linked file keeps its
+ # mode and content
+ assert entered
+ assert stat.S_IMODE(target.stat().st_mode) == 0o644
+ assert target.read_text(encoding="utf-8") == "unrelated\n"
+
+
+ os.name == "nt" or not hasattr(os, "link"),
+ reason="POSIX hard links required",
+)
[email protected]("victim_suffix", ["", "-wal", "-shm"])
+def test_sidecar_hard_linked_to_another_users_database_is_refused_unopened(
+ tmp_path: Path, victim_suffix: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given this user's lock file hard-linked to ANOTHER user's database
+ # file in the same directory
+ db_path = tmp_path / "ldr_user_attacker.db"
+ db_path.write_bytes(b"attacker")
+ victim = tmp_path / f"ldr_user_victim.db{victim_suffix}"
+ victim.write_bytes(b"victim database")
+ lock_path = db_path.with_name(f"{db_path.name}.rekey.lock")
+ os.link(victim, lock_path)
+ real_open = os.open
+ opened: list[str] = []
+
+ def spy_open(path, *args, **kwargs):
+ opened.append(os.fspath(path))
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", spy_open)
+
+ # When the rekey lock is taken
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("sidecar lock acquired through another user's file")
+
+ # Then it is refused, typed, and the victim's inode is never opened
+ assert raised.value.errno == errno.EMLINK
+ assert (
+ raised.value.operation is rekey_lock_module.RekeySidecarOperation.OPEN
+ )
+ assert os.fspath(lock_path) not in opened
+ assert os.fspath(victim) not in opened
+ assert victim.read_bytes() == b"victim database"
+
+
+def test_plain_sidecar_is_locked(tmp_path: Path) -> None:
+ # Given an ordinary database directory with no links
+ db_path = tmp_path / "user.db"
+ db_path.write_bytes(b"database")
+ (tmp_path / "other.db").write_bytes(b"other")
+
+ # When the rekey lock is taken twice (created, then reused)
+ for _ in range(2):
+ with acquire_rekey_lock(db_path):
+ pass
+
+ # Then it is a single-link regular file owned by the sidecar name
+ status = db_path.with_name("user.db.rekey.lock").stat()
+ assert status.st_nlink == 1
+ assert stat.S_ISREG(status.st_mode)
+
+
+ os.name == "nt" or not hasattr(os, "link"),
+ reason="POSIX hard links required",
+)
[email protected]("database_suffix", ["", "-wal", "-shm"])
+def test_sidecar_hard_linked_to_the_database_is_refused_unopened(
+ tmp_path: Path, database_suffix: str, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given the lock file hard-linked to the database, its WAL or its shm
+ db_path = tmp_path / "user.db"
+ db_path.write_bytes(b"database")
+ target = db_path.with_name(f"user.db{database_suffix}")
+ if database_suffix:
+ target.write_bytes(b"sidecar")
+ lock_path = db_path.with_name("user.db.rekey.lock")
+ os.link(target, lock_path)
+ real_open = os.open
+ opened: list[str] = []
+
+ def spy_open(path, *args, **kwargs):
+ opened.append(os.fspath(path))
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", spy_open)
+
+ # When the rekey lock is taken
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("sidecar lock acquired through a link to the database")
+
+ # Then it is refused, typed, without ever opening (and so closing) a
+ # descriptor to the database file
+ assert raised.value.errno == errno.EMLINK
+ assert (
+ raised.value.operation is rekey_lock_module.RekeySidecarOperation.OPEN
+ )
+ assert os.fspath(lock_path) not in opened
+
+
+ os.name == "nt" or not hasattr(os, "mkfifo"),
+ reason="POSIX FIFOs required",
+)
+ "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]
+)
+def test_non_regular_sidecar_is_refused_and_its_fd_closed(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, lock_suffix: str
+) -> None:
+ # Given a FIFO planted at the sidecar name
+ db_path = tmp_path / "user.db"
+ sidecar = db_path.with_name(f"user.db{lock_suffix}")
+ os.mkfifo(sidecar, 0o600)
+ opened_fds: list[int] = []
+ real_open = os.open
+
+ def recording_open(path, flags, mode=0o777, *args, **kwargs):
+ fd = real_open(path, flags, mode, *args, **kwargs)
+ if Path(path) == sidecar:
+ opened_fds.append(fd)
+ return fd
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", recording_open)
+
+ # When the sidecar is opened for the rekey lock or authorization lease
+ if lock_suffix == ".rekey.lock":
+ acquire = acquire_rekey_lock(db_path)
+ else:
+ acquire = rekey_lock_module.acquire_authorization_lease(
+ db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED
+ )
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire:
+ pytest.fail("sidecar lock acquired on a FIFO")
+ monkeypatch.undo()
+
+ # Then the open is refused, typed, and the fd it opened is closed
+ assert (
+ raised.value.operation is rekey_lock_module.RekeySidecarOperation.OPEN
+ )
+ assert raised.value.errno == errno.EINVAL
+ assert "non-regular" in str(raised.value)
+ assert len(opened_fds) == 1
+ with pytest.raises(OSError) as closed:
+ os.fstat(opened_fds[0])
+ assert closed.value.errno == errno.EBADF
+ assert stat.S_ISFIFO(sidecar.lstat().st_mode)
+
+
+def test_missing_epoch_is_invalid_until_repaired_under_lock(
+ tmp_path: Path,
+) -> None:
+ # Given no durable epoch for an existing database path
+ db_path = tmp_path / "user.db"
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+ # When the caller repairs it while holding the stable sidecar lock
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+ # Then the persisted token is a canonical lowercase UUID
+ assert str(UUID(epoch)) == epoch
+ assert rekey_lock_module.read_rekey_epoch(db_path) == epoch
+
+
+def test_rekey_epoch_pending_commit_and_cleanup_are_explicit(
+ tmp_path: Path,
+) -> None:
+ # Given a stable committed epoch
+ db_path = tmp_path / "user.db"
+ with acquire_rekey_lock(db_path):
+ committed_epoch = rekey_lock_module.read_or_initialize_rekey_epoch(
+ db_path
+ )
+
+ # When a candidate is begun but not yet committed
+ pending_epoch = begin_rekey_epoch(db_path)
+ assert str(UUID(pending_epoch)) == pending_epoch
+ if os.name != "nt":
+ pending_path = db_path.with_name("user.db.rekey.pending")
+ assert stat.S_IMODE(pending_path.stat().st_mode) == 0o600
+
+ # Then cache projection fails closed while full state remains recoverable
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=committed_epoch,
+ pending_epoch=pending_epoch,
+ )
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+ # When the candidate is committed, pending metadata remains explicit
+ commit_rekey_epoch(db_path, pending_epoch)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=pending_epoch,
+ pending_epoch=pending_epoch,
+ )
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+ # When cleanup succeeds, the committed epoch becomes cache-visible
+ clear_pending_rekey_epoch(db_path, pending_epoch)
+
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=pending_epoch,
+ pending_epoch=None,
+ )
+ assert rekey_lock_module.read_rekey_epoch(db_path) == pending_epoch
+ if os.name != "nt":
+ pending_path = db_path.with_name("user.db.rekey.pending")
+ epoch_path = db_path.with_name("user.db.rekey.epoch")
+ assert not pending_path.exists()
+ assert stat.S_IMODE(epoch_path.stat().st_mode) == 0o600
+
+
+def test_malformed_epoch_fails_closed_under_lock(
+ tmp_path: Path,
+) -> None:
+ # Given a malformed durable epoch
+ db_path = tmp_path / "user.db"
+ epoch_path = db_path.with_name("user.db.rekey.epoch")
+ epoch_path.write_text("NOT-A-CANONICAL-UUID", encoding="ascii")
+ # When the caller attempts to read or repair it under the lock
+ with acquire_rekey_lock(db_path):
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_rekey_epoch_state(db_path)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+ # Then malformed content is preserved for explicit operator recovery
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+ assert epoch_path.read_text(encoding="ascii") == "NOT-A-CANONICAL-UUID"
+
+
+def _commit_first_epoch(db_path: Path) -> str:
+ with acquire_rekey_lock(db_path):
+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+
+def _run_bounded(
+ call: Callable[[], object], timeout: float = 5.0
+) -> tuple[threading.Thread, dict[str, object]]:
+ # Runs ``call`` on a daemon thread and waits at most ``timeout``, so a
+ # read that hangs fails the test instead of the CI job.
+ outcome: dict[str, object] = {}
+
+ def target() -> None:
+ try:
+ outcome["value"] = call()
+ except BaseException as error: # noqa: BLE001 - handed to the test
+ outcome["error"] = error
+
+ worker = threading.Thread(target=target, daemon=True)
+ worker.start()
+ worker.join(timeout)
+ return worker, outcome
+
+
+def _release_fifo_reader(fifo: Path) -> None:
+ # A reader blocked opening the FIFO is released by a writer that opens
+ # and closes it; with no reader waiting the open fails (ENXIO) and
+ # there is nothing to release.
+ try:
+ fd = os.open(fifo, os.O_WRONLY | os.O_NONBLOCK)
+ except OSError:
+ return
+ os.close(fd)
+
+
[email protected](not hasattr(os, "mkfifo"), reason="POSIX FIFOs required")
[email protected]("suffix", [".rekey.epoch", ".rekey.pending"])
[email protected]("planted_kind", ["fifo", "directory"])
+def test_a_non_regular_epoch_file_is_refused_without_reading_it(
+ tmp_path: Path, suffix: str, planted_kind: str
+) -> None:
+ # Given a FIFO or a directory planted at an epoch file name
+ db_path = tmp_path / "user.db"
+ _commit_first_epoch(db_path)
+ planted = db_path.with_name(f"user.db{suffix}")
+ planted.unlink(missing_ok=True)
+ if planted_kind == "fifo":
+ os.mkfifo(planted, 0o600)
+ else:
+ planted.mkdir()
+
+ # When the durable state is read under the sidecar, as every probe does
+ def read_under_the_sidecar() -> RekeyEpochState:
+ with acquire_rekey_lock(db_path):
+ return read_rekey_epoch_state(db_path)
+
+ worker, outcome = _run_bounded(read_under_the_sidecar)
+ finished = not worker.is_alive()
+ if planted_kind == "fifo":
+ _release_fifo_reader(planted)
+ worker.join(timeout=10)
+
+ # Then it fails at once as "nothing was read", leaves the file alone
+ # and releases the sidecar
+ assert finished, "the epoch read waited on a planted FIFO under the flock"
+ error = outcome.get("error")
+ assert isinstance(error, rekey_lock_module.RekeyEpochUnreadableError), (
+ outcome
+ )
+ assert error.errno == errno.EINVAL
+ if planted_kind == "fifo":
+ assert stat.S_ISFIFO(planted.lstat().st_mode)
+ else:
+ assert planted.is_dir()
+ with acquire_rekey_lock(db_path):
+ pass
+
+
[email protected](os.name == "nt", reason="POSIX symlinks required")
[email protected]("suffix", [".rekey.epoch", ".rekey.pending"])
+def test_a_symlinked_epoch_file_is_refused_not_followed(
+ tmp_path: Path, suffix: str
+) -> None:
+ # Given a link at an epoch file name to a file elsewhere holding the
+ # committed token itself, so following the link would vouch for it
+ db_path = tmp_path / "user.db"
+ committed = _commit_first_epoch(db_path)
+ target = tmp_path / "elsewhere" / "token"
+ target.parent.mkdir()
+ target.write_text(committed, encoding="ascii")
+ link = db_path.with_name(f"user.db{suffix}")
+ link.unlink(missing_ok=True)
+ link.symlink_to(target)
+
+ # When the durable state is read
+ with pytest.raises(rekey_lock_module.RekeyEpochUnreadableError) as raised:
+ read_rekey_epoch_state(db_path)
+
+ # Then the link is refused (O_NOFOLLOW), and both files are left as found
+ assert raised.value.errno == errno.ELOOP
+ assert link.is_symlink()
+ assert target.read_text(encoding="ascii") == committed
+
+
[email protected]("suffix", [".rekey.epoch", ".rekey.pending"])
+def test_an_oversized_epoch_file_is_refused_without_being_read(
+ tmp_path: Path, suffix: str
+) -> None:
+ # Given a regular file far larger than a token at an epoch file name
+ db_path = tmp_path / "user.db"
+ _commit_first_epoch(db_path)
+ size = 8 * 1024 * 1024
+ db_path.with_name(f"user.db{suffix}").write_bytes(b"0" * size)
+
+ # When the durable state is read
+ was_tracing = tracemalloc.is_tracing()
+ if not was_tracing:
+ tracemalloc.start()
+ try:
+ tracemalloc.reset_peak()
+ baseline, _ = tracemalloc.get_traced_memory()
+ with pytest.raises(RekeyRecoveryRequiredError) as raised:
+ read_rekey_epoch_state(db_path)
+ _, peak = tracemalloc.get_traced_memory()
+ finally:
+ if not was_tracing:
+ tracemalloc.stop()
+
+ # Then it does not vouch, like any malformed token (it is not an
+ # unreadable file), and was refused without being read into memory
+ assert not isinstance(
+ raised.value, rekey_lock_module.RekeyEpochUnreadableError
+ )
+ assert peak - baseline < size // 8
+
+
+ ("error_number", "tolerated"),
+ [
+ pytest.param(errno.EIO, False, id="EIO-real-failure"),
+ pytest.param(errno.EINVAL, True, id="EINVAL-unsupported"),
+ ],
+)
+def test_pending_transition_directory_fsync_failure_fails_closed(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ error_number: int,
+ tolerated: bool,
+) -> None:
+ """A directory-fsync error during pending publication is classified.
+
+ ``_sync_parent_directory`` treats the errnos a filesystem uses to say
+ "directory fsync is not supported here" (EINVAL, ENOTSUP/EOPNOTSUPP,
+ ENOSYS) as "as durable as this filesystem allows", because treating
+ them as fatal made every epoch publication fail on such mounts. A real
+ failure (EIO) still propagates. This test originally asserted that
+ EINVAL raised; that predates the tolerance and contradicted the
+ documented behaviour (and
+ ``test_unsupported_directory_fsync_does_not_fail_epoch_publication``),
+ so it now pins both sides on the pending transition: the real failure
+ fails closed with the visible pending file still fencing caches, and
+ the unsupported errno publishes the pending epoch normally.
+ """
+ if os.name == "nt":
+ pytest.skip("directory fsync is skipped entirely on Windows")
+ # Given a stable epoch and a filesystem whose directory fsync errors
+ db_path = tmp_path / "user.db"
+ with acquire_rekey_lock(db_path):
+ committed_epoch = rekey_lock_module.read_or_initialize_rekey_epoch(
+ db_path
+ )
+ real_fsync = os.fsync
+
+ def fail_directory_fsync(fd: int) -> None:
+ if stat.S_ISDIR(os.fstat(fd).st_mode):
+ raise OSError(error_number, "directory fsync error")
+ real_fsync(fd)
+
+ monkeypatch.setattr(os, "fsync", fail_directory_fsync)
+
+ # When pending publication cannot synchronize its parent directory
+ with acquire_rekey_lock(db_path):
+ if tolerated:
+ pending_epoch = begin_rekey_epoch(db_path)
+ else:
+ with pytest.raises(OSError) as raised:
+ begin_rekey_epoch(db_path)
+ assert raised.value.errno == error_number
+
+ # Then E1 is preserved and the pending file fences caches either way
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == committed_epoch
+ assert state.pending_epoch is not None
+ if tolerated:
+ assert state.pending_epoch == pending_epoch
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+
+def test_epoch_replace_failure_aborts_without_accepting_partial_state(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given an epoch replacement that the filesystem refuses
+ db_path = tmp_path / "user.db"
+ epoch_path = db_path.with_name("user.db.rekey.epoch")
+
+ def fail_replace(source: str | Path, _target: str | Path) -> None:
+ if os.name != "nt":
+ assert stat.S_IMODE(Path(source).stat().st_mode) == 0o600
+ raise OSError(errno.EIO, "replace failed")
+
+ monkeypatch.setattr(os, "replace", fail_replace)
+
+ # When repair attempts the atomic replace, then the write failure propagates
+ with acquire_rekey_lock(db_path):
+ with pytest.raises(OSError, match="replace failed"):
+ rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+ assert not epoch_path.exists()
+
+
+def test_pending_without_committed_epoch_fails_closed(tmp_path: Path) -> None:
+ # Given a canonical pending epoch without committed metadata
+ db_path = tmp_path / "user.db"
+ pending_path = db_path.with_name("user.db.rekey.pending")
+ pending_path.write_text(str(UUID(int=1)), encoding="ascii")
+
+ # When state is read, then the contradictory transition requires recovery
+ with pytest.raises(RekeyRecoveryRequiredError):
+ read_rekey_epoch_state(db_path)
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+
+def test_posix_noncontention_error_is_typed(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ if os.name == "nt":
+ pytest.skip("POSIX backend test")
+
+ # Given flock reports an unrelated I/O error
+ import fcntl
+
+ def fail_flock(_fd: int, _operation: int) -> None:
+ raise OSError(errno.EIO, "simulated I/O failure")
+
+ monkeypatch.setattr(fcntl, "flock", fail_flock)
+
+ # When the sidecar lock is acquired, then the I/O failure is normalized
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(tmp_path / "user.db"):
+ pytest.fail("lock unexpectedly acquired")
+
+ _assert_sidecar_io_error(
+ raised.value,
+ tmp_path / "user.db",
+ rekey_lock_module.RekeySidecarOperation.ACQUIRE,
+ )
+ assert not isinstance(raised.value, RekeyLockContentionError)
+
+
+def test_sidecar_open_failure_is_typed(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given sidecar creation fails with raw operating-system text
+ db_path = tmp_path / "user.db"
+
+ def fail_open(_path: Path, _flags: int, _mode: int) -> int:
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ # When the existing rekey lock opens its persistent sidecar
+ with monkeypatch.context() as scoped:
+ scoped.setattr(os, "open", fail_open)
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("sidecar unexpectedly opened")
+
+ # Then the failure carries only typed safe diagnostics
+ _assert_sidecar_io_error(
+ raised.value,
+ db_path,
+ rekey_lock_module.RekeySidecarOperation.OPEN,
+ )
+
+
+def test_sidecar_acquire_failure_is_typed(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given the platform backend reports a non-contention acquisition failure
+ db_path = tmp_path / "user.db"
+ acquire_name = (
+ "_acquire_windows_lock" if os.name == "nt" else "_acquire_posix_lock"
+ )
+
+ def fail_acquire(
+ _fd: int,
+ _lock_path: Path,
+ *,
+ mode: rekey_lock_module.AuthorizationLeaseMode | None = None,
+ blocking: bool = False,
+ ) -> None:
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ # When the existing rekey lock acquires its sidecar
+ with monkeypatch.context() as scoped:
+ scoped.setattr(rekey_lock_module, acquire_name, fail_acquire)
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("sidecar lock unexpectedly acquired")
+
+ # Then acquisition exposes only the typed sidecar failure
+ _assert_sidecar_io_error(
+ raised.value,
+ db_path,
+ rekey_lock_module.RekeySidecarOperation.ACQUIRE,
+ )
+
+
+def test_sidecar_unlock_failure_is_typed_and_closes_once(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given unlock fails and close calls are observable
+ db_path = tmp_path / "user.db"
+ release_name = (
+ "_release_windows_lock" if os.name == "nt" else "_release_posix_lock"
+ )
+ real_close = os.close
+ close_calls: list[int] = []
+
+ def fail_release(_fd: int) -> None:
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ def close_once(fd: int) -> None:
+ close_calls.append(fd)
+ real_close(fd)
+
+ # When the protected body succeeds but unlock fails
+ with monkeypatch.context() as scoped:
+ scoped.setattr(rekey_lock_module, release_name, fail_release)
+ scoped.setattr(os, "close", close_once)
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pass
+
+ # Then unlock is typed and close was attempted exactly once
+ _assert_sidecar_io_error(
+ raised.value,
+ db_path,
+ rekey_lock_module.RekeySidecarOperation.UNLOCK,
+ )
+ assert len(close_calls) == 1
+
+
+def test_sidecar_close_failure_is_typed_without_retry(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given close fails after actually closing the descriptor
+ db_path = tmp_path / "user.db"
+ real_close = os.close
+ close_calls: list[int] = []
+
+ def fail_close(fd: int) -> None:
+ close_calls.append(fd)
+ real_close(fd)
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ # When cleanup closes the existing rekey sidecar
+ with monkeypatch.context() as scoped:
+ scoped.setattr(os, "close", fail_close)
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pass
+
+ # Then close is typed and never retried
+ _assert_sidecar_io_error(
+ raised.value,
+ db_path,
+ rekey_lock_module.RekeySidecarOperation.CLOSE,
+ )
+ assert len(close_calls) == 1
+
+
+def test_protected_body_error_survives_unlock_and_close_failures(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given both cleanup stages fail with unsafe operating-system text
+ db_path = tmp_path / "user.db"
+ release_name = (
+ "_release_windows_lock" if os.name == "nt" else "_release_posix_lock"
+ )
+ real_close = os.close
+ close_calls: list[int] = []
+
+ def fail_release(_fd: int) -> None:
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ def fail_close(fd: int) -> None:
+ close_calls.append(fd)
+ real_close(fd)
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ # When the protected body raises before cleanup runs
+ with monkeypatch.context() as scoped:
+ scoped.setattr(rekey_lock_module, release_name, fail_release)
+ scoped.setattr(os, "close", fail_close)
+ with pytest.raises(_ProtectedBodyError) as raised:
+ with acquire_rekey_lock(db_path):
+ raise _ProtectedBodyError
+
+ # Then the body error wins and cleanup emits only safe typed notes
+ assert len(close_calls) == 1
+ assert len(raised.value.__notes__) == 2
+ assert all(str(db_path) not in note for note in raised.value.__notes__)
+ assert all(
+ "sensitive raw OS text" not in note for note in raised.value.__notes__
+ )
+ assert any("unlock" in note for note in raised.value.__notes__)
+ assert any("close" in note for note in raised.value.__notes__)
+
+
+class _CallerHandledError(Exception):
+ """An unrelated exception the caller is handling when it takes the lock."""
+
+
+def _fail_one_release_stage(
+ scoped: pytest.MonkeyPatch, stage: rekey_lock_module.RekeySidecarOperation
+) -> list[int]:
+ release_name = (
+ "_release_windows_lock" if os.name == "nt" else "_release_posix_lock"
+ )
+ real_close = os.close
+ close_calls: list[int] = []
+
+ def fail_release(_fd: int) -> None:
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ def fail_close(fd: int) -> None:
+ close_calls.append(fd)
+ real_close(fd)
+ raise OSError(errno.EIO, "sensitive raw OS text")
+
+ match stage:
+ case rekey_lock_module.RekeySidecarOperation.UNLOCK:
+ scoped.setattr(rekey_lock_module, release_name, fail_release)
+ case rekey_lock_module.RekeySidecarOperation.CLOSE:
+ scoped.setattr(os, "close", fail_close)
+ case _:
+ raise AssertionError(f"not a release stage: {stage}")
+ return close_calls
+
+
+ "stage",
+ [
+ rekey_lock_module.RekeySidecarOperation.UNLOCK,
+ rekey_lock_module.RekeySidecarOperation.CLOSE,
+ ],
+ ids=["unlock", "close"],
+)
+def test_release_failure_after_a_clean_body_raises_inside_a_callers_except_block(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ stage: rekey_lock_module.RekeySidecarOperation,
+) -> None:
+ # Given a caller that takes the lock while it handles an unrelated
+ # exception (a probe made from an error handler), and a release stage
+ # that fails
+ db_path = tmp_path / "user.db"
+ handled = _CallerHandledError("the caller's own error")
+
+ # When the protected body finishes cleanly
+ with monkeypatch.context() as scoped:
+ close_calls = _fail_one_release_stage(scoped, stage)
+ try:
+ raise handled
+ except _CallerHandledError:
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
+ with acquire_rekey_lock(db_path):
+ pass
+
+ # Then the release failure is raised, typed and scrubbed, as it is
+ # outside an except block. It used to be added as a note to the
+ # caller's unrelated exception (the one sys.exception() returned in the
+ # lock's cleanup) and never raised, so the caller saw a clean release.
+ assert raised.value.operation is stage
+ assert raised.value.errno == errno.EIO
+ assert str(db_path) not in str(raised.value)
+ assert "sensitive raw OS text" not in str(raised.value)
+ assert not getattr(handled, "__notes__", None)
+ if stage is rekey_lock_module.RekeySidecarOperation.CLOSE:
+ assert len(close_calls) == 1
+
+
+def test_body_error_inside_a_callers_except_block_carries_the_release_note(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a caller handling an unrelated exception and a failing unlock
+ db_path = tmp_path / "user.db"
+ handled = _CallerHandledError("the caller's own error")
+
+ # When the protected body itself raises
+ with monkeypatch.context() as scoped:
+ _fail_one_release_stage(
+ scoped, rekey_lock_module.RekeySidecarOperation.UNLOCK
+ )
+ try:
+ raise handled
+ except _CallerHandledError:
+ with pytest.raises(_ProtectedBodyError) as raised:
+ with acquire_rekey_lock(db_path):
+ raise _ProtectedBodyError
+
+ # Then the body's error wins and carries the release failure as a
+ # note; the caller's exception is left alone
+ assert len(raised.value.__notes__) == 1
+ assert "unlock" in raised.value.__notes__[0]
+ assert not getattr(handled, "__notes__", None)
+
+
+@final
+class _FakeMsvcrt(ModuleType):
+ LK_LOCK: ClassVar[int] = 0
+ LK_NBLCK: ClassVar[int] = 1
+ LK_RLCK: ClassVar[int] = 2
+ LK_NBRLCK: ClassVar[int] = 3
+ LK_UNLCK: ClassVar[int] = 4
+
+ def __init__(self, acquire_error: OSError | None = None) -> None:
+ super().__init__("msvcrt")
+ self.acquire_error = acquire_error
+ self.calls: list[tuple[int, int, int]] = []
+
+ def locking(self, fd: int, mode: int, size: int) -> None:
+ self.calls.append((os.lseek(fd, 0, os.SEEK_CUR), mode, size))
+ if (
+ mode in {self.LK_NBLCK, self.LK_NBRLCK}
+ and self.acquire_error is not None
+ ):
+ raise self.acquire_error
+
+
+def test_windows_backend_locks_one_byte_from_start(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given an empty persistent sidecar and a Windows locking backend
+ lock_path = tmp_path / "user.db.rekey.lock"
+ fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
+ backend = _FakeMsvcrt()
+ monkeypatch.setitem(sys.modules, "msvcrt", backend)
+
+ try:
+ # When the Windows lock is acquired and released
+ _acquire_windows_lock(fd, lock_path)
+ _release_windows_lock(fd)
+ finally:
+ os.close(fd)
+
+ # Then a persistent byte backs both operations at offset zero
+ assert lock_path.read_bytes() == b"\0"
+ assert backend.calls == [
+ (0, backend.LK_NBLCK, 1),
+ (0, backend.LK_UNLCK, 1),
+ ]
+
+
+def test_windows_backend_selects_shared_and_exclusive_modes(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a persistent sidecar and a Windows locking backend
+ lock_path = tmp_path / "user.db.rekey.authorization.lock"
+ fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
+ backend = _FakeMsvcrt()
+ monkeypatch.setitem(sys.modules, "msvcrt", backend)
+ modes = rekey_lock_module.AuthorizationLeaseMode
+
+ try:
+ # When shared nonblocking and exclusive blocking leases are acquired
+ _acquire_windows_lock(fd, lock_path, mode=modes.SHARED)
+ _release_windows_lock(fd)
+ _acquire_windows_lock(
+ fd, lock_path, mode=modes.EXCLUSIVE, blocking=True
+ )
+ _release_windows_lock(fd)
+ finally:
+ os.close(fd)
+
+ # Then Windows uses read-lock and write-lock modes respectively
+ assert [call[1] for call in backend.calls] == [
+ backend.LK_NBRLCK,
+ backend.LK_UNLCK,
+ backend.LK_LOCK,
+ backend.LK_UNLCK,
+ ]
+
+
+def test_windows_backend_translates_only_contention(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given the Windows backend reports lock contention
+ lock_path = tmp_path / "user.db.rekey.lock"
+ fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
+ backend = _FakeMsvcrt(OSError(errno.EACCES, "region already locked"))
+ monkeypatch.setitem(sys.modules, "msvcrt", backend)
+
+ try:
+ # When the one-byte lock is attempted, then contention is typed
+ with pytest.raises(RekeyLockContentionError) as raised:
+ _acquire_windows_lock(fd, lock_path)
+ finally:
+ os.close(fd)
+
+ assert raised.value.lock_path == lock_path
+
+
+def test_windows_backend_preserves_unrelated_io_errors(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given the Windows backend reports an unrelated I/O error
+ lock_path = tmp_path / "user.db.rekey.lock"
+ fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
+ backend = _FakeMsvcrt(OSError(errno.EIO, "simulated I/O failure"))
+ monkeypatch.setitem(sys.modules, "msvcrt", backend)
+
+ try:
+ # When the one-byte lock is attempted, then the original error propagates
+ with pytest.raises(OSError) as raised:
+ _acquire_windows_lock(fd, lock_path)
+ finally:
+ os.close(fd)
+
+ assert raised.value.errno == errno.EIO
+ assert not isinstance(raised.value, RekeyLockContentionError)
+
+
+def test_epoch_token_is_fsynced_before_it_replaces_the_epoch_file(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a spy on fsync and replace in the epoch writer
+ db_path = tmp_path / "user.db"
+ events: list[tuple[str, str]] = []
+ real_fsync = os.fsync
+ real_replace = os.replace
+
+ def spy_fsync(fd: int) -> None:
+ status = os.fstat(fd)
+ if stat.S_ISREG(status.st_mode):
+ events.append(("fsync", os.readlink(f"/proc/self/fd/{fd}")))
+ real_fsync(fd)
+
+ def spy_replace(source, target) -> None:
+ events.append(("replace", os.fspath(source)))
+ real_replace(source, target)
+
+ if not Path("/proc/self/fd").is_dir():
+ pytest.skip("needs /proc/self/fd to name the synced file")
+ monkeypatch.setattr(rekey_lock_module.os, "fsync", spy_fsync)
+ monkeypatch.setattr(rekey_lock_module.os, "replace", spy_replace)
+
+ # When an epoch is published
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+ # Then the temporary token file was fsynced before it was renamed over
+ # the epoch file
+ replaced = [path for kind, path in events if kind == "replace"]
+ assert len(replaced) == 1
+ synced_before = events[: events.index(("replace", replaced[0]))]
+ assert ("fsync", replaced[0]) in synced_before
+
+
+class TestEpochTransitionGuards:
+ """Each epoch transition refuses a state it was not written for."""
+
+ _ONE = str(UUID(int=1))
+ _TWO = str(UUID(int=2))
+ _THREE = str(UUID(int=3))
+
+ def _state(self, db_path: Path, committed, pending) -> None:
+ for suffix, token in (
+ (".rekey.epoch", committed),
+ (".rekey.pending", pending),
+ ):
+ path = db_path.with_name(f"{db_path.name}{suffix}")
+ if token is None:
+ path.unlink(missing_ok=True)
+ else:
+ path.write_text(token, encoding="ascii")
+
+ def test_begin_refuses_without_a_committed_epoch(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ with pytest.raises(RekeyRecoveryRequiredError):
+ begin_rekey_epoch(db_path)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(None, None)
+
+ def test_begin_refuses_while_a_generation_is_pending(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._ONE, self._TWO)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ begin_rekey_epoch(db_path)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._ONE, self._TWO
+ )
+
+ def test_commit_refuses_a_candidate_that_is_not_pending(
+ self, tmp_path: Path
+ ):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._ONE, self._TWO)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ commit_rekey_epoch(db_path, self._THREE)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._ONE, self._TWO
+ )
+
+ def test_commit_refuses_without_a_pending_generation(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._ONE, None)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ commit_rekey_epoch(db_path, self._TWO)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._ONE, None
+ )
+
+ def test_commit_refuses_a_non_canonical_candidate(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._ONE, self._TWO)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ commit_rekey_epoch(db_path, self._TWO.replace("-", ""))
+
+ def test_clear_refuses_before_the_pending_generation_is_committed(
+ self, tmp_path: Path
+ ):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._ONE, self._TWO)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ clear_pending_rekey_epoch(db_path, self._TWO)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._ONE, self._TWO
+ )
+
+ def test_clear_refuses_a_different_pending_generation(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._TWO, self._THREE)
+ with pytest.raises(RekeyRecoveryRequiredError):
+ clear_pending_rekey_epoch(db_path, self._TWO)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._TWO, self._THREE
+ )
+
+ def test_clear_is_idempotent_once_cleared(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._TWO, None)
+ clear_pending_rekey_epoch(db_path, self._TWO)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._TWO, None
+ )
+
+ def test_clear_removes_a_committed_pending_generation(self, tmp_path: Path):
+ db_path = tmp_path / "user.db"
+ self._state(db_path, self._TWO, self._TWO)
+ clear_pending_rekey_epoch(db_path, self._TWO)
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ self._TWO, None
+ )
+
+
+ os.name == "nt" or not hasattr(os, "link"),
+ reason="POSIX hard links required",
+)
+def test_hard_link_guard_cannot_identify_an_inode_after_its_other_name_is_removed(
+ tmp_path: Path,
+) -> None:
+ """Document the trusted-directory boundary: current names, not history."""
+ db_path = tmp_path / "user.db"
+ other_path = tmp_path / "other-user.db"
+ other_path.write_bytes(b"file whose original name will be removed")
+ lock_path = db_path.with_name(f"{db_path.name}.rekey.lock")
+ os.link(other_path, lock_path)
+
+ # Control: while the original name is present, the guard refuses it.
+ with pytest.raises(rekey_lock_module.RekeySidecarIOError) as refused:
+ with acquire_rekey_lock(db_path):
+ pytest.fail("a same-directory hard link was accepted")
+ assert refused.value.errno == errno.EMLINK
+
+ # Removing that name erases the evidence available to the guard. This
+ # is a documented limit, not protection against writers of the data dir.
+ other_path.unlink()
+ assert lock_path.stat().st_nlink == 1
+ with acquire_rekey_lock(db_path):
+ assert (
+ lock_path.read_bytes()
+ == b"file whose original name will be removed"
+ )
diff --git a/tests/database/test_rekey_serialization.py b/tests/database/test_rekey_serialization.py
new file mode 100644
index 0000000000..51bd384c48
--- /dev/null
+++ b/tests/database/test_rekey_serialization.py
@@ -0,0 +1,1632 @@
+from __future__ import annotations
+
+import ast
+import errno
+import multiprocessing
+import os
+import stat
+import threading
+import time
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import contextmanager
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Final
+from unittest.mock import MagicMock, call, patch
+
+import pytest
+
+from local_deep_research.database import encrypted_db
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochState,
+ RekeyEpochUnreadableError,
+ RekeyIndeterminateError,
+ RekeyLockContentionError,
+ RekeyRecoveryRequiredError,
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+ RekeyStage,
+ RekeyUnavailableError,
+ acquire_rekey_lock,
+ begin_rekey_epoch,
+ commit_rekey_epoch,
+ read_rekey_epoch_state,
+)
+
+
+def _new_manager(data_dir: Path) -> DatabaseManager:
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = data_dir
+ return manager
+
+
+def manager(tmp_path: Path) -> DatabaseManager:
+ return _new_manager(tmp_path)
+
+
+def _existing_db(manager: DatabaseManager, username: str) -> Path:
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ return db_path
+
+
+def _initialize_epoch(db_path: Path) -> str:
+ with acquire_rekey_lock(db_path):
+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+
+type _QueryRow = tuple[int, ...] | tuple[str, ...]
+
+
+@dataclass(frozen=True, slots=True)
+class _RawVerification:
+ synchronous: int = 2
+ quick_rows: tuple[_QueryRow, ...] = (("ok",),)
+ cipher_rows: tuple[_QueryRow, ...] = ()
+ checkpoint: tuple[int, int, int] = (0, 0, 0)
+ schema_readable: bool = True
+
+
+_SUCCESSFUL_RAW: Final = _RawVerification()
+
+
+def _query_result(
+ *,
+ row: _QueryRow | None = None,
+ rows: tuple[_QueryRow, ...] = (),
+) -> MagicMock:
+ result = MagicMock()
+ result.fetchone.return_value = row
+ result.__iter__.return_value = iter(rows)
+ return result
+
+
+def _raw_connection(
+ verification: _RawVerification = _SUCCESSFUL_RAW,
+ *,
+ events: list[str] | None = None,
+ label: str = "raw",
+) -> MagicMock:
+ connection = MagicMock()
+
+ def execute(statement: str) -> MagicMock:
+ if events is not None:
+ events.append(f"{label}:{statement}")
+ if statement == "PRAGMA synchronous = FULL":
+ return _query_result()
+ if statement == "PRAGMA synchronous":
+ return _query_result(row=(verification.synchronous,))
+ if statement == "SELECT count(*) FROM sqlite_master":
+ if not verification.schema_readable:
+ raise RuntimeError("file is not a database")
+ return _query_result(row=(1,))
+ if statement == "PRAGMA quick_check":
+ return _query_result(rows=verification.quick_rows)
+ if statement == "PRAGMA cipher_integrity_check":
+ return _query_result(rows=verification.cipher_rows)
+ if statement == "PRAGMA wal_checkpoint(TRUNCATE)":
+ return _query_result(row=verification.checkpoint)
+ if statement == "PRAGMA wal_checkpoint(PASSIVE)":
+ return _query_result(row=verification.checkpoint)
+ if statement.startswith("PRAGMA busy_timeout = "):
+ return _query_result()
+ raise AssertionError(f"unexpected raw SQL: {statement}")
+
+ connection.execute.side_effect = execute
+ return connection
+
+
+def test_same_user_contention_raises_before_database_work(
+ manager: DatabaseManager,
+) -> None:
+ # Given one same-user rekey paused on its raw verified connection
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+ rekey_started = threading.Event()
+ release_rekey = threading.Event()
+
+ def controlled_rekey(connection, _password, *, db_path) -> None:
+ assert connection is old_connection
+ rekey_started.set()
+ assert release_rekey.wait(timeout=5)
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ) as make_raw,
+ patch.object(manager, "open_user_database") as cached_open,
+ patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey",
+ side_effect=controlled_rekey,
+ ) as rekey,
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ winner = executor.submit(
+ manager.change_password, "alice", "old-password", "new-password"
+ )
+ assert rekey_started.wait(timeout=5)
+
+ try:
+ # When a second request for Alice arrives during the rekey
+ with pytest.raises(RekeyLockContentionError):
+ manager.change_password(
+ "alice", "other-old-password", "other-new-password"
+ )
+ finally:
+ release_rekey.set()
+
+ # Then only the winner reaches raw verification and rekey
+ assert winner.result(timeout=5) is True
+ assert make_raw.call_args_list == [
+ call(db_path, "old-password"),
+ call(db_path, "new-password"),
+ ]
+ cached_open.assert_not_called()
+ rekey.assert_called_once_with(
+ old_connection, "new-password", db_path=db_path
+ )
+ old_connection.close.assert_called_once_with()
+ new_connection.close.assert_called_once_with()
+
+
+def test_in_process_sidecar_holder_contention_raises_before_database_work(
+ manager: DatabaseManager,
+) -> None:
+ # Given this thread already holds Alice's sidecar lock, so the
+ # change_password below loses on the in-process gate after its bounded
+ # local wait (the cross-process flock path is pinned separately by
+ # test_cross_process_flock_holder_is_typed_contention_not_a_hang)
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ with (
+ patch.object(manager, "_make_sqlcipher_connection") as make_raw,
+ acquire_rekey_lock(db_path),
+ ):
+ # When this manager attempts Alice's password change
+ with pytest.raises(RekeyLockContentionError):
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then it never touches the shared database lifecycle
+ make_raw.assert_not_called()
+
+
+def test_rekey_locks_remain_held_through_raw_connection_close(
+ manager: DatabaseManager,
+) -> None:
+ # Given a successful rekey paused while closing its raw connection
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+ close_started = threading.Event()
+ release_close = threading.Event()
+
+ def controlled_close() -> None:
+ close_started.set()
+ assert release_close.wait(timeout=5)
+
+ old_connection.close.side_effect = controlled_close
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch("local_deep_research.database.encrypted_db.set_sqlcipher_rekey"),
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ winner = executor.submit(
+ manager.change_password, "alice", "old-password", "new-password"
+ )
+ assert close_started.wait(timeout=5)
+
+ try:
+ # When another request arrives during final raw cleanup
+ with pytest.raises(RekeyLockContentionError):
+ manager.change_password(
+ "alice", "other-old-password", "other-new-password"
+ )
+ finally:
+ release_close.set()
+
+ # Then cleanup completes before either serialization lock is released
+ assert winner.result(timeout=5) is True
+
+
+def test_different_users_rekey_independently(manager: DatabaseManager) -> None:
+ # Given Alice's raw rekey is paused while Bob has a separate database
+ alice_path = _existing_db(manager, "alice")
+ bob_path = _existing_db(manager, "bob")
+ _initialize_epoch(alice_path)
+ _initialize_epoch(bob_path)
+ alice_old_connection = _raw_connection(label="alice-old")
+ alice_new_connection = _raw_connection(label="alice-new")
+ bob_old_connection = _raw_connection(label="bob-old")
+ bob_new_connection = _raw_connection(label="bob-new")
+ alice_started = threading.Event()
+ release_alice = threading.Event()
+ bob_rekeyed = threading.Event()
+
+ def make_raw(_db_path: Path, password: str) -> MagicMock:
+ return {
+ "alice-old": alice_old_connection,
+ "alice-new": alice_new_connection,
+ "bob-old": bob_old_connection,
+ "bob-new": bob_new_connection,
+ }[password]
+
+ def controlled_rekey(connection, _password, *, db_path) -> None:
+ if connection is alice_old_connection:
+ alice_started.set()
+ assert release_alice.wait(timeout=5)
+ else:
+ assert connection is bob_old_connection
+ bob_rekeyed.set()
+
+ with (
+ patch.object(
+ manager, "_make_sqlcipher_connection", side_effect=make_raw
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey",
+ side_effect=controlled_rekey,
+ ),
+ ThreadPoolExecutor(max_workers=2) as executor,
+ ):
+ alice = executor.submit(
+ manager.change_password, "alice", "alice-old", "alice-new"
+ )
+ assert alice_started.wait(timeout=5)
+
+ try:
+ # When Bob changes his password while Alice owns Alice's locks
+ bob = executor.submit(
+ manager.change_password, "bob", "bob-old", "bob-new"
+ )
+
+ # Then Bob completes without waiting for Alice
+ assert bob.result(timeout=5) is True
+ assert bob_rekeyed.is_set()
+ assert not alice.done()
+ finally:
+ release_alice.set()
+
+ assert alice.result(timeout=5) is True
+
+
+def test_wrong_old_password_does_not_advance_epoch(
+ manager: DatabaseManager,
+) -> None:
+ # Given raw SQLCipher verification rejects the supplied old password
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=ValueError("invalid database key"),
+ ) as make_raw,
+ patch.object(manager, "open_user_database") as cached_open,
+ patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey"
+ ) as rekey,
+ ):
+ # When Alice attempts a password change
+ result = manager.change_password(
+ "alice", "incorrect-password", "new-password"
+ )
+
+ # Then verification stays uncached and the durable epoch remains unchanged
+ assert result is False
+ make_raw.assert_called_once_with(db_path, "incorrect-password")
+ cached_open.assert_not_called()
+ rekey.assert_not_called()
+ assert rekey_lock_module.read_rekey_epoch(db_path) == original_epoch
+
+
+def test_encryption_unavailable_returns_false(
+ manager: DatabaseManager,
+) -> None:
+ # Given this process runs in the documented unencrypted fallback mode
+ manager.has_encryption = False
+
+ # When Alice requests a password change
+ changed = manager.change_password("alice", "old-password", "new-password")
+
+ # Then the documented boolean contract holds: unencrypted mode is a
+ # static deployment mode (nothing to rekey), not an operational
+ # failure, so it answers False with a warning rather than a typed 503
+ # (tests/security/test_db_connection_unencrypted_mode.py pins the same).
+ assert changed is False
+
+
+def test_missing_database_is_typed_unavailable(
+ manager: DatabaseManager,
+) -> None:
+ # Given Alice has no database file
+ db_path = manager._get_user_db_path("alice")
+
+ # When Alice requests a password change
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then absence is operationally distinct from old-password rejection
+ assert raised.value.db_path == db_path
+ assert raised.value.stage is RekeyStage.PREPARATION
+
+
+def test_pre_pending_checkpoint_failure_is_typed_unavailable(
+ manager: DatabaseManager,
+) -> None:
+ # Given the old key opens but its pre-rekey checkpoint fails
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ nested_message = "old-password checkpoint detail"
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ return_value=old_connection,
+ ),
+ patch.object(
+ manager,
+ "_pre_rekey_checkpoint",
+ side_effect=OSError(nested_message),
+ ),
+ ):
+ # When preparation cannot establish a durable checkpoint
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then the typed error contains no nested SQLCipher or password text
+ assert raised.value.db_path == db_path
+ assert raised.value.stage is RekeyStage.PREPARATION
+ assert nested_message not in str(raised.value)
+ old_connection.close.assert_called_once_with()
+
+
+def test_cached_engine_disposal_failure_is_typed_unavailable(
+ manager: DatabaseManager,
+) -> None:
+ # Given preparation detaches a cached engine that cannot be disposed
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ engine.dispose.side_effect = OSError("old-password disposal detail")
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(engine, "old-password", epoch),
+ )
+
+ # When Alice requests a password change
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then disposal is an operational preparation failure, not bad credentials
+ assert raised.value.db_path == db_path
+ assert raised.value.stage is RekeyStage.PREPARATION
+ assert "old-password" not in str(raised.value)
+ engine.dispose.assert_called_once_with()
+
+
+def test_sidecar_io_failure_is_typed_unavailable(
+ manager: DatabaseManager,
+) -> None:
+ # Given the database exists but its process sidecar cannot be opened
+ db_path = _existing_db(manager, "alice")
+
+ with patch.object(
+ encrypted_db,
+ "acquire_rekey_lock",
+ side_effect=OSError("old-password sidecar detail"),
+ ):
+ # When Alice requests a password change
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then sidecar I/O is unavailable without carrying nested secret text
+ assert raised.value.db_path == db_path
+ assert raised.value.stage is RekeyStage.PREPARATION
+ assert "old-password" not in str(raised.value)
+
+
+def test_pending_publication_failure_without_pending_is_unavailable(
+ manager: DatabaseManager,
+) -> None:
+ # Given pending publication will fail before making its marker visible
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ return_value=old_connection,
+ ),
+ patch.object(
+ encrypted_db,
+ "begin_rekey_epoch",
+ side_effect=OSError("pending publication detail"),
+ ),
+ ):
+ # When the failed operation is reconciled against stable E1
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then the operation is unavailable because no candidate became durable
+ assert raised.value.stage is RekeyStage.PENDING_PUBLICATION
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=original_epoch,
+ pending_epoch=None,
+ )
+
+
+def test_pending_publication_failure_with_visible_pending_is_indeterminate(
+ manager: DatabaseManager,
+) -> None:
+ # Given pending os.replace succeeds before its directory fsync fails
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ pending_path = db_path.with_name(f"{db_path.name}.rekey.pending")
+ old_connection = _raw_connection(label="old")
+ real_sync_parent = rekey_lock_module._sync_parent_directory
+
+ def fail_pending_directory_sync(path: Path) -> None:
+ if path == pending_path:
+ raise OSError("pending directory fsync detail")
+ real_sync_parent(path)
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ return_value=old_connection,
+ ),
+ patch.object(
+ rekey_lock_module,
+ "_sync_parent_directory",
+ side_effect=fail_pending_directory_sync,
+ ),
+ ):
+ # When the pending publication wrapper reports failure
+ with pytest.raises(RekeyIndeterminateError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then its visible E1+pending-E2 state remains explicit for recovery
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+ assert raised.value.stage is RekeyStage.PENDING_PUBLICATION
+ assert raised.value.state == state
+
+
+def test_rekey_publishes_after_fresh_candidate_verification(
+ manager: DatabaseManager,
+) -> None:
+ # Given distinct old-key and new-key raw connections
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ events: list[str] = []
+ old_connection = _raw_connection(events=events, label="old")
+ new_connection = _raw_connection(events=events, label="new")
+ real_commit = commit_rekey_epoch
+
+ def make_raw(_db_path: Path, password: str) -> MagicMock:
+ events.append(f"open:{password}")
+ return old_connection if password == "old-password" else new_connection
+
+ def spy_rekey(connection, _password, *, db_path) -> None:
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+ events.append("rekey")
+
+ def spy_commit(path: Path, candidate_epoch: str) -> None:
+ assert (
+ connection_events := [
+ event for event in events if event.startswith("new:")
+ ]
+ )
+ assert connection_events == [
+ "new:PRAGMA synchronous = FULL",
+ "new:PRAGMA synchronous",
+ "new:SELECT count(*) FROM sqlite_master",
+ "new:PRAGMA quick_check",
+ "new:PRAGMA busy_timeout = "
+ f"{encrypted_db._gated_checkpoint_busy_timeout_ms()}",
+ "new:PRAGMA wal_checkpoint(TRUNCATE)",
+ "new:PRAGMA cipher_integrity_check",
+ ]
+ assert events[-1] == "new:PRAGMA cipher_integrity_check"
+ events.append("commit")
+ real_commit(path, candidate_epoch)
+
+ with (
+ patch.object(
+ manager, "_make_sqlcipher_connection", side_effect=make_raw
+ ) as make_raw,
+ patch.object(manager, "open_user_database") as cached_open,
+ patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey",
+ side_effect=spy_rekey,
+ ) as rekey,
+ patch.object(
+ encrypted_db, "commit_rekey_epoch", side_effect=spy_commit
+ ),
+ ):
+ # When the password changes
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then only the verified candidate is published and pending metadata is cleared
+ assert result is True
+ assert make_raw.call_args_list == [
+ call(db_path, "old-password"),
+ call(db_path, "new-password"),
+ ]
+ rekey.assert_called_once_with(
+ old_connection, "new-password", db_path=db_path
+ )
+ old_connection.close.assert_called_once_with()
+ new_connection.close.assert_called_once_with()
+ cached_open.assert_not_called()
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch != original_epoch
+ assert events.index("new:PRAGMA cipher_integrity_check") < events.index(
+ "commit"
+ )
+
+
+def test_rekey_failure_preserves_committed_epoch_and_pending_candidate(
+ manager: DatabaseManager,
+) -> None:
+ # Given a cached old-key engine and a rekey operation that will fail
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_engine = MagicMock()
+ entry = manager._make_connection_cache_entry(
+ old_engine, "old-password", original_epoch
+ )
+ manager._cache_connection("alice", entry)
+ old_connection = _raw_connection(label="old")
+
+ with (
+ patch.object(manager, "_checkpoint_wal") as checkpoint,
+ patch.object(
+ manager, "_make_sqlcipher_connection", return_value=old_connection
+ ),
+ patch(
+ "local_deep_research.database.encrypted_db.set_sqlcipher_rekey",
+ side_effect=RuntimeError("simulated rekey failure"),
+ ),
+ ):
+ # When rekey fails after pending metadata is durable
+ with pytest.raises(RekeyIndeterminateError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then E1 remains committed, E2 remains pending, and no old cache survives
+ state = read_rekey_epoch_state(db_path)
+ assert raised.value.stage is RekeyStage.PHYSICAL_REKEY
+ assert raised.value.state == state
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+ assert state.pending_epoch != original_epoch
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+ assert "alice" not in manager.connections
+ assert "alice" not in manager._password_verifiers
+ assert "alice" not in manager._connection_epochs
+ checkpoint.assert_not_called()
+ old_engine.dispose.assert_called_once_with()
+ old_connection.close.assert_called_once_with()
+
+
+ "verification",
+ [
+ _RawVerification(quick_rows=(("corrupt",),)),
+ _RawVerification(cipher_rows=(("hmac failure",),)),
+ ],
+ ids=["quick-check", "cipher-integrity"],
+)
+def test_integrity_scan_findings_after_proven_key_publish(
+ manager: DatabaseManager,
+ verification: _RawVerification,
+) -> None:
+ # Given a candidate whose key decrypts the schema but whose integrity
+ # scans report damage (a database that was already damaged fails them
+ # under whichever key is active)
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(verification, label="new")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(encrypted_db.logger, "warning") as warning,
+ ):
+ # When the password change verifies the candidate
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then the proven key is published and the finding is logged loudly
+ assert result is True
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch != original_epoch
+ assert any(
+ "after the active key was proven" in str(call.args[0])
+ for call in warning.call_args_list
+ )
+
+
+ "verification",
+ [_RawVerification(synchronous=1)],
+ ids=["synchronous"],
+)
+def test_candidate_verification_failure_keeps_pending_state(
+ manager: DatabaseManager,
+ verification: _RawVerification,
+) -> None:
+ # Given a rekey whose fresh candidate connection fails a strict durability check
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(
+ verification,
+ label="new",
+ )
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ ):
+ # When the password change verifies the candidate
+ with pytest.raises(RekeyIndeterminateError) as raised:
+ manager.change_password("alice", "old-password", "new-password")
+
+ # Then publication is withheld and the pending generation fences all callers
+ state = read_rekey_epoch_state(db_path)
+ assert raised.value.stage is RekeyStage.CANDIDATE_VERIFICATION
+ assert raised.value.state == state
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+
+def test_busy_checkpoint_after_proven_candidate_publishes(
+ manager: DatabaseManager,
+) -> None:
+ # Given a candidate that passes the key proof (schema read)
+ # but whose post-rekey checkpoint is kept busy by a concurrent reader
+ # (moved out of the failure matrix above: a busy reader is not ambiguity)
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ events: list[str] = []
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(
+ _RawVerification(checkpoint=(1, 1, 0)), events=events, label="new"
+ )
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ ):
+ # When the password change verifies the candidate
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then the verified generation is published and pending state cleared,
+ # without the cipher scan that stale old-key WAL frames would fail
+ assert result is True
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch != original_epoch
+ assert "new:PRAGMA cipher_integrity_check" not in events
+
+
+def test_commit_failure_after_proven_key_is_success_with_pending_left(
+ manager: DatabaseManager,
+) -> None:
+ # Given a verified candidate whose committed publication fails before E2
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=OSError("commit publication detail"),
+ ),
+ ):
+ # When committed publication reports failure
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then the proven new key is a completed change, and E1+pending-E2
+ # stays on disk so caches stay fenced until the next open commits it
+ assert result is True
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+
+
+def test_commit_wrapper_failure_after_e2_publication_returns_success(
+ manager: DatabaseManager,
+) -> None:
+ # Given the commit wrapper publishes E2 before reporting an I/O failure
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ def publish_then_raise(path: Path, candidate_epoch: str) -> None:
+ commit_rekey_epoch(path, candidate_epoch)
+ raise OSError("post-replace directory fsync detail")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=publish_then_raise,
+ ),
+ ):
+ # When change_password reconciles publication while holding the sidecar
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then verified E2 is success and its pending marker is cleaned up
+ state = read_rekey_epoch_state(db_path)
+ assert result is True
+ assert state.committed_epoch != original_epoch
+ assert state.pending_epoch is None
+
+
+def test_commit_reconciliation_cleanup_failure_preserves_explicit_state(
+ manager: DatabaseManager,
+) -> None:
+ # Given E2 publishes before the wrapper and recovery cleanup both fail
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ def publish_then_raise(path: Path, candidate_epoch: str) -> None:
+ commit_rekey_epoch(path, candidate_epoch)
+ raise OSError("post-replace directory fsync detail")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=publish_then_raise,
+ ),
+ patch.object(
+ encrypted_db,
+ "clear_pending_rekey_epoch",
+ side_effect=OSError("cleanup detail"),
+ ),
+ ):
+ # When change_password reconciles committed publication
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then success is final while E2+pending-E2 remains recoverable
+ state = read_rekey_epoch_state(db_path)
+ assert result is True
+ assert state.committed_epoch != original_epoch
+ assert state.pending_epoch == state.committed_epoch
+
+
+def test_unreadable_commit_state_reread_after_proven_key_is_success(
+ manager: DatabaseManager,
+) -> None:
+ # Given a verified candidate whose commit failure cannot be reconciled
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(
+ encrypted_db,
+ "read_rekey_epoch_state",
+ side_effect=[
+ RekeyEpochState(original_epoch, None),
+ RekeyRecoveryRequiredError(db_path),
+ ],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=OSError("commit publication detail"),
+ ),
+ ):
+ # When commit reconciliation cannot parse durable state
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then the proven key still makes it a completed change, and the
+ # pending metadata remains untouched for the next open's recovery
+ assert result is True
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == original_epoch
+ assert state.pending_epoch is not None
+
+
+def test_candidate_close_failure_after_commit_returns_success(
+ manager: DatabaseManager,
+) -> None:
+ # Given a fully verifiable candidate whose final raw close fails
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+ new_connection.close.side_effect = OSError("simulated close failure")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ ):
+ # When durable verification and epoch publication complete first
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then cleanup cannot reverse the committed manager outcome
+ state = read_rekey_epoch_state(db_path)
+ assert result is True
+ assert state.pending_epoch is None
+ assert state.committed_epoch != original_epoch
+
+
+def test_pending_cleanup_failure_after_commit_returns_success(
+ manager: DatabaseManager,
+) -> None:
+ # Given a verified candidate whose pending cleanup fails after publication
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "clear_pending_rekey_epoch",
+ side_effect=OSError("simulated pending cleanup failure"),
+ ),
+ ):
+ # When the committed epoch is durable before cleanup runs
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then success is final and E2+pending-E2 remains explicit for recovery
+ state = read_rekey_epoch_state(db_path)
+ assert result is True
+ assert state.committed_epoch != original_epoch
+ assert state.pending_epoch == state.committed_epoch
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+
+
+def test_sidecar_release_failure_after_commit_returns_success(
+ manager: DatabaseManager,
+) -> None:
+ # Given a successful rekey whose sidecar release reports an I/O error
+ db_path = _existing_db(manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ old_connection = _raw_connection(label="old")
+ new_connection = _raw_connection(label="new")
+
+ @contextmanager
+ def release_failure(
+ path: Path,
+ *,
+ blocking: bool = False,
+ local_wait: float | None = None,
+ contention_retry: float = 0.0,
+ ):
+ with acquire_rekey_lock(
+ path,
+ blocking=blocking,
+ local_wait=local_wait,
+ contention_retry=contention_retry,
+ ):
+ yield
+ raise OSError("simulated sidecar release failure")
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(encrypted_db, "set_sqlcipher_rekey"),
+ patch.object(
+ encrypted_db,
+ "acquire_rekey_lock",
+ side_effect=release_failure,
+ ),
+ ):
+ # When release fails only after committed publication and cleanup
+ result = manager.change_password(
+ "alice", "old-password", "new-password"
+ )
+
+ # Then the manager preserves success and the stable committed generation
+ state = read_rekey_epoch_state(db_path)
+ assert result is True
+ assert state.pending_epoch is None
+ assert state.committed_epoch != original_epoch
+
+
[email protected]("candidate_already_committed", [False, True])
+def test_cold_open_recovers_verified_pending_candidate(
+ manager: DatabaseManager,
+ candidate_already_committed: bool,
+) -> None:
+ # Given either an ambiguous E1+pending-E2 or cleanup-only E2+pending-E2 state
+ db_path = _existing_db(manager, "alice")
+ with acquire_rekey_lock(db_path):
+ original_epoch = rekey_lock_module.read_or_initialize_rekey_epoch(
+ db_path
+ )
+ pending_epoch = begin_rekey_epoch(db_path)
+ if candidate_already_committed:
+ commit_rekey_epoch(db_path, pending_epoch)
+ recovery_connection = _raw_connection(label="recovery")
+ engine = MagicMock()
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ return_value=recovery_connection,
+ ) as make_raw,
+ patch.object(
+ manager, "_open_user_database_cold", return_value=engine
+ ) as cold_open,
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ # When cold open validates the supplied password against the surviving key
+ opened = manager.open_user_database("alice", "surviving-password")
+
+ # Then E2 is finalized before migrations and cache publication
+ assert opened is engine
+ make_raw.assert_called_once_with(db_path, "surviving-password")
+ cold_open.assert_called_once()
+ state = read_rekey_epoch_state(db_path)
+ assert state == RekeyEpochState(
+ committed_epoch=pending_epoch,
+ pending_epoch=None,
+ )
+ if not candidate_already_committed:
+ assert state.committed_epoch != original_epoch
+
+
+def test_cold_open_wrong_key_leaves_pending_state_untouched(
+ manager: DatabaseManager,
+) -> None:
+ # Given an interrupted rekey and a password that cannot open its surviving key
+ db_path = _existing_db(manager, "alice")
+ with acquire_rekey_lock(db_path):
+ original_epoch = rekey_lock_module.read_or_initialize_rekey_epoch(
+ db_path
+ )
+ pending_epoch = begin_rekey_epoch(db_path)
+ secret = "wrong-recovery-password"
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=ValueError(secret),
+ ),
+ patch.object(manager, "_open_user_database_cold") as cold_open,
+ ):
+ # When cold-open recovery cannot open the surviving key with it
+ opened = manager.open_user_database("alice", secret)
+
+ # Then it is an ordinary failed login (None, which the login route
+ # counts toward the lockout) and the candidate is left untouched
+ assert opened is None
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=original_epoch,
+ pending_epoch=pending_epoch,
+ )
+ assert rekey_lock_module.read_rekey_epoch(db_path) is None
+ cold_open.assert_not_called()
+
+
+def test_change_password_wrong_key_during_pending_returns_false(
+ manager: DatabaseManager,
+) -> None:
+ # Given an interrupted rekey and a current password that opens neither key
+ db_path = _existing_db(manager, "alice")
+ with acquire_rekey_lock(db_path):
+ original_epoch = rekey_lock_module.read_or_initialize_rekey_epoch(
+ db_path
+ )
+ pending_epoch = begin_rekey_epoch(db_path)
+
+ with (
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=ValueError("wrong key"),
+ ) as make_raw,
+ patch.object(encrypted_db, "set_sqlcipher_rekey") as rekey,
+ ):
+ # When a password change is attempted with that password
+ result = manager.change_password(
+ "alice", "wrong-password", "new-password"
+ )
+
+ # Then it is the ordinary wrong-current-password outcome, nothing moved
+ assert result is False
+ make_raw.assert_called_once_with(db_path, "wrong-password")
+ rekey.assert_not_called()
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(
+ committed_epoch=original_epoch,
+ pending_epoch=pending_epoch,
+ )
+
+
+def test_rekey_lock_registry_is_distinct_from_init_locks(
+ manager: DatabaseManager,
+) -> None:
+ # Given both per-user lifecycle locks are requested for Alice
+ init_lock = manager._get_init_lock("alice")
+
+ # When the rekey lock is resolved
+ rekey_lock = manager._get_rekey_lock("alice")
+
+ # Then rekey serialization does not reuse the cold-open lock
+ assert rekey_lock is not init_lock
+
+
+def _hold_sidecar_in_child(db_path: str, acquired, release) -> None:
+ with acquire_rekey_lock(Path(db_path)):
+ acquired.set()
+ if not release.wait(timeout=30):
+ raise RuntimeError("parent did not release the sidecar holder")
+
+
+def test_cross_process_flock_holder_is_typed_contention_not_a_hang(
+ manager: DatabaseManager,
+) -> None:
+ # Given a separate process holding Alice's sidecar flock (the in-process
+ # gate is free, so change_password reaches the flock itself)
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ context = multiprocessing.get_context("spawn")
+ acquired = context.Event()
+ release = context.Event()
+ holder = context.Process(
+ target=_hold_sidecar_in_child,
+ args=(str(db_path), acquired, release),
+ )
+ holder.start()
+ try:
+ assert acquired.wait(timeout=20), "child did not take the sidecar"
+ with (
+ patch.object(manager, "_make_sqlcipher_connection") as make_raw,
+ ThreadPoolExecutor(max_workers=1) as executor,
+ ):
+ # When this process attempts Alice's password change
+ started = time.monotonic()
+ attempt = executor.submit(
+ manager.change_password,
+ "alice",
+ "old-password",
+ "new-password",
+ )
+ try:
+ # Then it fails with the typed contention error after the
+ # short retry window -- a blocking flock would hang here
+ # until the holder exits
+ with pytest.raises(RekeyLockContentionError):
+ attempt.result(timeout=8)
+ elapsed = time.monotonic() - started
+ finally:
+ release.set()
+ assert elapsed >= encrypted_db._REKEY_CONTENTION_RETRY_SECONDS * 0.9
+ make_raw.assert_not_called()
+ finally:
+ release.set()
+ holder.join(timeout=10)
+ if holder.is_alive():
+ holder.terminate()
+ holder.join(timeout=5)
+ assert holder.exitcode == 0
+
+
+def test_disposal_checkpoint_busy_wait_is_bounded_and_restored(
+ manager: DatabaseManager,
+) -> None:
+ # Given a pooled connection whose busy timeout is the configured one
+ engine = MagicMock()
+ connection = engine.connect.return_value.__enter__.return_value
+ statements: list[str] = []
+
+ def execute(statement):
+ statements.append(str(statement))
+ return MagicMock()
+
+ connection.execute.side_effect = execute
+
+ # When the gated disposal checkpoint runs
+ manager._checkpoint_wal(engine, "for alice")
+
+ # Then the TRUNCATE waits at most the gated bound, and the connection
+ # gets its configured busy timeout back before it returns to the pool
+ bound = encrypted_db._gated_checkpoint_busy_timeout_ms()
+ configured = encrypted_db.get_sqlcipher_busy_timeout_ms()
+ assert bound <= encrypted_db._GATED_CHECKPOINT_BUSY_TIMEOUT_MS
+ assert statements == [
+ f"PRAGMA busy_timeout = {bound}",
+ "PRAGMA wal_checkpoint(TRUNCATE)",
+ f"PRAGMA busy_timeout = {configured}",
+ ]
+
+
+def test_post_rekey_checkpoint_busy_wait_is_bounded(
+ manager: DatabaseManager,
+) -> None:
+ # Given a fresh new-key connection
+ events: list[str] = []
+ connection = _raw_connection(events=events, label="new")
+
+ # When the post-rekey checkpoint runs while the sidecar gate is held
+ assert manager._best_effort_rekey_checkpoint(connection) is True
+
+ # Then its busy wait is bounded before the TRUNCATE is issued
+ assert events == [
+ "new:PRAGMA busy_timeout = "
+ f"{encrypted_db._gated_checkpoint_busy_timeout_ms()}",
+ "new:PRAGMA wal_checkpoint(TRUNCATE)",
+ ]
+
+
+def _busy_timeout_f_string_executions(source: str) -> list[int]:
+ """Lines where an f-string ``PRAGMA busy_timeout`` reaches execute()."""
+ found = []
+ for node in ast.walk(ast.parse(source)):
+ if not (
+ isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "execute"
+ and node.args
+ ):
+ continue
+ statement = node.args[0]
+ if (
+ isinstance(statement, ast.Call)
+ and isinstance(statement.func, ast.Name)
+ and statement.func.id == "text"
+ and statement.args
+ ):
+ statement = statement.args[0]
+ if isinstance(statement, ast.JoinedStr) and any(
+ isinstance(part, ast.Constant) and "busy_timeout" in str(part.value)
+ for part in statement.values
+ ):
+ found.append(node.lineno)
+ return found
+
+
+def test_busy_timeout_pragmas_are_built_from_an_int_template() -> None:
+ # Given the modules that set a connection's busy timeout
+ from local_deep_research.database import sqlcipher_utils
+
+ # When their execute() calls are scanned
+ offenders = {
+ module.__name__: _busy_timeout_f_string_executions(
+ Path(module.__file__).read_text(encoding="utf-8")
+ )
+ for module in (encrypted_db, sqlcipher_utils)
+ }
+
+ # Then no PRAGMA busy_timeout is an f-string handed to execute() (the
+ # repo Semgrep rule sql-execute-with-string-format); each one comes
+ # from a fixed "{:d}" template formatted with an int
+ assert offenders == {
+ encrypted_db.__name__: [],
+ sqlcipher_utils.__name__: [],
+ }
+
+
+def test_gated_checkpoint_pragma_carries_only_an_integer(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # Given the gated checkpoint bound
+ bound = encrypted_db._gated_checkpoint_busy_timeout_ms()
+
+ # Then its statement carries exactly that integer
+ assert encrypted_db._gated_checkpoint_busy_timeout_pragma() == (
+ f"PRAGMA busy_timeout = {bound}"
+ )
+
+ # And a value that is not an integer never reaches the statement
+ monkeypatch.setattr(
+ encrypted_db,
+ "_gated_checkpoint_busy_timeout_ms",
+ lambda: "5000; PRAGMA key = 'x'",
+ )
+ with pytest.raises(ValueError):
+ encrypted_db._gated_checkpoint_busy_timeout_pragma()
+
+
+def test_pre_rekey_checkpoint_is_passive_and_never_refuses(
+ manager: DatabaseManager,
+) -> None:
+ # Given an old-key connection whose WAL a reader keeps pinned (busy)
+ events: list[str] = []
+ connection = _raw_connection(
+ _RawVerification(checkpoint=(1, 7, 3)), events=events, label="old"
+ )
+
+ # When the pre-rekey checkpoint runs, it neither waits nor refuses
+ manager._pre_rekey_checkpoint(connection)
+
+ # Then only a PASSIVE checkpoint (no busy handler) was issued
+ assert events == ["old:PRAGMA wal_checkpoint(PASSIVE)"]
+
+
+ "error_number", [errno.EINVAL, getattr(errno, "ENOTSUP", errno.EINVAL)]
+)
+def test_unsupported_directory_fsync_does_not_fail_epoch_publication(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ # Given a filesystem that cannot fsync directories at all
+ if os.name == "nt":
+ pytest.skip("directory fsync is skipped entirely on Windows")
+ db_path = tmp_path / "alice.db"
+ db_path.touch()
+ real_fsync = os.fsync
+
+ def no_directory_fsync(fd: int) -> None:
+ if _is_directory_fd(fd):
+ raise OSError(error_number, "unsupported")
+ real_fsync(fd)
+
+ monkeypatch.setattr(os, "fsync", no_directory_fsync)
+
+ # When the first epoch is published there
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+ # Then the rename is as durable as that filesystem allows, not an error
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(epoch, None)
+
+
+def test_failing_directory_fsync_still_fails_epoch_publication(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a directory fsync that genuinely fails (EIO)
+ if os.name == "nt":
+ pytest.skip("directory fsync is skipped entirely on Windows")
+ db_path = tmp_path / "alice.db"
+ db_path.touch()
+ real_fsync = os.fsync
+
+ def failing_directory_fsync(fd: int) -> None:
+ if _is_directory_fd(fd):
+ raise OSError(errno.EIO, "I/O error")
+ real_fsync(fd)
+
+ monkeypatch.setattr(os, "fsync", failing_directory_fsync)
+
+ # When the first epoch is published, the failure still propagates
+ with acquire_rekey_lock(db_path):
+ with pytest.raises(OSError) as raised:
+ rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ assert raised.value.errno == errno.EIO
+
+
+def _is_directory_fd(fd: int) -> bool:
+ return stat.S_ISDIR(os.fstat(fd).st_mode)
+
+
+def _make_epoch_file_unreadable(db_path: Path) -> None:
+ # A directory where the epoch file should be: read_text raises
+ # IsADirectoryError, an OSError other than FileNotFoundError, so the
+ # reader raises RekeyEpochUnreadableError (as for EMFILE or EIO).
+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ epoch_file.unlink()
+ epoch_file.mkdir()
+
+
+ ("disturb", "expected_error"),
+ [
+ pytest.param(
+ _make_epoch_file_unreadable,
+ RekeyEpochUnreadableError,
+ id="epoch-file-unreadable",
+ ),
+ pytest.param(
+ rekey_lock_module.begin_rekey_epoch,
+ RekeyRecoveryRequiredError,
+ id="pending-epoch-appeared",
+ ),
+ pytest.param(
+ rekey_lock_module.advance_rekey_epoch,
+ RekeyRecoveryRequiredError,
+ id="committed-epoch-moved",
+ ),
+ ],
+)
+def test_post_open_epoch_recheck_that_cannot_vouch_fails_closed(
+ manager: DatabaseManager, disturb, expected_error
+) -> None:
+ # Given a cold open that proved the password, after which the re-read
+ # of the epoch fails, finds a pending epoch, or finds a different
+ # committed epoch (only a writer bypassing the held sidecar can cause
+ # the last two)
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ engine = MagicMock()
+
+ def cold_open_then_disturb(*_args, **_kwargs) -> MagicMock:
+ disturb(db_path)
+ return engine
+
+ with (
+ patch.object(
+ manager,
+ "_open_user_database_cold",
+ side_effect=cold_open_then_disturb,
+ ),
+ patch.object(manager, "_run_phase2_rekey") as phase2,
+ ):
+ # When the user logs in, nothing is known against the credential:
+ # the open raises the RekeyOperationalError the login route answers
+ # with 503 (lockout untouched), never the None a wrong password
+ # returns (401, counted toward lockout)
+ with pytest.raises(expected_error):
+ manager.open_user_database("alice", "password")
+
+ # Then the engine is still discarded unpublished
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+ phase2.assert_not_called()
+
+
+def test_cold_open_epoch_publication_io_failure_is_typed(
+ manager: DatabaseManager, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Given a database whose first epoch (post-upgrade) cannot be written
+ db_path = _existing_db(manager, "alice")
+
+ def full_disk(*_args, **_kwargs):
+ raise OSError(errno.ENOSPC, "No space left on device")
+
+ monkeypatch.setattr(rekey_lock_module.tempfile, "mkstemp", full_disk)
+
+ with patch.object(manager, "_open_user_database_cold") as cold_open:
+ # When the user logs in, the I/O failure is a typed sidecar error
+ # (the login route answers 503 without counting it toward the
+ # lockout), not a raw OSError that escapes as a server error
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.open_user_database("alice", "password")
+
+ assert raised.value.db_path == db_path
+ assert raised.value.operation is RekeySidecarOperation.PUBLISH
+ assert raised.value.errno == errno.ENOSPC
+ assert raised.value.__cause__ is None
+ assert raised.value.__suppress_context__ is True
+ cold_open.assert_not_called()
+
+
+def test_concurrent_cold_opens_fail_within_one_bound_while_sidecar_held(
+ manager: DatabaseManager,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # Given a held sidecar (a long rekey) and a slow key derivation
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ bound = 0.4
+ derivation = 0.2
+ monkeypatch.setattr(encrypted_db, "_SIDECAR_WAIT_TIMEOUT_SECONDS", bound)
+
+ def slow_derivation(_password, **_kwargs) -> bytes:
+ time.sleep(derivation)
+ return b"\x00" * 32
+
+ monkeypatch.setattr(encrypted_db, "get_key_from_password", slow_derivation)
+ holder_ready = threading.Event()
+ release_holder = threading.Event()
+
+ def hold_sidecar() -> None:
+ with acquire_rekey_lock(db_path):
+ holder_ready.set()
+ assert release_holder.wait(timeout=30)
+
+ holder = threading.Thread(target=hold_sidecar, daemon=True)
+ holder.start()
+ assert holder_ready.wait(timeout=5)
+ openers = 6
+
+ def open_once() -> tuple[str, float]:
+ start = time.monotonic()
+ try:
+ manager.open_user_database("alice", "password")
+ except RekeyLockContentionError:
+ return "contention", time.monotonic() - start
+ return "returned", time.monotonic() - start
+
+ try:
+ # When several opens of that user arrive at once
+ started = time.monotonic()
+ with ThreadPoolExecutor(max_workers=openers) as pool:
+ outcomes = list(pool.map(lambda _: open_once(), range(openers)))
+ elapsed = time.monotonic() - started
+ finally:
+ release_holder.set()
+ holder.join(timeout=5)
+
+ # Then every one fails with the typed contention error within about one
+ # bound (plus one derivation), not one after another behind init_lock
+ assert [kind for kind, _ in outcomes] == ["contention"] * openers
+ serial = openers * (bound + derivation)
+ assert elapsed < 2 * (bound + derivation) + 0.5 < serial
+ assert (
+ max(duration for _, duration in outcomes)
+ < 2 * (bound + derivation) + 0.5
+ )
+
+
+def test_recovery_that_cannot_prove_the_key_does_not_commit(
+ manager: DatabaseManager,
+) -> None:
+ # Given a pending generation and a password whose connection opens but
+ # cannot decrypt the schema page (the key proof fails)
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ begin_rekey_epoch(db_path)
+ state = read_rekey_epoch_state(db_path)
+ unproven = _raw_connection(_RawVerification(schema_readable=False))
+
+ with patch.object(
+ manager, "_make_sqlcipher_connection", return_value=unproven
+ ):
+ # When recovery runs with it
+ with pytest.raises(RekeyRecoveryRequiredError):
+ manager._recover_pending_rekey(db_path, "password", state)
+
+ # Then nothing is committed or cleared
+ assert read_rekey_epoch_state(db_path) == state
+ unproven.close.assert_called_once_with()
+
+
+def test_recovery_runs_the_key_proof_before_committing(
+ manager: DatabaseManager,
+) -> None:
+ # Given a pending generation and a password that proves the key
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ pending = begin_rekey_epoch(db_path)
+ state = read_rekey_epoch_state(db_path)
+ events: list[str] = []
+ proven = _raw_connection(events=events, label="recovery")
+ real_commit = commit_rekey_epoch
+
+ def commit_after_proof(path: Path, candidate: str) -> None:
+ assert "recovery:SELECT count(*) FROM sqlite_master" in events
+ events.append("commit")
+ real_commit(path, candidate)
+
+ with (
+ patch.object(
+ manager, "_make_sqlcipher_connection", return_value=proven
+ ),
+ patch.object(
+ encrypted_db, "commit_rekey_epoch", side_effect=commit_after_proof
+ ),
+ ):
+ # When recovery runs
+ assert manager._recover_pending_rekey(db_path, "password", state)
+
+ # Then the key proof ran first and the pending epoch is committed
+ assert "commit" in events
+ assert read_rekey_epoch_state(db_path) == RekeyEpochState(pending, None)
+
+
+def test_lifecycle_lock_timeout_spends_the_request_wait_budget(
+ manager: DatabaseManager,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ # Given a cached engine whose lifecycle lock another claim holds
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(MagicMock(), "password", epoch),
+ )
+ monkeypatch.setattr(encrypted_db, "_SIDECAR_WAIT_TIMEOUT_SECONDS", 0.2)
+ lifecycle_lock = manager._get_rekey_lock("alice")
+ assert lifecycle_lock.acquire(timeout=1)
+ try:
+ with encrypted_db.sidecar_wait_budget():
+ assert encrypted_db._sidecar_wait_seconds() == 0.2
+ # When a claim gives up on the lifecycle lock
+ with pytest.raises(RekeyLockContentionError):
+ manager.get_session("alice")
+ # Then the request's later waits try once without waiting
+ assert encrypted_db._sidecar_wait_seconds() == 0.0
+ finally:
+ lifecycle_lock.release()
diff --git a/tests/database/test_rekey_teardown_fencing.py b/tests/database/test_rekey_teardown_fencing.py
new file mode 100644
index 0000000000..7948125868
--- /dev/null
+++ b/tests/database/test_rekey_teardown_fencing.py
@@ -0,0 +1,500 @@
+from __future__ import annotations
+
+import threading
+from collections.abc import Generator
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal, assert_never
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import acquire_rekey_lock
+from local_deep_research.web.auth import connection_cleanup as cleanup_module
+from local_deep_research.web.auth.connection_cleanup import (
+ cleanup_idle_connections,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class PausedRekey:
+ manager: DatabaseManager
+ stale_engine: MagicMock
+
+
+def _new_manager(data_dir: Path) -> DatabaseManager:
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = data_dir
+ return manager
+
+
+def _existing_db(manager: DatabaseManager, username: str) -> Path:
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ return db_path
+
+
+def _initialize_epoch(db_path: Path) -> str:
+ with acquire_rekey_lock(db_path):
+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+
+def _cache_current_engine(
+ manager: DatabaseManager, username: str = "alice"
+) -> tuple[Path, MagicMock]:
+ db_path = _existing_db(manager, username)
+ epoch = _initialize_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(engine, "password", epoch),
+ )
+ return db_path, engine
+
+
+def paused_rekey(tmp_path: Path) -> Generator[PausedRekey, None, None]:
+ stale_manager = _new_manager(tmp_path)
+ db_path = _existing_db(stale_manager, "alice")
+ original_epoch = _initialize_epoch(db_path)
+ stale_engine = MagicMock()
+ stale_manager._cache_connection(
+ "alice",
+ stale_manager._make_connection_cache_entry(
+ stale_engine, "old-password", original_epoch
+ ),
+ )
+ sidecar_acquired = threading.Event()
+ release_sidecar = threading.Event()
+
+ def hold_pending_sidecar() -> None:
+ with acquire_rekey_lock(db_path):
+ pending_epoch = rekey_lock_module.begin_rekey_epoch(db_path)
+ sidecar_acquired.set()
+ assert release_sidecar.wait(timeout=5)
+ rekey_lock_module.commit_rekey_epoch(db_path, pending_epoch)
+ rekey_lock_module.clear_pending_rekey_epoch(db_path, pending_epoch)
+
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ sidecar_owner = executor.submit(hold_pending_sidecar)
+ assert sidecar_acquired.wait(timeout=5)
+ try:
+ yield PausedRekey(stale_manager, stale_engine)
+ finally:
+ release_sidecar.set()
+ sidecar_owner.result(timeout=5)
+
+
+def test_close_user_database_disposes_only_during_rekey(
+ paused_rekey: PausedRekey,
+) -> None:
+ # Given a cached old-key engine while another worker holds the rekey sidecar
+ manager = paused_rekey.manager
+ stale_engine = paused_rekey.stale_engine
+
+ with patch.object(
+ manager, "_checkpoint_wal", wraps=manager._checkpoint_wal
+ ) as checkpoint:
+ # When explicit close runs during the paused rekey
+ manager.close_user_database("alice")
+
+ # Then close detaches and disposes without opening the stale engine
+ checkpoint.assert_not_called()
+ stale_engine.connect.assert_not_called()
+ stale_engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+
+
+def test_close_all_databases_disposes_only_during_rekey(
+ paused_rekey: PausedRekey,
+) -> None:
+ # Given a cached old-key engine while another worker holds the rekey sidecar
+ manager = paused_rekey.manager
+ stale_engine = paused_rekey.stale_engine
+
+ with patch.object(
+ manager, "_checkpoint_wal", wraps=manager._checkpoint_wal
+ ) as checkpoint:
+ # When shutdown closes every cached database
+ manager.close_all_databases()
+
+ # Then shutdown disposes without opening the stale engine
+ checkpoint.assert_not_called()
+ stale_engine.connect.assert_not_called()
+ stale_engine.dispose.assert_called_once_with()
+ assert not manager.connections
+ assert not manager._password_verifiers
+ assert not manager._connection_epochs
+
+
+def test_periodic_pool_reset_skips_live_rekey_sidecar(
+ paused_rekey: PausedRekey,
+) -> None:
+ # Given an active user whose cached old-key engine needs a periodic pool reset
+ manager = paused_rekey.manager
+ stale_engine = paused_rekey.stale_engine
+ stale_engine.pool.checkedout.return_value = 0
+ session_manager = MagicMock()
+ session_manager.get_active_usernames.return_value = {"alice"}
+ previous_dispose_time = cleanup_module._last_dispose_time
+ cleanup_module._last_dispose_time = -1e9
+
+ try:
+ with (
+ patch(
+ "local_deep_research.web.auth.connection_cleanup.get_usernames_with_active_research",
+ return_value=set(),
+ ),
+ patch(
+ "local_deep_research.web.auth.connection_cleanup.cleanup_dead_threads"
+ ),
+ patch(
+ "local_deep_research.web.auth.connection_cleanup._count_open_fds",
+ return_value=0,
+ ),
+ patch.object(
+ manager, "_checkpoint_wal", wraps=manager._checkpoint_wal
+ ) as checkpoint,
+ ):
+ # When the periodic scheduler resets cached pools during rekey
+ cleanup_idle_connections(session_manager, manager)
+ finally:
+ cleanup_module._last_dispose_time = previous_dispose_time
+
+ # Then live sidecar contention skips reset without touching the cached engine
+ checkpoint.assert_not_called()
+ stale_engine.connect.assert_not_called()
+ stale_engine.dispose.assert_not_called()
+ assert manager.connections["alice"] is stale_engine
+
+
+def test_periodic_pool_reset_counts_only_successful_disposals(
+ tmp_path: Path,
+) -> None:
+ # Given two current-generation engines where one pool disposal fails
+ manager = _new_manager(tmp_path)
+ engines = {"alice": MagicMock(), "bob": MagicMock()}
+ engines["alice"].dispose.side_effect = RuntimeError("dispose failed")
+ for username, engine in engines.items():
+ db_path = _existing_db(manager, username)
+ epoch = _initialize_epoch(db_path)
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(engine, "password", epoch),
+ )
+
+ with patch.object(manager, "_checkpoint_wal") as checkpoint:
+ # When the manager resets all current cached pools
+ disposed = manager.reset_connection_pools()
+
+ # Then both are attempted, only the successful reset is counted, and caches stay
+ assert disposed == 1
+ assert checkpoint.call_count == 2
+ engines["alice"].dispose.assert_called_once_with()
+ engines["bob"].dispose.assert_called_once_with()
+ assert manager.get_connected_usernames() == {"alice", "bob"}
+
+
+def test_periodic_pool_reset_skips_replaced_engine_identity(
+ tmp_path: Path,
+) -> None:
+ # Given a cached generation that is replaced after the reset snapshot
+ manager = _new_manager(tmp_path)
+ db_path = _existing_db(manager, "alice")
+ epoch = _initialize_epoch(db_path)
+ old_engine = MagicMock()
+ replacement_engine = MagicMock()
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(old_engine, "password", epoch),
+ )
+ real_acquire = acquire_rekey_lock
+
+ def replace_before_sidecar(path: Path, *, blocking: bool = False):
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(
+ replacement_engine, "password", epoch
+ ),
+ )
+ return real_acquire(path, blocking=blocking)
+
+ with (
+ patch(
+ "local_deep_research.database.encrypted_db.acquire_rekey_lock",
+ side_effect=replace_before_sidecar,
+ ),
+ patch.object(manager, "_checkpoint_wal") as checkpoint,
+ ):
+ # When periodic reset reaches the snapshotted generation
+ disposed = manager.reset_connection_pools()
+
+ # Then it does not checkpoint or dispose either mismatched identity
+ assert disposed == 0
+ checkpoint.assert_not_called()
+ old_engine.dispose.assert_not_called()
+ replacement_engine.dispose.assert_not_called()
+ assert manager.connections["alice"] is replacement_engine
+
+
+def test_periodic_reset_skips_lifecycle_contention(tmp_path: Path) -> None:
+ # Given another accessor owns the stable lifecycle lock
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ lifecycle_lock = manager._get_rekey_lock("alice")
+ lifecycle_lock.acquire()
+
+ try:
+ # When periodic reset attempts nonblocking ownership
+ disposed = manager.reset_connection_pools()
+ finally:
+ lifecycle_lock.release()
+
+ # Then contention skips without checkpointing or disposing
+ assert disposed == 0
+ engine.connect.assert_not_called()
+ engine.dispose.assert_not_called()
+ assert manager.connections["alice"] is engine
+
+
+def test_periodic_reset_skips_sidecar_contention(tmp_path: Path) -> None:
+ # Given another process owns the stable sidecar lock
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+
+ with acquire_rekey_lock(db_path):
+ # When periodic reset attempts nonblocking ownership
+ disposed = manager.reset_connection_pools()
+
+ # Then contention skips without checkpointing or disposing
+ assert disposed == 0
+ engine.connect.assert_not_called()
+ engine.dispose.assert_not_called()
+ assert manager.connections["alice"] is engine
+
+
+def test_accessor_winner_makes_concurrent_reset_skip(tmp_path: Path) -> None:
+ # Given an accessor owns the exact current generation inside its claim
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ accessor_started = threading.Event()
+ release_accessor = threading.Event()
+
+ def hold_claim() -> bool:
+ with manager._acquire_current_connection_generation(
+ "alice"
+ ) as generation:
+ accessor_started.set()
+ assert release_accessor.wait(timeout=5)
+ return generation is not None
+
+ with ThreadPoolExecutor(max_workers=1) as executor:
+ accessor = executor.submit(hold_claim)
+ assert accessor_started.wait(timeout=5)
+ try:
+ # When reset races the accessor's lifecycle ownership
+ disposed = manager.reset_connection_pools()
+ finally:
+ release_accessor.set()
+ assert accessor.result(timeout=5) is True
+
+ # Then reset skips and never disposes the generation in active use
+ assert disposed == 0
+ engine.dispose.assert_not_called()
+
+
+def test_integrity_scan_runs_after_the_claim_is_released(
+ tmp_path: Path,
+) -> None:
+ # Given an integrity check paused inside its (slow) database scan
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ connection = engine.connect.return_value.__enter__.return_value
+ quick_check = MagicMock()
+ quick_check.fetchone.return_value = ("ok",)
+ cipher_check = MagicMock()
+ cipher_check.__iter__.return_value = iter(())
+ scan_started = threading.Event()
+ release_scan = threading.Event()
+
+ def execute(_statement):
+ if not scan_started.is_set():
+ scan_started.set()
+ assert release_scan.wait(timeout=10)
+ return quick_check
+ return cipher_check
+
+ connection.execute.side_effect = execute
+
+ with ThreadPoolExecutor(max_workers=2) as executor:
+ integrity = executor.submit(manager.check_database_integrity, "alice")
+ assert scan_started.wait(timeout=5)
+ try:
+ # When another probe for the same user (the shape the shared
+ # log-drain thread and every request take) runs meanwhile
+ probe = executor.submit(manager.is_user_connected, "alice")
+ # Then it completes without waiting for the scan: the scan no
+ # longer holds the lifecycle lock, sidecar gate or flock
+ probe_connected = probe.result(timeout=3)
+ lifecycle_free = manager._get_rekey_lock("alice").acquire(
+ blocking=False
+ )
+ if lifecycle_free:
+ manager._get_rekey_lock("alice").release()
+ finally:
+ release_scan.set()
+ assert integrity.result(timeout=5) is True
+
+ assert probe_connected is True
+ assert lifecycle_free is True
+ # The scan still ran on the engine the claim validated
+ engine.connect.assert_called_once_with()
+ engine.dispose.assert_not_called()
+
+
+def test_reset_winner_serializes_following_accessor(tmp_path: Path) -> None:
+ # Given reset owns the current generation and is paused in disposal
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ reset_started = threading.Event()
+ release_reset = threading.Event()
+ reset_action_complete = threading.Event()
+
+ def controlled_dispose() -> None:
+ reset_started.set()
+ assert release_reset.wait(timeout=5)
+ reset_action_complete.set()
+
+ engine.dispose.side_effect = controlled_dispose
+
+ with (
+ patch.object(manager, "_checkpoint_wal"),
+ ThreadPoolExecutor(max_workers=2) as executor,
+ ):
+ reset = executor.submit(manager.reset_connection_pools)
+ assert reset_started.wait(timeout=5)
+ real_claim = manager._claim_connection_generation
+
+ def claim_after_reset(generation, state):
+ assert reset_action_complete.is_set()
+ return real_claim(generation, state)
+
+ with patch.object(
+ manager,
+ "_claim_connection_generation",
+ side_effect=claim_after_reset,
+ ):
+ # When an accessor snapshots while reset still owns lifecycle
+ connected = executor.submit(manager.is_user_connected, "alice")
+ release_reset.set()
+ assert reset.result(timeout=5) == 1
+ assert connected.result(timeout=5) is True
+
+ # Then exactly one reset occurred and the current generation stayed attached
+ engine.dispose.assert_called_once_with()
+ assert manager.connections["alice"] is engine
+
+
[email protected]("transition", ["stale", "pending"])
+def test_stale_or_pending_reset_detaches_before_single_disposal(
+ tmp_path: Path,
+ transition: Literal["stale", "pending"],
+) -> None:
+ # Given reset snapshots a cache generation fenced by durable state
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ with acquire_rekey_lock(db_path):
+ match transition:
+ case "stale":
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ case "pending":
+ rekey_lock_module.begin_rekey_epoch(db_path)
+ case unreachable:
+ assert_never(unreachable)
+
+ with patch.object(manager, "_checkpoint_wal") as checkpoint:
+ # When periodic reset runs twice against that fenced generation
+ first_disposed = manager.reset_connection_pools()
+ second_disposed = manager.reset_connection_pools()
+
+ # Then exact detachment gives one owner one dispose without checkpoint
+ assert first_disposed == 1
+ assert second_disposed == 0
+ checkpoint.assert_not_called()
+ engine.connect.assert_not_called()
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+
+
+def _make_epoch_file_unreadable(db_path: Path) -> None:
+ # A directory where the epoch file should be: the sidecar still opens,
+ # but reading the epoch raises IsADirectoryError, which the reader
+ # reports as RekeyEpochUnreadableError (the same outcome as EMFILE
+ # right after the sidecar took the last fd, or EIO).
+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ epoch_file.unlink()
+ epoch_file.mkdir()
+
+
+def test_periodic_reset_keeps_engine_when_epoch_read_fails(
+ tmp_path: Path,
+) -> None:
+ # Given a current cached engine whose epoch file cannot be read
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ _make_epoch_file_unreadable(db_path)
+
+ with patch.object(manager, "_checkpoint_wal") as checkpoint:
+ # When the periodic reset sweeps it
+ disposed = manager.reset_connection_pools()
+
+ # Then nothing was read, so nothing proves it stale: the sweep skips it
+ # and keeps the engine (the next successful probe re-checks the epoch)
+ assert disposed == 0
+ checkpoint.assert_not_called()
+ engine.connect.assert_not_called()
+ engine.dispose.assert_not_called()
+ assert manager.connections["alice"] is engine
+
+
+def test_close_still_disposes_when_epoch_read_fails(tmp_path: Path) -> None:
+ # Given a current cached engine whose epoch file cannot be read
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ _make_epoch_file_unreadable(db_path)
+
+ with patch.object(manager, "_checkpoint_wal") as checkpoint:
+ # When the user's database is closed explicitly
+ manager.close_user_database("alice")
+
+ # Then it is still detached and disposed, without a WAL checkpoint
+ checkpoint.assert_not_called()
+ engine.connect.assert_not_called()
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+
+
+def test_periodic_reset_detaches_engine_when_epoch_is_malformed(
+ tmp_path: Path,
+) -> None:
+ # Given a current cached engine whose epoch file was read but is not
+ # a canonical epoch token (state that does not vouch, not a read fault)
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text("not-a-uuid")
+
+ with patch.object(manager, "_checkpoint_wal") as checkpoint:
+ disposed = manager.reset_connection_pools()
+
+ # Then the generation is stale: detached and disposed once, unchecked
+ assert disposed == 1
+ checkpoint.assert_not_called()
+ engine.connect.assert_not_called()
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
diff --git a/tests/database/test_sidecar_fault_probe.py b/tests/database/test_sidecar_fault_probe.py
new file mode 100644
index 0000000000..5765ca43da
--- /dev/null
+++ b/tests/database/test_sidecar_fault_probe.py
@@ -0,0 +1,1289 @@
+"""A sidecar fault on the cache probe fails closed and keeps the engine.
+
+The request fails closed, but the cached engine is kept: evicting on a
+transient fault (an EMFILE spike) would dispose every user's engine.
+The same holds when the sidecar opens but the epoch read then fails;
+only an epoch that was read and does not vouch detaches the engine.
+
+Every cache accessor claims the cached generation under the rekey
+sidecar. When the sidecar cannot be opened (fd exhaustion, EROFS,
+EACCES, a vanished data directory), the lock module raises the typed
+``RekeySidecarIOError`` -- not an ``OSError`` -- so the probe's
+untrusted-state branch never saw it and every caller had to catch it
+itself (``get_session``, the log sink and the metrics session did not).
+``is_user_connected`` and ``_password_matches_cached`` now read it as
+"not connected". The fault is injected at the real ``os.open`` of the
+sidecar file.
+
+Three callers must not read such a fault as a plain miss.
+``probe_user_connection`` reports it as UNAVAILABLE, so request
+authentication can answer a retryable 503 instead of a 401 that logs the
+user out. ``open_user_database`` raises it before deriving a key: its cold
+path needs the same sidecar, so falling through only added a PBKDF2
+derivation to every attempt for as long as the fault lasted. The metrics
+session falls through to that raise and lets it out too. Its callers
+decide what it means: the metrics writer reports its own "no session"
+failure, and ``get_user_db_session`` raises
+``DatabaseSessionUnavailableError``, which request handlers answer with a
+retryable 503. Folding it into a None made it the same
+``DatabaseSessionError`` a wrong password produces, and the home page
+answered that by logging the user out. ``get_session`` raises it too:
+``get_settings_manager`` reads its None as "this user has no database"
+and answers every setting from the defaults file, so during the fault a
+stored ``strict`` egress scope read as the default.
+
+Releasing the sidecar after a claim can fail as well. That raises for
+every caller when the claim's body finished cleanly, even when the probe
+runs inside a caller's except block, and is added as a note to the body's
+own exception when the body raised.
+"""
+
+from __future__ import annotations
+
+import errno
+import os
+import stat
+import threading
+import uuid
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+from contextvars import copy_context
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from local_deep_research.database import encrypted_db as encrypted_db_module
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database import (
+ session_context as session_context_module,
+)
+from local_deep_research.database import (
+ thread_local_session as thread_local_session_module,
+)
+from local_deep_research.database import thread_metrics as thread_metrics_module
+from local_deep_research.database.encrypted_db import (
+ DatabaseManager,
+ UserConnectionStatus,
+)
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochUnreadableError,
+ RekeyLockContentionError,
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+ acquire_rekey_lock,
+)
+from local_deep_research.database.sqlcipher_utils import get_salt_file_path
+from local_deep_research.utilities import db_utils, log_utils
+
+
+def _new_manager(data_dir: Path) -> DatabaseManager:
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = data_dir
+ return manager
+
+
+ "accessor", ["is_user_connected", "get_session", "_password_matches_cached"]
+)
+def test_sidecar_open_fault_fails_closed_without_evicting(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, accessor: str
+) -> None:
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+ real_open = rekey_lock_module.os.open
+
+ def emfile_on_sidecar(path, flags, *args, **kwargs):
+ if str(path).endswith(".rekey.lock"):
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", emfile_on_sidecar)
+
+ if accessor == "is_user_connected":
+ result = manager.is_user_connected("alice")
+ assert result is False
+ elif accessor == "get_session":
+ # Raised, not None: get_settings_manager reads None as "this user
+ # has no database" and answers every setting from the defaults
+ # file (see the get_settings_manager test below).
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.get_session("alice")
+ assert raised.value.operation is RekeySidecarOperation.OPEN
+ assert raised.value.errno == errno.EMFILE
+ else:
+ assert manager._password_matches_cached("alice", "password") is False
+
+ # This request failed closed, but the healthy engine was NOT evicted:
+ # an fd-exhaustion spike must not dispose every user's engine at once.
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ # Once the fault clears, the next probe re-takes the sidecar, re-checks
+ # the epoch and hands the same engine out again.
+ monkeypatch.setattr(rekey_lock_module.os, "open", real_open)
+ assert manager.is_user_connected("alice") is True
+
+
+def test_engine_kept_through_a_fault_is_still_fenced_by_a_later_rekey(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # The engine survives the faulted probe only because every later
+ # hand-out re-validates it: a rekey committed during the fault must
+ # detach it on the first successful probe.
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+ real_open = rekey_lock_module.os.open
+
+ def emfile_on_sidecar(path, flags, *args, **kwargs):
+ if str(path).endswith(".rekey.lock"):
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", emfile_on_sidecar)
+ assert manager.is_user_connected("alice") is False
+ # Kept through the fault (the precondition this test is about).
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+ monkeypatch.setattr(rekey_lock_module.os, "open", real_open)
+
+ # Another worker commits a rekey while the engine sits in the cache.
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+
+ assert manager.is_user_connected("alice") is False
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
+
+
+def _fail_epoch_reads(monkeypatch: pytest.MonkeyPatch, error_number: int):
+ # The epoch reader opens each token file with os.open (no-follow,
+ # non-blocking); fail that open for the token files only. Their names
+ # end in the suffix, while mkstemp's temporaries carry a random tail,
+ # so a publication is unaffected. Returns the call that clears it.
+ real_open = rekey_lock_module.os.open
+
+ def failing_open(path, flags, *args, **kwargs):
+ if os.fspath(path).endswith((".rekey.epoch", ".rekey.pending")):
+ raise OSError(error_number, "injected epoch read failure")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", failing_open)
+ return lambda: monkeypatch.setattr(rekey_lock_module.os, "open", real_open)
+
+
[email protected]("error_number", [errno.EMFILE, errno.EIO])
+def test_epoch_read_fault_after_the_sidecar_opens_keeps_the_engine(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ # The sidecar opens, then the epoch read fails (EMFILE right after the
+ # sidecar took the last fd). Nothing was read, so this must behave like
+ # a sidecar fault: the request sees "not connected" (get_session, for
+ # which that would mean "no database", gets the typed error), and the
+ # engine stays.
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+
+ clear_fault = _fail_epoch_reads(monkeypatch, error_number)
+ assert manager.is_user_connected("alice") is False
+ with pytest.raises(RekeyEpochUnreadableError) as raised:
+ manager.get_session("alice")
+ assert raised.value.errno == error_number
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ clear_fault()
+ assert manager.is_user_connected("alice") is True
+
+
+def test_epoch_advanced_during_an_epoch_read_fault_still_detaches(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+
+ clear_fault = _fail_epoch_reads(monkeypatch, errno.EMFILE)
+ assert manager.is_user_connected("alice") is False
+ assert manager.connections["alice"] is engine
+ clear_fault()
+
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+
+ assert manager.is_user_connected("alice") is False
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
+
+
+def test_malformed_epoch_on_the_probe_still_detaches(tmp_path: Path) -> None:
+ # Only "could not read" keeps the engine. An epoch that WAS read and
+ # cannot vouch for the generation (a malformed token) still detaches
+ # and disposes it.
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text(
+ "NOT-A-CANONICAL-UUID", encoding="ascii"
+ )
+
+ assert manager.is_user_connected("alice") is False
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
+
+
+def _cache_current_engine(
+ manager: DatabaseManager,
+ password: str = "password",
+ *,
+ username: str = "alice",
+) -> tuple[Path, MagicMock]:
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+ engine = MagicMock()
+ manager._cache_connection(
+ username, manager._make_connection_cache_entry(engine, password, epoch)
+ )
+ return db_path, engine
+
+
+def _fail_sidecar_opens(monkeypatch: pytest.MonkeyPatch, db_path: Path):
+ # EMFILE for this database's sidecar only: os.open is process-wide, and
+ # any other thread's files must keep opening normally.
+ sidecar = os.fspath(db_path.with_name(f"{db_path.name}.rekey.lock"))
+ real_open = rekey_lock_module.os.open
+
+ def emfile_on_sidecar(path, flags, *args, **kwargs):
+ if os.fspath(path) == sidecar:
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", emfile_on_sidecar)
+ return real_open
+
+
+def _watch_key_derivation(
+ monkeypatch: pytest.MonkeyPatch, watched: Path
+) -> list[Path]:
+ # Records each PBKDF2 derivation a cold open of THIS database starts and
+ # answers it with a well-formed key, so code that does derive carries on
+ # and the test can report that it did. Every other database passes
+ # straight through, so another thread in the process cannot skew it.
+ real_derive = encrypted_db_module.get_key_from_password
+ derived: list[Path] = []
+
+ def derive(password, db_path=None):
+ if db_path is not None and Path(db_path) == watched:
+ derived.append(Path(db_path))
+ return bytes(64)
+ return real_derive(password, db_path=db_path)
+
+ monkeypatch.setattr(encrypted_db_module, "get_key_from_password", derive)
+ return derived
+
+
+def test_probe_user_connection_tells_a_sidecar_fault_from_a_miss(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ manager = _new_manager(tmp_path)
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.NOT_CONNECTED
+ )
+ db_path, engine = _cache_current_engine(manager)
+ assert (
+ manager.probe_user_connection("alice") is UserConnectionStatus.CONNECTED
+ )
+
+ real_open = _fail_sidecar_opens(monkeypatch, db_path)
+ # is_user_connected cannot tell the fault from a miss; the probe can.
+ assert manager.is_user_connected("alice") is False
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.UNAVAILABLE
+ )
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", real_open)
+ assert (
+ manager.probe_user_connection("alice") is UserConnectionStatus.CONNECTED
+ )
+
+ # A stale engine is a miss, not a fault: detached and disposed.
+ with acquire_rekey_lock(db_path):
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.NOT_CONNECTED
+ )
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
+
+
[email protected]("error_number", [errno.EMFILE, errno.EIO])
+def test_probe_user_connection_reports_an_epoch_read_fault_as_unavailable(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+
+ clear_fault = _fail_epoch_reads(monkeypatch, error_number)
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.UNAVAILABLE
+ )
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ clear_fault()
+ assert (
+ manager.probe_user_connection("alice") is UserConnectionStatus.CONNECTED
+ )
+
+
+def test_probe_user_connection_reports_a_malformed_epoch_as_a_miss(
+ tmp_path: Path,
+) -> None:
+ # An epoch that was read and does not vouch is a verdict, not a fault.
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_text(
+ "NOT-A-CANONICAL-UUID", encoding="ascii"
+ )
+
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.NOT_CONNECTED
+ )
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
+
+
[email protected](os.name == "nt", reason="injects a POSIX flock unlock")
+def test_probe_user_connection_reports_a_release_fault_without_raising(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # The claim succeeds but releasing the sidecar fails. is_user_connected
+ # lets that RekeySidecarIOError(UNLOCK) escape; the probe never raises.
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+
+ def failing_unlock(fd: int) -> None:
+ raise OSError(errno.EIO, "injected unlock failure")
+
+ monkeypatch.setattr(
+ rekey_lock_module, "_release_posix_lock", failing_unlock
+ )
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.is_user_connected("alice")
+ assert raised.value.operation is RekeySidecarOperation.UNLOCK
+
+ assert (
+ manager.probe_user_connection("alice")
+ is UserConnectionStatus.UNAVAILABLE
+ )
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
+class _CallerHandledError(Exception):
+ """An unrelated exception a caller is handling when it probes."""
+
+
+def _fail_posix_unlock(monkeypatch: pytest.MonkeyPatch) -> None:
+ def failing_unlock(fd: int) -> None:
+ raise OSError(errno.EIO, "injected unlock failure")
+
+ monkeypatch.setattr(
+ rekey_lock_module, "_release_posix_lock", failing_unlock
+ )
+
+
[email protected](os.name == "nt", reason="injects a POSIX flock unlock")
+def test_a_release_fault_is_reported_when_probed_inside_an_except_block(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # A probe can run inside a caller's error handler: render_template
+ # opens a database session for its egress-scope read, and routes render
+ # their error pages from except branches. The lock's cleanup used to
+ # take the exception that CALLER was handling for the claim's own
+ # failure (sys.exception()), attach the unlock failure to it as a note
+ # and report a clean release: CONNECTED, where the probe outside an
+ # except block answers UNAVAILABLE.
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ _fail_posix_unlock(monkeypatch)
+
+ handled = _CallerHandledError("the caller's own error")
+ try:
+ raise handled
+ except _CallerHandledError:
+ status = manager.probe_user_connection("alice")
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.is_user_connected("alice")
+
+ assert status is UserConnectionStatus.UNAVAILABLE
+ assert raised.value.operation is RekeySidecarOperation.UNLOCK
+ assert not getattr(handled, "__notes__", None)
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
[email protected](os.name == "nt", reason="injects a POSIX flock unlock")
+def test_an_error_inside_the_claim_keeps_a_release_fault_as_a_note(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # The claim enters and leaves the sidecar lock by hand, so that it can
+ # tell a failed acquire from a failure in its own body. It hands the
+ # body's exception to the lock the way a with statement does, so the
+ # unlock failure becomes a note on that exception instead of replacing
+ # it with a RekeySidecarIOError.
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ _fail_posix_unlock(monkeypatch)
+
+ class _ClaimBodyError(Exception):
+ pass
+
+ def broken_sessionmaker(**_kwargs):
+ raise _ClaimBodyError("the claim's own failure")
+
+ monkeypatch.setattr(
+ encrypted_db_module, "sessionmaker", broken_sessionmaker
+ )
+
+ with pytest.raises(_ClaimBodyError) as raised:
+ manager.get_session("alice")
+
+ notes = getattr(raised.value, "__notes__", [])
+ assert len(notes) == 1
+ assert "unlock" in notes[0]
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
[email protected]("fault", ["sidecar-open", "epoch-read"])
+def test_get_settings_manager_does_not_answer_a_fault_with_defaults(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, fault: str
+) -> None:
+ # get_settings_manager without a session reaches the user's cached
+ # engine through db_manager.get_session, and answers that method's None
+ # ("this user has no database") from the defaults file. A sidecar or
+ # epoch fault used to come back as that None, so every stored setting
+ # read as its default for as long as the fault lasted: a "strict" or
+ # "private_only" egress scope as "adaptive", a custom library
+ # storage_path as the default directory. It must reach the caller.
+ manager = _new_manager(tmp_path)
+ username = f"settings_{uuid.uuid4().hex[:8]}"
+ db_path, engine = _cache_current_engine(manager, username=username)
+ monkeypatch.setattr(db_utils, "db_manager", manager)
+ expected: type[Exception]
+ if fault == "sidecar-open":
+ _fail_sidecar_opens(monkeypatch, db_path)
+ expected = RekeySidecarIOError
+ else:
+ _fail_epoch_reads(monkeypatch, errno.EIO)
+ expected = RekeyEpochUnreadableError
+
+ with pytest.raises(expected):
+ db_utils.get_settings_manager(username=username)
+
+ assert manager.connections[username] is engine
+ engine.dispose.assert_not_called()
+ # The defaults fallback itself is unchanged for a user with no engine.
+ no_engine = db_utils.get_settings_manager(
+ username=f"no_engine_{uuid.uuid4().hex[:8]}"
+ )
+ assert no_engine.db_session is None
+
+
+def test_open_user_database_fails_fast_on_a_sidecar_fault(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # With an engine cached, the fault is raised before any key is derived.
+ # Falling through to the cold path paid a PBKDF2 derivation only to
+ # fail on the same sidecar, on every request, socket handshake and
+ # background session for as long as the fault lasted.
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ real_open = _fail_sidecar_opens(monkeypatch, db_path)
+
+ for password in ("password", "wrong-password"):
+ # Never a credential verdict (None): the login route answers this
+ # typed error with a 503 that is not counted toward lockout.
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.open_user_database("alice", password)
+ assert raised.value.operation is RekeySidecarOperation.OPEN
+ assert raised.value.errno == errno.EMFILE
+
+ assert derived == []
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ # Once the fault clears, the kept engine is handed out again.
+ monkeypatch.setattr(rekey_lock_module.os, "open", real_open)
+ assert manager.open_user_database("alice", "password") is engine
+ assert derived == []
+
+
+def test_metrics_session_raises_the_typed_fault_and_the_writer_reports_none(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # create_thread_safe_session_for_metrics falls through to
+ # open_user_database, which raises the typed fault before deriving a
+ # key. The metrics session lets it out rather than answering None: a
+ # None reached get_user_db_session as the DatabaseSessionError a wrong
+ # password produces (see the get_user_db_session test below).
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ _fail_sidecar_opens(monkeypatch, db_path)
+
+ with pytest.raises(RekeySidecarIOError) as raised:
+ manager.create_thread_safe_session_for_metrics("alice", "password")
+ assert raised.value.operation is RekeySidecarOperation.OPEN
+ assert raised.value.errno == errno.EMFILE
+ assert derived == []
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+ # The metrics writer, whose callers do not know the typed error,
+ # reports its own "no session" failure instead (thread_metrics used to
+ # re-raise whatever the metrics session raised).
+ monkeypatch.setattr(thread_metrics_module, "db_manager", manager)
+ writer = thread_metrics_module.ThreadSafeMetricsWriter()
+ writer.set_user_password("alice", "password")
+ with pytest.raises(ValueError, match="Failed to create session") as failed:
+ with writer.get_session("alice"):
+ pytest.fail("a session was handed out during the fault")
+ # Raised after the fault was handled, so the fault (whose frames hold
+ # the password) is not chained onto it.
+ assert failed.value.__context__ is None
+ assert derived == []
+
+
+def _fail_sidecar_opens_after(
+ monkeypatch: pytest.MonkeyPatch, db_path: Path, healthy_opens: int
+) -> list[int]:
+ # An fd-exhaustion spike that starts part-way through a request: the
+ # first ``healthy_opens`` opens of this database's sidecar succeed,
+ # every later one fails with EMFILE. Other files are untouched.
+ sidecar = os.fspath(db_path.with_name(f"{db_path.name}.rekey.lock"))
+ real_open = rekey_lock_module.os.open
+ failed: list[int] = []
+ opened = 0
+
+ def flaky_sidecar(path, flags, *args, **kwargs):
+ nonlocal opened
+ if os.fspath(path) == sidecar:
+ opened += 1
+ if opened > healthy_opens:
+ failed.append(opened)
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock_module.os, "open", flaky_sidecar)
+ return failed
+
+
+ "healthy_opens",
+ [0, 1],
+ ids=["open-claim-faults", "metrics-claim-faults-after-open-succeeded"],
+)
+def test_get_user_db_session_reports_a_sidecar_fault_as_unavailable(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, healthy_opens: int
+) -> None:
+ # On a thread without a session, get_user_db_session claims the cached
+ # engine twice: in open_user_database, then in
+ # create_thread_safe_session_for_metrics. A fluctuating fd-exhaustion
+ # spike can fail either claim, the second one even after the first
+ # (and require_auth's probe before it) succeeded. Each used to come
+ # back as None, i.e. the plain DatabaseSessionError a wrong password
+ # produces, which "/" answered by clearing the session. It must be the
+ # typed subclass that request handlers answer with a retryable 503.
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ sessions = thread_local_session_module.ThreadLocalSessionManager()
+ monkeypatch.setattr(thread_local_session_module, "db_manager", manager)
+ monkeypatch.setattr(
+ thread_local_session_module, "thread_session_manager", sessions
+ )
+ monkeypatch.setattr(
+ session_context_module, "thread_session_manager", sessions
+ )
+ failed = _fail_sidecar_opens_after(monkeypatch, db_path, healthy_opens)
+
+ with pytest.raises(session_context_module.DatabaseSessionError) as raised:
+ with session_context_module.get_user_db_session("alice", "password"):
+ pytest.fail("a session was handed out during the fault")
+
+ assert failed, "the fault was never hit: measuring the wrong branch"
+ # Compared by name so a tree without the subclass reports what it
+ # raised instead (the plain error), rather than an AttributeError.
+ assert type(raised.value).__name__ == "DatabaseSessionUnavailableError", (
+ "a sidecar fault must not read like a failed open: got "
+ f"{type(raised.value).__name__}({raised.value})"
+ )
+ assert isinstance(
+ raised.value, session_context_module.DatabaseSessionUnavailableError
+ )
+ assert raised.value.reason == "RekeySidecarIOError"
+ # Neither chained nor kept as context: the frames the fault came
+ # through hold the plaintext password.
+ assert raised.value.__cause__ is None
+ assert raised.value.__context__ is None
+ # Failed before any key derivation, kept the engine, cached nothing.
+ assert derived == []
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+ assert sessions.get_current_session() is None
+ assert sessions._thread_credentials == {}
+
+
+# A sidecar held past the claim's bound. The holder never releases on its
+# own: another process's rekey or cold open of a large database, a stopped
+# process, a hung mount. Every claim used to wait for it with a blocking
+# flock, so every request for the user and the shared log-drain thread
+# (which persists every user's log entries) hung for as long as it held it.
+# The bound is shortened here; each wait below is bounded by the test too,
+# so a regression fails instead of hanging the job.
+_BOUND = 0.2
+_JOIN_SECONDS = 10.0
+
+
+@contextmanager
+def _sidecar_flock_held_elsewhere(db_path: Path) -> Iterator[None]:
+ # A second open file description of the sidecar: flock makes it contend
+ # with the manager's own descriptor exactly as another process's would,
+ # and it never takes this process's gate, like a foreign holder.
+ import fcntl
+
+ fd = os.open(
+ db_path.with_name(f"{db_path.name}.rekey.lock"),
+ os.O_RDWR | os.O_CREAT,
+ 0o600,
+ )
+ try:
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ yield
+ finally:
+ os.close(fd)
+
+
+def _run_bounded(
+ call: Callable[[], object], timeout: float = 5.0
+) -> tuple[threading.Thread, dict[str, object]]:
+ outcome: dict[str, object] = {}
+
+ def target() -> None:
+ try:
+ outcome["value"] = call()
+ except BaseException as error: # noqa: BLE001 - handed to the test
+ outcome["error"] = error
+
+ worker = threading.Thread(target=target, daemon=True)
+ worker.start()
+ worker.join(timeout)
+ return worker, outcome
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+ ("accessor", "expected"),
+ [
+ ("is_user_connected", False),
+ ("probe_user_connection", UserConnectionStatus.UNAVAILABLE),
+ ("get_session", RekeyLockContentionError),
+ ("open_user_database", RekeyLockContentionError),
+ ],
+)
+def test_a_sidecar_held_past_the_bound_fails_the_claim_closed(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ accessor: str,
+ expected: object,
+) -> None:
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ arguments = ("alice", "password")[
+ : 2 if accessor == "open_user_database" else 1
+ ]
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(
+ lambda: getattr(manager, accessor)(*arguments)
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, f"{accessor} waited for the holder instead of the bound"
+ if isinstance(expected, type):
+ assert isinstance(outcome.get("error"), expected), outcome
+ else:
+ assert outcome.get("value") is expected, outcome
+ # A retryable fault, not a verdict: nothing derived, nothing evicted,
+ # and the same engine is handed out once the holder is gone.
+ assert derived == []
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+ assert (
+ manager.probe_user_connection("alice") is UserConnectionStatus.CONNECTED
+ )
+
+
+ ("accessor", "expected"),
+ [
+ ("probe_user_connection", UserConnectionStatus.UNAVAILABLE),
+ ("get_session", RekeyLockContentionError),
+ ],
+)
+def test_a_claim_queued_behind_another_claim_shares_the_bound(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ accessor: str,
+ expected: object,
+) -> None:
+ # Each claim takes the user's lifecycle lock before the sidecar and
+ # keeps it while it waits for the sidecar. Bounding only the sidecar
+ # wait would leave the Nth queued request waiting N bounds, so the
+ # lifecycle lock counts against the same budget.
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ _, engine = _cache_current_engine(manager)
+ lifecycle_lock = manager._get_rekey_lock("alice")
+ assert lifecycle_lock.acquire(timeout=5)
+ try:
+ worker, outcome = _run_bounded(
+ lambda: getattr(manager, accessor)("alice")
+ )
+ finished = not worker.is_alive()
+ finally:
+ lifecycle_lock.release()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, f"{accessor} queued on the lifecycle lock past the bound"
+ if isinstance(expected, type):
+ assert isinstance(outcome.get("error"), expected), outcome
+ else:
+ assert outcome.get("value") is expected, outcome
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+ assert (
+ manager.probe_user_connection("alice") is UserConnectionStatus.CONNECTED
+ )
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_a_cold_open_gives_up_on_a_sidecar_held_past_the_bound(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ derived = _watch_key_derivation(monkeypatch, db_path)
+
+ with (
+ patch.object(manager, "_open_user_database_cold") as cold_open,
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(
+ lambda: manager.open_user_database("alice", "password")
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "the cold open waited for the holder instead of the bound"
+ # The typed fault the login route answers with 503 (not counted toward
+ # lockout); the key was derived first, as every cold open does.
+ assert isinstance(outcome.get("error"), RekeyLockContentionError), outcome
+ assert derived == [db_path]
+ cold_open.assert_not_called()
+ assert "alice" not in manager.connections
+
+
+def _spy_sidecar_waits(
+ monkeypatch: pytest.MonkeyPatch, db_path: Path
+) -> list[float]:
+ # The wait each request-path take of this database's sidecar (claim,
+ # cold open) is granted: its ``timeout``. A held sidecar makes each such
+ # wait last its full timeout, so the list shows how long a sequence of
+ # them waited for the holder without timing anything.
+ real_acquire = encrypted_db_module.acquire_rekey_lock
+ granted: list[float] = []
+
+ def acquire(path, **kwargs):
+ if Path(path) == db_path and kwargs.get("timeout") is not None:
+ granted.append(kwargs["timeout"])
+ return real_acquire(path, **kwargs)
+
+ monkeypatch.setattr(encrypted_db_module, "acquire_rekey_lock", acquire)
+ return granted
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_a_request_waits_out_a_held_sidecar_once(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # One authenticated request claims the user's sidecar up to four times:
+ # DatabaseMiddleware's probe and reopen, then require_auth's probe and
+ # its classifying probe. Each used to wait out the bound on its own, so
+ # a held sidecar cost the request four bounds (about 20 s) before its
+ # 503. Inside one request's budget the first wait that runs out spends
+ # it, and each later wait tries once and gives up at once.
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ granted = _spy_sidecar_waits(monkeypatch, db_path)
+
+ def one_request() -> list[object]:
+ outcomes: list[object] = []
+ with encrypted_db_module.sidecar_wait_budget():
+ outcomes.append(manager.is_user_connected("alice"))
+ try:
+ manager.open_user_database("alice", "password")
+ except RekeyLockContentionError as error:
+ outcomes.append(type(error))
+ outcomes.append(manager.is_user_connected("alice"))
+ outcomes.append(manager.probe_user_connection("alice"))
+ return outcomes
+
+ for _request in range(2):
+ granted.clear()
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(one_request)
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "a request waited for the holder instead of the bound"
+ assert outcome.get("value") == [
+ False,
+ RekeyLockContentionError,
+ False,
+ UserConnectionStatus.UNAVAILABLE,
+ ], outcome
+ # One bound for the request, and a fresh one for the next request.
+ assert len(granted) == 4, granted
+ assert 0 < granted[0] <= _BOUND, granted
+ assert granted[1:] == [0.0, 0.0, 0.0], granted
+
+ # Outside every request (the log drain, for one) each wait gets the
+ # bound: one request's budget does not reach them.
+ granted.clear()
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(
+ lambda: [manager.is_user_connected("alice") for _ in range(2)]
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+ assert finished
+ assert outcome.get("value") == [False, False], outcome
+ assert len(granted) == 2 and all(0 < wait for wait in granted), granted
+
+ # A retryable fault, not a verdict: nothing derived, nothing evicted.
+ assert derived == []
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
[email protected]("request_spent", [False, True])
[email protected]("request_raises", [False, True])
+def test_background_context_does_not_keep_a_finished_request_wait_budget(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+ request_spent: bool,
+ request_raises: bool,
+) -> None:
+ # Research and indexing workers copy the request context, then keep
+ # running after its scope exits. Each later background claim must get
+ # its own bounded wait, whether the request exhausted its budget or
+ # left it unused, and whether it returned normally or raised.
+ monkeypatch.setattr(
+ encrypted_db_module, "_SIDECAR_WAIT_TIMEOUT_SECONDS", _BOUND
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ granted = _spy_sidecar_waits(monkeypatch, db_path)
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ try:
+ with encrypted_db_module.sidecar_wait_budget():
+ background_context = copy_context()
+ if request_spent:
+ assert not manager.is_user_connected("alice")
+ if request_raises:
+ raise RuntimeError("request failed after spawning work")
+ except RuntimeError as error:
+ assert request_raises
+ assert str(error) == "request failed after spawning work"
+
+ granted.clear()
+ worker, outcome = _run_bounded(
+ lambda: background_context.run(
+ lambda: [
+ manager.probe_user_connection("alice") for _ in range(2)
+ ]
+ )
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "a background claim waited beyond its own bound"
+ assert outcome.get("value") == [UserConnectionStatus.UNAVAILABLE] * 2, (
+ outcome
+ )
+ assert len(granted) == 2 and all(0 < wait <= _BOUND for wait in granted), (
+ granted
+ )
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_running_background_context_stops_sharing_when_request_finishes(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ monkeypatch.setattr(
+ encrypted_db_module, "_SIDECAR_WAIT_TIMEOUT_SECONDS", _BOUND
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ granted = _spy_sidecar_waits(monkeypatch, db_path)
+ first_claim_finished = threading.Event()
+ request_finished = threading.Event()
+
+ def background() -> list[UserConnectionStatus]:
+ outcomes = [manager.probe_user_connection("alice")]
+ first_claim_finished.set()
+ assert request_finished.wait(timeout=_JOIN_SECONDS)
+ outcomes.extend(
+ manager.probe_user_connection("alice") for _ in range(2)
+ )
+ return outcomes
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ try:
+ with encrypted_db_module.sidecar_wait_budget():
+ background_context = copy_context()
+ worker, outcome = _run_bounded(
+ lambda: background_context.run(background), timeout=0
+ )
+ assert first_claim_finished.wait(timeout=_JOIN_SECONDS)
+ # While the request lives, the worker's exhausted wait is
+ # shared with its caller, just like a run_db_sync worker.
+ assert not manager.is_user_connected("alice")
+ finally:
+ request_finished.set()
+ worker.join(timeout=_JOIN_SECONDS)
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "the background job did not finish its bounded claims"
+ assert outcome.get("value") == [UserConnectionStatus.UNAVAILABLE] * 3, (
+ outcome
+ )
+ assert len(granted) == 4, granted
+ assert 0 < granted[0] <= _BOUND and granted[1] == 0.0, granted
+ assert all(0 < wait <= _BOUND for wait in granted[2:]), granted
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_a_spent_request_budget_returns_once_a_wait_gets_the_sidecar(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Giving up at once is for a sidecar still held past the bound. Once a
+ # later wait of the same request gets it (the holder let go), the
+ # request's next wait may wait out the bound again instead of failing
+ # on the first brief hold of another probe.
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ granted = _spy_sidecar_waits(monkeypatch, db_path)
+ holder_seen = threading.Event()
+ holder_gone = threading.Event()
+
+ def one_request() -> list[UserConnectionStatus]:
+ with encrypted_db_module.sidecar_wait_budget():
+ outcomes = [
+ manager.probe_user_connection("alice"),
+ manager.probe_user_connection("alice"),
+ ]
+ holder_seen.set()
+ assert holder_gone.wait(timeout=_JOIN_SECONDS)
+ outcomes.append(manager.probe_user_connection("alice"))
+ outcomes.append(manager.probe_user_connection("alice"))
+ return outcomes
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(one_request, timeout=0)
+ assert holder_seen.wait(timeout=_JOIN_SECONDS)
+ holder_gone.set()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert not worker.is_alive()
+ assert outcome.get("value") == [
+ UserConnectionStatus.UNAVAILABLE,
+ UserConnectionStatus.UNAVAILABLE,
+ UserConnectionStatus.CONNECTED,
+ UserConnectionStatus.CONNECTED,
+ ], outcome
+ assert len(granted) == 4, granted
+ assert 0 < granted[0] <= _BOUND, granted
+ assert granted[1:3] == [0.0, 0.0], granted
+ assert 0 < granted[3] <= _BOUND, granted
+ assert manager.connections["alice"] is engine
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_a_cold_open_that_runs_out_spends_the_request_budget(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # With no engine cached the probes never take the sidecar, so the
+ # request's only wait is the reopen's cold open. When it runs out, that
+ # is the request's wait: a later open in the same request gives up at
+ # once rather than waiting out a second bound.
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("alice")
+ db_path.touch()
+ derived = _watch_key_derivation(monkeypatch, db_path)
+ granted = _spy_sidecar_waits(monkeypatch, db_path)
+
+ def one_request() -> list[object]:
+ outcomes: list[object] = []
+ with encrypted_db_module.sidecar_wait_budget():
+ for _attempt in range(2):
+ try:
+ manager.open_user_database("alice", "password")
+ except RekeyLockContentionError as error:
+ outcomes.append(type(error))
+ return outcomes
+
+ with (
+ patch.object(manager, "_open_user_database_cold") as cold_open,
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(one_request)
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "the cold opens waited for the holder past one bound"
+ assert outcome.get("value") == [
+ RekeyLockContentionError,
+ RekeyLockContentionError,
+ ], outcome
+ assert len(granted) == 2, granted
+ assert 0 < granted[0] <= _BOUND, granted
+ assert granted[1] == 0.0, granted
+ # Every cold open derives the key before it takes the sidecar.
+ assert derived == [db_path, db_path]
+ cold_open.assert_not_called()
+ assert "alice" not in manager.connections
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_a_sidecar_held_past_the_bound_costs_the_log_drain_one_bound(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # The single log-drain thread claims the user's engine for each entry
+ # it persists. With the claim bounded, an entry for a user whose
+ # sidecar is held costs it at most the bound, not the holder's time,
+ # and is dropped (the file and stderr sinks keep it).
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ monkeypatch.setattr(encrypted_db_module, "db_manager", manager)
+ entry = {
+ "username": "alice",
+ "timestamp": None,
+ "message": "queued while the sidecar is held",
+ "module": "test",
+ "function": "test",
+ "line_no": 1,
+ "level": "INFO",
+ "research_id": None,
+ }
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, _ = _run_bounded(
+ lambda: log_utils._write_log_to_database(entry)
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "the log drain waited for one user's sidecar holder"
+ engine.connect.assert_not_called()
+ assert manager.connections["alice"] is engine
+
+
[email protected](os.name == "nt", reason="POSIX flock required")
+def test_registration_gives_up_on_a_sidecar_held_past_the_bound(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ pytest.importorskip("sqlcipher3")
+ monkeypatch.setattr(
+ encrypted_db_module,
+ "_SIDECAR_WAIT_TIMEOUT_SECONDS",
+ _BOUND,
+ raising=False,
+ )
+ manager = _new_manager(tmp_path)
+ db_path = manager._get_user_db_path("bob")
+
+ with _sidecar_flock_held_elsewhere(db_path):
+ worker, outcome = _run_bounded(
+ lambda: manager.create_user_database("bob", "BobPassword1!"),
+ timeout=30,
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=30)
+
+ assert finished, "registration waited for the holder instead of the bound"
+ # Failed clean: the partial database is removed so the username can
+ # register again, and the stable lock inode stays.
+ assert isinstance(outcome.get("error"), RekeyLockContentionError), outcome
+ assert not db_path.exists()
+ assert not get_salt_file_path(db_path).exists()
+ assert not db_path.with_name(f"{db_path.name}.rekey.epoch").exists()
+ assert db_path.with_name(f"{db_path.name}.rekey.lock").exists()
+ assert "bob" not in manager.connections
+
+
+def _release_fifo_reader(fifo: Path) -> None:
+ # A reader blocked opening the FIFO is released by a writer that opens
+ # and closes it; with no reader waiting the open fails (ENXIO) and
+ # there is nothing to release.
+ try:
+ fd = os.open(fifo, os.O_WRONLY | os.O_NONBLOCK)
+ except OSError:
+ return
+ os.close(fd)
+
+
[email protected](not hasattr(os, "mkfifo"), reason="POSIX FIFOs required")
[email protected]("suffix", [".rekey.epoch", ".rekey.pending"])
+def test_a_planted_fifo_cannot_hang_the_probe_or_the_shutdown(
+ tmp_path: Path, suffix: str
+) -> None:
+ # Every probe reads the epoch files while it holds the user's sidecar
+ # flock. A FIFO planted at one of their names made that read wait for a
+ # writer, so the probe, every later probe for the user and shutdown's
+ # close_all_databases hung behind it.
+ manager = _new_manager(tmp_path)
+ db_path, engine = _cache_current_engine(manager)
+ planted = db_path.with_name(f"{db_path.name}{suffix}")
+ planted.unlink(missing_ok=True)
+ os.mkfifo(planted, 0o600)
+
+ worker, outcome = _run_bounded(
+ lambda: manager.probe_user_connection("alice")
+ )
+ finished = not worker.is_alive()
+ _release_fifo_reader(planted)
+ worker.join(timeout=_JOIN_SECONDS)
+
+ assert finished, "the probe waited on a planted FIFO under the flock"
+ # Nothing was read: a retryable fault, and the engine is kept
+ assert outcome.get("value") is UserConnectionStatus.UNAVAILABLE, outcome
+ assert manager.connections["alice"] is engine
+ engine.dispose.assert_not_called()
+ assert stat.S_ISFIFO(planted.lstat().st_mode)
+
+ # The sidecar was released: the next probe answers too, and shutdown
+ # disposes the engine instead of waiting on the FIFO.
+ worker, outcome = _run_bounded(
+ lambda: manager.probe_user_connection("alice")
+ )
+ assert not worker.is_alive()
+ assert outcome.get("value") is UserConnectionStatus.UNAVAILABLE, outcome
+ worker, outcome = _run_bounded(manager.close_all_databases)
+ assert not worker.is_alive()
+ assert "error" not in outcome, outcome
+ assert "alice" not in manager.connections
+ engine.dispose.assert_called_once_with()
diff --git a/tests/database/test_sqlcipher_connection_factory.py b/tests/database/test_sqlcipher_connection_factory.py
index 32cabbd896..1879e5ba61 100644
--- a/tests/database/test_sqlcipher_connection_factory.py
+++ b/tests/database/test_sqlcipher_connection_factory.py
@@ -46,9 +46,7 @@ def mock_sqlcipher():
f"{_EDB}.get_sqlcipher_module", return_value=mock_module
) as mock_get,
patch(f"{_EDB}.set_sqlcipher_key") as mock_set_key,
- patch(
- f"{_EDB}.verify_sqlcipher_connection", return_value=True
- ) as mock_verify,
+ patch(f"{_EDB}.check_sqlcipher_key", return_value=True) as mock_verify,
patch(f"{_EDB}.apply_sqlcipher_pragmas") as mock_pragmas,
patch(f"{_EDB}.apply_performance_pragmas") as mock_perf,
):
@@ -166,6 +164,28 @@ class TestMakeSqlcipherConnection:
assert call_order == ["key", "pragmas", "verify", "performance"]
+ def test_busy_timeout_is_set_before_verification(
+ self, manager, mock_sqlcipher
+ ):
+ """A lock held elsewhere must be waited out, not read as a wrong key."""
+ from local_deep_research.database.sqlcipher_utils import (
+ get_sqlcipher_busy_timeout_ms,
+ )
+
+ busy_set_at_verify = []
+ cursor = mock_sqlcipher["cursor"]
+ pragma = f"PRAGMA busy_timeout = {get_sqlcipher_busy_timeout_ms()}"
+ mock_sqlcipher["verify"].side_effect = lambda *a: (
+ busy_set_at_verify.append(
+ any(c.args == (pragma,) for c in cursor.execute.call_args_list)
+ )
+ or True
+ )
+
+ manager._make_sqlcipher_connection(Path("/tmp/test.db"), "secret")
+
+ assert busy_set_at_verify == [True]
+
def test_raises_on_failed_verification(self, manager, mock_sqlcipher):
"""Must raise ValueError when verification fails."""
mock_sqlcipher["verify"].return_value = False
@@ -326,7 +346,7 @@ class TestMakeSqlcipherConnection:
def test_closes_cursor_and_conn_on_verify_exception(
self, manager, mock_sqlcipher
):
- """Cleanup must happen when verify_sqlcipher_connection raises (not just returns False)."""
+ """Cleanup must happen when check_sqlcipher_key raises (not just returns False)."""
mock_sqlcipher["verify"].side_effect = RuntimeError("verify crashed")
with pytest.raises(RuntimeError, match="verify crashed"):
diff --git a/tests/database/test_sqlcipher_open_close_serialization.py b/tests/database/test_sqlcipher_open_close_serialization.py
new file mode 100644
index 0000000000..c54f995260
--- /dev/null
+++ b/tests/database/test_sqlcipher_open_close_serialization.py
@@ -0,0 +1,115 @@
+"""In-process serialization of native SQLCipher open and close.
+
+The SQLite 3.51.1 bundled in ``sqlcipher3-binary`` 0.6.0 deadlocks when one
+thread opens a WAL-mode database file while another closes a connection to
+the same file. The driver keeps the GIL while it closes, so the deadlock
+froze a whole pytest worker in CI
+(``test_metrics_entry_wrong_password_storm_racing_close_and_reopen``).
+The shared ``sqlcipher_compat.connect_sqlcipher`` lifecycle guard serializes
+native open and close on affected builds, including connection finalization.
+
+The stress test runs in a subprocess with a hard timeout: an in-process
+deadlock that holds the GIL cannot be bounded by the test itself.
+"""
+
+import os
+import subprocess
+import sys
+import textwrap
+from pathlib import Path
+
+import pytest
+
+from local_deep_research.database import sqlcipher_utils
+
+
+SRC_DIR = Path(sqlcipher_utils.__file__).resolve().parents[2]
+STRESS_TIMEOUT_S = 45
+
+_STRESS_SCRIPT = textwrap.dedent(
+ """
+ import sys, tempfile, threading
+ from pathlib import Path
+ from local_deep_research.database.sqlcipher_utils import (
+ create_sqlcipher_connection,
+ )
+
+ good, bad = "ab" * 32, "cd" * 32
+ db = Path(tempfile.mkdtemp()) / "stress.db"
+ conn = create_sqlcipher_connection(db, hex_key=good, creation_mode=True)
+ conn.execute("CREATE TABLE t (a INTEGER)")
+ conn.execute("INSERT INTO t VALUES (1)")
+ conn.commit()
+ conn.close()
+
+ iterations = int(sys.argv[1])
+ errors = []
+
+ def right_key():
+ try:
+ for _ in range(iterations):
+ c = create_sqlcipher_connection(
+ db, hex_key=good,
+ connect_kwargs={"check_same_thread": False},
+ )
+ c.execute("SELECT a FROM t").fetchall()
+ c.close()
+ except Exception as exc:
+ errors.append(repr(exc))
+
+ def wrong_key():
+ try:
+ for _ in range(iterations):
+ try:
+ create_sqlcipher_connection(
+ db, hex_key=bad,
+ connect_kwargs={"check_same_thread": False},
+ )
+ except Exception: # wrong key: DatabaseError or ValueError
+ pass
+ else:
+ errors.append("wrong key opened the database")
+ except Exception as exc:
+ errors.append(repr(exc))
+
+ threads = [
+ threading.Thread(target=right_key),
+ threading.Thread(target=right_key),
+ threading.Thread(target=wrong_key),
+ ]
+ for t in threads:
+ t.start()
+ for t in threads:
+ t.join()
+ print("ERRORS" if errors else "OK", errors, flush=True)
+ """
+)
+
+
+def test_concurrent_open_and_close_of_one_file_does_not_deadlock(tmp_path):
+ """Three threads open and close the same WAL database (two with the
+ right key, one with a wrong key). Without the lifecycle guard this
+ freezes the process within the first few hundred cycles."""
+ pytest.importorskip("sqlcipher3", reason="requires SQLCipher")
+ script = tmp_path / "stress.py"
+ script.write_text(_STRESS_SCRIPT)
+ env = dict(os.environ)
+ env["PYTHONPATH"] = os.pathsep.join(
+ [str(SRC_DIR), env.get("PYTHONPATH", "")]
+ ).rstrip(os.pathsep)
+ try:
+ result = subprocess.run(
+ [sys.executable, str(script), "400"],
+ capture_output=True,
+ text=True,
+ timeout=STRESS_TIMEOUT_S,
+ env=env,
+ cwd=tmp_path,
+ )
+ except subprocess.TimeoutExpired:
+ pytest.fail(
+ "concurrent open/close of one SQLCipher file deadlocked "
+ f"(no exit within {STRESS_TIMEOUT_S}s)"
+ )
+ assert result.returncode == 0, result.stderr[-2000:]
+ assert result.stdout.strip().startswith("OK"), result.stdout[-2000:]
diff --git a/tests/database/test_sqlcipher_utils.py b/tests/database/test_sqlcipher_utils.py
index 6ad363756d..21a241a5cb 100644
--- a/tests/database/test_sqlcipher_utils.py
+++ b/tests/database/test_sqlcipher_utils.py
@@ -569,6 +569,50 @@ class TestApplyPerformancePragmas:
assert "DELETE" in journal_call
+class TestBusyTimeoutPragma:
+ """``PRAGMA busy_timeout`` is built from a fixed template and an int.
+
+ PRAGMA takes no bind parameters. Both executions (before key
+ verification in ``encrypted_db`` and in ``apply_performance_pragmas``)
+ go through ``sqlcipher_busy_timeout_pragma`` so the statement is never
+ an f-string passed straight to ``execute`` and can only carry an
+ integer literal.
+ """
+
+ _SETTING = "local_deep_research.database.sqlcipher_utils.get_env_setting"
+
+ def test_statement_carries_the_effective_integer(self):
+ from local_deep_research.database.sqlcipher_utils import (
+ sqlcipher_busy_timeout_pragma,
+ )
+
+ with patch(self._SETTING, return_value="1234"):
+ assert sqlcipher_busy_timeout_pragma() == (
+ "PRAGMA busy_timeout = 1234"
+ )
+
+ def test_non_integer_value_never_reaches_the_statement(self):
+ from local_deep_research.database.sqlcipher_utils import (
+ sqlcipher_busy_timeout_pragma,
+ )
+
+ with patch(self._SETTING, return_value="5000; PRAGMA key = 'x'"):
+ with pytest.raises(ValueError):
+ sqlcipher_busy_timeout_pragma()
+
+ def test_performance_pragmas_execute_the_built_statement(self):
+ from local_deep_research.database.sqlcipher_utils import (
+ apply_performance_pragmas,
+ )
+
+ cursor = Mock()
+ with patch(self._SETTING, side_effect=lambda key, default: default):
+ apply_performance_pragmas(cursor)
+
+ executed = [c.args[0] for c in cursor.execute.call_args_list]
+ assert "PRAGMA busy_timeout = 30000" in executed
+
+
class TestVerifySqlcipherConnection:
"""Tests for verify_sqlcipher_connection function."""
diff --git a/tests/js/security/url-source-flows-xss.test.js b/tests/js/security/url-source-flows-xss.test.js
index 54caa1f9f5..9a1424298e 100644
--- a/tests/js/security/url-source-flows-xss.test.js
+++ b/tests/js/security/url-source-flows-xss.test.js
@@ -13,10 +13,10 @@
* - components/library_search_ui.js:67/:90/:107/:124 preserves existing
* query parameters with URLSearchParams, then :74/:97/:114/:131 assigns
* the serialized result back to location.search. No DOM/fetch/storage sink.
- * - components/chat.js:281-288 reads `q`, removes the query string with
- * history.replaceState, assigns the decoded value to the chat input, then
- * handleSend renders it through textContent (:1114) and sends it as JSON
- * request data (:684-687). It never enters a fetch URL.
+ * - components/chat.js reads `q`, removes the query string with
+ * history.replaceState and pre-fills an empty chat without taking focus.
+ * Explicit Send renders it through textContent and sends it as JSON
+ * request data. It never enters a fetch URL or submits on page load.
* - components/logpanel.js:481 only checks for the literal `debug=logs`;
* matching controls a delayed toggle click and does not render the URL.
*
@@ -458,7 +458,7 @@ describe('rerunConfig sessionStorage -> form -> request', { concurrent: false },
describe('components/chat.js location.search q flow', { concurrent: false }, () => {
test.each(SPECIAL_VALUES)(
- 'renders and submits q as inert data, never as URL structure: %s',
+ 'drafts q and explicitly submits it as inert data, never as URL structure: %s',
async (payload) => {
vi.resetModules();
installChatPage();
@@ -483,13 +483,7 @@ describe('components/chat.js location.search q flow', { concurrent: false }, ()
if (url === '/api/chat/sessions/source-session/messages') {
return jsonResponse({
success: true,
- messages: [{
- id: 'benign-existing-message',
- role: 'user',
- message_type: 'chat',
- content: 'Benign existing chat message',
- created_at: '2026-08-20T10:00:00Z',
- }],
+ messages: [],
has_more: false,
in_progress_research_id: null,
});
@@ -507,6 +501,19 @@ describe('components/chat.js location.search q flow', { concurrent: false }, ()
await import('@js/components/chat.js');
await vi.waitFor(() => {
expect(document.getElementById('chat-input').dataset.initComplete).toBe('true');
+ });
+
+ const input = document.getElementById('chat-input');
+ expect(input.value).toBe(payload);
+ expect(document.activeElement).not.toBe(input);
+ expect(document.getElementById('send-btn').disabled).toBe(false);
+ expect(chatFetch.mock.calls.some(([, options]) => options?.method === 'POST')).toBe(false);
+ expect(document.querySelector('.ldr-chat-message-user')).toBeNull();
+ expect(window.location.search).toBe('');
+ expectInert(document.body);
+
+ document.getElementById('send-btn').click();
+ await vi.waitFor(() => {
expect(chatFetch.mock.calls.some(
([url, options]) => url === '/api/chat/sessions/source-session/messages'
&& options?.method === 'POST'
@@ -515,17 +522,18 @@ describe('components/chat.js location.search q flow', { concurrent: false }, ()
const userMessages = document.querySelectorAll('.ldr-chat-message-user .ldr-chat-message-text');
const renderedPayload = userMessages[userMessages.length - 1];
- const sendCall = chatFetch.mock.calls.find(
+ const sendCalls = chatFetch.mock.calls.filter(
([url, options]) => url === '/api/chat/sessions/source-session/messages'
&& options?.method === 'POST'
);
- expect(userMessages.length).toBeGreaterThanOrEqual(2);
+ expect(sendCalls).toHaveLength(1);
+ const [sendCall] = sendCalls;
+ expect(userMessages).toHaveLength(1);
expect(renderedPayload.textContent).toBe(payload);
- expect(document.getElementById('chat-messages').textContent)
- .toContain('Benign existing chat message');
expect(document.getElementById('chat-title').textContent).toBe('Benign chat session');
expect(sendCall[0]).toBe('/api/chat/sessions/source-session/messages');
expect(sendCall[0]).not.toContain(payload);
+ expect(sendCall[1].headers['X-CSRFToken']).toBe('test-csrf');
expect(sendCall[1].body).toBe(JSON.stringify({
content: payload,
trigger_research: true,
diff --git a/tests/security/test_auth_attack_scenarios.py b/tests/security/test_auth_attack_scenarios.py
index d90d614768..579b873767 100644
--- a/tests/security/test_auth_attack_scenarios.py
+++ b/tests/security/test_auth_attack_scenarios.py
@@ -454,9 +454,9 @@ class TestPasswordChangeRevokesTheOtherDevice:
must stay locked out after the account is reopened.
The second half is the part with no equivalent anywhere on this
- branch. ``change_password`` ends with
- ``destroy_all_user_sessions`` + ``clear_all_for_user`` +
- ``close_user_database``, and a test that stops there cannot tell
+ branch. The route runs ``destroy_all_user_sessions`` +
+ ``clear_all_for_user`` after a rekey that has already detached the
+ user's cached engine, and a test that stops there cannot tell
which of those three produced the 401. Two of them are undone by the
very next login — including the legitimate one the user performs
thirty seconds later with their new password. Only the destroyed
diff --git a/tests/security/test_auth_db_session_dependency_error_paths.py b/tests/security/test_auth_db_session_dependency_error_paths.py
index 02fae219b1..a4b1c5fcac 100644
--- a/tests/security/test_auth_db_session_dependency_error_paths.py
+++ b/tests/security/test_auth_db_session_dependency_error_paths.py
@@ -31,6 +31,9 @@ throw point under explicit fixture control:
propagate untouched — if the ``except`` ever widens to ``Exception``
(a favourite "robustness" refactor), bug signals would disappear into
401s and login loops, so that widening must fail this pin.
+4. Its ``DatabaseSessionUnavailableError`` subclass (a rekey operational
+ fault, not a credential problem) maps to a retryable 503 with
+ ``Retry-After`` and leaves the session alone.
Both entry routes into the ``except`` are covered: the error raised while
*opening* the session (``__enter__``), and the error thrown *back into the
@@ -163,3 +166,66 @@ class TestNonCredentialFailures:
with pytest.raises(RuntimeError, match="boom"):
next(get_db_session_dep(request, "alice"))
+
+
+def _session_unavailable():
+ # Imported here so this module still collects against a tree that
+ # predates the subclass (the fail-on-old run).
+ from local_deep_research.database.session_context import (
+ DatabaseSessionUnavailableError,
+ )
+
+ return DatabaseSessionUnavailableError("RekeySidecarIOError")
+
+
+class TestOperationalFaultKeepsTheSession:
+ """A rekey sidecar fault is not a reason to log the user out.
+
+ ``get_user_db_session`` raises ``DatabaseSessionUnavailableError`` (a
+ ``DatabaseSessionError`` subclass) when the database could not be
+ opened or claimed because of a rekey operational fault, e.g. an
+ fd-exhaustion spike that began after ``require_auth``'s probe passed.
+ Caught by the base-class clause, it cleared the cookie and answered
+ 401. It must get ``require_auth``'s retryable 503 and leave the session
+ as it was, from either entry route into the ``except``.
+ """
+
+ @staticmethod
+ def _assert_retryable_503(excinfo, session_dict, expected):
+ assert excinfo.value.status_code == 503
+ assert excinfo.value.headers == {"Retry-After": "5"}
+ assert session_dict == expected, (
+ "a transient sidecar fault cleared a recoverable session"
+ )
+
+ def test_open_failure_answers_503_and_keeps_the_session(self, monkeypatch):
+ _patch_ctx(
+ monkeypatch,
+ _FakeSessionContext(enter_exc=_session_unavailable()),
+ )
+ session_dict = {"username": "alice", "session_id": "sid-3"}
+ request = _request_with_session(session_dict)
+
+ with pytest.raises(HTTPException) as excinfo:
+ next(get_db_session_dep(request, "alice"))
+
+ self._assert_retryable_503(
+ excinfo, session_dict, {"username": "alice", "session_id": "sid-3"}
+ )
+
+ def test_handler_time_failure_answers_503_and_keeps_the_session(
+ self, monkeypatch
+ ):
+ ctx = _patch_ctx(monkeypatch, _FakeSessionContext(session=object()))
+ session_dict = {"username": "alice", "session_id": "sid-4"}
+ request = _request_with_session(session_dict)
+
+ gen = get_db_session_dep(request, "alice")
+ next(gen)
+ with pytest.raises(HTTPException) as excinfo:
+ gen.throw(_session_unavailable())
+
+ self._assert_retryable_503(
+ excinfo, session_dict, {"username": "alice", "session_id": "sid-4"}
+ )
+ assert ctx.exited
diff --git a/tests/security/test_change_password_evicts_stale_engine.py b/tests/security/test_change_password_evicts_stale_engine.py
index c8d7551fe4..5c40863678 100644
--- a/tests/security/test_change_password_evicts_stale_engine.py
+++ b/tests/security/test_change_password_evicts_stale_engine.py
@@ -1,40 +1,50 @@
"""Regression coverage for the change_password stale-cached-engine window.
-``change_password`` cold-opens the user's database with the OLD password
-(``open_user_database`` caches that engine together with a verifier for the old
-password), rekeys the SQLCipher file to the NEW password, and then closes the
-connection. Before the fix the close happened ONLY in the ``finally`` block, so
-between the rekey and the finally the cache held an engine whose creator closure
-still derives the now-invalid OLD key while its verifier still matched the OLD
-password. In that window a concurrent ``open_user_database(username, old)`` would
-pass the verifier check and be handed the stale-key engine. ``change_password``
-now runs under a per-user lock, but that lock only excludes a second
-``change_password``; a concurrent ``open_user_database`` is not held back by it,
-so the eviction is still what closes this window.
-
-The fix evicts the connection (and its verifier) immediately AFTER the rekey,
-INSIDE the ``try``, via ``close_user_database_if_stale`` -- a conditional
-variant that only drops the cached engine when its verifier still matches the
-OLD password (so a concurrent ``open_user_database(new_password)`` racing the
-rekey, which legitimately succeeds against the now-new_password-keyed file and
-caches a fresh engine with the NEW verifier, is left alone). The finally-close
-calls the same conditional helper as a backstop for the exception path.
-
-This test pins that the eviction is performed by the in-try close, NOT merely by
-the finally. It neutralises the finally-close and proves the stale engine/verifier
-are already gone, then proves the old password can no longer open the database and
-the new password can. Because the finally is neutralised, a build that relied on
-the finally alone would leave the stale engine cached and fail here.
-
-Surfaced by a multi-agent review of PR #5596.
+Originally ``change_password`` cold-opened the user's database with the OLD
+password (caching that engine together with a verifier for the old password),
+rekeyed the SQLCipher file to the NEW password, and only then closed the
+connection, so between the rekey and the close the cache held an engine whose
+creator closure still derived the now-invalid OLD key while its verifier still
+matched the OLD password; a concurrent ``open_user_database(username, old)``
+could be handed that stale-key engine.
+
+``change_password`` now never caches a connection of its own. Holding the
+user's on-disk rekey sidecar lock for the whole staged sequence (the same lock
+every cold open and every cache claim takes), it detaches and disposes any
+cached engine BEFORE the rekey, rekeys through one uncached raw SQLCipher
+connection, and publishes a new durable rekey epoch; an engine cached under a
+superseded epoch is detached by the next claim instead of being handed out.
+``test_change_password_evicts_cached_engine_before_raw_rekey`` pins that the
+engine, its verifier and its epoch are all gone at the moment of the rekey and
+that afterwards only the new password opens the database.
+
+Surfaced by a multi-agent review of PR #5596. Extended to cover durable rekey
+epochs, interrupted-rekey recovery, and fenced cache teardown.
"""
+import builtins
+import io
+import os
+import threading
+import time
import uuid
+from contextlib import contextmanager
+from pathlib import Path
+from unittest.mock import MagicMock, patch
import pytest
from local_deep_research.database import encrypted_db
+from local_deep_research.database import rekey_lock as rekey_lock_module
from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ RekeyIndeterminateError,
+ RekeyStage,
+ RekeyUnavailableError,
+ acquire_rekey_lock,
+ read_or_initialize_rekey_epoch,
+ read_rekey_epoch_state,
+)
@pytest.fixture
@@ -49,29 +59,10 @@ def manager(tmp_path, monkeypatch):
mgr.close_all_databases()
-class _LoggerProxy:
- """Delegates to the real loguru logger but reports every ``info`` message.
-
- Used to observe WHEN change_password emits its "Password changed" success
- line, which is logged after the in-try eviction but before the finally-close
- -- letting the close spy tell the two closes apart.
- """
-
- def __init__(self, real, on_info):
- self._real = real
- self._on_info = on_info
-
- def info(self, message, *args, **kwargs):
- self._on_info(message)
- return self._real.info(message, *args, **kwargs)
-
- def __getattr__(self, name): # warning/debug/opt/... pass straight through
- return getattr(self._real, name)
-
-
-def test_change_password_evicts_stale_engine_in_try_not_only_finally(
+def test_change_password_evicts_cached_engine_before_raw_rekey(
manager, monkeypatch
):
+ # Given a cached old-password engine for a real encrypted database
if not manager.has_encryption:
pytest.skip("SQLCipher not available; rekey path is a no-op")
@@ -82,107 +73,648 @@ def test_change_password_evicts_stale_engine_in_try_not_only_finally(
engine = manager.create_user_database(username, old)
assert engine is not None
assert manager.is_user_connected(username)
+ assert username in manager._password_verifiers
+ assert username in manager._connection_epochs
- state = {
- "rekeyed": False,
- "success_logged": False,
- "intry_close": False,
- "intry_engine_present": None,
- "intry_verifier_present": None,
- "gone_after_intry": None,
- "finally_neutralized": False,
- }
-
- # Flag the exact moment the rekey completes: everything after this and
- # before the "Password changed" log is the in-try region the fix targets.
real_rekey = encrypted_db.set_sqlcipher_rekey
+ cached_state_at_rekey = None
def spy_rekey(conn, new_password, **kwargs):
+ nonlocal cached_state_at_rekey
+ cached_state_at_rekey = (
+ username in manager.connections,
+ username in manager._password_verifiers,
+ username in manager._connection_epochs,
+ )
real_rekey(conn, new_password, **kwargs)
- state["rekeyed"] = True
monkeypatch.setattr(encrypted_db, "set_sqlcipher_rekey", spy_rekey)
- # The success line is logged AFTER the in-try close but BEFORE the finally.
- def on_info(message):
- if "Password changed" in str(message):
- state["success_logged"] = True
+ # When the password changes through one uncached raw SQLCipher connection
+ assert manager.change_password(username, old, new) is True
- monkeypatch.setattr(
- encrypted_db, "logger", _LoggerProxy(encrypted_db.logger, on_info)
+ # Then old-key cache state was gone before rekey and credentials rotated
+ assert cached_state_at_rekey == (False, False, False)
+ assert username not in manager.connections
+ assert username not in manager._password_verifiers
+ assert username not in manager._connection_epochs
+
+ assert manager.open_user_database(username, old) is None, (
+ "the old password must not open the re-keyed database"
+ )
+ reopened = manager.open_user_database(username, new)
+ assert reopened is not None
+ assert manager._password_matches_cached(username, new)
+ assert not manager._password_matches_cached(username, old)
+
+
+def test_pending_recovery_accepts_real_surviving_old_password(
+ manager, monkeypatch
+):
+ # Given SQLCipher leaves the old key intact after rekey fails post-pending
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"recoverold_{uuid.uuid4().hex[:8]}"
+ old = "OldRecoveryHorse1!" # noqa: S105
+ new = "NewRecoveryHorse2!" # noqa: S105
+ engine = manager.create_user_database(username, old)
+ assert engine is not None
+ db_path = manager._get_user_db_path(username)
+ original_epoch = read_rekey_epoch_state(db_path).committed_epoch
+ real_rekey = encrypted_db.set_sqlcipher_rekey
+
+ def fail_rekey(_connection, _new_password, *, db_path):
+ raise RuntimeError("simulated pre-rekey failure")
+
+ monkeypatch.setattr(encrypted_db, "set_sqlcipher_rekey", fail_rekey)
+
+ # When password change fails after its pending generation is durable
+ with pytest.raises(RekeyIndeterminateError) as raised:
+ manager.change_password(username, old, new)
+ assert raised.value.stage is RekeyStage.PHYSICAL_REKEY
+ interrupted = read_rekey_epoch_state(db_path)
+ assert interrupted.committed_epoch == original_epoch
+ assert interrupted.pending_epoch is not None
+ monkeypatch.setattr(encrypted_db, "set_sqlcipher_rekey", real_rekey)
+
+ # Then a fresh real old-key open verifies and finalizes the pending generation
+ reopened = manager.open_user_database(username, old)
+ assert reopened is not None
+ recovered = read_rekey_epoch_state(db_path)
+ assert recovered.committed_epoch == interrupted.pending_epoch
+ assert recovered.pending_epoch is None
+ assert manager.open_user_database(username, new) is None
+
+
+def test_pending_recovery_accepts_real_surviving_new_password(manager):
+ # Given SQLCipher applies the new key but committed metadata publication fails
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"recovernew_{uuid.uuid4().hex[:8]}"
+ old = "OldRecoveryHorse3!" # noqa: S105
+ new = "NewRecoveryHorse4!" # noqa: S105
+ engine = manager.create_user_database(username, old)
+ assert engine is not None
+ db_path = manager._get_user_db_path(username)
+ original_epoch = read_rekey_epoch_state(db_path).committed_epoch
+
+ with patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=OSError("simulated epoch publication failure"),
+ ):
+ # When durable candidate verification succeeds but E2 cannot publish,
+ # the change is complete: the new key is proven, so it is reported
+ # as such (the route then revokes every old-password credential)
+ assert manager.change_password(username, old, new) is True
+
+ interrupted = read_rekey_epoch_state(db_path)
+ assert interrupted.committed_epoch == original_epoch
+ assert interrupted.pending_epoch is not None
+
+ # Then the wrong old key is an ordinary failed login (None, counted by
+ # the login route's lockout) and leaves recovery state untouched
+ assert manager.open_user_database(username, old) is None
+ assert read_rekey_epoch_state(db_path) == interrupted
+
+ # And the surviving real new key verifies before recovery publishes E2
+ reopened = manager.open_user_database(username, new)
+ assert reopened is not None
+ recovered = read_rekey_epoch_state(db_path)
+ assert recovered.committed_epoch == interrupted.pending_epoch
+ assert recovered.pending_epoch is None
+ assert manager.open_user_database(username, old) is None
+
+
+def test_unpublished_epoch_after_proven_key_fences_stale_engine_and_purges_backups(
+ manager, monkeypatch
+):
+ # Given a real database with a cached old-password engine and an
+ # old-key backup
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ from local_deep_research.database.backup.backup_service import (
+ BackupService,
)
- # Spy the conditional close used by change_password's in-try and
- # finally paths. (The unconditional close_user_database at the top of
- # change_password is unrelated to the stale-engine window -- it just
- # clears any pre-existing engine -- so we spy on the conditional variant
- # the post-rekey path actually uses.) The spy captures the state at the
- # in-try call and NEUTRALISES the finally call, forcing the in-try call
- # to be the only thing that could have removed the stale engine/verifier.
- real_close_stale = manager.close_user_database_if_stale
-
- def spy_close_stale(user, expected_old_password):
- if state["rekeyed"] and not state["success_logged"]:
- # The in-try eviction (post-rekey, pre-success-log). The stale
- # engine + old verifier must still be cached right now -- that is
- # the very window being closed -- and must be gone once we evict.
- state["intry_close"] = True
- state["intry_engine_present"] = user in manager.connections
- state["intry_verifier_present"] = (
- user in manager._password_verifiers
- )
- real_close_stale(user, expected_old_password)
- state["gone_after_intry"] = (
- user not in manager.connections
- and user not in manager._password_verifiers
- )
- return
- if state["rekeyed"] and state["success_logged"]:
- # The finally backstop. Neutralise it: if the in-try close did its
- # job the stale state is already gone; if the code relied on the
- # finally alone, skipping it here leaves the stale engine cached and
- # the assertions below fail.
- state["finally_neutralized"] = True
- return
- # Should not happen: change_password only calls this method
- # post-rekey. Run normally just in case.
- real_close_stale(user, expected_old_password)
+ from local_deep_research.config.paths import get_encrypted_database_path
+
+ # The backup service resolves the database through the data directory,
+ # so the manager must write where that resolution looks.
+ manager.data_dir = get_encrypted_database_path()
+ username = f"unpub_{uuid.uuid4().hex[:8]}"
+ old = "OldUnpublishedHorse1!" # noqa: S105
+ new = "NewUnpublishedHorse2!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ db_path = manager._get_user_db_path(username)
+ old_backup = BackupService(username=username, password=old).create_backup(
+ force=True, prune=False
+ )
+ assert old_backup.success, old_backup.error
+ stale = manager.open_user_database(username, old)
+ assert stale is not None
+ original_epoch = read_rekey_epoch_state(db_path).committed_epoch
+
+ # When the epoch publication fails after the new key was proven
+ with patch.object(
+ encrypted_db,
+ "commit_rekey_epoch",
+ side_effect=OSError(28, "No space left on device"),
+ ):
+ assert manager.change_password(username, old, new) is True
+
+ # Then the raw file opens only with the new password
+ with pytest.raises(ValueError):
+ DatabaseManager._make_sqlcipher_connection(db_path, old)
+ DatabaseManager._make_sqlcipher_connection(db_path, new).close()
+
+ # And the pending generation fences any engine cached under the old
+ # epoch: re-publish the stale engine as if a racing claim had kept it
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(stale, old, original_epoch),
+ )
+ assert manager.get_session(username) is None
+ assert manager.open_user_database(username, old) is None
+ pending = read_rekey_epoch_state(db_path).pending_epoch
+ assert pending is not None
+
+ # And the next open with the new password commits the pending epoch and
+ # purges the backup the superseded key still opens
+ assert old_backup.backup_path.exists()
+ assert manager.open_user_database(username, new) is not None
+ state = read_rekey_epoch_state(db_path)
+ assert state.committed_epoch == pending
+ assert state.pending_epoch is None
+ assert not old_backup.backup_path.exists()
+
+
+def test_epoch_replace_directory_fsync_failure_reconciles_real_rekey(
+ manager, monkeypatch
+):
+ # Given a real SQLCipher database whose E2 replace becomes visible before fsync fails
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"replacefsync_{uuid.uuid4().hex[:8]}"
+ old = "OldReplaceHorse5!" # noqa: S105
+ new = "NewReplaceHorse6!" # noqa: S105
+ engine = manager.create_user_database(username, old)
+ assert engine is not None
+ db_path = manager._get_user_db_path(username)
+ epoch_path = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ original_epoch = read_rekey_epoch_state(db_path).committed_epoch
+ real_sync_parent = rekey_lock_module._sync_parent_directory
+
+ def fail_after_epoch_replace(path):
+ if path == epoch_path:
+ raise OSError("simulated directory fsync failure")
+ real_sync_parent(path)
monkeypatch.setattr(
- manager, "close_user_database_if_stale", spy_close_stale
+ rekey_lock_module, "_sync_parent_directory", fail_after_epoch_replace
)
+ cleanup = MagicMock(side_effect=OSError("simulated cleanup failure"))
+ monkeypatch.setattr(encrypted_db, "clear_pending_rekey_epoch", cleanup)
+ # When committed publication raises after os.replace exposes E2
assert manager.change_password(username, old, new) is True
- # 1. The eviction happened INSIDE the try, right after the rekey...
- assert state["intry_close"], (
- "change_password must evict the cached engine in-try after the rekey, "
- "not only in the finally"
+ # Then success and E2+pending-E2 reflect the real surviving new key
+ interrupted = read_rekey_epoch_state(db_path)
+ assert interrupted.committed_epoch != original_epoch
+ assert interrupted.pending_epoch == interrupted.committed_epoch
+ cleanup.assert_called_once_with(db_path, interrupted.pending_epoch)
+ with pytest.raises(ValueError):
+ manager._make_sqlcipher_connection(db_path, old)
+ new_connection = manager._make_sqlcipher_connection(db_path, new)
+ new_connection.close()
+ assert read_rekey_epoch_state(db_path) == interrupted
+
+
+def test_change_password_does_not_checkpoint_stale_cache_after_epoch_advance(
+ tmp_path,
+):
+ # Given a cached engine whose epoch advances as password change takes the sidecar
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = tmp_path
+ username = "alice"
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ original_epoch = read_or_initialize_rekey_epoch(db_path)
+ stale_engine = MagicMock()
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(
+ stale_engine, "old-password", original_epoch
+ ),
)
- # 2. ...and at that instant the stale-key engine + old verifier were still
- # cached, i.e. the window is real and this close is what closes it.
- assert state["intry_engine_present"] is True
- assert state["intry_verifier_present"] is True
- assert state["gone_after_intry"] is True
- # 3. The finally ran but we neutralised it, so anything still gone below is
- # attributable to the in-try close, not the finally backstop.
- assert state["finally_neutralized"] is True
-
- # 4. Stale engine and old verifier are gone even with the finally disabled.
- assert username not in manager.connections
- assert username not in manager._password_verifiers
+ old_connection = MagicMock()
+ new_connection = MagicMock()
+
+ @contextmanager
+ def advance_epoch_before_yield(
+ path, *, blocking=False, local_wait=None, contention_retry=0.0
+ ):
+ with acquire_rekey_lock(
+ path,
+ blocking=blocking,
+ local_wait=local_wait,
+ contention_retry=contention_retry,
+ ):
+ rekey_lock_module.advance_rekey_epoch(path)
+ yield
+
+ with (
+ patch.object(
+ encrypted_db,
+ "acquire_rekey_lock",
+ advance_epoch_before_yield,
+ ),
+ patch.object(
+ manager,
+ "_make_sqlcipher_connection",
+ side_effect=[old_connection, new_connection],
+ ),
+ patch.object(manager, "_require_full_synchronous"),
+ patch.object(manager, "_pre_rekey_checkpoint"),
+ patch.object(manager, "_verify_rekey_candidate"),
+ patch.object(manager, "_checkpoint_wal") as checkpoint,
+ patch.object(encrypted_db, "set_sqlcipher_rekey") as rekey,
+ ):
+ # When the raw password-change path rekeys after acquiring that sidecar
+ changed = manager.change_password(
+ username, "old-password", "new-password"
+ )
+
+ # Then stale cache disposal never opens the old-key engine to checkpoint it
+ assert changed is True
+ checkpoint.assert_not_called()
+ stale_engine.connect.assert_not_called()
+ stale_engine.dispose.assert_called_once_with()
+ rekey.assert_called_once_with(
+ old_connection, "new-password", db_path=db_path
+ )
+ old_connection.close.assert_called_once_with()
+ new_connection.close.assert_called_once_with()
+
+
+def test_live_reader_busy_checkpoint_does_not_strand_committed_rekey(
+ manager, monkeypatch
+):
+ # Given a real old-key reader that opens its read transaction after the
+ # pre-rekey checkpoint, so it pins the WAL through candidate verification
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"livereader_{uuid.uuid4().hex[:8]}"
+ old = "OldReaderHorse5!" # noqa: S105
+ new = "NewReaderHorse6!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ real_make = DatabaseManager._make_sqlcipher_connection
+
+ def make_short_busy(path, password, **kwargs):
+ # Only shortens the busy wait (db_config.busy_timeout_ms, 30 s by
+ # default) of any statement that cannot finish while the reader
+ # lives; every SQLCipher call stays real.
+ connection = real_make(path, password, **kwargs)
+ connection.execute("PRAGMA busy_timeout = 200")
+ return connection
+
+ monkeypatch.setattr(manager, "_make_sqlcipher_connection", make_short_busy)
+ readers = []
+ real_begin = encrypted_db.begin_rekey_epoch
+
+ def begin_with_live_reader(path):
+ reader = real_make(path, old, isolation_level=None)
+ reader.execute("BEGIN")
+ reader.execute("SELECT count(*) FROM sqlite_master").fetchone()
+ readers.append(reader)
+ return real_begin(path)
- # 5. The old password can no longer open the re-keyed database, and the new
- # password can. (These re-open, so restore the real close first.)
monkeypatch.setattr(
- manager, "close_user_database_if_stale", real_close_stale
+ encrypted_db, "begin_rekey_epoch", begin_with_live_reader
)
- monkeypatch.setattr(encrypted_db, "set_sqlcipher_rekey", real_rekey)
+ try:
+ # When the password changes while that reader still holds the WAL
+ assert manager.change_password(username, old, new) is True
+
+ # Then the new key is committed and active, and the old key is an
+ # ordinary wrong password, all while the reader is still alive
+ assert len(readers) == 1
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch is not None
+ assert manager.open_user_database(username, old) is None
+ assert manager.open_user_database(username, new) is not None
+ assert read_rekey_epoch_state(db_path) == state
+ finally:
+ for reader in readers:
+ reader.close()
+
+
+def test_reader_pinned_before_change_password_neither_refuses_nor_stalls(
+ manager, monkeypatch
+):
+ # Given an old-key reader whose read transaction is already open over a
+ # non-empty WAL when the password change starts, and a busy timeout far
+ # longer than any rekey step should wait while holding the sidecar gate
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ busy_timeout_ms = 8000
+ monkeypatch.setenv("LDR_DB_CONFIG_BUSY_TIMEOUT_MS", str(busy_timeout_ms))
+ username = f"pinnedreader_{uuid.uuid4().hex[:8]}"
+ old = "OldPinnedHorse5!" # noqa: S105
+ new = "NewPinnedHorse6!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ make = DatabaseManager._make_sqlcipher_connection
+ writer = make(db_path, old, isolation_level=None)
+ reader = make(db_path, old, isolation_level=None)
+ try:
+ writer.execute("CREATE TABLE pinned_probe (value TEXT)")
+ writer.executemany(
+ "INSERT INTO pinned_probe VALUES (?)",
+ [(f"row-{index}",) for index in range(50)],
+ )
+ writer.close()
+ reader.execute("BEGIN")
+ assert reader.execute(
+ "SELECT count(*) FROM pinned_probe"
+ ).fetchone() == (50,)
+
+ # When the password changes while that reader keeps the WAL pinned
+ started = time.monotonic()
+ changed = manager.change_password(username, old, new)
+ elapsed = time.monotonic() - started
+
+ # Then the rekey commits instead of refusing at the pre-rekey
+ # checkpoint, and no checkpoint waited out the full busy timeout
+ assert changed is True
+ assert elapsed < busy_timeout_ms / 1000 * 0.75
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch is not None
+ assert manager.open_user_database(username, old) is None
+ assert manager.open_user_database(username, new) is not None
+ finally:
+ writer.close()
+ reader.close()
+
+
+def _sqlite_file_names(db_path):
+ return {
+ db_path.name,
+ f"{db_path.name}-wal",
+ f"{db_path.name}-shm",
+ f"{db_path.name}-journal",
+ }
- assert manager.open_user_database(username, old) is None, (
- "the old password must not open the re-keyed database"
+
+def test_change_password_opens_no_raw_descriptor_on_the_live_database(
+ manager, monkeypatch
+):
+ # Given a connection to the user's database that stays open in this
+ # process across the password change (a checked-out session, a pinned
+ # reader). Closing ANY descriptor to that file would release every
+ # POSIX fcntl lock this process holds on it, SQLite's WAL-mode SHARED
+ # lock included, behind SQLite's back (sqlite.org/howtocorrupt.html
+ # section 2.2); another process could then delete the WAL under it.
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"nofdclose_{uuid.uuid4().hex[:8]}"
+ old = "OldDescriptorHorse1!" # noqa: S105
+ new = "NewDescriptorHorse2!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ guarded = _sqlite_file_names(db_path)
+ live = DatabaseManager._make_sqlcipher_connection(
+ db_path, old, isolation_level=None
)
- reopened = manager.open_user_database(username, new)
- assert reopened is not None
- assert manager._password_matches_cached(username, new)
- assert not manager._password_matches_cached(username, old)
+ opened = []
+ # Every Python-level way to get a descriptor on a path. ``builtins.open``
+ # and ``io.open`` are the same function object but two separate name
+ # bindings: patching ``io.open`` alone does NOT intercept a bare
+ # ``open(db_path, "r+b")`` (which resolves through ``builtins``), so a
+ # regression that reintroduced the raw fsync that way would slip past.
+ # ``Path.open`` is hooked explicitly too rather than relying on its
+ # current implementation delegating to ``io.open``.
+ real_os_open = os.open
+ real_io_open = io.open
+ real_builtins_open = builtins.open
+ real_path_open = Path.open
+
+ def _record(file):
+ if not isinstance(file, int):
+ opened.append(Path(os.fsdecode(file)).name)
+
+ def recording_os_open(path, *args, **kwargs):
+ _record(path)
+ return real_os_open(path, *args, **kwargs)
+
+ def recording_io_open(file, *args, **kwargs):
+ _record(file)
+ return real_io_open(file, *args, **kwargs)
+
+ def recording_builtins_open(file, *args, **kwargs):
+ _record(file)
+ return real_builtins_open(file, *args, **kwargs)
+
+ def recording_path_open(self, *args, **kwargs):
+ _record(self)
+ return real_path_open(self, *args, **kwargs)
+
+ try:
+ live.execute("SELECT count(*) FROM sqlite_master").fetchone()
+ with monkeypatch.context() as patcher:
+ patcher.setattr(os, "open", recording_os_open)
+ patcher.setattr(io, "open", recording_io_open)
+ patcher.setattr(builtins, "open", recording_builtins_open)
+ patcher.setattr(Path, "open", recording_path_open)
+
+ # When the password changes while that connection is open
+ changed = manager.change_password(username, old, new)
+ finally:
+ live.close()
+
+ # Then the change succeeded without any Python-level open (and so no
+ # close) of the database, its WAL, SHM or journal; SQLite's own
+ # syncing under synchronous=FULL provides the durability
+ assert changed is True
+ assert [name for name in opened if name in guarded] == []
+ assert any(name.startswith(f"{db_path.name}.") for name in opened), (
+ "the recorder must have seen the sidecar opens, or it hooked nothing"
+ )
+ assert manager.open_user_database(username, new) is not None
+
+
+def test_rekey_that_silently_did_not_apply_fails_cleanly_under_old_key(
+ manager, monkeypatch
+):
+ # Given a writer holding a write transaction on the user's database
+ # for longer than the busy timeout: SQLCipher's PRAGMA rekey then
+ # answers "ok" without raising and leaves the file under the OLD key
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ monkeypatch.setenv("LDR_DB_CONFIG_BUSY_TIMEOUT_MS", "300")
+ username = f"silentrekey_{uuid.uuid4().hex[:8]}"
+ old = "OldSilentHorse3!" # noqa: S105
+ new = "NewSilentHorse4!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ original_epoch = read_rekey_epoch_state(db_path).committed_epoch
+ writer = DatabaseManager._make_sqlcipher_connection(
+ db_path, old, isolation_level=None
+ )
+ try:
+ writer.execute("CREATE TABLE silent_probe (value TEXT)")
+ writer.execute("BEGIN IMMEDIATE")
+ writer.execute("INSERT INTO silent_probe VALUES ('held')")
+
+ # When the password change runs into that writer
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password(username, old, new)
+ finally:
+ writer.rollback()
+ writer.close()
+
+ # Then it fails as retryable, not indeterminate: no pending state is
+ # left for a login to repair, the old password still opens the
+ # database, and the new one (which never took effect) does not
+ assert raised.value.stage is RekeyStage.PHYSICAL_REKEY
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch is not None
+ assert state.committed_epoch != original_epoch
+ assert manager.open_user_database(username, new) is None
+ assert manager.open_user_database(username, old) is not None
+
+
+def test_integrity_check_after_busy_checkpoint_rekey_is_not_corrupted(
+ manager, monkeypatch
+):
+ # Given a password change whose post-rekey checkpoint a live old-key
+ # reader kept busy, so the WAL holds new-key frames the main file lacks
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ username = f"integbusy_{uuid.uuid4().hex[:8]}"
+ old = "OldIntegrityHorse7!" # noqa: S105
+ new = "NewIntegrityHorse8!" # noqa: S105
+ assert manager.create_user_database(username, old) is not None
+ manager.close_user_database(username)
+ real_make = DatabaseManager._make_sqlcipher_connection
+
+ def make_short_busy(path, password, **kwargs):
+ connection = real_make(path, password, **kwargs)
+ connection.execute("PRAGMA busy_timeout = 200")
+ return connection
+
+ monkeypatch.setattr(manager, "_make_sqlcipher_connection", make_short_busy)
+ readers = []
+ real_begin = encrypted_db.begin_rekey_epoch
+
+ def begin_with_live_reader(path):
+ reader = real_make(path, old, isolation_level=None)
+ reader.execute("BEGIN")
+ reader.execute("SELECT count(*) FROM sqlite_master").fetchone()
+ readers.append(reader)
+ return real_begin(path)
+
+ monkeypatch.setattr(
+ encrypted_db, "begin_rekey_epoch", begin_with_live_reader
+ )
+ try:
+ assert manager.change_password(username, old, new) is True
+ assert manager.open_user_database(username, new) is not None
+
+ # When the integrity check runs while the reader still pins the WAL
+ healthy = manager.check_database_integrity(username)
+ finally:
+ for reader in readers:
+ reader.close()
+
+ # Then the un-checkpointed new-key frames are not reported as corruption
+ assert len(readers) == 1
+ assert healthy is True
+ # And once the reader is gone the full check (cipher scan included)
+ # still passes on the same engine
+ assert manager.check_database_integrity(username) is True
+
+
+def test_busy_verification_during_pending_recovery_is_not_a_wrong_password(
+ manager, monkeypatch
+):
+ # Given a rollback-journal database left in pending-rekey state, and
+ # another connection holding its EXCLUSIVE lock as the user logs in
+ if not manager.has_encryption:
+ pytest.skip("SQLCipher not available; rekey path is a no-op")
+ monkeypatch.setenv("LDR_DB_CONFIG_JOURNAL_MODE", "DELETE")
+ username = f"busypend_{uuid.uuid4().hex[:8]}"
+ password = "PendingBusyHorse9!" # noqa: S105
+ assert manager.create_user_database(username, password) is not None
+ manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ with acquire_rekey_lock(db_path):
+ read_or_initialize_rekey_epoch(db_path)
+ pending_epoch = rekey_lock_module.begin_rekey_epoch(db_path)
+ holder = DatabaseManager._make_sqlcipher_connection(
+ db_path, password, isolation_level=None
+ )
+ holder.execute("BEGIN EXCLUSIVE")
+
+ # The driver's own default lock wait would outlast the hold below; drop
+ # it to zero so only a busy timeout set before verification waits.
+ real_module = encrypted_db.get_sqlcipher_module()
+
+ class _NoDefaultLockWait:
+ def __getattr__(self, name):
+ return getattr(real_module, name)
+
+ @staticmethod
+ def connect(*args, **kwargs):
+ return real_module.connect(*args, timeout=0, **kwargs)
+
+ monkeypatch.setattr(
+ encrypted_db, "get_sqlcipher_module", lambda: _NoDefaultLockWait()
+ )
+ verifying = threading.Event()
+ real_verify = encrypted_db.check_sqlcipher_key
+
+ def verify_marking_start(cursor):
+ verifying.set()
+ return real_verify(cursor)
+
+ monkeypatch.setattr(
+ encrypted_db, "check_sqlcipher_key", verify_marking_start
+ )
+
+ def release_once_verification_waits():
+ if verifying.wait(timeout=30):
+ # Hold the real lock while the verification read waits on it
+ # (SQLite's busy handler runs on wall time; no clock to fake).
+ time.sleep(0.5) # allow: unmarked-sleep
+ holder.execute("COMMIT")
+
+ releaser = threading.Thread(target=release_once_verification_waits)
+ releaser.start()
+ try:
+ # When the CORRECT password logs in while the lock is still held
+ engine = manager.open_user_database(username, password)
+ finally:
+ releaser.join(timeout=30)
+ holder.close()
+
+ # Then the lock is waited out rather than read as a wrong password, and
+ # the pending generation is finalized
+ assert verifying.is_set()
+ assert engine is not None
+ state = read_rekey_epoch_state(db_path)
+ assert state.pending_epoch is None
+ assert state.committed_epoch == pending_epoch
diff --git a/tests/security/test_db_connection_change_password_race.py b/tests/security/test_db_connection_change_password_race.py
index d689c7ab97..d34003fa2c 100644
--- a/tests/security/test_db_connection_change_password_race.py
+++ b/tests/security/test_db_connection_change_password_race.py
@@ -1,31 +1,33 @@
"""change_password rekey racing concurrent open_user_database(old_password).
-Incident context (PR #5596): the fix's eviction comment in ``change_password``
-warns about the exact race these tests drive at the real-database level:
-
- open_user_database() above cold-opened with old_password and cached that
- engine together with a verifier for old_password. The rekey just
- invalidated that engine's key [...] yet the cached verifier still matches
- old_password -- so until this eviction a concurrent
- open_user_database(old_password) would pass the verifier check and be
- handed the stale-key engine.
-
-``change_password`` now runs under a per-user lock, but that lock excludes
-only a second ``change_password``: a concurrent ``open_user_database`` still
-runs alongside the rekey, so the window above is still closed by the
-eviction alone.
-
-``test_change_password_evicts_stale_engine.py`` pins that the eviction runs
-inside the try (via monkeypatched spies on close/rekey/log); it never runs
-the race with real threads. ``test_login_cached_connection_password_extra.
-py::test_password_change_invalidates_old_verifier_and_arms_new`` pins only
-the sequential after-state. Neither proves the end-to-end concurrent
-property against a real SQLCipher file:
-
-* WHILE change_password runs, threads calling open_user_database(old) may
- receive engines (legitimately -- the old key is still valid until the
- rekey lands), but every such engine must be DEAD once the rekey
- completes: it must never yield a queryable session afterward.
+Incident context (PR #5596): ``change_password`` used to cold-open the
+database with the old password, cache that engine with a verifier for the
+old password, rekey the file, and evict the engine only afterwards. Between
+the rekey and the eviction a concurrent ``open_user_database(old_password)``
+passed the verifier check and was handed the stale-key engine.
+
+The durable rekey epoch closes that window structurally. ``change_password``
+holds the user's on-disk rekey sidecar lock for its whole staged sequence and
+detaches any cached engine BEFORE the rekey; every cold open and every cache
+claim takes the same lock, and a cached generation is only handed out while
+its recorded epoch is the committed durable epoch. #6565's in-process
+generation counter and post-rekey conditional eviction
+(``_get_rekey_epoch`` / ``_bump_rekey_epoch`` /
+``_cache_connection(expected_epoch=...)`` / ``close_user_database_if_stale``)
+are superseded by that fence; the tests that pinned them below now pin the
+stronger behaviour, and each docstring says what changed.
+
+``test_change_password_evicts_stale_engine.py`` pins the ordering (cache
+empty at the moment of the rekey) via monkeypatched spies;
+``test_login_cached_connection_password_extra.py::
+test_password_change_invalidates_old_verifier_and_arms_new`` pins the
+sequential after-state. This module proves the end-to-end concurrent
+properties against a real SQLCipher file:
+
+* threads calling open_user_database(old) before the rekey starts may
+ receive engines (the old key is still valid then), but every such engine
+ must be DEAD once the rekey completes: it must never yield a queryable
+ session afterward.
* AFTER change_password returns: the old key must fail (and cache nothing),
the new key must open and read data written under previous passwords.
* Under a stress loop of chained rekeys, no snapshot of the manager may
@@ -45,6 +47,7 @@ import pytest
from sqlalchemy import text
from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import read_rekey_epoch
# Module-level gate. Without a working SQLCipher build every test below
# is a no-op, and the dozens of in-body ``pytest.skip`` calls let a CI
@@ -201,20 +204,18 @@ def test_old_password_open_racing_change_password_never_usable_after_rekey(
DEAD once change_password returns: a SELECT on it must fail.
STRICTLY WEAKER than the spy test it complements, and deliberately so.
- Delete ``encrypted_db.py::change_password``'s IN-TRY eviction and keep
- only the ``finally`` close, and every assertion here still passes: the
- grabbed engine is dead after the rekey either way, because the rekey
- invalidated the key its creator closure derives. That is a property of
- SQLCipher rekey, not of where the eviction sits.
+ Move ``change_password``'s cache detach from before the rekey to after
+ it, and every assertion here still passes: the grabbed engine is dead
+ after the rekey either way, because the rekey invalidated the key its
+ creator closure derives. That is a property of SQLCipher rekey, not of
+ where the detach sits.
What this test does pin, under real threads rather than spies: no
engine handed out under the OLD password survives a completed
change_password, the old password no longer opens afterwards, and the
- new one reads the pre-change data. The PLACEMENT of the eviction --
- inside the try, closing the window a concurrent old-password open could
- otherwise use -- is pinned by
- ``tests/security/test_change_password_evicts_stale_engine.py``, which
- fails when that in-try close is removed.
+ new one reads the pre-change data. The ORDERING of the detach (cache
+ empty at the moment of the rekey) is pinned by
+ ``tests/security/test_change_password_evicts_stale_engine.py``.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
@@ -315,18 +316,16 @@ def test_stress_change_password_chain_vs_open_hammers(manager):
* every engine obtained with the FINAL password is keyed with the
final password: disposed first (to force a fresh connection through
its creator, where SQLCipher key verification runs), it still opens
- and reads the canary. More than one such engine can legitimately
- exist -- a final-password engine cold-opened between the last
- rekey's commit and change_password's trailing close_user_database is
- evicted by that close (it evicts by username), and a later open
- caches a new one -- so engine identity is NOT the invariant, the
- key is;
+ and reads the canary. Engine identity is NOT the invariant, the key
+ is: any eviction between two grabs (a logout, the idle pool reset)
+ legitimately makes a later open cache a second final-password
+ engine;
* no intermediate or original password can open at the end;
* the final password opens and reads the canary written at creation;
* the engine/verifier PAIRING invariant, sampled under the connections
lock after each change: the set of usernames in ``connections``
always equals the set in ``_password_verifiers``. Either map CAN
- legitimately be (re-)populated between the change_password eviction
+ legitimately be (re-)populated between change_password's detach
and the snapshot by a concurrent open_user_database republishing
engine+verifier atomically via _cache_connection -- that is not a
publication gap. The engine-without-verifier / verifier-without-
@@ -470,11 +469,10 @@ def test_stress_change_password_chain_vs_open_hammers(manager):
# page, and an engine rekeyed in place keeps a live connection
# that already carries the new key even though the engine's own
# creator still derives an earlier password. Several distinct
- # final-password engines can legitimately exist: one cold-opened
- # after the last rekey commits but before change_password's
- # trailing close_user_database is evicted by that close (it
- # evicts by username), and a later open caches a new one. Its
- # creator still derives the final key, so it is not mis-keyed.
+ # final-password engines can legitimately exist: any eviction
+ # between two grabs (a logout, the idle pool reset) makes a later
+ # open cache a new one. Each creator still derives the final key,
+ # so none is mis-keyed.
# Once disposed, an engine keyed with any earlier password fails
# SQLCipher verification against the rekeyed file and trips this.
# (A final-password open cannot succeed before the final rekey: the
@@ -515,23 +513,22 @@ def test_stress_change_password_chain_vs_open_hammers(manager):
def test_change_password_does_not_evict_concurrent_new_password_engine(
manager,
):
- """Direct pin for the post-rekey conditional eviction.
-
- ``test_stress_change_password_chain_vs_open_hammers`` proves the
- end-to-end invariant under racing threads (every final-password engine
- the hammer grabs is the surviving one). This test proves the same
- invariant at the unit level by hand-feeding a fresh engine with the
- NEW password into the cache BEFORE the rekey, then observing that the
- conditional close used by ``change_password`` leaves it alone.
-
- A concurrent ``open_user_database(new_password)`` racing the rekey
- publishes exactly this state -- file keyed with new_password, cache
- holding an engine with the new verifier. The unconditional
- ``close_user_database`` would drop it and break the invariant; the
- conditional ``close_user_database_if_stale(old_password)`` must not.
- Without the gate, every racing hammer cold-open re-opens a fresh
- engine after the rekey, forcing a needless cold-open for the next
- caller AND breaking the test_stress engine-identity assertion above.
+ """A new-password engine published after the rekey is never evicted.
+
+ #6565 pinned this through ``close_user_database_if_stale``, the
+ post-rekey conditional eviction that let an engine with the NEW
+ verifier survive while dropping one with the OLD verifier. The durable
+ rekey epoch supersedes that helper and it no longer exists:
+ ``change_password`` detaches the cached engine BEFORE the rekey while
+ holding the sidecar lock every cold open also takes, and runs no
+ eviction afterwards, so nothing it does can clobber a new-password
+ engine published once it releases the sidecar. An engine with the OLD
+ verifier can only exist under a superseded epoch, and the cache claim
+ detaches it instead of handing it out. This test pins both halves of
+ the old contract against the stronger mechanism: the new-password
+ engine survives later opens (cache hits and a failed old-password
+ cold open alike), and a stale old-verifier engine planted under the
+ pre-rekey epoch is never returned and is detached on its first claim.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
@@ -542,206 +539,241 @@ def test_change_password_does_not_evict_concurrent_new_password_engine(
engine = manager.create_user_database(username, old)
_write_canary(engine, "gap-B-cond")
manager.close_user_database(username)
+ db_path = manager._get_user_db_path(username)
+ pre_rekey_epoch = read_rekey_epoch(db_path)
+ assert pre_rekey_epoch is not None
- # Simulate the race outcome: change_password runs through to completion
- # with no concurrent open. The cache is empty afterwards (its in-try
- # conditional close evicted the stale engine it opened). Hand-feed the
- # state the racing open would have produced -- cache holds a fresh
- # engine with the NEW verifier -- then call the conditional close the
- # FINALLY path uses and confirm the fresh engine survives.
assert manager.change_password(username, old, new) is True
assert username not in manager.connections
+ assert read_rekey_epoch(db_path) not in (None, pre_rekey_epoch)
- # Cold-open with the new password -- this is the "concurrent open"
- # outcome the conditional close must not undo.
+ # The new-password engine a concurrent open would publish once the
+ # sidecar is released.
fresh = manager.open_user_database(username, new)
assert fresh is not None
- surviving_before = manager.connections.get(username)
- assert surviving_before is fresh
-
- # The FINALLY path of change_password runs unconditionally; in the
- # success path it is followed by a return True, so under the live code
- # there is no chance to observe this on the success branch -- the
- # exception branch calls it too. Call the helper directly here to pin
- # the no-op property: passing the OLD password must NOT drop the
- # engine, because its verifier matches the NEW password, not the
- # old one.
- manager.close_user_database_if_stale(username, old)
- surviving_after = manager.connections.get(username)
- assert surviving_after is fresh, (
- "close_user_database_if_stale(old_password) must leave an engine "
- "with a different (new_password) verifier cached -- otherwise a "
- "concurrent open_user_database(new_password) racing the rekey has "
- "its surviving engine clobbered and the test_stress invariant "
- "breaks"
+ assert manager.connections.get(username) is fresh
+
+ # Neither a cache hit nor a failed old-password cold open evicts it.
+ assert manager.open_user_database(username, new) is fresh
+ assert manager.open_user_database(username, old) is None
+ assert manager.connections.get(username) is fresh, (
+ "a failed old-password open must leave the surviving new-password "
+ "engine cached"
)
assert manager._password_matches_cached(username, new) is True
- # And, symmetrically: the conditional close DOES drop a stale engine
- # whose verifier still matches the OLD password. Force the cache into
- # that state directly -- the helper only checks the verifier, so any
- # engine with the old verifier is treated as stale regardless of
- # whether its closure matches the current file key (which is the
- # security-relevant property: an old-verifier engine could be returned
- # to a caller presenting old_password without a SQLCipher round-trip).
- stale_engine = manager.connections.get(username)
- assert stale_engine is not None
- manager._cache_connection(username, stale_engine, old)
- assert manager._password_matches_cached(username, old)
- manager.close_user_database_if_stale(username, old)
- assert username not in manager.connections, (
- "close_user_database_if_stale(old_password) must drop a stale "
- "engine whose verifier still matches the old password -- that is "
- "the whole point of the helper"
+ # Symmetric half: an engine carrying the OLD verifier under the
+ # pre-rekey epoch (what a stale publish would plant) is never handed
+ # out; its first claim detaches it.
+ from sqlalchemy import create_engine
+
+ stale = create_engine("sqlite://")
+ try:
+ replaced = manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(stale, old, pre_rekey_epoch),
+ )
+ assert replaced is fresh
+ replaced.dispose()
+ assert manager.open_user_database(username, old) is None, (
+ "an old-verifier engine cached under a superseded epoch must "
+ "never be returned to the old password"
+ )
+ assert username not in manager.connections, (
+ "the stale generation must be detached on its first claim"
+ )
+ assert username not in manager._password_verifiers
+ assert username not in manager._connection_epochs
+ finally:
+ stale.dispose()
+
+
+def test_change_password_keeps_engine_published_after_sidecar_release(
+ manager, monkeypatch
+):
+ """Nothing change_password runs after releasing the sidecar evicts.
+
+ The test above publishes the new-password engine only after
+ ``change_password`` has returned, so an eviction added to the tail of
+ ``change_password`` (after the sidecar ``with`` block, or in its
+ ``finally``) would still pass it. Here the racing
+ ``open_user_database(new)`` runs, on another thread, at the exact
+ moment ``change_password`` releases the rekey sidecar, while
+ ``change_password`` itself is still on the stack. The engine it
+ publishes must still be cached once ``change_password`` returns.
+ """
+ if not manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
+
+ import contextlib
+
+ from local_deep_research.database import encrypted_db as encrypted_db_mod
+
+ username = f"chgpw_tail_{uuid.uuid4().hex[:8]}"
+ old = "OldCorrectHorse1!" # noqa: S105
+ new = "NewCorrectHorse2!" # noqa: S105
+ engine = manager.create_user_database(username, old)
+ _write_canary(engine, "gap-B-tail")
+ manager.close_user_database(username)
+
+ real_acquire = encrypted_db_mod.acquire_rekey_lock
+ published = {}
+
+ def _race_open_new():
+ published["engine"] = manager.open_user_database(username, new)
+
+ @contextlib.contextmanager
+ def _acquire_then_race(*args, **kwargs):
+ with real_acquire(*args, **kwargs) as handle:
+ yield handle
+ # Only change_password passes contention_retry; every open and
+ # disposal path acquires without it.
+ if "contention_retry" in kwargs and "engine" not in published:
+ racer = threading.Thread(target=_race_open_new)
+ racer.start()
+ racer.join(timeout=JOIN_TIMEOUT_S)
+ assert not racer.is_alive(), "racing open deadlocked"
+
+ monkeypatch.setattr(
+ encrypted_db_mod, "acquire_rekey_lock", _acquire_then_race
)
- assert username not in manager._password_verifiers
+
+ assert manager.change_password(username, old, new) is True
+
+ fresh = published.get("engine")
+ assert fresh is not None, (
+ "premise: the racing new-password open must have published an "
+ "engine while change_password was still running"
+ )
+ assert manager.connections.get(username) is fresh, (
+ "change_password evicted the new-password engine a concurrent "
+ "open published after the sidecar was released"
+ )
+ usable, detail = _engine_usable(fresh)
+ assert usable, detail
+ assert manager.open_user_database(username, new) is fresh
def test_cache_connection_disposes_overwritten_engine(manager, monkeypatch):
- """Direct pin for the orphan-dispose property of ``_cache_connection``.
-
- When ``open_user_database(new_password)`` publishes a fresh engine and
- overwrites the engine ``change_password`` cached moments earlier, the
- overwritten engine reference is lost to the cache but its SQLCipher
- pool and file handle still hold the OLD hex-key closure. Without an
- explicit ``dispose()`` on the orphan, that pool lingers until GC,
- leaking one stale connection per overwrite on exactly the path the
- conditional-eviction fix makes common (concurrent new-password opens
- against a rekeying database).
-
- Spy ``Engine.dispose`` so we can observe the call directly -- a pure
- "engine is gone" check would miss the leak because the orphan is also
- gone from the cache the instant it is overwritten, regardless of
- whether anyone disposes it.
-
- The publish uses ``_cache_connection`` directly rather than going
- through ``open_user_database(new_password)``: the file is still keyed
- with ``old_password`` in this test (no rekey happened), so a real
- new-password open would fail at the SQLCipher layer. ``_cache_connection``
- is the exact publication primitive ``open_user_database`` calls after a
- successful SQLCipher round-trip, so this drives the property under
- test in isolation.
+ """The engine a publish overwrites is disposed exactly once.
+
+ #6565 disposed the orphan inside ``_cache_connection`` under
+ ``_connections_lock``. With the durable epoch, ``_cache_connection``
+ publishes ``(engine, verifier, epoch)`` atomically and RETURNS the
+ overwritten engine instead (``None`` for a first publish or a
+ same-identity republish), and every caller disposes it outside
+ ``_connections_lock`` through ``_dispose_detached_connection`` (a WAL
+ checkpoint plus ``dispose``). The leak #6565 fixed stays fixed; the
+ dispose just moves to the caller so a slow checkpoint never runs under
+ the global cache lock. This pins the primitive's return contract and,
+ through the real ``open_user_database`` publish, that the orphan is
+ disposed exactly once and the published engine is not disposed.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
+ from sqlalchemy import create_engine
+ from sqlalchemy.engine import Engine as _Engine
+
+ import local_deep_research.database.initialize as _init_module
+
username = f"chgpw_orphan_{uuid.uuid4().hex[:8]}"
old = "OldCorrectHorse1!" # noqa: S105
- new = "NewCorrectHorse2!" # noqa: S105
-
- bootstrap_engine = manager.create_user_database(username, old)
- assert bootstrap_engine is not None
- # User file is created; close so we can drive the cache by hand below
- # without triggering an in-loop open.
+ bootstrap = manager.create_user_database(username, old)
+ assert bootstrap is not None
manager.close_user_database(username)
+ epoch = read_rekey_epoch(manager._get_user_db_path(username))
+ assert epoch is not None
- # Spy Engine.dispose. The spy records every call site by id(self) so
- # we can identify which engine was disposed when multiple engines
- # are alive at once.
- from sqlalchemy.engine import Engine as _Engine
+ # Primitive contract.
+ first = create_engine("sqlite://")
+ second = create_engine("sqlite://")
+ try:
+ entry = manager._make_connection_cache_entry(first, old, epoch)
+ assert manager._cache_connection(username, entry) is None, (
+ "a first publish has nothing to hand back for disposal"
+ )
+ assert manager._cache_connection(username, entry) is None, (
+ "a same-identity republish must not hand back the live engine"
+ )
+ assert (
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(second, old, epoch),
+ )
+ is first
+ ), "an overwrite must hand back the engine it displaced"
+ assert manager.connections[username] is second
+ finally:
+ manager._detach_connection(username)
+ first.dispose()
+ second.dispose()
- real_engine_dispose = _Engine.dispose
- dispose_calls = []
+ # Real caller: an engine planted while the cold open runs is the one
+ # its publish overwrites.
+ real_dispose = _Engine.dispose
+ dispose_calls: list[int] = []
def spy_dispose(self):
dispose_calls.append(id(self))
- return real_engine_dispose(self)
+ return real_dispose(self)
monkeypatch.setattr(_Engine, "dispose", spy_dispose)
+ planted = create_engine("sqlite://")
+ real_initialize = _init_module.initialize_database
- # Stage the cache into the post-rekey / pre-conditional-close state:
- # change_password has cold-opened with old_password (caching the old
- # engine) and is about to do the in-try conditional close. A
- # concurrent open_user_database(new_password) racing the rekey
- # publishes a fresh engine and overwrites the old one. Hand-feed that
- # state directly so the overwrite path runs deterministically -- use
- # a fresh real Engine for the "new" engine so dispose has work to do.
- published_old = manager.open_user_database(username, old)
- assert published_old is not None
- assert manager.connections[username] is published_old
-
- # Construct a synthetic "racing" engine -- the publication primitive
- # ``_cache_connection`` accepts any Engine; the SQLCipher key
- # derivation that opens it is not relevant to this test, only the
- # overwrite + dispose behavior is. Reuse the user file URL so
- # dispose() has a non-trivial pool to close.
- from sqlalchemy import create_engine
+ planted_once = []
- fresh = create_engine(
- f"sqlite:///{manager._get_user_db_path(username).as_posix()}"
- )
+ def plant_then_initialize(engine):
+ if not planted_once:
+ planted_once.append(True)
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(planted, old, epoch),
+ )
+ return real_initialize(engine)
- # Reset the spy -- open_user_database above may have triggered
- # disposes elsewhere we do not care about. The property we are
- # pinning is "the OVERWRITE disposes the previous engine".
- dispose_calls.clear()
-
- # Drive the race outcome: _cache_connection overwrites the cache.
- manager._cache_connection(username, fresh, new)
- assert manager.connections[username] is fresh
-
- # The orphan (published_old) MUST have been disposed as part of the
- # cache swap. dispose_calls records the old engine exactly once.
- assert id(published_old) in dispose_calls, (
- "_cache_connection must dispose the previous engine it is "
- "overwriting; an orphaned SQLCipher pool + file handle leaks on "
- "exactly the concurrent-new-password-open path this PR made common"
- )
- assert dispose_calls.count(id(published_old)) == 1, (
- f"the orphan engine was disposed "
- f"{dispose_calls.count(id(published_old))} times -- exactly once "
- "is the contract"
+ monkeypatch.setattr(
+ _init_module, "initialize_database", plant_then_initialize
)
-
- # Symmetry: an in-place re-publish with the SAME engine object must
- # NOT trigger an additional dispose (the conditional guards against
- # ``orphan is engine`` to avoid disposing a still-live pool when a
- # caller caches the same engine twice, which would close connections
- # a live user is mid-query on).
- before = dispose_calls.count(id(fresh))
- manager._cache_connection(username, fresh, new)
- assert dispose_calls.count(id(fresh)) == before, (
- "_cache_connection must not dispose a same-identity re-publish; "
- "the in-place republish happens on every reopen-with-same-password "
- "and disposing the live pool would terminate in-flight queries"
+ published = manager.open_user_database(username, old)
+ assert published is not None
+ assert manager.connections[username] is published
+ assert dispose_calls.count(id(planted)) == 1, (
+ "open_user_database must dispose the engine its publish displaced "
+ f"exactly once (saw {dispose_calls.count(id(planted))})"
)
-
- # First-time publish (no previous engine) MUST NOT trigger any
- # dispose -- otherwise every cold-open would race to close a pool
- # that did not exist yet.
- username_two = f"chgpw_orphan2_{uuid.uuid4().hex[:8]}"
- fresh_two = manager.create_user_database(username_two, old)
- assert fresh_two is not None
- assert manager.connections[username_two] is fresh_two
- # No overwrite happened on this username, so dispose_calls should
- # not have grown for fresh_two since the last measurement.
- after = dispose_calls.count(id(fresh_two))
- # (The create_user_database path also caches; a same-identity
- # republish via _cache_connection must be a no-op for dispose.)
- manager._cache_connection(username_two, fresh_two, old)
- assert dispose_calls.count(id(fresh_two)) == after, (
- "_cache_connection must not dispose a same-identity republish on "
- "a brand-new entry either"
+ assert id(published) not in dispose_calls, (
+ "the published engine itself must not be disposed"
)
-def test_verifier_matches_degrades_on_corrupt_entry(manager, monkeypatch):
- """Pin the malformed-verifier-entry safety net.
-
- ``_verifier_matches`` is called from inside ``change_password``'s
- ``finally`` block -- a raise here would replace the actual
- ``change_password`` outcome with a spurious ``TypeError`` /
- ``ValueError`` and the operator would see a confusing "password
- change failed" instead of the underlying cause. The invariants that
- produce well-formed entries are unit-locked (``_make_verifier`` and
- ``_cache_connection``), so a corrupt entry can only arrive through
- external dict mutation; degrading to no-match is safer than letting
- that exception escape.
-
- Without the safety net the assertion below raises before
- ``change_password`` returns, masking the real outcome (a clean
- rekey); with it, the conditional close no-ops, the rekey has
- already landed, and ``change_password`` returns True.
+ "corrupt_entry",
+ ["not-a-tuple", ("x", "y"), (b"salt", 123)],
+ ids=["non-tuple", "str-members", "int-digest"],
+)
+def test_verifier_matches_degrades_on_corrupt_entry(manager, corrupt_entry):
+ """Pin the malformed-verifier-entry safety net on every verifier check.
+
+ Two code paths compare a password against a cached verifier:
+ ``_verifier_matches`` (the cold-open path's ``_cached_engine_trusted``,
+ under the sidecar lock) and ``_connection_generation_password_matches``
+ (the cache-hit claim behind ``open_user_database``'s fast path and
+ ``_password_matches_cached``). Both go through
+ ``_verifier_entry_matches``, so a corrupt entry degrades to no-match on
+ both instead of raising ``TypeError`` / ``ValueError`` out of a login.
+ The invariants that produce well-formed entries are unit-locked, so a
+ corrupt entry can only arrive through external dict mutation; the
+ no-match falls through to a real key-validating cold open, which
+ republishes a well-formed verifier.
+
+ (#6565 introduced the guard for ``_verifier_matches`` because its
+ post-rekey conditional close called it from ``change_password``'s
+ ``finally``. That close no longer exists, ``change_password`` detaches
+ the cache without consulting the verifier, and the guard now matters
+ for the two open paths above; the generation path had no guard and
+ raised before this test covered it.)
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
@@ -753,202 +785,141 @@ def test_verifier_matches_degrades_on_corrupt_entry(manager, monkeypatch):
_write_canary(engine, "gap-B-corrupt")
manager.close_user_database(username)
- # Stage a corrupt verifier: a non-tuple value where the cache expects
- # (salt, expected). Run change_password; the in-try conditional close
- # would see the corrupt value via _verifier_matches and degrade to
- # False, leaving any new engine cached, then the finally-close
- # degrades again, returning True. Without the safety net the unpacking
- # raises TypeError out of the helper and the caller sees a confusing
- # failure.
- # Inject AFTER the create + close above so the rekey flow runs against
- # a clean cache and the corruption only matters for the conditional
- # close path.
published = manager.open_user_database(username, old)
assert published is not None
- # Corrupt the verifier entry directly. The helper MUST treat this as
- # no-match (return False), not raise.
with manager._connections_lock:
- manager._password_verifiers[username] = "not-a-tuple" # type: ignore[assignment]
+ manager._password_verifiers[username] = corrupt_entry # type: ignore[assignment]
- # The helper degrades to False rather than raising.
+ # The lock-held helper degrades to no-match.
with manager._connections_lock:
- result = manager._verifier_matches(username, old)
- assert result is False, (
- "_verifier_matches must degrade to no-match on a malformed "
- "verifier entry; a raise here propagates out of "
- "change_password's finally and masks the real rekey outcome"
- )
+ assert manager._verifier_matches(username, old) is False
+ # So does the generation claim path.
+ assert manager._password_matches_cached(username, old) is False
- # End-to-end: change_password still returns True with a corrupt
- # verifier in the cache, because the conditional close degrades to
- # no-match (does not evict), and the in-try rekey already succeeded.
- # Restore a valid verifier first so the second pass (in the absence
- # of the safety net) would otherwise succeed -- we are now proving the
- # conditional close path tolerates the corruption it might encounter.
- # Keep the corruption in place so the finally-close also exercises it.
+ # A login with the correct password falls through to a key-validating
+ # cold open instead of raising, and republishes a well-formed verifier.
+ reopened = manager.open_user_database(username, old)
+ assert reopened is not None
+ assert reopened is not published
+ assert manager.connections.get(username) is reopened
+ assert manager._password_matches_cached(username, old) is True
+
+ # change_password detaches the cache without consulting the verifier,
+ # so a corrupt entry cannot mask its outcome either.
+ with manager._connections_lock:
+ manager._password_verifiers[username] = corrupt_entry # type: ignore[assignment]
assert manager.change_password(username, old, new) is True
- # The fresh new-password open afterwards must still work; the corrupt
- # entry did not leak into the post-rekey verifier (it was overwritten
- # by the in-try cold-open inside change_password before the corruption
- # could affect the result). Restoring the corruption before the
- # change_password above was deliberate -- change_password's own
- # open_user_database(old_password) cache_connection call wrote a
- # fresh verifier for old_password, so the cache now has a valid one
- # again, which is what the finally-close consulted.
+ assert username not in manager._password_verifiers
+ assert manager.open_user_database(username, new) is not None
def test_change_password_eviction_window_handles_concurrent_publish_deterministically(
- manager,
+ manager, monkeypatch
):
- """Deterministic (non-threaded) probe for the exact window the
- threaded stress test races on.
-
- ``test_stress_change_password_chain_vs_open_hammers`` is the proof
- under real threads, but its discriminating window is µs-scale: on
- fast hardware every observed hammer grab lands AFTER the final
- eviction, so the test could pass pre-fix. This probe drives the
- race outcome directly -- rekeys the file first so a fresh new-
- password open succeeds, then publishes it on top of the cold-open
- change_password would have cached -- and asserts the surviving-
- engine identity, the orphan-engine disposition, and the verifier
- mismatch guard together.
-
- The probe is intentionally NOT a threading test: every property
- here is about a specific cache state and can be set up without
- concurrency. It does not depend on the scheduler interleaving
- correctly and so cannot become vacuous on faster hardware. The
- threaded test stays as the end-to-end proof; this probe is the
- pin that survives on any machine.
+ """A new-password open racing the rekey waits for it, then publishes.
+
+ #6565 pinned this window by staging its outcome by hand (rekeying the
+ file outside ``change_password``, publishing a new-password engine on
+ top of the stale one) and checking that its post-rekey conditional
+ close left the racing engine alone. The durable rekey epoch removes the
+ window instead of arbitrating it: ``change_password`` holds the user's
+ sidecar lock across detach, rekey and epoch commit, and every cold open
+ takes the same lock, so a racing ``open_user_database(new)`` cannot
+ run against the half-rekeyed file at all. It blocks until the rekey
+ commits, then opens against the new key and publishes the only engine.
+
+ Deterministic, not a stress loop: the racing open is started from
+ inside the rekey itself and must still be blocked after a bounded
+ wait (a fenced open is ALWAYS blocked there, because the sidecar is
+ held, so this cannot flake under the correct code). Without the fence
+ it would cold-open against the still-old-keyed file and return None,
+ which the final assertions catch.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
+ from local_deep_research.database import encrypted_db
+
username = f"chgpw_det_{uuid.uuid4().hex[:8]}"
old = "OldCorrectHorse1!" # noqa: S105
new = "NewCorrectHorse2!" # noqa: S105
-
- bootstrap_engine = manager.create_user_database(username, old)
- _write_canary(bootstrap_engine, "gap-B-det")
+ bootstrap = manager.create_user_database(username, old)
+ _write_canary(bootstrap, "gap-B-det")
manager.close_user_database(username)
- del bootstrap_engine
-
- # Drive the cache state change_password's in-try conditional close
- # observes when a concurrent new-password open wins the race:
- # 1. ``change_password`` cold-opens with old_password -> cache
- # holds ``stale`` with the old verifier.
- # 2. ``change_password`` rekeys the file to new_password.
- # 3. A racing ``open_user_database(new_password)`` succeeds (file
- # IS keyed with new_password now) and publishes a fresh engine
- # + new verifier on top of ``stale``.
- # 4. ``change_password``'s in-try conditional close sees a verifier
- # that no longer matches old_password -> no-op.
- # 5. Property: the racing engine survives in the cache, the
- # overwritten engine has been disposed exactly once.
- from sqlalchemy.engine import Engine as _Engine
-
- real_engine_dispose = _Engine.dispose
- dispose_calls = []
-
- def spy_dispose(self):
- dispose_calls.append(id(self))
- return real_engine_dispose(self)
+ del bootstrap
- import pytest as _pytest
+ racing_result = {}
+ racing_errors = {}
- mp = _pytest.MonkeyPatch()
- mp.setattr(_Engine, "dispose", spy_dispose)
- try:
- stale = manager.open_user_database(username, old)
- assert stale is not None
- assert manager.connections[username] is stale
- assert manager._password_matches_cached(username, old) is True
-
- # Rekey the file in-place -- mirrors what change_password does
- # internally -- so the racing open below can succeed against
- # new_password.
- from local_deep_research.database.encrypted_db import (
- set_sqlcipher_rekey,
- )
+ def racing_open():
+ try:
+ racing_result["engine"] = manager.open_user_database(username, new)
+ except Exception as exc: # noqa: BLE001 - surfaced below
+ racing_errors["open"] = exc
- with stale.connect() as conn:
- set_sqlcipher_rekey(
- conn,
- new,
- db_path=manager._get_user_db_path(username),
- )
+ racer = threading.Thread(target=racing_open, name="gap-b-det-racer")
+ observed = {}
+ real_rekey = encrypted_db.set_sqlcipher_rekey
- # The racing new-password open: publishes ``surviving`` on top
- # of ``stale``. This is the cache state the in-try conditional
- # close will consult. The overwrite path inside
- # ``_cache_connection`` MUST have already disposed ``stale`` at
- # this point -- that is item 1 of the latest review feedback.
- surviving = manager.open_user_database(username, new)
- assert surviving is not None
- assert manager.connections[username] is surviving
- assert manager._password_matches_cached(username, new) is True
- assert not manager._password_matches_cached(username, old)
-
- # Property 1: the racing publish disposed ``stale`` exactly once.
- assert id(stale) in dispose_calls, (
- "the overwritten engine (stale) must have been disposed by "
- "_cache_connection's overwrite path, not leaked until GC"
- )
- assert dispose_calls.count(id(stale)) == 1, (
- "stale engine must be disposed exactly once on the overwrite"
- )
+ def rekey_with_racing_open(conn, new_password, **kwargs):
+ observed["cache_at_rekey"] = username in manager.connections
+ racer.start()
+ racer.join(timeout=0.3)
+ observed["racer_blocked"] = racer.is_alive()
+ return real_rekey(conn, new_password, **kwargs)
- # Reset the dispose baseline so we can prove the conditional
- # close itself triggers NO dispose (the verifier mismatch makes
- # it a no-op on the cache, and the orphan was already cleaned up
- # by the overwrite path above).
- dispose_calls.clear()
-
- # Property 2: the conditional close change_password's in-try
- # would call is a no-op on the cache because the verifier no
- # longer matches.
- manager.close_user_database_if_stale(username, old)
- assert manager.connections[username] is surviving, (
- "conditional close must leave the racing new-password engine "
- "cached -- this is the test_stress invariant"
- )
- assert id(surviving) not in dispose_calls, (
- "the conditional close must not touch the racing engine; the "
- "verifier mismatch already routed it to no-op"
- )
+ monkeypatch.setattr(
+ encrypted_db, "set_sqlcipher_rekey", rekey_with_racing_open
+ )
+ assert manager.change_password(username, old, new) is True
+ racer.join(timeout=JOIN_TIMEOUT_S)
+ assert not racer.is_alive(), "the racing open deadlocked on the sidecar"
+ assert not racing_errors, f"the racing open raised: {racing_errors}"
- # End-to-end: the file still answers the new password (the
- # rekey succeeded) and the cache still serves it (the
- # conditional close left it alone).
- with surviving.connect() as conn:
- note = conn.execute(
- text("SELECT note FROM gap_b_canary WHERE id = 1")
- ).scalar()
- assert note == "gap-B-det", "rekey must preserve data"
- finally:
- mp.undo()
+ assert observed == {"cache_at_rekey": False, "racer_blocked": True}, (
+ "the racing new-password open must be held off for the whole rekey "
+ f"by the sidecar lock: {observed}"
+ )
+ surviving = racing_result.get("engine")
+ assert surviving is not None, (
+ "the racing new-password open must succeed once the rekey commits"
+ )
+ assert manager.connections.get(username) is surviving
+ assert manager._password_matches_cached(username, new) is True
+ assert manager.open_user_database(username, old) is None
+ assert manager.connections.get(username) is surviving
+ with surviving.connect() as conn:
+ note = conn.execute(
+ text("SELECT note FROM gap_b_canary WHERE id = 1")
+ ).scalar()
+ assert note == "gap-B-det", "rekey must preserve data"
def test_rekey_epoch_refuses_stale_old_publish(manager):
- """Pin the publish-time epoch guard that closes the window
- eviction-at-close cannot.
-
- Interleave (from review at ``c02cf47c``):
- 1. ``change_password(old->new)`` cold-opens with old and caches E1;
- 2. an external unconditional close (logout / idle sweep) evicts E1;
- 3. a concurrent ``open_user_database(old)`` that missed the fast path
- cold-opens with the OLD key while the file is still old-keyed, and
- parks just before ``_cache_connection``;
- 4. the rekey lands; both conditional closes no-op on the emptied cache;
- 5. the parked open publishes E2 keyed OLD with a matching old verifier.
-
- Without the epoch, step 5 plants a stale engine that
- ``open_user_database(old)`` is then handed without a SQLCipher
- round-trip. With it, the parked publish snapshots the pre-rekey
- generation, the rekey bumps it, and the publish is refused (fail CLOSED).
+ """An engine published under a pre-rekey epoch is never handed out.
+
+ Interleave #6565 targeted (from review at ``c02cf47c``): an
+ ``open_user_database(old)`` verifies against the still-old-keyed file,
+ parks before publishing, the rekey lands, and the parked open then
+ plants an OLD-key engine with a matching old verifier. #6565 refused
+ that publish with an in-process integer generation compared at publish
+ time (``_get_rekey_epoch`` / ``_bump_rekey_epoch`` /
+ ``_cache_connection(expected_epoch=...)``), which only fenced rekeys
+ run by the same process. The durable rekey epoch supersedes it, and
+ those helpers no longer exist: the parked open cannot straddle the
+ rekey because it holds the sidecar lock from its epoch read to its
+ publish, and even an engine that does get cached under a superseded
+ epoch is never trusted, because every cache claim re-reads the durable
+ epoch under the sidecar and detaches a generation whose epoch is not
+ the committed one. This test plants exactly the stale state the
+ interleave would produce and asserts the old password still cannot
+ use it, the stale generation is dropped, and the new password works.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
+ from sqlalchemy import create_engine
+
username = f"chgpw_epoch_{uuid.uuid4().hex[:8]}"
old = "OldCorrectHorse1!" # noqa: S105
new = "NewCorrectHorse2!" # noqa: S105
@@ -957,70 +928,51 @@ def test_rekey_epoch_refuses_stale_old_publish(manager):
_write_canary(bootstrap, "gap-B-epoch")
manager.close_user_database(username)
del bootstrap
+ db_path = manager._get_user_db_path(username)
- # Parked-open simulation: snapshot the pre-rekey generation, exactly as
- # _open_user_database_cold does before its SQLCipher round-trip.
- stale_epoch = manager._get_rekey_epoch(username)
- assert stale_epoch == 0
+ # The parked open read this epoch before its SQLCipher round-trip.
+ stale_epoch = read_rekey_epoch(db_path)
+ assert stale_epoch is not None
assert manager.change_password(username, old, new) is True
- assert manager._get_rekey_epoch(username) == stale_epoch + 1
- # change_password evicted everything; cache empty (the external-close
- # shape from the interleave leaves the same empty cache).
+ current_epoch = read_rekey_epoch(db_path)
+ assert current_epoch not in (None, stale_epoch)
assert username not in manager.connections
- # Step 5: the parked OLD-key open tries to publish after the rekey.
- from sqlalchemy import create_engine
-
- parked = create_engine(
- f"sqlite:///{manager._get_user_db_path(username).as_posix()}"
- )
+ # The parked OLD-key open's publish lands after the rekey.
+ parked = create_engine(f"sqlite:///{db_path.as_posix()}")
try:
- published = manager._cache_connection(
- username, parked, old, expected_epoch=stale_epoch
+ assert (
+ manager._cache_connection(
+ username,
+ manager._make_connection_cache_entry(parked, old, stale_epoch),
+ )
+ is None
)
- assert published is False, (
- "a publish whose pre-verification epoch predates the completed "
- "rekey must be refused -- otherwise a stale OLD-key engine is "
- "cached after change_password"
+ assert manager.is_user_connected(username) is False, (
+ "a generation cached under a superseded epoch must not count as "
+ "connected"
)
assert username not in manager.connections, (
- "refused stale publish must leave the cache empty"
+ "the claim must detach the stale generation"
)
+ assert manager.open_user_database(username, old) is None
+ assert username not in manager.connections
assert username not in manager._password_verifiers
finally:
parked.dispose()
- # End-to-end: old is dead, new reads pre-change data.
- assert manager.open_user_database(username, old) is None
+ # End-to-end: the new password opens, reads pre-change data, and is
+ # published under the current epoch.
reopened = manager.open_user_database(username, new)
assert reopened is not None
+ assert manager._connection_epochs.get(username) == current_epoch
with reopened.connect() as conn:
note = conn.execute(
text("SELECT note FROM gap_b_canary WHERE id = 1")
).scalar()
assert note == "gap-B-epoch"
- # Symmetry: a publish snapshotting the CURRENT generation succeeds.
- current_epoch = manager._get_rekey_epoch(username)
- from sqlalchemy import create_engine as _create_engine
-
- fresh = _create_engine(
- f"sqlite:///{manager._get_user_db_path(username).as_posix()}"
- )
- try:
- assert (
- manager._cache_connection(
- username, fresh, new, expected_epoch=current_epoch
- )
- is True
- )
- assert manager.connections[username] is fresh
- finally:
- # Leave the manager clean for the fixture teardown.
- manager.close_user_database(username)
- fresh.dispose()
-
def test_verifier_matches_degrades_on_wrong_typed_members(manager):
"""Pin the widened malformed-verifier guard.
@@ -1028,8 +980,9 @@ def test_verifier_matches_degrades_on_wrong_typed_members(manager):
The unpack-only guard degraded ``"not-a-tuple"`` but a two-tuple with
wrong-typed members -- ``("x", "y")`` or ``(b"salt", 123)`` -- still
raised ``TypeError`` out of ``_verifier_matches`` (via ``salt + bytes``
- or ``hmac.compare_digest(bytes, int)``), which escapes through
- ``change_password``'s ``finally`` and masks the real rekey outcome.
+ or ``hmac.compare_digest(bytes, int)``), which would escape from the
+ login path that consulted the verifier instead of falling through to a
+ key-validating open.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
@@ -1113,85 +1066,86 @@ def test_engine_usable_rejects_mis_keyed_engine():
def test_open_cold_refuses_stale_publish_and_disposes(manager, monkeypatch):
- """Pin the epoch plumbing through ``_open_user_database_cold``.
-
- ``test_rekey_epoch_refuses_stale_old_publish`` drives the
- ``_cache_connection`` primitive directly. This test drives the real
- caller: a rekey landing between the pre-verification epoch snapshot
- and the cache publish must make ``_open_user_database_cold`` refuse
- the publish (return None, cache empty, engine disposed).
-
- A caller that forgets ``expected_epoch`` (unconditional publish via
- the ``None`` default) returns the engine and populates the cache
- here, so this test fails without the plumbing.
+ """A rekey cannot land between a cold open's verification and publish.
+
+ #6565 pinned this by bumping its in-process generation from inside
+ ``initialize_database`` and asserting that ``_open_user_database_cold``
+ refused the publish. With the durable epoch, the publish moved to
+ ``open_user_database`` (``_open_user_database_cold`` builds and
+ verifies the engine only), and the whole cold open runs under the
+ sidecar lock that ``change_password`` must hold for its rekey. So the
+ stronger property is pinned here with a REAL ``change_password``
+ started from inside the cold open: it must stay blocked until the
+ open has published, and must then detach and dispose that engine
+ before rekeying, leaving the old password dead and the old engine
+ unusable. (A writer that bypasses the sidecar and moves the epoch
+ mid-open is covered by
+ ``tests/database/test_rekey_serialization.py::``
+ ``test_post_open_epoch_recheck_that_cannot_vouch_fails_closed``.)
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
- import time as _time
-
- from sqlalchemy.engine import Engine as _Engine
-
import local_deep_research.database.initialize as _init_module
username = f"chgpw_cold_{uuid.uuid4().hex[:8]}"
- password = "ColdCorrectHorse1!" # noqa: S105
+ old = "ColdCorrectHorse1!" # noqa: S105
+ new = "ColdCorrectHorse2!" # noqa: S105
- bootstrap = manager.create_user_database(username, password)
+ bootstrap = manager.create_user_database(username, old)
_write_canary(bootstrap, "gap-B-cold")
manager.close_user_database(username)
del bootstrap
- assert manager._get_rekey_epoch(username) == 0
+ outcome = {}
+ errors = {}
+
+ def rekeyer():
+ try:
+ outcome["changed"] = manager.change_password(username, old, new)
+ except Exception as exc: # noqa: BLE001 - surfaced below
+ errors["rekeyer"] = exc
- # Simulate a rekey landing mid-cold-open: the snapshot in
- # _open_user_database_cold runs before this hook, the publish runs
- # after it, so the publish must observe the bumped generation.
+ rekey_thread = threading.Thread(target=rekeyer, name="gap-b-cold-rekey")
+ observed = {}
real_initialize = _init_module.initialize_database
- def _bump_then_initialize(engine):
- manager._bump_rekey_epoch(username)
+ def rekey_during_initialize(engine):
+ if not observed:
+ rekey_thread.start()
+ rekey_thread.join(timeout=0.3)
+ observed["rekey_blocked"] = rekey_thread.is_alive()
return real_initialize(engine)
monkeypatch.setattr(
- _init_module, "initialize_database", _bump_then_initialize
+ _init_module, "initialize_database", rekey_during_initialize
)
- real_dispose = _Engine.dispose
- dispose_calls: list[int] = []
-
- def _spy_dispose(self):
- dispose_calls.append(id(self))
- return real_dispose(self)
-
- monkeypatch.setattr(_Engine, "dispose", _spy_dispose)
- dispose_calls.clear()
-
- result = manager._open_user_database_cold(
- username, password, _time.perf_counter()
- )
- assert result is None, (
- "a cold-open straddling a rekey must be refused (fail CLOSED) "
- "-- without expected_epoch plumbing the stale publish succeeds"
- )
- assert username not in manager.connections, (
- "refused stale publish must leave the cache empty"
+ opened = manager.open_user_database(username, old)
+ rekey_thread.join(timeout=JOIN_TIMEOUT_S)
+ assert not rekey_thread.is_alive(), "change_password deadlocked"
+ assert not errors, f"change_password raised: {errors}"
+ assert observed == {"rekey_blocked": True}, (
+ "change_password must wait for the in-flight cold open to publish"
)
- assert username not in manager._password_verifiers, (
- "refused stale publish must not arm a verifier"
- )
- assert dispose_calls, (
- "the refused stale engine must be disposed, not leaked until GC"
+ assert opened is not None, (
+ "the cold open verified the then-current key and must publish"
)
+ assert outcome.get("changed") is True
- # Control without the mid-open rekey: the same cold-open publishes.
- monkeypatch.setattr(_init_module, "initialize_database", real_initialize)
- manager.close_user_database(username)
- control = manager._open_user_database_cold(
- username, password, _time.perf_counter()
+ # The rekey detached the engine the open published, so the old
+ # password neither finds it cached nor opens the rekeyed file.
+ assert manager.connections.get(username) is not opened
+ assert manager.open_user_database(username, old) is None
+ usable, detail = _engine_usable(opened)
+ assert not usable, (
+ "the engine published under the old key must be unusable after "
+ f"the rekey ({detail})"
)
- assert control is not None
- try:
- assert manager.connections.get(username) is control
- finally:
- manager.close_user_database(username)
+ reopened = manager.open_user_database(username, new)
+ assert reopened is not None
+ with reopened.connect() as conn:
+ note = conn.execute(
+ text("SELECT note FROM gap_b_canary WHERE id = 1")
+ ).scalar()
+ assert note == "gap-B-cold"
diff --git a/tests/security/test_db_connection_close_all_race.py b/tests/security/test_db_connection_close_all_race.py
index 7031330ee2..fbfea08c18 100644
--- a/tests/security/test_db_connection_close_all_race.py
+++ b/tests/security/test_db_connection_close_all_race.py
@@ -6,7 +6,7 @@ credential-verified open put it there". ``close_all_databases`` is the
shutdown-scale eviction: it clears ``connections`` and
``_password_verifiers`` wholesale, all under ``_connections_lock``
(``_init_locks`` is deliberately retained, like
-``_password_change_locks``, so an in-flight open keeps its lock). Racing
+``_rekey_locks``, so an in-flight open keeps its lock). Racing
it against open/close churn must never tear that model:
* O1 single user: a churn thread cycling open(correct)+close while a
diff --git a/tests/security/test_db_connection_create_safety.py b/tests/security/test_db_connection_create_safety.py
index 7b227f321f..7bf84fd99b 100644
--- a/tests/security/test_db_connection_create_safety.py
+++ b/tests/security/test_db_connection_create_safety.py
@@ -237,6 +237,10 @@ def test_orphaned_salt_recovery_creates_usable_database(manager):
stem + "-wal",
stem + "-shm",
stem + "-journal",
+ # Durable rekey-fencing sidecars (PR #6662): the owner-only stable
+ # lock inode and the committed epoch projection beside the db.
+ stem + ".rekey.lock",
+ stem + ".rekey.epoch",
}
assert artifacts <= allowed, (
f"unexpected artifacts after recovery: {artifacts}"
diff --git a/tests/security/test_db_connection_readonly_hostile_fs.py b/tests/security/test_db_connection_readonly_hostile_fs.py
index 42cdf77019..bd3e67d6d9 100644
--- a/tests/security/test_db_connection_readonly_hostile_fs.py
+++ b/tests/security/test_db_connection_readonly_hostile_fs.py
@@ -28,6 +28,9 @@ import pytest
from sqlalchemy import text
from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ DatabaseKeyCheckUnavailableError,
+)
# Module-level gate. Without a working SQLCipher build every test below
# is a no-op, and the dozens of in-body ``pytest.skip`` calls let a CI
@@ -222,6 +225,11 @@ def test_nonwritable_data_dir_refuses_open_and_restore_reopens(manager):
"""R2: data dir at 0o500 (no write) -- WAL sidecars cannot be created,
the creator fails, the open refuses cleanly; restoring 0o700 reopens.
+ The refusal is the typed ``DatabaseKeyCheckUnavailableError`` (a
+ ``RekeyOperationalError``: login answers 503 without a lockout strike),
+ never ``None``: ``None`` means "wrong password", and the password here
+ is correct -- only the directory is unusable.
+
Permissions are restored in ``finally`` so pytest's tmp_path cleanup
(which must delete files inside this dir) never breaks."""
if not manager.has_encryption:
@@ -236,11 +244,8 @@ def test_nonwritable_data_dir_refuses_open_and_restore_reopens(manager):
os.chmod(data_dir, 0o500)
try:
assert stat.S_IMODE(os.stat(data_dir).st_mode) == 0o500
- result = manager.open_user_database(username, password)
- assert result is None, (
- "open succeeded with a non-writable data directory -- WAL "
- "sidecar creation should have failed the creator"
- )
+ with pytest.raises(DatabaseKeyCheckUnavailableError):
+ manager.open_user_database(username, password)
_assert_nothing_published(manager, username)
finally:
os.chmod(data_dir, original_mode)
diff --git a/tests/security/test_db_connection_rekey_duel.py b/tests/security/test_db_connection_rekey_duel.py
index 36aebb9324..4e6292a5fb 100644
--- a/tests/security/test_db_connection_rekey_duel.py
+++ b/tests/security/test_db_connection_rekey_duel.py
@@ -93,8 +93,13 @@ def test_double_rekey_duel_existential_invariants(manager, run_id):
barrier = threading.Barrier(2)
errors = {}
outcomes = {}
+ contention = {}
pairing_violations = []
+ from local_deep_research.database.rekey_lock import (
+ RekeyLockContentionError,
+ )
+
def dueler(label, new_password):
try:
barrier.wait(timeout=BARRIER_TIMEOUT_S)
@@ -106,6 +111,13 @@ def test_double_rekey_duel_existential_invariants(manager, run_id):
verifiers = set(manager._password_verifiers)
if engines != verifiers:
pairing_violations.append(f"{label}: {engines} vs {verifiers}")
+ except RekeyLockContentionError as exc:
+ # The serialized-rekey contract: the loser is told loudly that
+ # another rekey lifecycle owns the lock (the web layer maps
+ # this to 409). It is a classified outcome, not an error --
+ # the pre-serialization duel could only distinguish loser by
+ # open(old) failing after the fact.
+ contention[label] = exc
except Exception as exc: # noqa: BLE001 - surfaced via `errors` below
errors[label] = exc
@@ -122,13 +134,12 @@ def test_double_rekey_duel_existential_invariants(manager, run_id):
assert not pairing_violations, (
f"engine/verifier pairing broke mid-duel: {pairing_violations}"
)
- # Race witness: both duelers must have recorded an outcome. The former
- # assertion here was `set(outcomes.values()) <= {True, False}`, which
- # holds for ANY bool-returning function and for an empty dict -- it did
- # not distinguish "the duel ran" from "neither thread got a slice".
- assert set(outcomes) == {"A", "B"}, (
- f"a dueling thread recorded no outcome: {outcomes} -- the duel did "
- "not actually run"
+ # Race witness: both duelers must have recorded a classified outcome
+ # (a boolean from change_password, or the typed contention error the
+ # serialized-rekey contract prescribes for the loser).
+ assert set(outcomes) | set(contention) == {"A", "B"}, (
+ f"a dueling thread recorded no outcome: outcomes={outcomes} "
+ f"contention={contention} -- the duel did not actually run"
)
reported_true = {label for label, changed in outcomes.items() if changed}
assert len(reported_true) <= 1, (
diff --git a/tests/security/test_db_connection_unencrypted_mode.py b/tests/security/test_db_connection_unencrypted_mode.py
index 47e582398e..2d1feae16c 100644
--- a/tests/security/test_db_connection_unencrypted_mode.py
+++ b/tests/security/test_db_connection_unencrypted_mode.py
@@ -120,10 +120,16 @@ def test_unencrypted_mode_full_flow_is_plaintext_and_passwordless(
is reopened
)
- # The cache trusts directly in this mode (documented short-circuit).
+ # The cache trusts directly in this mode (documented short-circuit);
+ # the epoch must match the durable projection for the trust to engage.
with unenc_manager._connections_lock:
+ current_epoch = unenc_manager._connection_epochs.get(username)
+ assert current_epoch is not None
assert (
- unenc_manager._cached_engine_trusted(username, "any-string") is True
+ unenc_manager._cached_engine_trusted(
+ username, "any-string", current_epoch
+ )
+ is True
)
# get_session works and reads data.
diff --git a/tests/security/test_db_connection_username_trust.py b/tests/security/test_db_connection_username_trust.py
index 074e2cf434..517558f430 100644
--- a/tests/security/test_db_connection_username_trust.py
+++ b/tests/security/test_db_connection_username_trust.py
@@ -59,7 +59,7 @@ ADVERSARIAL_USERNAMES = [
]
HASHED_FILENAME_RE = re.compile(
- r"^(ldr_user_[0-9a-f]{16}\.db)(\.salt|-wal|-shm|-journal)?$"
+ r"^(ldr_user_[0-9a-f]{16}\.db)(\.salt|-wal|-shm|-journal|\.rekey\.lock|\.rekey\.epoch|\.rekey\.pending)?$"
)
diff --git a/tests/security/test_injection_and_template_safety.py b/tests/security/test_injection_and_template_safety.py
index 076e517bbb..63da2a99d4 100644
--- a/tests/security/test_injection_and_template_safety.py
+++ b/tests/security/test_injection_and_template_safety.py
@@ -167,8 +167,13 @@ REVIEWED_RAW_SQL_MODULES = {
# settings. PRAGMA takes no bind parameters.
"database/sqlcipher_utils.py",
# check_database_integrity picks one of two literal PRAGMAs
- # (quick_check / integrity_check) on has_encryption; no request value
- # reaches the string. PRAGMA takes no bind parameters.
+ # (quick_check / integrity_check) on has_encryption, and
+ # _make_sqlcipher_connection sets busy_timeout via
+ # sqlcipher_busy_timeout_pragma() (an int() of the range-checked
+ # db_config.busy_timeout_ms env setting), and the gated WAL
+ # checkpoints via _gated_checkpoint_busy_timeout_pragma() (an int()
+ # of the lower of that setting and a 1000 ms constant); no request
+ # value reaches any of these strings. PRAGMA takes no bind parameters.
"database/encrypted_db.py",
# PRAGMA user_version from a module-level int constant
"journal_quality/db.py",
diff --git a/tests/security/test_login_cached_connection_password_extra.py b/tests/security/test_login_cached_connection_password_extra.py
index 8a86e919a0..6fd757141d 100644
--- a/tests/security/test_login_cached_connection_password_extra.py
+++ b/tests/security/test_login_cached_connection_password_extra.py
@@ -90,9 +90,9 @@ def test_phase2_rekey_reentry_reuses_cached_engine(manager, monkeypatch):
cold_passwords = []
original_cold = manager._open_user_database_cold
- def counting_cold(u, p, start):
+ def counting_cold(u, p, start, prederived_hex_key=None):
cold_passwords.append(p)
- return original_cold(u, p, start)
+ return original_cold(u, p, start, prederived_hex_key)
monkeypatch.setattr(manager, "_open_user_database_cold", counting_cold)
diff --git a/tests/security/test_metrics_hostile_input_fastapi.py b/tests/security/test_metrics_hostile_input_fastapi.py
index d7ca377296..5a8dc1f3ef 100644
--- a/tests/security/test_metrics_hostile_input_fastapi.py
+++ b/tests/security/test_metrics_hostile_input_fastapi.py
@@ -860,6 +860,97 @@ def test_journal_download_allowed_under_resolvable_public_scope(
assert resp.json()["success"] is True
+def test_journal_download_does_not_read_the_default_scope_during_a_sidecar_fault(
+ app, monkeypatch
+):
+ """A stored ``strict`` must never be replaced by the default scope.
+
+ The route reads ``policy.egress_scope`` through
+ ``get_settings_manager(username=...)``, which reaches the user's cached
+ engine through ``db_manager.get_session``. A rekey sidecar fault there
+ (an fd-exhaustion spike that starts after ``require_auth``'s probe
+ passed) used to come back as that method's None, which
+ ``get_settings_manager`` answers as "this user has no database" from
+ the defaults file: the scope read as the default ``adaptive`` and the
+ public download went ahead against the stored ``strict``. The request
+ must fail closed with the retryable 503 instead, and the stored scope
+ must still refuse the download once the fault clears.
+
+ The fault is injected at the real ``os.open`` of this user's sidecar,
+ only while the route reads the scope.
+ """
+ import errno
+
+ from local_deep_research.database import encrypted_db, rekey_lock
+ from local_deep_research.journal_quality import downloader
+ from local_deep_research.web.routers import metrics as metrics_routes
+
+ username = _user("egressfault")
+ client = _register_and_login(app, username)
+ _set_scope(username, "strict")
+ downloads = []
+
+ def record_download(force=False):
+ downloads.append(force)
+ return True, "SHOULD-NOT-RUN"
+
+ monkeypatch.setattr(downloader, "download_journal_data", record_download)
+ monkeypatch.setattr(
+ downloader, "get_download_state", lambda: {"counts": None}
+ )
+
+ user_db = encrypted_db.db_manager._get_user_db_path(username)
+ sidecar = os.fspath(user_db.with_name(f"{user_db.name}.rekey.lock"))
+ real_open = rekey_lock.os.open
+ spike = {"on": False, "hits": 0}
+
+ def emfile_during_spike(path, flags, *args, **kwargs):
+ if spike["on"] and os.fspath(path) == sidecar:
+ spike["hits"] += 1
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ real_resolve = metrics_routes._resolve_persisted_egress_scope
+
+ def resolve_during_spike(name):
+ # require_auth (and DatabaseMiddleware's probe before it) passed:
+ # the route body is running.
+ spike["on"] = True
+ try:
+ return real_resolve(name)
+ finally:
+ spike["on"] = False
+
+ monkeypatch.setattr(rekey_lock.os, "open", emfile_during_spike)
+ monkeypatch.setattr(
+ metrics_routes, "_resolve_persisted_egress_scope", resolve_during_spike
+ )
+ resp = client.post(DOWNLOAD, json={})
+
+ assert spike["hits"] >= 1, (
+ "the scope read never reached the sidecar: measuring the wrong branch"
+ )
+ assert downloads == [], (
+ "the public download ran during a sidecar fault, i.e. with the "
+ f"default scope instead of the stored 'strict': {resp.text[:300]}"
+ )
+ assert resp.status_code == 503, (
+ f"expected the retryable 503, got {resp.status_code}: {resp.text[:300]}"
+ )
+ assert resp.headers.get("retry-after") == "5"
+ assert resp.json() == {
+ "detail": "Your database is temporarily unavailable. "
+ "Please try again in a moment."
+ }
+ assert "errno" not in resp.text
+
+ # Once the fault clears, the stored scope is read again and refuses.
+ monkeypatch.setattr(rekey_lock.os, "open", real_open)
+ resp = client.post(DOWNLOAD, json={})
+ assert resp.status_code == 403, resp.text[:300]
+ assert downloads == []
+
+
def test_stale_unprotected_scope_with_gate_off_is_treated_as_adaptive(
app, monkeypatch
):
diff --git a/tests/security/test_password_leakage.py b/tests/security/test_password_leakage.py
index 8a15b4bf66..3f00a6b6c6 100644
--- a/tests/security/test_password_leakage.py
+++ b/tests/security/test_password_leakage.py
@@ -25,6 +25,8 @@ from urllib.parse import quote, quote_plus
import pytest
+from local_deep_research.database.rekey_lock import RekeyUnavailableError
+
_LEAKED_PASSWORD = "ldr-master-pw-SHOULD-NEVER-LEAK-987654321"
_LEAKED_OLD_PASSWORD = "ldr-old-master-pw-ALSO-NEVER-LEAK-123"
@@ -84,6 +86,9 @@ class TestEncryptedDBPasswordLeakage:
manager = DatabaseManager.__new__(DatabaseManager)
manager.has_encryption = True
manager.connections = {}
+ manager._password_verifiers = {}
+ manager._connection_epochs = {}
+ manager._rekey_locks = {}
manager._connections_lock = MagicMock()
manager._connections_lock.__enter__ = lambda *a: None
manager._connections_lock.__exit__ = lambda *a: None
@@ -102,14 +107,21 @@ class TestEncryptedDBPasswordLeakage:
f"sqlcipher rekey failed: old={_LEAKED_OLD_PASSWORD} "
f"new={_LEAKED_PASSWORD} target={fake_db}"
)
- manager.open_user_database = MagicMock(side_effect=exc)
+ manager._make_sqlcipher_connection = MagicMock(side_effect=exc)
+ manager.open_user_database = MagicMock(
+ side_effect=AssertionError("cached open path used during rekey")
+ )
with loguru_caplog_full.at_level("DEBUG"):
- result = manager.change_password(
- "alice", _LEAKED_OLD_PASSWORD, _LEAKED_PASSWORD
- )
+ with pytest.raises(RekeyUnavailableError) as raised:
+ manager.change_password(
+ "alice", _LEAKED_OLD_PASSWORD, _LEAKED_PASSWORD
+ )
- assert result is False
+ manager._make_sqlcipher_connection.assert_called_once_with(
+ fake_db, _LEAKED_OLD_PASSWORD
+ )
+ manager.open_user_database.assert_not_called()
_assert_no_leak(
loguru_caplog_full.text, _LEAKED_PASSWORD, "change_password new_pw"
)
@@ -118,8 +130,13 @@ class TestEncryptedDBPasswordLeakage:
_LEAKED_OLD_PASSWORD,
"change_password old_pw",
)
- assert "Failed to change password" in loguru_caplog_full.text, (
- "test did not exercise the except branch — check the fixtures"
+ _assert_no_leak(
+ str(raised.value), _LEAKED_PASSWORD, "change_password error"
+ )
+ _assert_no_leak(
+ str(raised.value),
+ _LEAKED_OLD_PASSWORD,
+ "change_password old_pw error",
)
def test_open_user_database_outer_catch_does_not_leak(
diff --git a/tests/security/test_password_verifier_primitive.py b/tests/security/test_password_verifier_primitive.py
index 3c8ece7ca2..81edad7ad7 100644
--- a/tests/security/test_password_verifier_primitive.py
+++ b/tests/security/test_password_verifier_primitive.py
@@ -14,30 +14,40 @@ green in environments without SQLCipher installed.
Each ``DatabaseManager`` under test is built via ``__new__`` so
``DatabaseManager.__init__`` (which probes for SQLCipher and can raise
``RuntimeError`` when it isn't installed and unencrypted mode isn't
-explicitly allowed) never runs. Only the handful of attributes the verifier
-methods actually touch are set: ``_password_verifiers``, ``_verifier_key``,
-``_connections_lock``, ``connections`` and ``has_encryption``.
+explicitly allowed) never runs. The fixture initializes the verifier, durable
+epoch, and isolated data-directory state those methods touch.
"""
import hmac
import secrets
import threading
-from unittest.mock import patch
+from pathlib import Path
+from unittest.mock import MagicMock, patch
import pytest
from local_deep_research.database.encrypted_db import DatabaseManager
+from local_deep_research.database.rekey_lock import (
+ acquire_rekey_lock,
+ read_or_initialize_rekey_epoch,
+)
-def make_manager(has_encryption: bool = True) -> DatabaseManager:
+def make_manager(
+ data_dir: Path, has_encryption: bool = True
+) -> DatabaseManager:
"""Build a bare ``DatabaseManager`` with only the verifier-related state
initialised, bypassing ``__init__`` (and therefore the SQLCipher probe)
entirely."""
mgr = DatabaseManager.__new__(DatabaseManager)
mgr.connections = {}
mgr._password_verifiers = {}
+ mgr._connection_epochs = {}
+ mgr._rekey_locks = {}
mgr._verifier_key = secrets.token_bytes(32)
mgr._connections_lock = threading.RLock()
+ mgr._data_dir_override = data_dir
+ mgr._initialized_data_dirs = set()
mgr.has_encryption = has_encryption
return mgr
@@ -47,8 +57,8 @@ def make_manager(has_encryption: bool = True) -> DatabaseManager:
# ---------------------------------------------------------------------------
-def test_verifier_matches_own_password_and_rejects_others():
- mgr = make_manager()
+def test_verifier_matches_own_password_and_rejects_others(tmp_path: Path):
+ mgr = make_manager(tmp_path)
username = "user_own_password"
password = "CorrectHorseBattery1!" # noqa: S105
@@ -69,8 +79,8 @@ def test_verifier_matches_own_password_and_rejects_others():
# ---------------------------------------------------------------------------
-def test_same_password_different_usernames_are_unlinkable():
- mgr = make_manager()
+def test_same_password_different_usernames_are_unlinkable(tmp_path: Path):
+ mgr = make_manager(tmp_path)
password = "SharedPassword1!" # noqa: S105
mgr._record_password_verifier("alice", password)
@@ -98,8 +108,8 @@ def test_same_password_different_usernames_are_unlinkable():
assert cross != digest_a
-def test_make_verifier_produces_fresh_salt_each_call():
- mgr = make_manager()
+def test_make_verifier_produces_fresh_salt_each_call(tmp_path: Path):
+ mgr = make_manager(tmp_path)
salt1, digest1 = mgr._make_verifier("SamePassword1!")
salt2, digest2 = mgr._make_verifier("SamePassword1!")
assert salt1 != salt2, "two calls must not reuse a salt"
@@ -132,8 +142,8 @@ UNUSUAL_PASSWORDS = {
}
-def test_unusual_passwords_each_match_only_their_own_username():
- mgr = make_manager()
+def test_unusual_passwords_each_match_only_their_own_username(tmp_path: Path):
+ mgr = make_manager(tmp_path)
for name, password in UNUSUAL_PASSWORDS.items():
mgr._record_password_verifier(f"user_{name}", password)
@@ -187,19 +197,25 @@ def test_unusual_passwords_each_match_only_their_own_username():
# ---------------------------------------------------------------------------
-def test_fail_closed_when_no_verifier_was_ever_recorded():
- mgr = make_manager()
+def test_fail_closed_when_no_verifier_was_ever_recorded(tmp_path: Path):
+ mgr = make_manager(tmp_path)
with mgr._connections_lock:
assert mgr._verifier_matches("never_registered", "anything") is False
assert mgr._verifier_matches("never_registered", "") is False
assert mgr._password_matches_cached("never_registered", "anything") is False
-def test_fail_closed_after_verifier_is_popped():
- mgr = make_manager()
+def test_fail_closed_after_verifier_is_popped(tmp_path: Path):
+ mgr = make_manager(tmp_path)
username = "user_popped"
password = "CorrectHorseBattery1!" # noqa: S105
+ db_path = mgr._get_user_db_path(username)
+ db_path.touch()
+ with acquire_rekey_lock(db_path):
+ epoch = read_or_initialize_rekey_epoch(db_path)
+ mgr.connections[username] = MagicMock()
mgr._record_password_verifier(username, password)
+ assert mgr._connection_epochs[username] == epoch
# Sanity: it matches before the pop.
assert mgr._password_matches_cached(username, password) is True
@@ -217,8 +233,10 @@ def test_fail_closed_after_verifier_is_popped():
# ---------------------------------------------------------------------------
-def test_rerecording_replaces_verifier_and_invalidates_old_password():
- mgr = make_manager()
+def test_rerecording_replaces_verifier_and_invalidates_old_password(
+ tmp_path: Path,
+):
+ mgr = make_manager(tmp_path)
username = "user_rerecord"
old_password = "OldPassword1!" # noqa: S105
new_password = "NewPassword2!" # noqa: S105
@@ -238,10 +256,10 @@ def test_rerecording_replaces_verifier_and_invalidates_old_password():
assert mgr._verifier_matches(username, old_password) is False
-def test_rerecording_same_password_still_rotates_salt():
+def test_rerecording_same_password_still_rotates_salt(tmp_path: Path):
"""Even re-recording the SAME password must not silently keep the old
salt/digest -- _record_password_verifier always calls _make_verifier."""
- mgr = make_manager()
+ mgr = make_manager(tmp_path)
username = "user_same_pw_rerecord"
password = "SamePassword1!" # noqa: S105
@@ -262,32 +280,65 @@ def test_rerecording_same_password_still_rotates_salt():
# ---------------------------------------------------------------------------
-def test_cached_engine_trusted_unencrypted_mode_ignores_password(): # DevSkim: ignore DS101155 - "trusted" is a test-name word; no TLS/certificate code here
- mgr = make_manager(has_encryption=False)
+def test_cached_engine_trusted_unencrypted_mode_ignores_password(
+ tmp_path: Path,
+): # DevSkim: ignore DS101155 - "trusted" is a test-name word; no TLS/certificate code here
+ mgr = make_manager(tmp_path, has_encryption=False)
username = "user_unencrypted"
+ current_epoch = "current-epoch"
+ mgr._connection_epochs[username] = current_epoch
+ mgr._connection_epochs["some-other-user"] = current_epoch
# No verifier is ever recorded in this test -- unencrypted mode must not
# need one.
with mgr._connections_lock:
- assert mgr._cached_engine_trusted(username, "any-password") is True
- assert mgr._cached_engine_trusted(username, "") is True
- assert mgr._cached_engine_trusted(username, "unencrypted-mode") is True
- assert mgr._cached_engine_trusted("some-other-user", "dummy") is True
+ assert (
+ mgr._cached_engine_trusted(username, "any-password", current_epoch)
+ is True
+ )
+ assert mgr._cached_engine_trusted(username, "", current_epoch) is True
+ assert (
+ mgr._cached_engine_trusted(
+ username, "unencrypted-mode", current_epoch
+ )
+ is True
+ )
+ assert (
+ mgr._cached_engine_trusted(
+ "some-other-user", "dummy", current_epoch
+ )
+ is True
+ )
-def test_cached_engine_trusted_encrypted_mode_defers_to_verifier():
- mgr = make_manager(has_encryption=True)
+def test_cached_engine_trusted_encrypted_mode_defers_to_verifier(
+ tmp_path: Path,
+):
+ mgr = make_manager(tmp_path, has_encryption=True)
username = "user_encrypted"
password = "CorrectHorseBattery1!" # noqa: S105
mgr._record_password_verifier(username, password)
+ current_epoch = "current-epoch"
+ mgr._connection_epochs[username] = current_epoch
with mgr._connections_lock:
- assert mgr._cached_engine_trusted(username, password) is True
- assert mgr._cached_engine_trusted(username, "wrong-password") is False
+ assert (
+ mgr._cached_engine_trusted(username, password, current_epoch)
+ is True
+ )
+ assert (
+ mgr._cached_engine_trusted(
+ username, "wrong-password", current_epoch
+ )
+ is False
+ )
# And fails closed once the verifier is gone, same as _verifier_matches.
mgr._password_verifiers.pop(username, None)
with mgr._connections_lock:
- assert mgr._cached_engine_trusted(username, password) is False
+ assert (
+ mgr._cached_engine_trusted(username, password, current_epoch)
+ is False
+ )
# ---------------------------------------------------------------------------
@@ -296,8 +347,8 @@ def test_cached_engine_trusted_encrypted_mode_defers_to_verifier():
# ---------------------------------------------------------------------------
-def test_verifier_matches_uses_hmac_compare_digest():
- mgr = make_manager()
+def test_verifier_matches_uses_hmac_compare_digest(tmp_path: Path):
+ mgr = make_manager(tmp_path)
username = "user_constant_time"
password = "CorrectHorseBattery1!" # noqa: S105
mgr._record_password_verifier(username, password)
@@ -326,13 +377,13 @@ def test_verifier_matches_uses_hmac_compare_digest():
assert mismatch_args[1] == expected_digest
-def test_verifier_matches_never_uses_plain_equality_shortcut():
+def test_verifier_matches_never_uses_plain_equality_shortcut(tmp_path: Path):
"""Guard against a regression to `computed == expected`: patch
hmac.compare_digest to always return False and confirm the verifier
result follows the patched function rather than the underlying bytes
equality (which would still be True for a genuinely matching password).
"""
- mgr = make_manager()
+ mgr = make_manager(tmp_path)
username = "user_forced_false"
password = "CorrectHorseBattery1!" # noqa: S105
mgr._record_password_verifier(username, password)
@@ -349,11 +400,13 @@ def test_verifier_matches_never_uses_plain_equality_shortcut():
@pytest.mark.parametrize("has_encryption", [True, False])
-def test_fresh_managers_do_not_share_verifier_keys(has_encryption):
+def test_fresh_managers_do_not_share_verifier_keys(
+ has_encryption, tmp_path: Path
+):
"""Sanity check that each manager gets its own random verifier key, so
a digest computed by one manager is meaningless to another."""
- mgr1 = make_manager(has_encryption=has_encryption)
- mgr2 = make_manager(has_encryption=has_encryption)
+ mgr1 = make_manager(tmp_path / "first", has_encryption=has_encryption)
+ mgr2 = make_manager(tmp_path / "second", has_encryption=has_encryption)
assert mgr1._verifier_key != mgr2._verifier_key
username = "user_cross_manager"
diff --git a/tests/security/test_sqlcipher_key_lifecycle.py b/tests/security/test_sqlcipher_key_lifecycle.py
index d0236ab794..64a6acd01b 100644
--- a/tests/security/test_sqlcipher_key_lifecycle.py
+++ b/tests/security/test_sqlcipher_key_lifecycle.py
@@ -57,6 +57,7 @@ from __future__ import annotations
import ast
import hashlib
import re
+import secrets
import threading
from datetime import timedelta
from importlib.util import find_spec
@@ -652,7 +653,11 @@ def _bare_manager(tmp_path) -> DatabaseManager:
mgr = DatabaseManager.__new__(DatabaseManager)
mgr.connections = {}
mgr._password_verifiers = {}
+ mgr._connection_epochs = {}
mgr._init_locks = {}
+ mgr._rekey_locks = {}
+ mgr._password_change_locks = {}
+ mgr._verifier_key = secrets.token_bytes(32)
mgr._connections_lock = threading.RLock()
mgr._initialized_data_dirs = set()
mgr._data_dir_override = None
@@ -719,59 +724,108 @@ class TestRekeyOnPasswordChange:
self, tmp_path, monkeypatch
):
"""POSITIVE CONTROL for the failure case below."""
- mgr, calls = self._arm(tmp_path, monkeypatch, raise_on_exit=False)
+ mgr, calls = self._arm(tmp_path, monkeypatch, raise_after_rekey=False)
assert mgr.change_password("u", "old", "new") is True
assert calls == ["new"]
def test_change_password_reports_false_after_the_rekey_already_landed(
self, tmp_path, monkeypatch
):
- """FINDING: ``change_password``'s ``except`` swallows failures that
- occur AFTER ``PRAGMA rekey`` has already succeeded, and returns the
- same ``False`` it uses for "wrong password" / "no such database".
-
- There is no compensating action: the file's key HAS changed, the
- session store still holds the OLD passphrase, and the auth database
- keeps no password hash (login IS an attempted decrypt). The route
- then maps this ``False`` onto "Current password is incorrect" with a
- 401 (pinned by the test below), so the user is told their old password
- was wrong — while the only passphrase that can now open their database
- is the new one they were told did not take effect.
-
- Narrow but not theoretical: the window is the ``with
- engine.connect()`` exit (rollback/close) plus the success log, between
- the rekey call and the ``return True``.
+ """FINDING (now FIXED by the serialized rekey lifecycle): a failure
+ AFTER ``PRAGMA rekey`` has already landed used to be swallowed into
+ the same ``False`` the route flashed as "Current password is
+ incorrect" — telling the user their old password was wrong while the
+ only passphrase that could open the database was the new one.
+
+ The staged lifecycle classifies this as INDETERMINATE: the pending
+ epoch was durably published before the physical rekey, so every
+ subsequent access is fenced until verified recovery, and the route
+ answers 503 instead of a wrong-password 401. This pins the typed
+ classification, not the old boolean.
"""
- mgr, calls = self._arm(tmp_path, monkeypatch, raise_on_exit=True)
+ from local_deep_research.database.rekey_lock import (
+ RekeyIndeterminateError,
+ )
+
+ mgr, calls = self._arm(tmp_path, monkeypatch, raise_after_rekey=True)
- result = mgr.change_password("u", "old", "new")
+ with pytest.raises(RekeyIndeterminateError):
+ mgr.change_password("u", "old", "new")
assert calls == ["new"], "the rekey must have actually been issued"
- assert result is False, (
- "change_password now distinguishes a post-rekey failure from a "
- "bad password — re-read the route's False branch"
- )
- @staticmethod
- def _arm(tmp_path, monkeypatch, *, raise_on_exit: bool):
- """Wire a manager whose rekey is observed and whose connection exit
- can be made to fail. No SQLCipher, no PBKDF2."""
+ class _PragmaConnection:
+ """Raw-connection stand-in answering the lifecycle's pragma probes."""
+
+ def __init__(self):
+ self.closed = False
+
+ def execute(self, sql, *args, **kwargs):
+ sql = sql.strip().upper()
+ if sql.startswith("PRAGMA SYNCHRONOUS ="):
+ return None
+ if sql.startswith("PRAGMA BUSY_TIMEOUT ="):
+ # The post-rekey checkpoint bounds its busy wait first.
+ return None
+ if sql == "PRAGMA SYNCHRONOUS":
+
+ class _R:
+ @staticmethod
+ def fetchone():
+ return (2,)
+
+ return _R()
+ if sql.startswith("PRAGMA WAL_CHECKPOINT"):
+
+ class _C:
+ @staticmethod
+ def fetchone():
+ return (0, 0, 0)
+
+ return _C()
+ if sql == "SELECT COUNT(*) FROM SQLITE_MASTER":
+
+ class _S:
+ @staticmethod
+ def fetchone():
+ return (1,)
+
+ return _S()
+ if sql.startswith("PRAGMA QUICK_CHECK"):
+ return [("ok",)]
+ if sql.startswith("PRAGMA CIPHER_INTEGRITY_CHECK"):
+ return []
+ raise AssertionError(f"unexpected probe: {sql}")
+
+ def commit(self):
+ pass
+
+ def close(self):
+ self.closed = True
+
+ @classmethod
+ def _arm(cls, tmp_path, monkeypatch, *, raise_after_rekey: bool):
+ """Wire a manager whose rekey is observed and whose post-rekey
+ phase can be made to fail. No SQLCipher, no PBKDF2 — raw
+ connections are pragma stand-ins, matching the staged lifecycle's
+ seams (_make_sqlcipher_connection + set_sqlcipher_rekey)."""
mgr = _bare_manager(tmp_path)
db_path = mgr._get_user_db_path("u")
db_path.write_bytes(b"")
-
- engine = _FakeEngine(raise_on_exit)
monkeypatch.setattr(
- mgr, "open_user_database", lambda username, password: engine
+ mgr,
+ "_make_sqlcipher_connection",
+ lambda path, password: cls._PragmaConnection(),
)
- monkeypatch.setattr(mgr, "close_user_database", lambda username: None)
calls: list[str] = []
- monkeypatch.setattr(
- encrypted_db,
- "set_sqlcipher_rekey",
- lambda conn, new_password, **kw: calls.append(new_password),
- )
+
+ def rekey(connection, new_password, *, db_path):
+ calls.append(new_password)
+ if raise_after_rekey:
+ raise OSError("simulated failure after the rekey landed")
+
+ monkeypatch.setattr(encrypted_db, "set_sqlcipher_rekey", rekey)
return mgr, calls
def test_route_reports_a_post_rekey_failure_as_a_wrong_password(self):
diff --git a/tests/web/auth/test_connection_cleanup_fallback.py b/tests/web/auth/test_connection_cleanup_fallback.py
index dca9189e74..8029e8f59e 100644
--- a/tests/web/auth/test_connection_cleanup_fallback.py
+++ b/tests/web/auth/test_connection_cleanup_fallback.py
@@ -5,8 +5,8 @@ and error-handling paths not reached by test_connection_cleanup.py.
Targets uncovered lines in src/local_deep_research/web/auth/connection_cleanup.py:
- 56-68: _count_open_fds fallback via resource.getrlimit + os.fstat loop
- 107->113: scheduler.is_running == False branch
-- 157-167: periodic pool dispose block (_last_dispose_time gate, engine.dispose
- loop with exception swallowed, disposed count log)
+- 157-167: periodic pool reset block (_last_dispose_time gate, delegation to
+ db.reset_connection_pools() with exception swallowed, reset count log)
- 179-182: engine.pool.checkedout() raising inside FD monitor
- 190-195: high-FD (>800) warning and outer FD-monitor exception handler
"""
@@ -171,18 +171,16 @@ class TestPeriodicPoolDispose:
self, _mock_research, _mock_sched, sm, db
):
# _last_dispose_time = 0 (from autouse fixture), so the interval
- # elapsed gate is satisfied and the dispose block runs.
- bad_engine = MagicMock()
- bad_engine.dispose.side_effect = RuntimeError("dispose failed")
- good_engine = MagicMock()
-
- db.connections = {"alice": bad_engine, "bob": good_engine}
+ # elapsed gate is satisfied and the dispose block runs. The sweep
+ # delegates to the fenced DatabaseManager.reset_connection_pools()
+ # (rekey-sidecar-aware checkpoint+dispose per engine; per-engine
+ # failures are swallowed and logged there, type name only).
+ db.reset_connection_pools.return_value = 2
cleanup_idle_connections(sm, db)
- # Both dispose calls attempted despite the first raising.
- bad_engine.dispose.assert_called_once()
- good_engine.dispose.assert_called_once()
+ # Delegated exactly once, and the count feeds the info log.
+ db.reset_connection_pools.assert_called_once_with()
# The timer was updated so a rapid second call would NOT re-dispose.
assert cc_module._last_dispose_time > 0
@@ -197,28 +195,24 @@ class TestPeriodicPoolDispose:
self, _mock_research, _mock_sched, sm, db
):
"""The pool-dispose workaround for ADR-0004's WAL/SHM handle leak
- depends on this loop succeeding. Pre-fix dispose() failures were
- logged at DEBUG, which hid silent drift if checkpoint/dispose
- repeatedly failed (disk pressure, lock starvation). Failures must
- surface at WARNING so an operator sees them, but only the
- exception TYPE NAME (no value) — the value can carry sensitive
- locals.
+ depends on this sweep succeeding. A whole-sweep failure must
+ surface at WARNING so an operator sees it, but only the exception
+ TYPE NAME (no value) — the value can carry sensitive locals — and
+ must not take the periodic scheduler down. Per-engine failure
+ logging (user + type name) lives inside
+ DatabaseManager.reset_connection_pools and is covered by its own
+ suite.
"""
- bad_engine = MagicMock()
- bad_engine.dispose.side_effect = RuntimeError(
+ db.reset_connection_pools.side_effect = RuntimeError(
"potentially-sensitive details"
)
- db.connections = {"alice": bad_engine}
with patch.object(cc_module.logger, "warning") as mock_warn:
+ # Must not raise: the scheduler calls this periodically.
cleanup_idle_connections(sm, db)
- # The failure path emitted a warning that names the user and the
- # exception TYPE, not the exception value.
warning_calls = [str(call) for call in mock_warn.call_args_list]
- assert any(
- "alice" in call and "RuntimeError" in call for call in warning_calls
- )
+ assert any("RuntimeError" in call for call in warning_calls)
# The exception's message text must not appear in any warning —
# only the type name is logged.
assert not any(
@@ -235,16 +229,27 @@ class TestPeriodicPoolDispose:
def test_skip_dispose_when_interval_not_elapsed(
self, _mock_research, _mock_sched, sm, db
):
- """If _last_dispose_time is recent, the dispose block is skipped."""
+ """If _last_dispose_time is recent, the pool-reset block is skipped.
+
+ The sweep no longer disposes engines itself; it delegates to
+ ``db.reset_connection_pools()`` (sidecar + lifecycle lock, WAL
+ checkpoint, dispose), so that call is what the interval gate must
+ suppress. Asserting ``engine.dispose`` would pass even with the
+ gate removed.
+ """
import time
cc_module._last_dispose_time = time.monotonic() # just now
+ before = cc_module._last_dispose_time
engine = MagicMock()
db.connections = {"alice": engine}
cleanup_idle_connections(sm, db)
+ db.reset_connection_pools.assert_not_called()
engine.dispose.assert_not_called()
+ # The timer was not advanced either.
+ assert cc_module._last_dispose_time == before
class TestFdMonitoringErrorPaths:
diff --git a/tests/web/dependencies/test_stale_session_cleanup.py b/tests/web/dependencies/test_stale_session_cleanup.py
index 0caf7d398f..189e458e35 100644
--- a/tests/web/dependencies/test_stale_session_cleanup.py
+++ b/tests/web/dependencies/test_stale_session_cleanup.py
@@ -15,9 +15,19 @@ those sessions are stale, not dead, and must survive. Clearing them would log
users out mid-session — which is why these tests assert both directions.
"""
+import errno
+from pathlib import Path
+
import pytest
from fastapi import HTTPException
+from local_deep_research.database.encrypted_db import UserConnectionStatus
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochUnreadableError,
+ RekeyLockContentionError,
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+)
from local_deep_research.web.dependencies import auth as auth_dep
@@ -29,13 +39,24 @@ class _FakeRequest:
class _FakeDBManager:
- def __init__(self, connected=False, has_encryption=True):
+ def __init__(self, connected=False, has_encryption=True, probe=None):
self._connected = connected
self.has_encryption = has_encryption
+ # What probe_user_connection reports once is_user_connected has said
+ # False. Defaults to the real manager's answer when no usable engine
+ # is cached; UNAVAILABLE models a kept engine behind a sidecar fault.
+ self._probe = probe or (
+ UserConnectionStatus.CONNECTED
+ if connected
+ else UserConnectionStatus.NOT_CONNECTED
+ )
def is_user_connected(self, username):
return self._connected
+ def probe_user_connection(self, username):
+ return self._probe
+
class _FakePasswordStore:
def __init__(self, password=None):
@@ -240,3 +261,243 @@ def test_no_username_raises_without_touching_session(monkeypatch):
assert exc.value.status_code == 401
assert exc.value.detail == "Authentication required"
+
+
+class _SidecarReleaseFaultDBManager(_FakeDBManager):
+ """``is_user_connected`` raising the one sidecar fault it lets escape.
+
+ The probe reports a sidecar that cannot be opened or locked, and an
+ epoch read that fails, as a plain False (``probe_user_connection``
+ tells those apart). A failure to RELEASE the sidecar after the claim
+ raises ``RekeySidecarIOError`` instead: a ``RekeyOperationalError``,
+ not an OSError, so it reaches the caller.
+ """
+
+ def is_user_connected(self, username):
+ raise RekeySidecarIOError(
+ Path("alice.db"), RekeySidecarOperation.UNLOCK, errno.EIO
+ )
+
+
+def _assert_retryable_503_keeping_the_session(exc, request, sessions):
+ assert exc.value.status_code == 503
+ assert exc.value.headers == {
+ "Retry-After": str(auth_dep.DATABASE_UNAVAILABLE_RETRY_AFTER_SECONDS)
+ }
+ assert request.session == {"username": "alice", "session_id": "s1"}, (
+ "a transient sidecar fault cleared a recoverable session"
+ )
+ assert sessions.destroyed == []
+
+
[email protected]_session_check
+def test_sidecar_release_fault_on_probe_answers_503_and_keeps_the_session(
+ monkeypatch, _live_server_session
+):
+ """A rekey-sidecar fault is not a verdict on the session.
+
+ Since the rekey-serialization work the probe takes the sidecar
+ (filesystem + blocking lock). Unhandled, its typed fault escaped
+ ``require_auth`` as a 500 on every request; degrading it to the 401
+ path instead let ``/`` (``end_rejected_session``) destroy the session
+ and its stored password, logging the user out after a transient
+ fault. It must be a retryable 503 that leaves the session alone.
+ """
+ monkeypatch.setattr(auth_dep, "db_manager", _SidecarReleaseFaultDBManager())
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+ request = _FakeRequest({"username": "alice", "session_id": "s1"})
+
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ _assert_retryable_503_keeping_the_session(
+ exc, request, _live_server_session
+ )
+
+
[email protected]_session_check
+def test_unverifiable_cached_engine_answers_503_and_keeps_the_session(
+ monkeypatch, _live_server_session
+):
+ """is_user_connected says False, the manager says the engine is kept.
+
+ That is a sidecar or epoch-read fault (an fd-exhaustion spike), not a
+ missing engine, so the request is retryable rather than a 401.
+ """
+ monkeypatch.setattr(
+ auth_dep,
+ "db_manager",
+ _FakeDBManager(connected=False, probe=UserConnectionStatus.UNAVAILABLE),
+ )
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+ request = _FakeRequest({"username": "alice", "session_id": "s1"})
+
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ _assert_retryable_503_keeping_the_session(
+ exc, request, _live_server_session
+ )
+
+
[email protected]_session_check
+ ("session_id", "password"),
+ [("s-revoked", "pw"), ("s1", None)],
+ ids=["revoked-server-side", "no-recovery-credential"],
+)
+def test_unverifiable_cached_engine_does_not_keep_a_dead_session(
+ monkeypatch, session_id, password
+):
+ """The 503 is only for a session that would otherwise be accepted."""
+ monkeypatch.setattr(
+ auth_dep,
+ "db_manager",
+ _FakeDBManager(connected=False, probe=UserConnectionStatus.UNAVAILABLE),
+ )
+ monkeypatch.setattr(
+ auth_dep,
+ "session_password_store",
+ _FakePasswordStore(password=password),
+ )
+ request = _FakeRequest({"username": "alice", "session_id": session_id})
+
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ assert exc.value.status_code == 401
+ assert request.session == {}
+
+
+def test_engine_found_by_the_second_probe_is_accepted(monkeypatch):
+ """The fault cleared, or another request reopened the database,
+ between the two probes. The later answer counts: the request is
+ served instead of 401ing into a logout at "/"."""
+ monkeypatch.setattr(
+ auth_dep,
+ "db_manager",
+ _FakeDBManager(connected=False, probe=UserConnectionStatus.CONNECTED),
+ )
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+ request = _FakeRequest({"username": "alice", "session_id": "s1"})
+
+ assert auth_dep.require_auth(request) == "alice"
+ assert request.session == {"username": "alice", "session_id": "s1"}
+
+
+class _ReopenFaultDBManager(_FakeDBManager):
+ """No engine is cached, and reopening the database meets ``fault``.
+
+ That is the shape of a cold open that cannot take the user's rekey
+ sidecar (another holder kept it past the wait bound, an fd-exhaustion
+ spike) or read its epoch, after the user's other session logged out or
+ the idle sweep closed the engine.
+ """
+
+ def __init__(self, fault):
+ super().__init__(connected=False)
+ self._fault = fault
+ self.opened = 0
+
+ def open_user_database(self, username, password):
+ self.opened += 1
+ raise self._fault
+
+
+_REOPEN_FAULTS = [
+ pytest.param(
+ lambda: RekeyLockContentionError(Path("alice.db.rekey.lock")),
+ id="sidecar-held-past-the-bound",
+ ),
+ pytest.param(
+ lambda: RekeySidecarIOError(
+ Path("alice.db"), RekeySidecarOperation.ACQUIRE, errno.EMFILE
+ ),
+ id="sidecar-emfile",
+ ),
+ pytest.param(
+ lambda: RekeyEpochUnreadableError(Path("alice.db"), errno.EIO),
+ id="epoch-unreadable",
+ ),
+]
+
+
[email protected]_session_check
[email protected]("make_fault", _REOPEN_FAULTS)
+def test_a_reopen_stopped_by_a_rekey_fault_answers_503_and_keeps_the_session(
+ monkeypatch, _live_server_session, make_fault
+):
+ """The missing engine is the reopen's fault, not the credential's.
+
+ ``ensure_user_database`` swallowed the fault, so ``require_auth`` saw
+ "not connected" with a recoverable session and answered 401, which
+ "/" turns into ``end_rejected_session``: a logout over a transient
+ fault. Recorded for the request, it is the retryable 503 instead.
+ """
+ manager = _ReopenFaultDBManager(make_fault())
+ monkeypatch.setattr(auth_dep, "db_manager", manager)
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+ request = _FakeRequest({"username": "alice", "session_id": "s1"})
+
+ with auth_dep.request_database_scope():
+ auth_dep.ensure_user_database(request)
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ assert manager.opened == 1, "the reopen was never attempted"
+ _assert_retryable_503_keeping_the_session(
+ exc, request, _live_server_session
+ )
+
+
[email protected]_session_check
+def test_a_reopen_fault_does_not_keep_a_revoked_session(monkeypatch):
+ """The 503 is only for a session that would otherwise be accepted."""
+ manager = _ReopenFaultDBManager(
+ RekeyLockContentionError(Path("alice.db.rekey.lock"))
+ )
+ monkeypatch.setattr(auth_dep, "db_manager", manager)
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+ request = _FakeRequest({"username": "alice", "session_id": "s-revoked"})
+
+ with auth_dep.request_database_scope():
+ auth_dep.ensure_user_database(request)
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ assert manager.opened == 1
+ assert exc.value.status_code == 401
+ assert request.session == {}
+
+
+def test_a_reopen_fault_stays_with_its_own_request(monkeypatch):
+ """The next request starts clean: its own reopen decides."""
+ manager = _ReopenFaultDBManager(
+ RekeyLockContentionError(Path("alice.db.rekey.lock"))
+ )
+ monkeypatch.setattr(auth_dep, "db_manager", manager)
+ monkeypatch.setattr(
+ auth_dep, "session_password_store", _FakePasswordStore(password="pw")
+ )
+
+ with auth_dep.request_database_scope():
+ auth_dep.ensure_user_database(
+ _FakeRequest({"username": "alice", "session_id": "s1"})
+ )
+
+ request = _FakeRequest({"username": "alice", "session_id": "s1"})
+ with auth_dep.request_database_scope():
+ with pytest.raises(HTTPException) as exc:
+ auth_dep.require_auth(request)
+
+ assert exc.value.status_code == 401
diff --git a/tests/web/routers/test_auth_disconnected_db_gate.py b/tests/web/routers/test_auth_disconnected_db_gate.py
index 080622dc60..7c39869f0c 100644
--- a/tests/web/routers/test_auth_disconnected_db_gate.py
+++ b/tests/web/routers/test_auth_disconnected_db_gate.py
@@ -151,9 +151,21 @@ def disconnected_db(monkeypatch):
database connection behind it is gone (e.g. evicted after idle timeout
or a server restart). This is the PoC from the regression report.
"""
- from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.encrypted_db import (
+ UserConnectionStatus,
+ db_manager,
+ )
monkeypatch.setattr(db_manager, "is_user_connected", lambda username: False)
+ # require_auth asks probe_user_connection to classify a negative answer
+ # (a kept engine behind a sidecar fault is a 503, not a 401) and takes
+ # its answer. The real probe would still find this test's live engine,
+ # so report what the real manager says once the connection is gone.
+ monkeypatch.setattr(
+ db_manager,
+ "probe_user_connection",
+ lambda username: UserConnectionStatus.NOT_CONNECTED,
+ )
class TestAuthCheckHonorsConnectivity:
diff --git a/tests/web/routers/test_auth_flow_gaps.py b/tests/web/routers/test_auth_flow_gaps.py
index c1e23fcbf6..7284df07ec 100644
--- a/tests/web/routers/test_auth_flow_gaps.py
+++ b/tests/web/routers/test_auth_flow_gaps.py
@@ -29,7 +29,10 @@ observe the real server-issued session_id without any mocking.
import base64
import contextlib
+import errno
import json
+import os
+import threading
import uuid
from unittest.mock import patch
@@ -103,6 +106,77 @@ def _session_dict(client: TestClient) -> dict:
return json.loads(base64.b64decode(padded))
+def _sidecar_flock_held_elsewhere(db_path):
+ """Hold the user's rekey sidecar flock from a second descriptor.
+
+ flock makes a second open file description contend with the server's
+ own descriptor exactly as another process's would, and it bypasses the
+ server's in-process gate, like a foreign holder (another worker's
+ password change of a large database, a stopped process).
+ """
+ import fcntl
+
+ fd = os.open(
+ db_path.with_name(f"{db_path.name}.rekey.lock"),
+ os.O_RDWR | os.O_CREAT,
+ 0o600,
+ )
+ try:
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ yield
+ finally:
+ os.close(fd)
+
+
+def _spy_sidecar_waits(monkeypatch, encrypted_db, db_path) -> list:
+ """Record the wait each request-path take of ``db_path``'s sidecar gets.
+
+ The cache claim, the cold open and the registration epoch take the
+ sidecar through ``encrypted_db.acquire_rekey_lock(..., timeout=...)``;
+ each entry is the ``timeout`` one of them was granted. A held sidecar
+ makes every such wait last its full timeout, so the sum bounds how long
+ the request waited for the holder without timing anything. Takes
+ without a timeout (disposals, password changes) and the log drain's own
+ probes are not request waits and are left out.
+ """
+ from pathlib import Path
+
+ real_acquire = encrypted_db.acquire_rekey_lock
+ granted: list = []
+
+ def acquire(path, **kwargs):
+ if (
+ Path(path) == db_path
+ and kwargs.get("timeout") is not None
+ and threading.current_thread().name != "log-queue-processor"
+ ):
+ granted.append(kwargs["timeout"])
+ return real_acquire(path, **kwargs)
+
+ monkeypatch.setattr(encrypted_db, "acquire_rekey_lock", acquire)
+ return granted
+
+
+def _run_bounded(call, timeout: float):
+ """Run ``call`` on a daemon thread and wait at most ``timeout`` seconds.
+
+ A request that hangs then fails the test instead of the CI job.
+ """
+ outcome: dict = {}
+
+ def target():
+ try:
+ outcome["value"] = call()
+ except BaseException as error: # noqa: BLE001 - handed to the test
+ outcome["error"] = error
+
+ worker = threading.Thread(target=target, daemon=True)
+ worker.start()
+ worker.join(timeout)
+ return worker, outcome
+
+
@pytest.fixture(scope="module")
def registered_user(app):
"""Register one real user for the whole module; yield (username, pw)."""
@@ -377,6 +451,146 @@ class TestChangePasswordDestroysOtherSessions:
assert post.headers.get("location", "").startswith("/auth/login")
+class TestChangePasswordIndeterminateOutcome:
+ """An indeterminate rekey may have activated the new key, so the route
+ must revoke every old-password session exactly as on success, and tell
+ the user to try the NEW password first. An unavailable rekey (old key
+ provably kept) keeps every session and the retry message.
+ """
+
+ def _two_sessions(self, app, user, pw):
+ from local_deep_research.database import temp_auth as temp_auth_module
+
+ client_a = _new_client(app)
+ assert _login(client_a, user, pw).status_code == 302
+ client_b = _new_client(app)
+ assert _login(client_b, user, pw).status_code == 302
+ session_b = _session_dict(client_b)
+ # Expire B's temp-auth token as a real session would after 10 s, so
+ # B cannot silently reopen the database with the old password.
+ token_b = session_b.get("temp_auth_token")
+ if token_b:
+ temp_auth_module.temp_auth_store.retrieve_auth(token_b)
+ assert client_b.get("/auth/check").status_code == 200
+ return client_a, client_b, session_b.get("session_id")
+
+ def _post_change(self, client, pw):
+ new_pw = "NewTestPassword456!" # noqa: S105
+ token = client.get("/auth/csrf-token").json().get("csrf_token", "")
+ return client.post(
+ "/auth/change-password",
+ data={
+ "current_password": pw,
+ "new_password": new_pw,
+ "confirm_password": new_pw,
+ "csrf_token": token,
+ },
+ follow_redirects=False,
+ )
+
+ def _forbid_backup_purge(self, monkeypatch):
+ class _NoBackupService:
+ def __init__(self, *_a, **_k):
+ raise AssertionError(
+ "no backup may be purged while neither key is proven"
+ )
+
+ monkeypatch.setattr(
+ "local_deep_research.database.backup.backup_service.BackupService",
+ _NoBackupService,
+ )
+
+ def test_indeterminate_change_revokes_every_session(
+ self, app, registered_user, monkeypatch
+ ):
+ from pathlib import Path
+
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.rekey_lock import (
+ RekeyIndeterminateError,
+ RekeyStage,
+ )
+ from local_deep_research.database.session_passwords import (
+ session_password_store,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+
+ user, pw = registered_user
+ client_a, client_b, sid_b = self._two_sessions(app, user, pw)
+ cleared: list = []
+ import local_deep_research.database.thread_local_session as tls
+
+ real_clear = tls.clear_user_credentials
+
+ def spy_clear(username):
+ cleared.append(username)
+ return real_clear(username)
+
+ monkeypatch.setattr(tls, "clear_user_credentials", spy_clear)
+
+ def indeterminate(*_a, **_k):
+ raise RekeyIndeterminateError(
+ Path("/nonexistent/user.db"),
+ RekeyStage.CANDIDATE_VERIFICATION,
+ None,
+ )
+
+ monkeypatch.setattr(db_manager, "change_password", indeterminate)
+ self._forbid_backup_purge(monkeypatch)
+
+ resp = self._post_change(client_a, pw)
+
+ assert resp.status_code == 302, resp.text[:300]
+ assert resp.headers.get("location") == "/auth/login"
+ # The other device's old-password session is revoked server-side
+ # and end to end.
+ assert session_manager.validate_session(sid_b) is None
+ assert session_password_store.get_session_password(user, sid_b) is None
+ assert client_b.get("/auth/check").status_code == 401
+ assert client_a.get("/auth/check").status_code == 401
+ assert cleared == [user]
+ # And the user is told the change may have landed.
+ page = client_a.get("/auth/login")
+ assert "may have completed" in page.text
+ assert "Try your NEW" in page.text
+ assert "could not be completed safely" not in page.text
+
+ def test_unavailable_change_keeps_sessions_and_retry_message(
+ self, app, registered_user, monkeypatch
+ ):
+ from pathlib import Path
+
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.rekey_lock import (
+ RekeyStage,
+ RekeyUnavailableError,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+
+ user, pw = registered_user
+ client_a, client_b, sid_b = self._two_sessions(app, user, pw)
+
+ def unavailable(*_a, **_k):
+ raise RekeyUnavailableError(
+ Path("/nonexistent/user.db"), RekeyStage.PHYSICAL_REKEY
+ )
+
+ monkeypatch.setattr(db_manager, "change_password", unavailable)
+ self._forbid_backup_purge(monkeypatch)
+
+ resp = self._post_change(client_a, pw)
+
+ assert resp.status_code == 503
+ assert "could not be completed safely" in resp.text
+ assert "may have completed" not in resp.text
+ assert session_manager.validate_session(sid_b) == user
+ assert client_b.get("/auth/check").status_code == 200
+
+
# ----------------------------------------------------------------------------
# Temp-auth token single use (HTTP layer)
# ----------------------------------------------------------------------------
@@ -608,3 +822,814 @@ class TestChangePasswordBackstop:
f"as a server error, got {resp.status_code}"
)
assert _REJECTED not in resp.text
+
+
+# ----------------------------------------------------------------------------
+# A rekey-sidecar fault is a retryable 503 that keeps the session
+# ----------------------------------------------------------------------------
+
+
+class TestAuthCheckSidecarFault:
+ """A fault on the connectivity probe is a 503, not a 500 or a 401.
+
+ Since the rekey-serialization work, ``is_user_connected`` takes the
+ cross-process rekey sidecar, and a failure to release it raises
+ ``RekeySidecarIOError`` (a ``RekeyOperationalError``, not an OSError)
+ straight at the caller. ``/auth/check`` is polled by XHR clients, so
+ an unhandled raise is a 500 on every poll for as long as the fault
+ lasts. A 401 is no better: the fault says nothing about the session,
+ and ``/`` turns a 401 into a logout. It must be a retryable 503 that
+ leaves the session usable once the fault clears.
+ """
+
+ def _logged_in_client(self, app, user: str, password: str):
+ # Same helper TestChangePasswordBackstop uses: this class was
+ # extracted without it, and the test 500'd on AttributeError
+ # before ever reaching the probe.
+ client = _new_client(app)
+ assert _login(client, user, password).status_code == 302
+ return client
+
+ def test_probe_sidecar_fault_answers_503_and_keeps_the_session(
+ self, app, registered_user
+ ):
+ from pathlib import Path
+
+ from local_deep_research.database.rekey_lock import (
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+ )
+
+ user, pw = registered_user
+ client = self._logged_in_client(app, user, pw)
+
+ with patch(
+ "local_deep_research.web.dependencies.auth.db_manager"
+ ) as mock_db:
+ mock_db.is_user_connected.side_effect = RekeySidecarIOError(
+ Path(f"{user}.db"), RekeySidecarOperation.UNLOCK, errno.EIO
+ )
+ resp = client.get("/auth/check")
+
+ assert mock_db.is_user_connected.called, (
+ "the request never reached the probe — measuring the wrong branch"
+ )
+ assert resp.status_code == 503, (
+ "a sidecar fault on the probe must be a retryable 503, "
+ f"got {resp.status_code}: {resp.text[:300]}"
+ )
+ assert resp.headers.get("retry-after") == "5"
+ assert "application/json" in resp.headers["content-type"]
+ assert _session_dict(client).get("username") == user
+
+ resp = client.get("/auth/check")
+ assert resp.status_code == 200, resp.text[:300]
+ assert resp.json() == {"authenticated": True, "username": user}
+
+
[email protected]_session_check
+class TestTransientSidecarFaultKeepsTheSession:
+ """An fd-exhaustion spike must not log a signed-in user out.
+
+ ``is_user_connected`` used to be a dictionary lookup, so a warm pool
+ kept serving through an fd spike. It now opens and flocks the rekey
+ sidecar. When that open fails the cached engine is kept, but the
+ request read as "not connected": ``require_auth`` answered 401, ``/``
+ turned that 401 into ``end_rejected_session`` (server-side session and
+ stored password destroyed, cookie cleared), and DatabaseMiddleware's
+ reopen with the stored password paid a full PBKDF2 derivation on
+ every request before failing on the same sidecar.
+
+ The fault is injected at the real ``os.open`` of the sidecar file, for
+ a user that really logged in: real SQLCipher engine, real server-side
+ session and stored password, and the real session check.
+ """
+
+ def test_sidecar_emfile_answers_503_and_the_session_survives(
+ self, app, registered_user, monkeypatch
+ ):
+ import os
+ from pathlib import Path
+
+ from local_deep_research.database import encrypted_db, rekey_lock
+ from local_deep_research.database.session_passwords import (
+ session_password_store,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+
+ user, pw = registered_user
+ client = _new_client(app)
+ assert _login(client, user, pw).status_code == 302
+ # One ordinary request consumes the post-login bootstrap token and
+ # stores this session's password, as the first page load does.
+ assert client.get("/auth/check").status_code == 200
+ session_id = _session_dict(client)["session_id"]
+ assert session_password_store.get_session_password(user, session_id)
+ engine = encrypted_db.db_manager.connections[user]
+
+ # Spy on key derivations for THIS user's database only, and fail only
+ # its sidecar: both seams are process-wide, and no other thread's
+ # database may count against (or be broken by) this test.
+ user_db = encrypted_db.db_manager._get_user_db_path(user)
+ sidecar = os.fspath(user_db.with_name(f"{user_db.name}.rekey.lock"))
+ real_derive = encrypted_db.get_key_from_password
+ derived = []
+
+ def derive(password, db_path=None):
+ if db_path is not None and Path(db_path) == user_db:
+ derived.append(db_path)
+ return real_derive(password, db_path=db_path)
+
+ monkeypatch.setattr(encrypted_db, "get_key_from_password", derive)
+ real_open = rekey_lock.os.open
+
+ def emfile_on_sidecar(path, flags, *args, **kwargs):
+ if os.fspath(path) == sidecar:
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ monkeypatch.setattr(rekey_lock.os, "open", emfile_on_sidecar)
+ try:
+ responses = {
+ path: client.get(
+ path,
+ headers={"Accept": "text/html"},
+ follow_redirects=False,
+ )
+ # The index (whose 401 handler ends the session), a page
+ # behind Depends(require_auth), and the XHR auth probe.
+ for path in ("/", "/auth/change-password", "/auth/check")
+ }
+ # A caller of "/" that asks for JSON.
+ json_index = client.get(
+ "/",
+ headers={"Accept": "application/json"},
+ follow_redirects=False,
+ )
+ finally:
+ monkeypatch.setattr(rekey_lock.os, "open", real_open)
+
+ for path, resp in responses.items():
+ assert resp.status_code == 503, (
+ f"{path} answered {resp.status_code} "
+ f"(location={resp.headers.get('location')!r}) during a "
+ "transient sidecar fault instead of a retryable 503"
+ )
+ assert resp.headers.get("retry-after") == "5", path
+ # A browser landing on "/" gets the branded error page, whose button
+ # retries "/", rather than a raw JSON body: /auth/login sends every
+ # signed-in cookie back to "/", so a lasting fault would otherwise
+ # leave the browser on that JSON. A JSON caller keeps the JSON 503.
+ index = responses["/"]
+ assert index.headers["content-type"].startswith("text/html"), (
+ index.text[:300]
+ )
+ assert 'data-status-code="503"' in index.text
+ assert 'href="/"' in index.text
+ assert json_index.status_code == 503, json_index.text[:300]
+ assert json_index.headers.get("retry-after") == "5"
+ assert json_index.json() == {
+ "detail": "Your database is temporarily unavailable. "
+ "Please try again in a moment."
+ }
+ # Nothing was torn down ...
+ assert session_manager.validate_session(session_id) == user
+ assert (
+ session_password_store.get_session_password(user, session_id) == pw
+ )
+ assert _session_dict(client).get("username") == user
+ assert _session_dict(client).get("session_id") == session_id
+ assert encrypted_db.db_manager.connections[user] is engine
+ # ... and no request paid for a key derivation it could not use.
+ assert derived == []
+
+ # Once the fault clears, the same session and engine carry on.
+ resp = client.get("/auth/check")
+ assert resp.status_code == 200, resp.text[:300]
+ assert resp.json() == {"authenticated": True, "username": user}
+ assert encrypted_db.db_manager.connections[user] is engine
+ assert derived == []
+
+ def test_spike_after_the_auth_probe_answers_503_and_the_session_survives(
+ self, app, registered_user, monkeypatch
+ ):
+ """The spike can start between two probes of the same request.
+
+ An fd-exhaustion spike fluctuates. ``require_auth``'s probe can
+ succeed and, a moment later, the index's own database session
+ (``get_user_db_session`` -> the thread-local session ->
+ ``open_user_database``'s claim) fail on the same sidecar. That
+ failure used to come back as the plain ``DatabaseSessionError`` a
+ wrong password produces, and the index's catch-all answered it by
+ clearing the cookie and redirecting to the login page, leaving the
+ server-side session and stored password orphaned. It must be the
+ same retryable 503 ``require_auth`` gives, with nothing torn down.
+ """
+ import os
+ from pathlib import Path
+
+ from local_deep_research.database import encrypted_db, rekey_lock
+ from local_deep_research.database.session_passwords import (
+ session_password_store,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+ from local_deep_research.web.dependencies import auth as auth_deps
+
+ user, pw = registered_user
+ client = _new_client(app)
+ assert _login(client, user, pw).status_code == 302
+ # One ordinary request consumes the post-login bootstrap token and
+ # stores this session's password, as the first page load does.
+ assert client.get("/auth/check").status_code == 200
+ session_id = _session_dict(client)["session_id"]
+ assert (
+ session_password_store.get_session_password(user, session_id) == pw
+ )
+ engine = encrypted_db.db_manager.connections[user]
+
+ # Both seams are process-wide: spy on key derivations for THIS
+ # user's database only, and fail only its sidecar.
+ user_db = encrypted_db.db_manager._get_user_db_path(user)
+ sidecar = os.fspath(user_db.with_name(f"{user_db.name}.rekey.lock"))
+ real_derive = encrypted_db.get_key_from_password
+ derived = []
+
+ def derive(password, db_path=None):
+ if db_path is not None and Path(db_path) == user_db:
+ derived.append(db_path)
+ return real_derive(password, db_path=db_path)
+
+ monkeypatch.setattr(encrypted_db, "get_key_from_password", derive)
+ real_open = rekey_lock.os.open
+ spike = {"on": False, "hits": 0}
+
+ def emfile_during_spike(path, flags, *args, **kwargs):
+ if spike["on"] and os.fspath(path) == sidecar:
+ spike["hits"] += 1
+ raise OSError(errno.EMFILE, "Too many open files")
+ return real_open(path, flags, *args, **kwargs)
+
+ # The index imports require_auth when it runs, so wrapping the module
+ # attribute starts the spike right after its probe has succeeded
+ # (DatabaseMiddleware's probe ran earlier still).
+ real_require_auth = auth_deps.require_auth
+ authenticated = []
+
+ def require_auth_then_spike(request):
+ username = real_require_auth(request)
+ authenticated.append(username)
+ spike["on"] = True
+ return username
+
+ monkeypatch.setattr(rekey_lock.os, "open", emfile_during_spike)
+ monkeypatch.setattr(auth_deps, "require_auth", require_auth_then_spike)
+ try:
+ resp = client.get(
+ "/", headers={"Accept": "text/html"}, follow_redirects=False
+ )
+ finally:
+ spike["on"] = False
+ monkeypatch.setattr(auth_deps, "require_auth", real_require_auth)
+ monkeypatch.setattr(rekey_lock.os, "open", real_open)
+
+ assert authenticated == [user], "require_auth's probe did not pass"
+ assert spike["hits"] >= 1, (
+ "the index's own session open never reached the sidecar: "
+ "measuring the wrong branch"
+ )
+ assert resp.status_code == 503, (
+ f"/ answered {resp.status_code} "
+ f"(location={resp.headers.get('location')!r}) when the spike hit "
+ "its session open after require_auth passed, instead of a "
+ "retryable 503"
+ )
+ assert resp.headers.get("retry-after") == "5"
+ assert "location" not in resp.headers
+ # Answered to this browser request as the branded error page that
+ # retries "/", not as a raw JSON body.
+ assert resp.headers["content-type"].startswith("text/html")
+ assert 'data-status-code="503"' in resp.text
+ # Nothing was torn down: cookie, server-side session, stored password
+ # and the cached engine all survive ...
+ assert _session_dict(client).get("username") == user
+ assert _session_dict(client).get("session_id") == session_id
+ assert session_manager.validate_session(session_id) == user
+ assert (
+ session_password_store.get_session_password(user, session_id) == pw
+ )
+ assert encrypted_db.db_manager.connections[user] is engine
+ # ... and the request failed before deriving a key it could not use.
+ assert derived == []
+
+ # Once the spike passes, the same session renders the page.
+ resp = client.get(
+ "/", headers={"Accept": "text/html"}, follow_redirects=False
+ )
+ assert resp.status_code == 200, resp.text[:300]
+ assert _session_dict(client).get("session_id") == session_id
+ assert encrypted_db.db_manager.connections[user] is engine
+ assert derived == []
+
+ @pytest.mark.skipif(os.name == "nt", reason="POSIX flock required")
+ def test_a_held_sidecar_answers_503_within_the_bound_and_keeps_the_session(
+ self, app, registered_user, monkeypatch
+ ):
+ """A holder that keeps the user's sidecar must not hang the request.
+
+ Every authenticated request claims the user's cached engine under
+ the rekey sidecar. A holder that keeps it (another worker's password
+ change of a large database, a stopped process, a hung mount) used to
+ hang each such request for as long as it held it: no timeout and no
+ 503. The claim now gives up after a bounded wait, and the request is
+ answered with the same retryable 503 as any other sidecar fault.
+
+ The bound is the request's, not each claim's. One request claims
+ the sidecar up to four times (DatabaseMiddleware's probe and its
+ reopen, then require_auth's probe and its classifying probe), and
+ each used to wait out the bound on its own: four bounds (about 20 s)
+ before a 503 that says ``Retry-After: 5``, on the shared threads
+ that serve every user. After the first wait runs out, the request's
+ later waits give up at once.
+ """
+ from local_deep_research.database import encrypted_db
+ from local_deep_research.database.session_passwords import (
+ session_password_store,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+
+ user, pw = registered_user
+ client = _new_client(app)
+ assert _login(client, user, pw).status_code == 302
+ assert client.get("/auth/check").status_code == 200
+ session_id = _session_dict(client)["session_id"]
+ engine = encrypted_db.db_manager.connections[user]
+ bound = 0.3
+ monkeypatch.setattr(
+ encrypted_db, "_SIDECAR_WAIT_TIMEOUT_SECONDS", bound, raising=False
+ )
+
+ user_db = encrypted_db.db_manager._get_user_db_path(user)
+ granted = _spy_sidecar_waits(monkeypatch, encrypted_db, user_db)
+ # The XHR probe, and the index, whose 401 handler ends the session.
+ for path in ("/auth/check", "/"):
+ granted.clear()
+ with _sidecar_flock_held_elsewhere(user_db):
+ worker, outcome = _run_bounded(
+ lambda path=path: client.get(
+ path,
+ headers={"Accept": "text/html"},
+ follow_redirects=False,
+ ),
+ timeout=20,
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=30)
+
+ assert finished, f"{path} waited for the sidecar holder"
+ assert "error" not in outcome, outcome
+ resp = outcome["value"]
+ assert resp.status_code == 503, (path, resp.text[:300])
+ assert resp.headers.get("retry-after") == "5", path
+ assert "location" not in resp.headers, path
+ # One bound for the whole request: the later claims did reach
+ # the sidecar, and were granted no wait of their own.
+ assert len(granted) >= 2, (path, granted)
+ assert 0 < granted[0] <= bound, (path, granted)
+ assert sum(granted) <= bound, (
+ f"{path} was granted {sum(granted):.2f} s of sidecar waits "
+ f"for one request against a {bound} s bound: {granted}"
+ )
+ # A retryable fault, not a verdict on the session: nothing torn
+ # down.
+ assert session_manager.validate_session(session_id) == user, path
+ assert (
+ session_password_store.get_session_password(user, session_id)
+ == pw
+ ), path
+ assert _session_dict(client).get("session_id") == session_id
+ assert encrypted_db.db_manager.connections[user] is engine
+
+ # Once the holder is gone, the same session and engine carry on.
+ resp = client.get("/auth/check")
+ assert resp.status_code == 200, resp.text[:300]
+ assert encrypted_db.db_manager.connections[user] is engine
+
+ @pytest.mark.skipif(os.name == "nt", reason="POSIX flock required")
+ def test_a_held_sidecar_with_no_cached_engine_answers_503_and_keeps_the_session(
+ self, app, registered_user, monkeypatch
+ ):
+ """A reopen stopped by a held sidecar is no verdict on the session.
+
+ With no engine cached (the user's other session logged out, the idle
+ sweep closed it), DatabaseMiddleware reopens the database with this
+ session's stored password. That cold open gives up on a held sidecar
+ with ``RekeyLockContentionError``, and ``ensure_user_database`` used
+ to swallow it: ``require_auth`` then saw "not connected" and
+ answered 401, and ``/`` turned that 401 into
+ ``end_rejected_session``, destroying the server-side session and
+ the stored password. It must be the retryable 503 with everything
+ kept, and the same session must reopen the database once the holder
+ is gone.
+ """
+ from local_deep_research.database import encrypted_db
+ from local_deep_research.database.session_passwords import (
+ session_password_store,
+ )
+ from local_deep_research.web.auth.session_manager import (
+ session_manager,
+ )
+
+ user, pw = registered_user
+ client = _new_client(app)
+ assert _login(client, user, pw).status_code == 302
+ # One ordinary request consumes the post-login bootstrap token and
+ # stores this session's password, as the first page load does.
+ assert client.get("/auth/check").status_code == 200
+ session_id = _session_dict(client)["session_id"]
+ assert (
+ session_password_store.get_session_password(user, session_id) == pw
+ )
+ encrypted_db.db_manager.close_user_database(user)
+ assert user not in encrypted_db.db_manager.connections
+ bound = 0.3
+ monkeypatch.setattr(
+ encrypted_db, "_SIDECAR_WAIT_TIMEOUT_SECONDS", bound, raising=False
+ )
+
+ user_db = encrypted_db.db_manager._get_user_db_path(user)
+ granted = _spy_sidecar_waits(monkeypatch, encrypted_db, user_db)
+ responses = {}
+ # The index (whose 401 handler ends the session), a page behind
+ # Depends(require_auth), the XHR probe, and "/" asked for JSON.
+ for path, accept in (
+ ("/", "text/html"),
+ ("/auth/change-password", "text/html"),
+ ("/auth/check", "text/html"),
+ ("/", "application/json"),
+ ):
+ granted.clear()
+ with _sidecar_flock_held_elsewhere(user_db):
+ worker, outcome = _run_bounded(
+ lambda path=path, accept=accept: client.get(
+ path,
+ headers={"Accept": accept},
+ follow_redirects=False,
+ ),
+ timeout=20,
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=30)
+
+ assert finished, f"{path} waited for the sidecar holder"
+ assert "error" not in outcome, outcome
+ resp = outcome["value"]
+ assert resp.status_code == 503, (
+ f"{path} ({accept}) answered {resp.status_code} "
+ f"(location={resp.headers.get('location')!r}) when the "
+ "reopen met a held sidecar, instead of a retryable 503"
+ )
+ assert resp.headers.get("retry-after") == "5", path
+ assert "location" not in resp.headers, path
+ # The reopen did reach the sidecar, and the request waited for
+ # it once.
+ assert granted, f"{path}: the reopen never took the sidecar"
+ assert sum(granted) <= bound, (path, granted)
+ responses[path, accept] = resp
+ # Nothing was torn down, and nothing was cached.
+ assert session_manager.validate_session(session_id) == user, path
+ assert (
+ session_password_store.get_session_password(user, session_id)
+ == pw
+ ), path
+ assert _session_dict(client).get("username") == user, path
+ assert _session_dict(client).get("session_id") == session_id
+ assert user not in encrypted_db.db_manager.connections
+
+ # A browser on "/" gets the branded page that retries "/"; a JSON
+ # caller gets the JSON 503.
+ index = responses["/", "text/html"]
+ assert index.headers["content-type"].startswith("text/html")
+ assert 'data-status-code="503"' in index.text
+ assert 'href="/"' in index.text
+ assert responses["/", "application/json"].json() == {
+ "detail": "Your database is temporarily unavailable. "
+ "Please try again in a moment."
+ }
+
+ # Once the holder is gone, the same session reopens the database
+ # with its stored password.
+ resp = client.get("/auth/check")
+ assert resp.status_code == 200, resp.text[:300]
+ assert resp.json() == {"authenticated": True, "username": user}
+ assert user in encrypted_db.db_manager.connections
+ assert _session_dict(client).get("session_id") == session_id
+
+ @pytest.mark.skipif(os.name == "nt", reason="POSIX flock required")
+ def test_login_while_the_sidecar_is_held_answers_503_without_a_lockout_strike(
+ self, app, registered_user, monkeypatch
+ ):
+ """The cold open's wait is bounded too, and is no password verdict.
+
+ A login opens the database cold under the same sidecar. It used to
+ hang for as long as the holder kept it; the open now gives up after
+ the bound with ``RekeyLockContentionError``, which the login route
+ answers with a 503 that does not count toward the account lockout:
+ the password was never checked.
+ """
+ from local_deep_research.database import encrypted_db
+ from local_deep_research.security.account_lockout import (
+ get_account_lockout_manager,
+ )
+
+ user, pw = registered_user
+ encrypted_db.db_manager.close_user_database(user)
+ assert user not in encrypted_db.db_manager.connections
+ monkeypatch.setattr(
+ encrypted_db, "_SIDECAR_WAIT_TIMEOUT_SECONDS", 0.3, raising=False
+ )
+ client = _new_client(app)
+ csrf_token = _csrf(client)
+ user_db = encrypted_db.db_manager._get_user_db_path(user)
+
+ with patch.object(
+ get_account_lockout_manager(), "record_failure"
+ ) as record_failure:
+ with _sidecar_flock_held_elsewhere(user_db):
+ worker, outcome = _run_bounded(
+ lambda: client.post(
+ "/auth/login",
+ data={
+ "username": user,
+ "password": pw,
+ "csrf_token": csrf_token,
+ },
+ follow_redirects=False,
+ ),
+ timeout=20,
+ )
+ finished = not worker.is_alive()
+ worker.join(timeout=30)
+
+ assert finished, "the login waited for the sidecar holder"
+ assert "error" not in outcome, outcome
+ resp = outcome["value"]
+ assert resp.status_code == 503, resp.text[:300]
+ record_failure.assert_not_called()
+ assert user not in encrypted_db.db_manager.connections
+
+ # Once the holder is gone, the same password logs in.
+ assert _login(client, user, pw).status_code == 302
+
+
+class TestLockedDatabaseIsNotAWrongPassword:
+ """A password check that met a locked database is no credential verdict."""
+
+ def test_login_against_a_locked_database_is_503_without_a_strike(
+ self, app, registered_user, monkeypatch
+ ):
+ from pathlib import Path
+
+ from local_deep_research.database.rekey_lock import (
+ DatabaseKeyCheckUnavailableError,
+ )
+ from local_deep_research.security.account_lockout import (
+ get_account_lockout_manager,
+ )
+ from local_deep_research.web.routers import auth as auth_router
+
+ user, pw = registered_user
+
+ def locked(*_a, **_k):
+ raise DatabaseKeyCheckUnavailableError(
+ Path("/nonexistent/user.db"), "OperationalError"
+ )
+
+ monkeypatch.setattr(
+ auth_router.db_manager, "open_user_database", locked
+ )
+ client = _new_client(app)
+ with patch.object(
+ get_account_lockout_manager(), "record_failure"
+ ) as record_failure:
+ resp = _login(client, user, pw)
+
+ assert resp.status_code == 503, resp.text[:300]
+ assert "Invalid username or password" not in resp.text
+ record_failure.assert_not_called()
+
+
+# ----------------------------------------------------------------------------
+# Change-password keeps a concurrent new-password engine (HTTP path)
+# ----------------------------------------------------------------------------
+
+
+class TestChangePasswordKeepsConcurrentNewPasswordEngine:
+ """The route must not evict the engine a racing new-password open cached.
+
+ ``DatabaseManager.change_password`` detaches the cached engine BEFORE
+ the rekey, under the rekey sidecar lock every cold open also takes, and
+ evicts nothing afterwards. The route used to follow a successful change
+ with an unconditional ``db_manager.close_user_database(username)``, so
+ an ``open_user_database(new_password)`` that published its engine after
+ the rekey committed (a fresh login with the new password, say) lost it
+ to the route and the next request had to cold-open again.
+
+ The real rekey runs here. The racing open is driven deterministically
+ from a wrapper around ``change_password``: it runs after the manager has
+ released the sidecar and returned True, and before the route's
+ post-change teardown, which is exactly where the old close landed.
+ """
+
+ def test_route_does_not_evict_the_new_password_engine(
+ self, app, monkeypatch
+ ):
+ from types import SimpleNamespace
+
+ from sqlalchemy import text
+
+ from local_deep_research.database.encrypted_db import db_manager
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
+
+ # A dedicated user: the real rekey below changes this account's
+ # password, so the module-scoped ``registered_user`` must not be it.
+ user = f"authgap_keep_{uuid.uuid4().hex[:8]}"
+ old_pw = "KeepEngineOld123!" # noqa: S105
+ new_pw = "KeepEngineNew456!" # noqa: S105
+ boot = _new_client(app)
+ reg = boot.post(
+ "/auth/register",
+ data={
+ "username": user,
+ "password": old_pw,
+ "confirm_password": old_pw,
+ "acknowledge": "true",
+ "csrf_token": _csrf(boot),
+ },
+ follow_redirects=False,
+ )
+ assert reg.status_code == 302, (
+ f"registration failed: {reg.status_code} {reg.text[:300]}"
+ )
+
+ client = _new_client(app)
+ assert _login(client, user, old_pw).status_code == 302
+
+ real_change_password = db_manager.change_password
+ published = {}
+
+ def _change_then_race(username, old, new):
+ result = real_change_password(username, old, new)
+ if result is True:
+ # The concurrent new-password open, landing after the
+ # manager released the sidecar.
+ published["engine"] = db_manager.open_user_database(
+ username, new
+ )
+ return result
+
+ monkeypatch.setattr(db_manager, "change_password", _change_then_race)
+
+ class _StubBackupService:
+ def __init__(self, *_a, **_k):
+ pass
+
+ def purge_and_refresh(self):
+ return SimpleNamespace(success=True, error=None)
+
+ monkeypatch.setattr(
+ "local_deep_research.database.backup.backup_service.BackupService",
+ _StubBackupService,
+ )
+
+ try:
+ token = client.get("/auth/csrf-token").json().get("csrf_token", "")
+ resp = client.post(
+ "/auth/change-password",
+ data={
+ "current_password": old_pw,
+ "new_password": new_pw,
+ "confirm_password": new_pw,
+ "csrf_token": token,
+ },
+ follow_redirects=False,
+ )
+ assert resp.status_code == 302, (
+ f"password change failed: {resp.status_code} {resp.text[:300]}"
+ )
+ assert resp.headers.get("location") == "/auth/login"
+
+ engine = published.get("engine")
+ assert engine is not None, (
+ "premise: the racing new-password open must have published "
+ "an engine, or nothing below measures the route"
+ )
+ assert db_manager.connections.get(user) is engine, (
+ "the password-change route evicted the engine a concurrent "
+ "new-password open had just cached"
+ )
+ assert db_manager.open_user_database(user, new_pw) is engine
+ with engine.connect() as conn:
+ conn.execute(
+ text("SELECT name FROM sqlite_master LIMIT 1")
+ ).all()
+ # And the rekey really happened: the old password is dead.
+ assert db_manager.open_user_database(user, old_pw) is None
+ finally:
+ db_manager.close_user_database(user)
+
+
+class TestChangePasswordContention:
+ def test_contended_change_answers_409_and_keeps_sessions(
+ self, app, registered_user, monkeypatch
+ ):
+ from pathlib import Path
+
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.rekey_lock import (
+ RekeyLockContentionError,
+ )
+
+ user, pw = registered_user
+ helper = TestChangePasswordIndeterminateOutcome()
+ client_a, client_b, _sid_b = helper._two_sessions(app, user, pw)
+
+ def contended(*_a, **_k):
+ raise RekeyLockContentionError(
+ Path("/nonexistent/user.db.rekey.lock")
+ )
+
+ monkeypatch.setattr(db_manager, "change_password", contended)
+ helper._forbid_backup_purge(monkeypatch)
+
+ resp = helper._post_change(client_a, pw)
+
+ assert resp.status_code == 409, resp.text[:300]
+ assert "already in progress" in resp.text
+ assert client_a.get("/auth/check").status_code == 200
+ assert client_b.get("/auth/check").status_code == 200
+
+
+class TestRevocationStepsAreIndependent:
+ """One failing revocation step must not skip the ones after it."""
+
+ def test_a_failing_early_step_does_not_skip_later_ones(self, monkeypatch):
+ from types import SimpleNamespace
+
+ import local_deep_research.database.session_passwords as sp
+ import local_deep_research.database.thread_local_session as tls
+ from local_deep_research.web.routers import auth as auth_router
+
+ calls: list = []
+
+ def failing_scheduler():
+ raise RuntimeError("scheduler down")
+
+ monkeypatch.setattr(
+ "local_deep_research.scheduler.background."
+ "get_background_job_scheduler",
+ failing_scheduler,
+ )
+
+ def failing_destroy(username):
+ calls.append(("destroy", username))
+ raise RuntimeError("session store down")
+
+ monkeypatch.setattr(
+ auth_router.session_manager,
+ "destroy_all_user_sessions",
+ failing_destroy,
+ )
+ monkeypatch.setattr(
+ sp.session_password_store,
+ "clear_all_for_user",
+ lambda username: calls.append(("passwords", username)),
+ )
+ monkeypatch.setattr(
+ tls,
+ "clear_user_credentials",
+ lambda username: calls.append(("thread", username)),
+ )
+ monkeypatch.setattr(
+ auth_router,
+ "_disconnect_user_sockets",
+ lambda username: calls.append(("sockets", username)),
+ )
+ request = SimpleNamespace(session={"username": "u", "session_id": "s"})
+
+ auth_router._revoke_user_credentials_after_rekey(request, "u")
+
+ assert calls == [
+ ("destroy", "u"),
+ ("passwords", "u"),
+ ("thread", "u"),
+ ("sockets", "u"),
+ ]
+ assert request.session == {}
diff --git a/tests/web/services/test_socketio_connect_gate.py b/tests/web/services/test_socketio_connect_gate.py
index 09e5e8937a..9bbb0706c8 100644
--- a/tests/web/services/test_socketio_connect_gate.py
+++ b/tests/web/services/test_socketio_connect_gate.py
@@ -10,10 +10,18 @@ session-cookie decoding and DB seams patched.
import asyncio
import threading
-from unittest.mock import Mock, patch
+from pathlib import Path
+from unittest.mock import AsyncMock, Mock, patch
import pytest
+from local_deep_research.database.rekey_lock import (
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+)
+from local_deep_research.web.dependencies.threadpool import (
+ run_db_sync as real_run_db_sync,
+)
from local_deep_research.web.services import socketio_asgi
from local_deep_research.web.services.socketio_asgi import connect
@@ -131,15 +139,32 @@ class TestConnectGate:
A cold open can create a thread-local DB session during phase-two rekey.
The connect path must sweep that session on the same pooled executor
worker before returning, rather than leave it attached for a later user.
+
+ Premise updated for the probe offload: the handshake now makes TWO
+ run_db_sync calls when it lazily opens — is_user_connected first, then
+ open_user_database — and run_db_sync sweeps its worker after EVERY
+ call, so a second sweep boundary (the probe's worker) is correct new
+ behavior, not a leak: the probe itself routes through the per-user
+ lifecycle lock and rekey sidecar and can leave thread-local state on
+ its worker. What this fence still pins, in call order: each offloaded
+ call sweeps on the very worker that ran it, and above all the open's
+ session is swept on the SAME thread that ran the open, before the
+ handshake returns. (Reverting run_db_sync to bare asyncio.to_thread
+ still fails: no sweep happens at all.)
"""
dbm = Mock()
- dbm.is_user_connected.return_value = False
+ probe_threads = []
open_threads = []
cleanup_threads = []
+ def record_probe_thread(username):
+ probe_threads.append(threading.get_ident())
+ return False
+
def record_open_thread(username, password):
open_threads.append(threading.get_ident())
+ dbm.is_user_connected.side_effect = record_probe_thread
dbm.open_user_database.side_effect = record_open_thread
with patch(
@@ -153,8 +178,15 @@ class TestConnectGate:
)
assert result is True
- assert cleanup_threads == open_threads
- assert cleanup_threads[0] != threading.get_ident()
+ # Probe first, then the open — and each sweep lands on the worker
+ # that ran its own call (executor may reuse one worker for both,
+ # hence list equality rather than a thread-uniqueness claim).
+ assert cleanup_threads == probe_threads + open_threads
+ # THE fence: the open's thread-local session is swept on the same
+ # executor worker that ran the open, before connect() returns.
+ assert cleanup_threads[-1] == open_threads[0]
+ # No sweep (and so no open) ever ran on the event-loop thread.
+ assert threading.get_ident() not in cleanup_threads
def test_lazy_db_open_failure_rejects_connection(self, sid_users):
dbm = Mock()
@@ -169,3 +201,70 @@ class TestConnectGate:
assert result is False
assert sid_users == {}
+
+
+def _sidecar_io_error() -> RekeySidecarIOError:
+ """A fault the rekey sidecar can really produce (fd exhaustion)."""
+ return RekeySidecarIOError(
+ Path("alice.db"), RekeySidecarOperation.ACQUIRE, 23
+ )
+
+
+class TestConnectProbeOffloadAndGuard:
+ """The is_user_connected probe must not run on the event loop.
+
+ Since the rekey-serialization work, the probe routes through
+ ``_acquire_current_connection_generation``: per-user lifecycle lock +
+ cross-process sidecar (filesystem I/O, blocking acquire). Called
+ directly inside ``async def connect`` — a single-worker deployment —
+ a rekey in progress would stall every user's sockets and async
+ routes for the rekey's duration, and a sidecar fault (not an OSError)
+ would raise straight out of the handshake. The probe therefore has to
+ travel through ``run_db_sync`` like the adjacent open_user_database
+ call, and a ``RekeyOperationalError`` has to degrade to "not
+ connected" (fail closed) rather than escape.
+ """
+
+ def test_probe_runs_off_the_event_loop_via_run_db_sync(self, sid_users):
+ dbm = Mock()
+ dbm.is_user_connected.return_value = True
+
+ spy = AsyncMock(wraps=real_run_db_sync)
+ with patch(
+ "local_deep_research.web.dependencies.threadpool.run_db_sync",
+ spy,
+ ):
+ result, _ = _connect(
+ session_data={"username": "alice", "session_id": "s1"},
+ db_manager=dbm,
+ )
+
+ assert result is True
+ # Connected, so the probe is the ONLY run_db_sync call the
+ # handshake makes — one await whose first arg is the probe.
+ spy.assert_awaited_once()
+ fn, arg = spy.await_args.args
+ # Identity, not __name__: production hands run_db_sync the
+ # manager's ``is_user_connected`` bound method unwrapped, but this
+ # test stands the manager in as a Mock, and a Mock child has no
+ # ``__name__`` (Mock refuses dunder attribute auto-creation) —
+ # so a name assertion can only ever see ''. Asserting identity
+ # is the mock analog of asserting the real bound method was
+ # passed, and also catches a wrapped/partial probe.
+ assert fn is dbm.is_user_connected
+ assert arg == "alice"
+
+ def test_sidecar_fault_on_probe_rejects_instead_of_raising(self, sid_users):
+ """A raising probe must fail closed through the no-connection
+ path, not blow up the handshake (or the event loop)."""
+ dbm = Mock()
+ dbm.is_user_connected.side_effect = _sidecar_io_error()
+
+ result, _ = _connect(
+ session_data={"username": "alice", "session_id": "s1"},
+ db_manager=dbm,
+ password=None,
+ )
+
+ assert result is False
+ assert sid_users == {}
diff --git a/tests/web/test_encoding_and_non_ascii.py b/tests/web/test_encoding_and_non_ascii.py
index 92f4baf62d..def87affc3 100644
--- a/tests/web/test_encoding_and_non_ascii.py
+++ b/tests/web/test_encoding_and_non_ascii.py
@@ -345,6 +345,12 @@ _DOES_NOT_CATCH_UNICODE_DECODE_ERROR = frozenset(
"RuntimeError",
"RecursionError",
"OSError",
+ # database/rekey_lock.py: ``class RekeyOperationalError(Exception)``,
+ # a direct Exception subclass, so not in UnicodeDecodeError's MRO.
+ # metrics.py's api_journal_data_download re-raises it from its outer
+ # ``try`` for the app's retryable 503; the body parse inside keeps
+ # its own ``except Exception`` and still answers 400.
+ "RekeyOperationalError",
}
)
diff --git a/tests/web/test_ensure_user_database_token_ordering.py b/tests/web/test_ensure_user_database_token_ordering.py
index de50e63c63..3275bd6e92 100644
--- a/tests/web/test_ensure_user_database_token_ordering.py
+++ b/tests/web/test_ensure_user_database_token_ordering.py
@@ -65,7 +65,9 @@ def _fake_request(session: dict):
class TestTempAuthTokenConsumedBeforeFastPath:
- def test_token_is_consumed_even_when_connection_already_open(self):
+ def test_token_is_consumed_even_when_connection_already_open(
+ self, tmp_path, monkeypatch
+ ):
"""The temp_auth_token must be consumed (removed from
temp_auth_store, popped from the session, promoted into
session_password_store) even when
@@ -84,16 +86,36 @@ class TestTempAuthTokenConsumedBeforeFastPath:
username = f"tokorder_{uuid.uuid4().hex[:10]}"
session_id = f"sess_{uuid.uuid4().hex[:10]}"
password = "correct-horse-battery-staple" # noqa: S105
- token = temp_auth_store.store_auth(username, password)
-
- # Simulate "the connection is already open" -- true for every
- # request after login. is_user_connected() only checks dict
- # membership (encrypted_db.py), so a sentinel is enough; nothing
- # else on the path under test touches the stored value.
- with db_manager._connections_lock:
- db_manager.connections[username] = object()
+ # Make "the connection is already open" REAL -- true for every
+ # request after login. is_user_connected() is no longer a dict
+ # membership check: it claims the cached generation, re-reads the
+ # durable rekey epoch beside the database file and detaches any
+ # entry whose recorded epoch is not the committed one. A planted
+ # sentinel (the old ``connections[username] = object()``) has no
+ # epoch file behind it, is detached on that first claim, and turns
+ # the fast path False -- after which the token block runs whatever
+ # the ordering is, and this test passes vacuously. So open the
+ # connection through the real create path (isolated data dir, the
+ # sanctioned low-KDF test knob) and assert the premise below.
+ monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")
+ # Save/restore the RAW override: ``monkeypatch.setattr(db_manager,
+ # "data_dir", ...)`` would, on undo, write the getter's resolved
+ # path back through the setter and pin ``_data_dir_override`` for
+ # the rest of the worker (same as tests/connected/conftest.py).
+ original_override = db_manager._data_dir_override
+ db_manager._data_dir_override = tmp_path
+ token = None
try:
+ db_manager.create_user_database(username, password)
+ # Store AFTER the slow create (SQLCipher + migrations) so the
+ # 10 s token TTL cannot expire on a loaded CI runner.
+ token = temp_auth_store.store_auth(username, password)
+ assert db_manager.is_user_connected(username) is True, (
+ "premise: the fast path must be live for this user, or "
+ "the token block runs regardless of its ordering and "
+ "nothing below pins it"
+ )
fake_session = {
"username": username,
"session_id": session_id,
@@ -125,10 +147,11 @@ class TestTempAuthTokenConsumedBeforeFastPath:
"written, meaning the token block never ran"
)
finally:
- with db_manager._connections_lock:
- db_manager.connections.pop(username, None)
+ db_manager.close_user_database(username)
session_password_store.clear_session(username, session_id)
# Idempotent no-op in the passing case (ensure_user_database
# already consumed it via retrieve_auth); cleans up after a
# failed run that left the token live.
- temp_auth_store.retrieve_auth(token)
+ if token is not None:
+ temp_auth_store.retrieve_auth(token)
+ db_manager._data_dir_override = original_override
diff --git a/tests/web/test_exception_handler_contract.py b/tests/web/test_exception_handler_contract.py
index 3eab1e9b06..60b134e845 100644
--- a/tests/web/test_exception_handler_contract.py
+++ b/tests/web/test_exception_handler_contract.py
@@ -526,3 +526,90 @@ class TestApiV1ErrorEnvelope:
assert resp.status_code == 405
assert resp.json() == {"detail": "Method Not Allowed"}
+
+
+# ---------------------------------------------------------------------------
+# A rekey operational fault left unhandled -> retryable 503
+# ---------------------------------------------------------------------------
+
+
+def _session_unavailable():
+ # Imported when the route runs, so this module still collects against
+ # a tree that predates the subclass (the fail-on-old run).
+ from local_deep_research.database.session_context import (
+ DatabaseSessionUnavailableError,
+ )
+
+ return DatabaseSessionUnavailableError("RekeySidecarIOError")
+
+
+def _raw_sidecar_fault():
+ import errno
+ from pathlib import Path
+
+ from local_deep_research.database.rekey_lock import (
+ RekeySidecarIOError,
+ RekeySidecarOperation,
+ )
+
+ return RekeySidecarIOError(
+ Path(f"{SECRET_MARKER}.db"), RekeySidecarOperation.OPEN, errno.EMFILE
+ )
+
+
[email protected](scope="module")
+def database_fault_client():
+ """Routes that leave a rekey operational fault unhandled.
+
+ ``get_user_db_session`` raises ``DatabaseSessionUnavailableError`` when
+ the database cannot be opened or claimed because of one (an
+ fd-exhaustion spike on the rekey sidecar, say), and a direct
+ ``db_manager`` call raises the ``RekeyOperationalError`` itself.
+ """
+ mini = FastAPI()
+ _register_exception_handlers(mini)
+ faults = {"session": _session_unavailable, "raw": _raw_sidecar_fault}
+
+ @mini.get("/page/{kind}")
+ def page(kind: str):
+ raise faults[kind]()
+
+ @mini.get("/api/v1/data/{kind}")
+ def api_v1_data(kind: str):
+ raise faults[kind]()
+
+ return TestClient(mini, raise_server_exceptions=False)
+
+
+class TestDatabaseUnavailableOverHTTP:
+ """Not a verdict on the session, so not the catch-all's 500 either.
+
+ Same retryable 503 and ``Retry-After`` that ``require_auth`` answers
+ when its own probe meets the fault, through the HTTPException handler
+ (so ``/api/v1`` keeps its ``error`` key). The fault's own text (it names
+ the database file and the errno) never reaches the body.
+ """
+
+ DETAIL = "Your database is temporarily unavailable. Please try again in a moment."
+
+ @pytest.mark.parametrize("kind", ["session", "raw"])
+ def test_page_answers_retryable_503(self, database_fault_client, kind):
+ resp = database_fault_client.get(
+ f"/page/{kind}", headers={"Accept": "text/html"}
+ )
+
+ assert resp.status_code == 503, resp.text[:300]
+ assert resp.headers["retry-after"] == "5"
+ assert resp.json() == {"detail": self.DETAIL}
+ _assert_scrubbed(resp)
+ assert "errno" not in resp.text
+
+ @pytest.mark.parametrize("kind", ["session", "raw"])
+ def test_api_v1_keeps_its_error_envelope(self, database_fault_client, kind):
+ resp = database_fault_client.get(
+ f"/api/v1/data/{kind}", headers={"Accept": "application/json"}
+ )
+
+ assert resp.status_code == 503, resp.text[:300]
+ assert resp.headers["retry-after"] == "5"
+ assert resp.json() == {"error": self.DETAIL, "detail": self.DETAIL}
diff --git a/tests/web/test_route_table_parity.py b/tests/web/test_route_table_parity.py
index b2b8a567a0..f6f5b169cf 100644
--- a/tests/web/test_route_table_parity.py
+++ b/tests/web/test_route_table_parity.py
@@ -1709,6 +1709,15 @@ EXPECTED_STATUS_CODES_GAINED = {
# Counterpart of the 302 entry above; same single behaviour.
("GET", "/auth/change-password"): [401],
("GET", "/news/health"): [401],
+ # The serialized rekey lifecycle (database/rekey_lock.py) maps typed
+ # operational failures to 503 so they do not masquerade as a form error
+ # or a 500. The 503 covers a retryable failure before anything was
+ # rekeyed (RekeyUnavailableError) and durable rekey state that needs the
+ # verified recovery path first (RekeyRecoveryRequiredError). An outcome
+ # left indeterminate AFTER the rekey ran (RekeyIndeterminateError)
+ # revokes every session and redirects to login like a success instead.
+ # web/routers/auth.py:change_password.
+ ("POST", "/auth/change-password"): [503],
# Intentional: an unimplemented endpoint answers 501 rather than letting
# a bare `except Exception` turn NotImplementedException into a 500.
("GET", "/news/api/categories"): [501],