Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

docs: record rekey lock domains and acquisition order (#7721)

Chanhyo Jung93af35d39b33
diff --git a/docs/decisions/0015-rekey-lock-architecture.md b/docs/decisions/0015-rekey-lock-architecture.md
new file mode 100644
index 0000000000..9979b84c74
--- /dev/null
+++ b/docs/decisions/0015-rekey-lock-architecture.md
@@ -0,0 +1,221 @@
+# ADR-0015: Rekey locks and connection generation publication
+
+**Date:** 2026-10-20 (Asia/Seoul)
+**Status:** Descriptive record of existing implementation
+
+## Context
+
+Password changes, cold opens and cached connection claims share a user's
+SQLCipher database across threads and server processes. Their exclusion
+boundaries differ. This record documents the implementation at
+`683a967df20fe98ec0ddc99668041e22b95ff78b`; it does not adopt a new lock
+policy or assert maintainer approval of a new design. The
+[rekey recovery guide](../security/rekey-recovery.md) remains the operational
+reference. [ADR-0004](0004-nullpool-for-sqlcipher.md) describes engine pooling;
+[ADR-0009](0009-per-user-isolation-boundary.md) describes user isolation.
+
+## Decision
+
+Record the following path-specific ownership and publication rules. Each
+path describes acquisition and release separately; these sequences do not
+define one universal chain of all application locks.
+
+### Lock map
+
+These are five named in-process domains in the manager/rekey boundary,
+not every application lock. The declarations and lookup helpers are in
+[DatabaseManager](../../src/local_deep_research/database/encrypted_db.py)
+(`__init__`, `_get_init_lock`, `_get_rekey_lock`,
+`_get_password_change_lock`) and
+[rekey_lock.py](../../src/local_deep_research/database/rekey_lock.py)
+(`_get_sidecar_gate`).
+
+| Domain | Key and scope | Role and lifetime |
+| --- | --- | --- |
+| `_init_locks` | Username; one `Lock` dictionary per manager | Serializes creation and cold-open construction/migration. Entries are created lazily and retained across close/reset so callers already holding a reference cannot race a replacement lock. |
+| `_rekey_locks` | Username; one `Lock` dictionary per manager | Lifecycle exclusion for cached-generation claims and disposal/reset. Entries are lazy and retained across close/reset. The name does not mean every password change takes this lock. |
+| `_password_change_locks` | Username; a separate `Lock` dictionary per manager | Nonblocking outer gate for concurrent `change_password` calls. Entries are lazy and retained; releasing a call's gate does not remove its entry. |
+| `_connections_lock` | One manager-wide `RLock`, created with the manager | Protects lock-registry lookup and short engine/verifier/epoch snapshots, identity claims, detach and publication. It is not a fourth per-user dictionary. |
+| `_sidecar_gates` | Actual sidecar `Path`; module-global dictionary of `Lock` objects | Serializes threads using the same sidecar, including different managers. Current storage is a strong dictionary with no entry removal. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release. |
+
+`_sidecar_gates_guard` only protects brief registry lookup/creation. It is
+released before waiting for the gate or OS lock. Manager lookup/snapshot
+similarly releases `_connections_lock` before outer waits; later cache
+bookkeeping reacquires it innermost for a short operation.
+
+The OS sidecar lock is a separate **inter-process** boundary.
+`acquire_rekey_lock` uses the database's `.rekey.lock` path, the local gate,
+then an open descriptor with POSIX `flock` or Windows locking of one byte
+at offset zero. The gate is non-reentrant: recursively acquiring the same
+sidecar is not supported. The lock file is kept rather than unlinked or
+replaced during ordinary release, preserving the inode/locked region used
+by other holders. `_open_sidecar` and `_refuse_foreign_sidecar` reject
+non-regular and already-unlinked files and check dangerous same-directory
+hard links; they do not make an untrusted data directory safe.
+
+`_acquire_sidecar_lock_locked` unlocks the OS lock if acquired, then closes
+the descriptor; `_acquire_sidecar_lock` finally releases the gate. Failed
+acquisition still closes an opened descriptor and releases the gate.
+Unlock/close failures raise a typed sidecar error after a clean body, or
+are attached as notes to the body's existing exception. OS retries retain
+the local gate, using the request's deadline or contention retry window.
+
+### Cached claims, disposal and reset
+
+In [encrypted_db.py](../../src/local_deep_research/database/encrypted_db.py),
+`_acquire_current_connection_generation` first snapshots engine, verifier
+and epoch under `_connections_lock`, then releases it. Acquisition is:
+
+1. Per-user lifecycle lock (`_rekey_locks`).
+2. Sidecar gate, then OS sidecar lock.
+3. Read durable epoch state, then briefly take `_connections_lock` in
+ `_claim_connection_generation` to compare the snapshot with current cache
+ identity and committed epoch.
+
+A current claim yields while lifecycle and sidecar ownership remain held.
+Keep its body to short bookkeeping (for example, verifier comparison or
+sessionmaker construction); database work such as integrity scans runs
+after leaving it. Release is OS unlock/close, gate, then lifecycle.
+A changed generation retries from a new snapshot **after** those releases,
+within the original wait budget. Missing/stale generations yield no usable
+connection; stale state is detached and disposed without checkpointing.
+An unreadable epoch or unavailable sidecar keeps the cached engine but
+returns no claim, or raises with `raise_unverifiable=True` after release.
+Release errors can themselves propagate. The bounded claim includes the
+lifecycle wait; request wait-budget accounting prevents repeated full waits
+after exhaustion until a successful sidecar acquisition.
+
+`close_user_database`, `close_all_databases` and `reset_connection_pools`
+snapshot before waiting. `_dispose_connection_generation` takes lifecycle
+then sidecar and delegates to
+`_dispose_connection_generation_under_sidecar_lock`, which performs the
+narrow identity/epoch claim. Only a current generation is checkpointed;
+a stale one is detached and disposed without a checkpoint, and a replaced
+or missing one is left alone. Explicit close can fall back on an exact
+identity detach and disposal without checkpoint after a fencing fault.
+`_reset_connection_generation` instead tries lifecycle and sidecar without
+blocking and skips on contention or operational faults. Its current pool
+is disposed without detaching the cache (`detach=False`). Both paths
+release sidecar before lifecycle, including failures. Registry identities
+survive these cache/pool operations.
+
+### Cold open, creation and reentry
+
+`open_user_database` performs its cached probe before acquiring init; that
+probe has the ownership described above. On a miss it takes the bounded
+init lock, derives the key and checks symlink/existence **before sidecar
+acquisition**. These cold-path preflight steps are inside init, not outside
+all locks. Under sidecar gate/OS exclusion it repeats path checks, handles
+pending recovery, establishes a stable epoch and checks cache bookkeeping.
+`_open_user_database_cold` then proves the connection and runs any
+pre-migration backup and migrations under the caller's init and sidecar
+ownership. Failed proof or migration does not publish a new cache entry.
+
+Before publication, `open_user_database` re-reads durable state: there must
+be no pending epoch and the same nonmissing committed epoch under which
+the engine was built. Non-vouching or changed state disposes the candidate
+without publishing; successful `_cache_connection` atomically stores the
+engine, verifier and epoch under `_connections_lock`. Sidecar releases
+before init on return or failure.
+
+Creation deliberately has a different boundary.
+`create_user_database` holds init through `_create_user_database_files`
+(exclusive file reservation) and `_initialize_new_user_database`
+(migrations and reservation identity checks). **Registration migration
+precedes sidecar acquisition.** Only afterwards does it acquire the
+sidecar, call `publish_new_database_epoch` for a fresh generation, and
+publish the engine/verifier/epoch. It releases sidecar before init.
+Migration failure disposes the engine and removes partial files;
+publication failure detaches/disposes and removes the partial database.
+This ordering is not a claim of cross-process exclusion over the entire
+registration operation.
+
+After successful cold publication, both init and sidecar have been
+released before `_purge_superseded_key_backups` or `_run_phase2_rekey` runs.
+Phase 2 calls
+[`rekey_user_indexes`](../../src/local_deep_research/vector_stores/legacy_rekey.py),
+whose session/integrity helpers can reenter `open_user_database`; callbacks
+that might invoke password change or another open must likewise occur
+after these non-reentrant locks are released. Do not move that work inside
+the cold-open ownership scope.
+
+Recovery cleanup intentionally reacquires the canonical database sidecar.
+`_purge_superseded_key_backups` requires exactly the expected committed
+epoch with no pending epoch and verifies the backup service's database
+path matches. It retains sidecar ownership through
+[`BackupService.purge_backups_under_superseded_key`](../../src/local_deep_research/database/backup/backup_service.py)
+and that service's per-user backup lock, releasing the backup lock before
+the sidecar. A newer epoch skips deletion; a fault only skips this
+best-effort cleanup. Pre-migration backup creation also nests the backup
+service lock under sidecar. Backup locking is an adjacent subsystem, not
+an additional claim that this map exhausts application locks. Short cache
+metadata operations and these service calls are local nesting, not a
+global lock hierarchy.
+
+### Password change and epoch publication
+
+`change_password` checks encryption availability and nonblank keys before
+its separate, nonblocking `_password_change_locks` gate. Under that gate,
+path preflight precedes `acquire_rekey_lock` (bounded local wait and a
+short OS contention retry). It does not universally acquire init or the
+lifecycle lock. Under sidecar ownership it resolves prior pending state,
+initializes the epoch if needed, detaches cache state under the short
+manager lock, disposes the detached engine without checkpointing, and
+opens the old key. No new cached engine is published by this method.
+
+After synchronous/checkpoint preparation,
+[`begin_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)
+publishes pending **before** physical rekey. The old connection is closed
+without further writes after rekey; a new-key connection must open and
+pass `_verify_rekey_candidate` before
+[`commit_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)
+publishes the pending generation as committed. Schema-page decryption is
+the key proof; subsequent integrity findings are warnings, not a demand
+that every page be healthy before committing the epoch.
+[`clear_pending_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)
+requires the matching committed generation before removing pending state.
+`_recover_pending_rekey` likewise proves the surviving key/schema before
+commit and clear; a wrong key leaves pending untouched. If new-key proof
+fails, `_resolve_unapplied_rekey` can prove and finalize the surviving old
+key, otherwise leaving an indeterminate outcome.
+
+Failures after pending may require recovery rather than imply the old
+password still works. Once the new key is proven, epoch publication
+failure can still report completion with pending left for the next valid
+open. Sidecar exits before the outer password-change gate is released in
+`finally`; leftover raw connections are closed in the intervening cleanup.
+Failure classification can read epoch state after sidecar release.
+
+Epochs coordinate generations; they do **not** authenticate passwords.
+A cache claim requires stable committed state plus current engine/verifier
+identity and matching cached epoch; encrypted cached opens additionally
+compare the supplied password verifier. Cold encrypted opens prove the key
+through SQLCipher. Plaintext fallback does not establish password proof.
+Revalidation applies per claim, not forever to already-issued SQLAlchemy
+sessions or checked-out connections. Direct lock-free cache access is not
+made safe by an epoch token.
+
+`read_rekey_epoch_state` reads pending first, then committed, including for
+exception classification and read-only projections that do not hold a
+sidecar. The unused application authorization projection in
+[rekey_epoch.py](../../src/local_deep_research/database/rekey_epoch.py) is not
+evidence of implemented web session fencing. Not every reader repairs
+state or owns a lock.
+
+## Consequences
+
+- Maintenance must preserve each path's release, retry and reentry points,
+ as well as the narrow cache publication conditions. A single diagram
+ asserting all locks always nest would conceal meaningful differences.
+- Epoch file publication uses a temporary UUID file, file flush/fsync,
+ atomic replacement and parent-directory sync where supported. Windows
+ skips directory sync; explicitly unsupported directory fsync is tolerated.
+ Publication can fail after replacement. This does not promise unconditional
+ crash durability, deadlock freedom or fencing of every existing session.
+- The stable sidecar is distinct from replaceable epoch files. Keep database,
+ WAL and salt recovery requirements in the unchanged recovery guide;
+ successful epoch publication does not prove all WAL pages reached the
+ main database file.
+- This documentation changes no runtime behavior and supplies no concurrency
+ safety proof. Separate pending gate-reclamation, rollback-retention and
+ authentication/session proposals are not implemented by this record.