~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs: record rekey lock domains and acquisition order (#7721)
Chanhyo Jung93af35d39b33
diff --git a/docs/decisions/0015-rekey-lock-architecture.md b/docs/decisions/0015-rekey-lock-architecture.mdnew file mode 100644index 0000000000..9979b84c74--- /dev/null+++ b/docs/decisions/0015-rekey-lock-architecture.md@@ -0,0 +1,221 @@+# ADR-0015: Rekey locks and connection generation publication++**Date:** 2026-10-20 (Asia/Seoul)+**Status:** Descriptive record of existing implementation++## Context++Password changes, cold opens and cached connection claims share a user's+SQLCipher database across threads and server processes. Their exclusion+boundaries differ. This record documents the implementation at+`683a967df20fe98ec0ddc99668041e22b95ff78b`; it does not adopt a new lock+policy or assert maintainer approval of a new design. The+[rekey recovery guide](../security/rekey-recovery.md) remains the operational+reference. [ADR-0004](0004-nullpool-for-sqlcipher.md) describes engine pooling;+[ADR-0009](0009-per-user-isolation-boundary.md) describes user isolation.++## Decision++Record the following path-specific ownership and publication rules. Each+path describes acquisition and release separately; these sequences do not+define one universal chain of all application locks.++### Lock map++These are five named in-process domains in the manager/rekey boundary,+not every application lock. The declarations and lookup helpers are in+[DatabaseManager](../../src/local_deep_research/database/encrypted_db.py)+(`__init__`, `_get_init_lock`, `_get_rekey_lock`,+`_get_password_change_lock`) and+[rekey_lock.py](../../src/local_deep_research/database/rekey_lock.py)+(`_get_sidecar_gate`).++| Domain | Key and scope | Role and lifetime |+| --- | --- | --- |+| `_init_locks` | Username; one `Lock` dictionary per manager | Serializes creation and cold-open construction/migration. Entries are created lazily and retained across close/reset so callers already holding a reference cannot race a replacement lock. |+| `_rekey_locks` | Username; one `Lock` dictionary per manager | Lifecycle exclusion for cached-generation claims and disposal/reset. Entries are lazy and retained across close/reset. The name does not mean every password change takes this lock. |+| `_password_change_locks` | Username; a separate `Lock` dictionary per manager | Nonblocking outer gate for concurrent `change_password` calls. Entries are lazy and retained; releasing a call's gate does not remove its entry. |+| `_connections_lock` | One manager-wide `RLock`, created with the manager | Protects lock-registry lookup and short engine/verifier/epoch snapshots, identity claims, detach and publication. It is not a fourth per-user dictionary. |+| `_sidecar_gates` | Actual sidecar `Path`; module-global dictionary of `Lock` objects | Serializes threads using the same sidecar, including different managers. Current storage is a strong dictionary with no entry removal. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release. |++`_sidecar_gates_guard` only protects brief registry lookup/creation. It is+released before waiting for the gate or OS lock. Manager lookup/snapshot+similarly releases `_connections_lock` before outer waits; later cache+bookkeeping reacquires it innermost for a short operation.++The OS sidecar lock is a separate **inter-process** boundary.+`acquire_rekey_lock` uses the database's `.rekey.lock` path, the local gate,+then an open descriptor with POSIX `flock` or Windows locking of one byte+at offset zero. The gate is non-reentrant: recursively acquiring the same+sidecar is not supported. The lock file is kept rather than unlinked or+replaced during ordinary release, preserving the inode/locked region used+by other holders. `_open_sidecar` and `_refuse_foreign_sidecar` reject+non-regular and already-unlinked files and check dangerous same-directory+hard links; they do not make an untrusted data directory safe.++`_acquire_sidecar_lock_locked` unlocks the OS lock if acquired, then closes+the descriptor; `_acquire_sidecar_lock` finally releases the gate. Failed+acquisition still closes an opened descriptor and releases the gate.+Unlock/close failures raise a typed sidecar error after a clean body, or+are attached as notes to the body's existing exception. OS retries retain+the local gate, using the request's deadline or contention retry window.++### Cached claims, disposal and reset++In [encrypted_db.py](../../src/local_deep_research/database/encrypted_db.py),+`_acquire_current_connection_generation` first snapshots engine, verifier+and epoch under `_connections_lock`, then releases it. Acquisition is:++1. Per-user lifecycle lock (`_rekey_locks`).+2. Sidecar gate, then OS sidecar lock.+3. Read durable epoch state, then briefly take `_connections_lock` in+ `_claim_connection_generation` to compare the snapshot with current cache+ identity and committed epoch.++A current claim yields while lifecycle and sidecar ownership remain held.+Keep its body to short bookkeeping (for example, verifier comparison or+sessionmaker construction); database work such as integrity scans runs+after leaving it. Release is OS unlock/close, gate, then lifecycle.+A changed generation retries from a new snapshot **after** those releases,+within the original wait budget. Missing/stale generations yield no usable+connection; stale state is detached and disposed without checkpointing.+An unreadable epoch or unavailable sidecar keeps the cached engine but+returns no claim, or raises with `raise_unverifiable=True` after release.+Release errors can themselves propagate. The bounded claim includes the+lifecycle wait; request wait-budget accounting prevents repeated full waits+after exhaustion until a successful sidecar acquisition.++`close_user_database`, `close_all_databases` and `reset_connection_pools`+snapshot before waiting. `_dispose_connection_generation` takes lifecycle+then sidecar and delegates to+`_dispose_connection_generation_under_sidecar_lock`, which performs the+narrow identity/epoch claim. Only a current generation is checkpointed;+a stale one is detached and disposed without a checkpoint, and a replaced+or missing one is left alone. Explicit close can fall back on an exact+identity detach and disposal without checkpoint after a fencing fault.+`_reset_connection_generation` instead tries lifecycle and sidecar without+blocking and skips on contention or operational faults. Its current pool+is disposed without detaching the cache (`detach=False`). Both paths+release sidecar before lifecycle, including failures. Registry identities+survive these cache/pool operations.++### Cold open, creation and reentry++`open_user_database` performs its cached probe before acquiring init; that+probe has the ownership described above. On a miss it takes the bounded+init lock, derives the key and checks symlink/existence **before sidecar+acquisition**. These cold-path preflight steps are inside init, not outside+all locks. Under sidecar gate/OS exclusion it repeats path checks, handles+pending recovery, establishes a stable epoch and checks cache bookkeeping.+`_open_user_database_cold` then proves the connection and runs any+pre-migration backup and migrations under the caller's init and sidecar+ownership. Failed proof or migration does not publish a new cache entry.++Before publication, `open_user_database` re-reads durable state: there must+be no pending epoch and the same nonmissing committed epoch under which+the engine was built. Non-vouching or changed state disposes the candidate+without publishing; successful `_cache_connection` atomically stores the+engine, verifier and epoch under `_connections_lock`. Sidecar releases+before init on return or failure.++Creation deliberately has a different boundary.+`create_user_database` holds init through `_create_user_database_files`+(exclusive file reservation) and `_initialize_new_user_database`+(migrations and reservation identity checks). **Registration migration+precedes sidecar acquisition.** Only afterwards does it acquire the+sidecar, call `publish_new_database_epoch` for a fresh generation, and+publish the engine/verifier/epoch. It releases sidecar before init.+Migration failure disposes the engine and removes partial files;+publication failure detaches/disposes and removes the partial database.+This ordering is not a claim of cross-process exclusion over the entire+registration operation.++After successful cold publication, both init and sidecar have been+released before `_purge_superseded_key_backups` or `_run_phase2_rekey` runs.+Phase 2 calls+[`rekey_user_indexes`](../../src/local_deep_research/vector_stores/legacy_rekey.py),+whose session/integrity helpers can reenter `open_user_database`; callbacks+that might invoke password change or another open must likewise occur+after these non-reentrant locks are released. Do not move that work inside+the cold-open ownership scope.++Recovery cleanup intentionally reacquires the canonical database sidecar.+`_purge_superseded_key_backups` requires exactly the expected committed+epoch with no pending epoch and verifies the backup service's database+path matches. It retains sidecar ownership through+[`BackupService.purge_backups_under_superseded_key`](../../src/local_deep_research/database/backup/backup_service.py)+and that service's per-user backup lock, releasing the backup lock before+the sidecar. A newer epoch skips deletion; a fault only skips this+best-effort cleanup. Pre-migration backup creation also nests the backup+service lock under sidecar. Backup locking is an adjacent subsystem, not+an additional claim that this map exhausts application locks. Short cache+metadata operations and these service calls are local nesting, not a+global lock hierarchy.++### Password change and epoch publication++`change_password` checks encryption availability and nonblank keys before+its separate, nonblocking `_password_change_locks` gate. Under that gate,+path preflight precedes `acquire_rekey_lock` (bounded local wait and a+short OS contention retry). It does not universally acquire init or the+lifecycle lock. Under sidecar ownership it resolves prior pending state,+initializes the epoch if needed, detaches cache state under the short+manager lock, disposes the detached engine without checkpointing, and+opens the old key. No new cached engine is published by this method.++After synchronous/checkpoint preparation,+[`begin_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)+publishes pending **before** physical rekey. The old connection is closed+without further writes after rekey; a new-key connection must open and+pass `_verify_rekey_candidate` before+[`commit_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)+publishes the pending generation as committed. Schema-page decryption is+the key proof; subsequent integrity findings are warnings, not a demand+that every page be healthy before committing the epoch.+[`clear_pending_rekey_epoch`](../../src/local_deep_research/database/rekey_lock.py)+requires the matching committed generation before removing pending state.+`_recover_pending_rekey` likewise proves the surviving key/schema before+commit and clear; a wrong key leaves pending untouched. If new-key proof+fails, `_resolve_unapplied_rekey` can prove and finalize the surviving old+key, otherwise leaving an indeterminate outcome.++Failures after pending may require recovery rather than imply the old+password still works. Once the new key is proven, epoch publication+failure can still report completion with pending left for the next valid+open. Sidecar exits before the outer password-change gate is released in+`finally`; leftover raw connections are closed in the intervening cleanup.+Failure classification can read epoch state after sidecar release.++Epochs coordinate generations; they do **not** authenticate passwords.+A cache claim requires stable committed state plus current engine/verifier+identity and matching cached epoch; encrypted cached opens additionally+compare the supplied password verifier. Cold encrypted opens prove the key+through SQLCipher. Plaintext fallback does not establish password proof.+Revalidation applies per claim, not forever to already-issued SQLAlchemy+sessions or checked-out connections. Direct lock-free cache access is not+made safe by an epoch token.++`read_rekey_epoch_state` reads pending first, then committed, including for+exception classification and read-only projections that do not hold a+sidecar. The unused application authorization projection in+[rekey_epoch.py](../../src/local_deep_research/database/rekey_epoch.py) is not+evidence of implemented web session fencing. Not every reader repairs+state or owns a lock.++## Consequences++- Maintenance must preserve each path's release, retry and reentry points,+ as well as the narrow cache publication conditions. A single diagram+ asserting all locks always nest would conceal meaningful differences.+- Epoch file publication uses a temporary UUID file, file flush/fsync,+ atomic replacement and parent-directory sync where supported. Windows+ skips directory sync; explicitly unsupported directory fsync is tolerated.+ Publication can fail after replacement. This does not promise unconditional+ crash durability, deadlock freedom or fencing of every existing session.+- The stable sidecar is distinct from replaceable epoch files. Keep database,+ WAL and salt recovery requirements in the unchanged recovery guide;+ successful epoch publication does not prove all WAL pages reached the+ main database file.+- This documentation changes no runtime behavior and supplies no concurrency+ safety proof. Separate pending gate-reclamation, rollback-retention and+ authentication/session proposals are not implemented by this record.