~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
feat(auth): fence server-side sessions with durable rekey epochs (#6750)
LearningCircuita95385be4c24
diff --git a/changelog.d/+post-login-credential-logging.security.md b/changelog.d/+post-login-credential-logging.security.mdnew file mode 100644index 0000000000..ed3bda0efe--- /dev/null+++ b/changelog.d/+post-login-credential-logging.security.md@@ -0,0 +1 @@+Post-login background-task and registration failure handlers no longer render a traceback, whose frame locals include the plaintext password, when loguru diagnostic logging is enabled: each now logs a warning naming the exception type instead, and the background-thread crash log uses the sanitized username.diff --git a/changelog.d/6662.security.md b/changelog.d/6662.security.mdindex e88591cd27..cd6c1d632c 100644--- a/changelog.d/6662.security.md+++ b/changelog.d/6662.security.md@@ -76,10 +76,10 @@ spending a full key derivation (PBKDF2) on a cold open that needs the samesidecar. The rekey epoch state is readpending-first, so a reader that does not hold the rekey sidecar lock cannotreport an epoch that a concurrent password-change commit superseded as-stable. In this release that covers only the classification of a-password-change failure raised after the lock was released: cache probes,-logins and password changes read the epoch under the lock, and session-validation does not read it yet.+stable. In this release the lock-free readers include session and socket+identity validation and the classification of a password-change failure+raised after the lock was released. Cache probes, logins and password+changes read the epoch under the lock.These waits for a user's rekey sidecar give up after 5 seconds: the cacheclaim behind every authenticated request (and behind `is_user_connected`,diff --git a/changelog.d/6750.security.md b/changelog.d/6750.security.mdnew file mode 100644index 0000000000..460bb71bb8--- /dev/null+++ b/changelog.d/6750.security.md@@ -0,0 +1,62 @@+Server-side login sessions are now bound to the database's durable rekey+epoch. A password change in any worker process ends every session issued+before it on its next check. A login that proved the old password just as+another worker committed a password change is refused (401) instead of+receiving a session. If the epoch cannot be read, or a password change is+pending, when the session is created (at login, or at the automatic+sign-in after registration) or when the login's database open re-reads+the epoch after verifying the password, or that re-read finds a different+committed epoch (possible only when something changed the epoch file+without taking the rekey lock), or that open recorded no epoch proof for+the user, the response is 503 rather than a session the server cannot+fence. A registration refused this way keeps the account it created, so+the user can sign in once the fault clears. None of these refusals+changes the account-lockout counter.++A session checked while a password change is pending (its new epoch+written but not yet committed) is removed, because the durable state+cannot vouch for it. This holds even when that change later fails with a+retryable error and the old password keeps working: the sessions removed+in the meantime stay removed, and their users sign in again.++If the epoch file cannot be read while a request is being authorized (for+example file-descriptor exhaustion), every request whose session cookie+names a signed-in user is refused with 503, including static files and+pages that need no login, except `POST /auth/logout`. The session is+kept, so a transient fault no longer logs every active user out. The+logout endpoint itself stays available during such a fault: it ends the+caller's own session and clears its cookie without opening the database.+The page's logout button first checks the session by fetching a fresh+CSRF token, which is refused like any other request, so during such a+fault it shows a message asking the user to try again. Clearing a cookie+whose database can no longer be reopened also destroys its server-side+session and disconnects its sockets during such a fault.++`SessionManager.has_active_sessions_for` no longer counts a session whose+epoch has been superseded. Only that check changed: the idle-connection+sweep's snapshot of active users (`get_active_usernames`) is still+time-based and keeps counting such a session until it idles out or its+next check removes it.++When a check removes a session (its epoch was superseded, the durable+epoch state cannot vouch for it, or it idled out), the live Socket.IO+connections opened with that session are disconnected too. A password+change ends every session of that user held by the worker process that+handled it and disconnects all of that user's sockets in that process at+once. When the password is changed in another worker, a socket opened with+a stolen cookie is disconnected only when its session is next checked (an+HTTP request, a socket connect, or a subscribe/unsubscribe carrying that+cookie); a socket that sends nothing keeps receiving the user's events in+its own worker until that session idles out, and progress events delivered+to a socket subscribed to a research keep the session from idling out. On+Windows, a reader holding an epoch file open briefly blocks a password+change or a new database from replacing or removing it (and the reverse);+both sides now retry that sharing violation a few times before failing,+and a read that still fails is refused with 503 as above.++The unused authorization-lease API (`acquire_authorization_lease`,+`AuthorizationLeaseMode`) and its `.rekey.authorization.lock` sidecar are+removed; the rekey lock is the only sidecar lock. When the backup refresh+after a password change cannot take the rekey lock (#7755), that is now+logged as a warning without a traceback, because the frame holds the old+and new passwords.diff --git a/docs/decisions/0015-rekey-lock-architecture.md b/docs/decisions/0015-rekey-lock-architecture.mdindex 9979b84c74..8e35706d01 100644--- a/docs/decisions/0015-rekey-lock-architecture.md+++ b/docs/decisions/0015-rekey-lock-architecture.md@@ -1,6 +1,6 @@# ADR-0015: Rekey locks and connection generation publication-**Date:** 2026-10-20 (Asia/Seoul)+**Date:** 2026-10-20 (Asia/Seoul; updated 2026-10-24)**Status:** Descriptive record of existing implementation## Context@@ -14,6 +14,12 @@ policy or assert maintainer approval of a new design. Thereference. [ADR-0004](0004-nullpool-for-sqlcipher.md) describes engine pooling;[ADR-0009](0009-per-user-isolation-boundary.md) describes user isolation.+> **Updated 2026-10-24:** the `_sidecar_gates` row describes the weak gate+> registry from #7705, and the epoch-projection paragraph at the end of+> *Password change and epoch publication* describes the web session+> fencing added by #6750, which also removed the unused authorization-lease+> API. The rest of the record was written against the commit above.+## DecisionRecord the following path-specific ownership and publication rules. Each@@ -36,7 +42,7 @@ not every application lock. The declarations and lookup helpers are in| `_rekey_locks` | Username; one `Lock` dictionary per manager | Lifecycle exclusion for cached-generation claims and disposal/reset. Entries are lazy and retained across close/reset. The name does not mean every password change takes this lock. || `_password_change_locks` | Username; a separate `Lock` dictionary per manager | Nonblocking outer gate for concurrent `change_password` calls. Entries are lazy and retained; releasing a call's gate does not remove its entry. || `_connections_lock` | One manager-wide `RLock`, created with the manager | Protects lock-registry lookup and short engine/verifier/epoch snapshots, identity claims, detach and publication. It is not a fourth per-user dictionary. |-| `_sidecar_gates` | Actual sidecar `Path`; module-global dictionary of `Lock` objects | Serializes threads using the same sidecar, including different managers. Current storage is a strong dictionary with no entry removal. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release. |+| `_sidecar_gates` | Actual sidecar `Path`; module-global `WeakValueDictionary` of `Lock` objects | Serializes threads using the same sidecar, including different managers. `_get_sidecar_gate` looks up or creates the entry under `_sidecar_gates_guard`. An entry is reclaimed once no holder or waiter references its gate, and the next caller creates a fresh one. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release, so a gate that is waited on or held cannot be reclaimed and replaced. |`_sidecar_gates_guard` only protects brief registry lookup/creation. It isreleased before waiting for the gate or OS lock. Manager lookup/snapshot@@ -197,10 +203,21 @@ made safe by an epoch token.`read_rekey_epoch_state` reads pending first, then committed, including forexception classification and read-only projections that do not hold a-sidecar. The unused application authorization projection in-[rekey_epoch.py](../../src/local_deep_research/database/rekey_epoch.py) is not-evidence of implemented web session fencing. Not every reader repairs-state or owns a lock.+sidecar. Web session fencing is built on one such projection,+`read_user_authorization_epoch` in+[authorization_epoch.py](../../src/local_deep_research/database/authorization_epoch.py):+[SessionManager](../../src/local_deep_research/web/auth/session_manager.py)+binds each session to the committed epoch when it is created (refusing a+login whose proven epoch is no longer committed) and re-projects the epoch+each time it validates the session (HTTP requests, Socket.IO connects,+subscribes and unsubscribes). A superseded epoch, a pending+rekey or state that cannot vouch for the session removes it; an unreadable+epoch file refuses that request (503 over HTTP) but keeps the session. The+projection takes no lock, so fencing applies per check, not across a+request already admitted. The former authorization-lease API+(`AuthorizationLeaseMode`, `acquire_authorization_lease`) and its+`.rekey.authorization.lock` sidecar were removed; `.rekey.lock` is the only+sidecar lock. Not every reader repairs state or owns a lock.## Consequences@@ -217,5 +234,6 @@ state or owns a lock.successful epoch publication does not prove all WAL pages reached themain database file.- This documentation changes no runtime behavior and supplies no concurrency- safety proof. Separate pending gate-reclamation, rollback-retention and- authentication/session proposals are not implemented by this record.+ safety proof. Gate reclamation (#7705) and web session fencing (#6750)+ have since been implemented and are described above; the separate+ rollback-retention proposal is not implemented by this record.diff --git a/pyproject.toml b/pyproject.tomlindex d711800858..d698c148c3 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -299,6 +299,7 @@ markers = ["fd_canary: heavy file-descriptor-leak regression tests (#3816). CPU-bound real-event-loop spins that exceeded the legacy 60s timeout under coverage + '-n auto' contention, and whose FD-count assertions are noisy under parallel execution; run in a dedicated serial, no-coverage CI step instead of the parallel suite.","lifespan: enters the app's real lifespan (startup/shutdown). At most ONE such test may run per process: socketio_asgi.init_lock() only assigns when _lock is None, so a second cycle would leave the asyncio.Lock bound to a dead loop. Do not add a second one or convert it into a fixture.","real_session_check: opt out of the autouse _legacy_bare_username_auth shim so the real server-side session-id check is exercised. For tests that must prove a destroyed/invalid session is rejected.",+ "real_durable_epoch: opt out of the autouse _durable_epoch_projection shim so the real durable rekey-epoch projection is exercised. For tests that must prove epoch-bound sessions fence on password change.","serial: tests that mutate PROCESS-GLOBAL state (os.umask, os.chdir, signal handlers) for the duration of an assertion. Correct on their own, but under '-n auto' any other file creation happening concurrently in the same worker process inherits the hostile global -- so they run in a dedicated serial CI step instead of the parallel suite. This marker says NOTHING about privilege by itself: if a serial test's assertions also require a genuinely unprivileged process (e.g. it self-skips with `if os.geteuid() == 0: pytest.skip(...)`), it MUST also carry the 'nonroot' marker explicitly -- do not rely on a CI step happening to invoke pytest under an unprivileged user (docker-tests.yml's serial step uses '--user ldruser', but that is CI wiring, not a property of the marker, and a lane that ran '-m serial' as root would silently no-op every assertion in such a test). docker-tests.yml runs 'serial' and 'nonroot' together in one step selecting '-m serial or nonroot' under '--user ldruser', so a module carrying both markers runs exactly once.","connected: marks the connected pytest suite (real per-user encrypted SQLCipher DBs; heavier scenarios run out of the default unit suite).","nonroot: tests whose assertions require a genuinely unprivileged process -- they start with `if os.geteuid() == 0: pytest.skip(...)` because permission bits (chmod/O_CREAT denial, read-only dirs, etc.) are not enforced for root. A test may carry 'nonroot' alone (does not mutate process-global state, e.g. tests/security/test_db_connection_create_hostile_dir.py, tests/security/test_db_connection_readonly_hostile_fs.py) or 'nonroot' together with 'serial' (also mutates process-global state, e.g. tests/security/test_db_connection_umask_perms.py). docker-tests.yml runs both markers together in one step, '-m serial or nonroot' under '--user ldruser', so a module carrying both runs exactly once regardless of which marker it also carries. The distinction matters because docker-tests.yml's default (main) container runs as root (it needs write access to the bind-mounted checkout for coverage/junit output), so nonroot-marked tests would silently no-op there unless excluded from the main step ('not serial and not nonroot') and run in that dedicated '--user ldruser' step instead. No exclusion is needed in security-tests.yml, which already executes as the non-root Actions runner user for every step (main and serial alike), so a serial+nonroot module's non-root requirement is satisfied there without a dedicated nonroot step or selector.",diff --git a/src/local_deep_research/database/authorization_epoch.py b/src/local_deep_research/database/authorization_epoch.pyindex 2ae4c7241a..9ee9a91303 100644--- a/src/local_deep_research/database/authorization_epoch.py+++ b/src/local_deep_research/database/authorization_epoch.py@@ -1,5 +1,8 @@"""Read-only authorization identity projection for encrypted user databases.+Session and socket identity validation use these projections to fence+identities issued before a password change.+The durable token, pending/committed epoch, and fencing state machine livesin ``rekey_lock``. These projections read without taking its sidecar lock;its pending-first reads prevent a concurrent commit and pending cleanupdiff --git a/src/local_deep_research/database/encrypted_db.py b/src/local_deep_research/database/encrypted_db.pyindex 93e357c8d2..c0d1ef8fc1 100644--- a/src/local_deep_research/database/encrypted_db.py+++ b/src/local_deep_research/database/encrypted_db.py@@ -139,6 +139,81 @@ type _ConnectionGenerationClaim = ()+class LoginProof:+ """The durable rekey epoch a successful ``open_user_database`` proved.++ Filled only inside :func:`capture_login_proof`. ``epoch`` is the+ committed epoch the password was verified against UNDER the rekey+ sidecar lock (the claimed cache generation's epoch, or the epoch the+ cold open ran under). A login binds its session to this value, not+ to whatever epoch is live by the time the session is created: a rekey+ in another worker can commit in between, and an old-password login+ straddling it must not come away with a new-epoch session.+ """++ __slots__ = ("username", "epoch")++ def __init__(self) -> None:+ self.username: str | None = None+ self.epoch: str | None = None++ def proven_epoch_for(self, username: str) -> str:+ """The epoch proved for ``username``, or refuse (fail closed).++ A successful open always records a proof for the user it opened,+ so a missing proof, one recorded for a different username, or one+ without an epoch means the caller cannot show which epoch the+ password was verified against; minting a session anyway would+ skip the straddle check silently.++ Raises:+ MissingLoginProofError: no usable proof for ``username``.+ """+ if self.username != username or self.epoch is None:+ raise MissingLoginProofError(username)+ return self.epoch+++class MissingLoginProofError(Exception):+ """A successful login open left no epoch proof for the logged-in user."""++ def __init__(self, username: str) -> None:+ self.username = username+ super().__init__("no login epoch proof was recorded for this login")+++# Per-thread proof collector. Only set while a caller is inside+# capture_login_proof(), so opens made by every other caller (request+# middleware, sockets, background threads) record nothing and nothing+# lingers on a pooled worker thread between requests.+_login_proof_collector = threading.local()+++@contextmanager+def capture_login_proof() -> Generator[LoginProof, None, None]:+ """Collect the epoch proved by ``open_user_database`` on this thread.++ The OUTERMOST successful open inside the block wins: the phase-2 RAG+ rekey re-enters ``open_user_database`` on the same thread before the+ outer call returns, and the outer call records last.+ """+ previous = getattr(_login_proof_collector, "proof", None)+ proof = LoginProof()+ _login_proof_collector.proof = proof+ try:+ yield proof+ finally:+ _login_proof_collector.proof = previous+++def _record_login_proof(username: str, epoch: str | None) -> None:+ proof = getattr(_login_proof_collector, "proof", None)+ if proof is None:+ return+ proof.username = username+ proof.epoch = epoch++@dataclass(frozen=True, slots=True)class _ConnectionGenerationUnverifiable:"""A generation is cached, but its epoch could not be checked.@@ -1965,17 +2040,11 @@ class DatabaseManager:) -> Optional[Engine]:"""Open a user's database, verifying the password only with SQLCipher.- With encryption enabled, a cold open validates the derived key- against the database; a cached open requires a matching in-memory- password verifier. An already-open database alone is not proof of- the caller's password.-- In the plaintext development fallback, neither cold nor cached- opens verify the account password. The nonblank-input check below- still applies, but a different password can open the same database.- A returned engine in that mode is not evidence of authentication.- Callers must not use this fallback for shared accounts that rely on- LDR password verification without adding independent authentication.+ With encryption enabled, cold opens validate the derived key and+ cached opens require a matching password verifier. The plaintext+ development fallback does not verify account passwords: its returned+ engine and epoch are not evidence of authentication. Shared plaintext+ accounts require independent authentication.When an engine is cached but its rekey sidecar cannot be opened orlocked, or its epoch cannot be read, the typed fault@@ -1988,6 +2057,39 @@ class DatabaseManager:raises it after the derivation (without waiting, in a request whose earlierwait ran out: ``sidecar_wait_budget``). None stays reserved for acredential or missing-database verdict.++ Inside :func:`capture_login_proof` the epoch the password was+ proved against is recorded for the caller (the login route binds+ the new session to it).+ """+ engine, proven_epoch, cold_opened = self._open_user_database_proving(+ username, password+ )+ # Phase 2 of the RAG vector-store cutover runs OUTSIDE init_lock (only+ # on the path that actually did the cold-open — the early returns+ # belong to a thread that already ran it). It re-enters+ # open_user_database() via get_user_db_session on a session-cache miss;+ # doing it while still holding the non-reentrant init_lock would+ # self-deadlock if a concurrent close_user_database() evicted the+ # freshly-cached connection between the cache write and that re-entry.+ if cold_opened:+ self._run_phase2_rekey(username, password)+ # Recorded AFTER phase 2 so this (outer) call's proof wins over any+ # proof the phase-2 re-entry recorded on this thread.+ _record_login_proof(+ username, proven_epoch if engine is not None else None+ )+ return engine++ def _open_user_database_proving(+ self, username: str, password: str+ ) -> tuple[Optional[Engine], Optional[str], bool]:+ """Open (or claim) the user's engine and report what was proved.++ Returns ``(engine, proven_epoch, cold_opened)``. ``proven_epoch`` is+ the committed durable epoch the password was verified against while+ the rekey sidecar lock was held; ``cold_opened`` is True only when+ this call built and cached a new engine (phase 2 must then run)."""open_start = time.perf_counter()@@ -2040,7 +2142,7 @@ class DatabaseManager:generation, password)):- return generation.engine+ return generation.engine, generation.epoch, False# Serialize the cold-open (engine build + migration) per user so two# concurrent first-opens never run alembic against the same database@@ -2059,10 +2161,10 @@ class DatabaseManager:password, db_path=db_path).hex()if _db_path_is_symlinked(db_path, username):- return None+ return None, None, Falseif not db_path.exists():logger.error(f"No database found for user {username}")- return None+ return None, None, False# The stable sidecar serializes cold opens with cross-process rekeys.# Epoch repair and publication happen only while this inode is locked.@@ -2078,10 +2180,10 @@ class DatabaseManager:# Re-checked under the lock (cheap now): the pre-gate above# ran without any exclusion.if _db_path_is_symlinked(db_path, username):- return None+ return None, None, Falseif not db_path.exists():logger.error(f"No database found for user {username}")- return None+ return None, None, Falsecleanup_stale_rekey_temps(db_path)rekey_state = read_rekey_epoch_state(db_path)if rekey_state.pending_epoch is not None:@@ -2095,7 +2197,7 @@ class DatabaseManager:if not self._recover_pending_rekey(db_path, password, rekey_state):- return None+ return None, None, Falserecovered_pending_rekey = Truetry:current_epoch = read_or_initialize_rekey_epoch(db_path)@@ -2114,7 +2216,7 @@ class DatabaseManager:if cached is not None and self._cached_engine_trusted(username, password, current_epoch):- return cached+ return cached, current_epoch, Falseif (cached is not Noneand not self._connection_epoch_matches(@@ -2130,7 +2232,7 @@ class DatabaseManager:username, password, open_start, prederived_hex_key)if engine is None:- return None+ return None, None, False# Re-read the durable state the password was just proved# under. No outcome other than the same stable committed@@ -2185,20 +2287,11 @@ class DatabaseManager:replaced = self._cache_connection(username, entry)self._dispose_detached_connection(username, replaced)- # Phase 2 of the RAG vector-store cutover runs OUTSIDE init_lock (only- # on the path that actually did the cold-open — the early returns above- # belong to a thread that already ran it). It re-enters- # open_user_database() via get_user_db_session on a session-cache miss;- # doing it while still holding the non-reentrant init_lock would- # self-deadlock if a concurrent close_user_database() evicted the- # freshly-cached connection between the cache write and that re-entry.- if engine is not None:- if recovered_pending_rekey:- self._purge_superseded_key_backups(- username, password, expected_epoch=current_epoch- )- self._run_phase2_rekey(username, password)- return engine+ if engine is not None and recovered_pending_rekey:+ self._purge_superseded_key_backups(+ username, password, expected_epoch=current_epoch+ )+ return engine, current_epoch, Truedef _purge_superseded_key_backups(self, username: str, password: str, *, expected_epoch: strdiff --git a/src/local_deep_research/database/rekey_lock.py b/src/local_deep_research/database/rekey_lock.pyindex 40ad123a31..f9dcb9dc04 100644--- a/src/local_deep_research/database/rekey_lock.py+++ b/src/local_deep_research/database/rekey_lock.py@@ -9,12 +9,12 @@ import tempfileimport threadingimport timeimport weakref-from collections.abc import Generator+from collections.abc import Callable, Generatorfrom contextlib import contextmanagerfrom dataclasses import dataclassfrom enum import StrEnum, uniquefrom pathlib import Path-from typing import assert_never+from typing import TypeVarfrom uuid import UUID, uuid4@@ -22,14 +22,6 @@ class RekeyOperationalError(Exception):"""Base for explicit operational failures during password rekey."""-@unique-class AuthorizationLeaseMode(StrEnum):- """Cross-process authorization lease compatibility modes."""-- SHARED = "shared"- EXCLUSIVE = "exclusive"--@uniqueclass RekeySidecarOperation(StrEnum):"""Safe lifecycle stages for sidecar I/O diagnostics."""@@ -172,9 +164,12 @@ class RekeyEpochUnreadableError(RekeyRecoveryRequiredError):Raised for every ``OSError`` other than ``FileNotFoundError`` whilereading ``.rekey.epoch`` / ``.rekey.pending`` (EMFILE, ENFILE, EAGAIN,EIO, EACCES, ...; EINTR never gets here, Python retries it itself per- PEP 475), and when the reader refuses a planted file without reading- it: a symlink (ELOOP) or anything that is not a regular file, such as- a FIFO or a directory (EINVAL). It is a ``RekeyRecoveryRequiredError``, so+ PEP 475; a PermissionError is first retried a bounded number of times+ as a possible Windows sharing violation, see+ ``_retry_sharing_violation``), and when the reader refuses a planted+ file without reading it: a symlink (ELOOP) or anything that is not a+ regular file, such as a FIFO or directory (EINVAL). It is a+ ``RekeyRecoveryRequiredError``, soevery existing handler keeps failing closed exactly as before; thesubclass only lets a caller that can retry later (session validation)tell "nothing could be read" apart from durable state that was read@@ -222,7 +217,6 @@ class RekeyEpochState:class _SidecarLockRequest:db_path: Pathlock_path: Path- mode: AuthorizationLeaseModeblocking: boolgate_wait: float | None = Nonecontention_retry: float = 0.0@@ -268,18 +262,11 @@ def _acquire_posix_lock(fd: int,lock_path: Path,*,- mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,blocking: bool = False,) -> None:import fcntl- match mode:- case AuthorizationLeaseMode.SHARED:- operation = fcntl.LOCK_SH- case AuthorizationLeaseMode.EXCLUSIVE:- operation = fcntl.LOCK_EX- case _ as unreachable:- assert_never(unreachable)+ operation = fcntl.LOCK_EXif not blocking:operation |= fcntl.LOCK_NBtry:@@ -304,7 +291,6 @@ def _acquire_windows_lock(fd: int,lock_path: Path,*,- mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,blocking: bool = False,) -> None:import msvcrt@@ -314,13 +300,7 @@ def _acquire_windows_lock(_ = os.write(fd, b"\0")_ = os.lseek(fd, 0, os.SEEK_SET)try:- match mode:- case AuthorizationLeaseMode.SHARED:- lock_mode = msvcrt.LK_RLCK if blocking else msvcrt.LK_NBRLCK- case AuthorizationLeaseMode.EXCLUSIVE:- lock_mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCK- case _ as unreachable:- assert_never(unreachable)+ lock_mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCKmsvcrt.locking(fd, lock_mode, 1)except OSError as error:if error.errno in {@@ -343,10 +323,6 @@ def _release_windows_lock(fd: int) -> None:msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)-def _authorization_lock_path(db_path: Path) -> Path:- return db_path.with_name(f"{db_path.name}.rekey.authorization.lock")--def _sidecar_io_error(lock_request: _SidecarLockRequest,operation: RekeySidecarOperation,@@ -541,14 +517,12 @@ def _acquire_os_lock_with_retry(_acquire_windows_lock(fd,lock_request.lock_path,- mode=lock_request.mode,blocking=lock_request.blocking,)else:_acquire_posix_lock(fd,lock_request.lock_path,- mode=lock_request.mode,blocking=lock_request.blocking,)except RekeyLockContentionError:@@ -655,7 +629,6 @@ def acquire_rekey_lock(lock_request = _SidecarLockRequest(db_path=db_path,lock_path=db_path.with_name(f"{db_path.name}.rekey.lock"),- mode=AuthorizationLeaseMode.EXCLUSIVE,blocking=blocking and timeout is None,gate_wait=local_wait,contention_retry=contention_retry,@@ -665,23 +638,6 @@ def acquire_rekey_lock(yield-@contextmanager-def acquire_authorization_lease(- db_path: Path,- mode: AuthorizationLeaseMode,- blocking: bool = True,-) -> Generator[None, None, None]:- """Hold a persistent shared or exclusive authorization sidecar lease."""- lock_request = _SidecarLockRequest(- db_path=db_path,- lock_path=_authorization_lock_path(db_path),- mode=mode,- blocking=blocking,- )- with _acquire_sidecar_lock(lock_request):- yield--def _epoch_path(db_path: Path) -> Path:return db_path.with_name(f"{db_path.name}.rekey.epoch")@@ -690,6 +646,37 @@ def _pending_epoch_path(db_path: Path) -> Path:return db_path.with_name(f"{db_path.name}.rekey.pending")+# Windows sharing violations between the lock-free epoch readers and the+# epoch writers. CPython opens files on Windows without FILE_SHARE_DELETE,+# so while a reader (session validation, on every authenticated request)+# holds ``.rekey.epoch`` / ``.rekey.pending`` open, the writer's+# ``os.replace`` onto it or ``unlink`` of it fails with PermissionError;+# and a reader that opens a file whose replace/unlink is in flight+# (delete-pending) gets PermissionError too. Both windows are a few+# microseconds, so each side retries a PermissionError a bounded number of+# times before giving up. Giving up keeps the pre-existing outcome: the+# reader raises ``RekeyEpochUnreadableError`` (fail closed: no epoch is+# ever returned from a failed read) and the writer re-raises. On POSIX a+# PermissionError here is a genuine permission problem; the retry then+# costs at most ``(attempts - 1) * delay`` before the same outcome.+_SHARING_VIOLATION_ATTEMPTS = 5+_SHARING_VIOLATION_DELAY = 0.01++_T = TypeVar("_T")+++def _retry_sharing_violation(operation: Callable[[], _T]) -> _T:+ """Run ``operation``, retrying a transient PermissionError (bounded)."""+ for attempt in range(1, _SHARING_VIOLATION_ATTEMPTS + 1):+ try:+ return operation()+ except PermissionError:+ if attempt >= _SHARING_VIOLATION_ATTEMPTS:+ raise+ time.sleep(_SHARING_VIOLATION_DELAY)+ raise AssertionError("unreachable") # pragma: no cover++# Every token this module writes is a canonical UUID string: 36 ASCII bytes._EPOCH_TOKEN_MAX_BYTES = len(str(UUID(int=0)))@@ -732,7 +719,7 @@ def _read_epoch_token(path: Path, db_path: Path) -> str | None:| getattr(os, "O_BINARY", 0))try:- fd = os.open(path, flags)+ fd = _retry_sharing_violation(lambda: os.open(path, flags))except FileNotFoundError:return Noneexcept OSError as error:@@ -773,8 +760,8 @@ def read_rekey_epoch_state(db_path: Path) -> RekeyEpochState:fails after that lock has been released (the read only picks whicherror to report). The read order matters for that reader and for theones that never take the lock: ``_record_password_verifier`` (only- tests call it) and the ``rekey_epoch`` authorization projection- (nothing in the application calls it yet). Such a reader may run while+ tests call it) and the ``authorization_epoch`` projection+ used by session and socket identity validation. Such a reader may run whilea rekey is publishing, so the two files are read in the order thatcannot tear into a stale-but-stable answer. Writers publish ``.pending`` beforethey change ``.epoch`` and remove it only after ``.epoch`` is@@ -914,8 +901,11 @@ def _write_epoch_token(path: Path, epoch: str, db_path: Path) -> None:stream.flush()os.fsync(stream.fileno())+ # A lock-free reader holding ``path`` open makes this fail with a+ # sharing violation on Windows: retry it (bounded), see+ # ``_retry_sharing_violation``.# nosemgrep: semgrep.rules.path-traversal-risk, reason: path is a fixed suffix on the sha256-named user database path and temporary_path is mkstemp output in its directory- os.replace(temporary_path, path)+ _retry_sharing_violation(lambda: os.replace(temporary_path, path))_sync_parent_directory(path)finally:if fd >= 0:@@ -955,7 +945,8 @@ def clear_pending_rekey_epoch(db_path: Path, pending_epoch: str) -> None:if state.committed_epoch != candidate or state.pending_epoch != candidate:raise RekeyRecoveryRequiredError(db_path)pending_path = _pending_epoch_path(db_path)- pending_path.unlink()+ # Same Windows sharing violation as the replace in _write_epoch_token.+ _retry_sharing_violation(pending_path.unlink)_sync_parent_directory(pending_path)@@ -993,7 +984,10 @@ def publish_new_database_epoch(db_path: Path) -> str:_write_epoch_token(_epoch_path(db_path), epoch, db_path)pending_path = _pending_epoch_path(db_path)try:- pending_path.unlink()+ # Same Windows sharing violation as clear_pending_rekey_epoch's+ # unlink: a lock-free reader holding the stale pending token open+ # must not fail the creation. FileNotFoundError is not retried.+ _retry_sharing_violation(pending_path.unlink)except FileNotFoundError:return epoch_sync_parent_directory(pending_path)diff --git a/src/local_deep_research/web/auth/connection_cleanup.py b/src/local_deep_research/web/auth/connection_cleanup.pyindex 94b3cd0715..d6af925d1a 100644--- a/src/local_deep_research/web/auth/connection_cleanup.py+++ b/src/local_deep_research/web/auth/connection_cleanup.py@@ -176,6 +176,8 @@ def _count_open_fds() -> int:def cleanup_idle_connections(session_manager, db_manager):"""Close db connections for users with no active sessions and no active research."""+ from .session_manager import SessionValidationUnavailableError+# 1. Purge expired sessions firstsession_manager.cleanup_expired_sessions()@@ -193,7 +195,16 @@ def cleanup_idle_connections(session_manager, db_manager):# 5. Double-check before closing (narrows race window)closed = 0for username in candidates:- if session_manager.has_active_sessions_for(username):+ try:+ has_active_sessions = session_manager.has_active_sessions_for(+ username+ )+ except SessionValidationUnavailableError:+ # Unknown activity is insufficient evidence for shared teardown.+ # Request authentication still refuses an unreadable epoch.+ logger.warning("Deferred idle cleanup: session state unavailable")+ continue+ if has_active_sessions:logger.debug(f"Skipped {username} (active session appeared since snapshot)")diff --git a/src/local_deep_research/web/auth/session_manager.py b/src/local_deep_research/web/auth/session_manager.pyindex 03d4528459..bd17c998a0 100644--- a/src/local_deep_research/web/auth/session_manager.py+++ b/src/local_deep_research/web/auth/session_manager.py@@ -6,15 +6,94 @@ Handles session creation, validation, and cleanup.import datetimeimport secretsimport threading+from dataclasses import dataclassfrom datetime import UTC-from typing import Dict, Optional, Set+from typing import Dict, Optional, Set, TypedDictfrom loguru import logger+from ...database.authorization_epoch import read_user_authorization_epoch+from ...database.rekey_lock import (+ RekeyEpochUnreadableError,+ RekeyOperationalError,+)from ...security import get_security_defaultfrom ...utilities.formatting import short_id+class _SessionData(TypedDict):+ username: str+ created_at: datetime.datetime+ last_access: datetime.datetime+ remember_me: bool+ rekey_epoch: str+++class StaleLoginProofError(Exception):+ """The login proved its password against a no-longer-current epoch.++ Raised by ``create_session`` when the durable epoch the login's+ ``open_user_database`` proved differs from the committed epoch at+ session creation: a rekey (password change, possibly in another+ worker) committed in between, so the password this login presented+ no longer opens the database and must not mint a session.+ """++ def __init__(self, username: str) -> None:+ self.username = username+ super().__init__("login proof is stale: the password changed")+++# Key under the ASGI ``scope["state"]`` (``request.state``) where+# ``DatabaseMiddleware`` (``_enforce_session_revocation``) records the+# ``(session_id, username)`` it validated for THIS request, so+# ``require_auth`` does not re-project the durable epoch a second time.+# ``scope["state"]`` is per request (Starlette/uvicorn copy it for each+# request) and is written only by server code after a successful+# validation; nothing client-controlled reaches it.+VALIDATED_SESSION_STATE_KEY = "ldr_validated_session"++# Set in ``scope["state"]`` by ``DatabaseMiddleware`` when it let a+# ``POST /auth/logout`` through WITHOUT validating the session, because the+# durable epoch could not be read. Only the logout route reads it.+UNVALIDATED_LOGOUT_STATE_KEY = "ldr_unvalidated_logout"+++class SessionValidationUnavailableError(Exception):+ """The session's durable epoch could not be read for THIS check.++ Raised by ``validate_session_identity`` / ``validate_session`` when+ the epoch file read itself failed (``RekeyEpochUnreadableError``:+ EMFILE, ENFILE, EAGAIN, EIO, ...). Nothing is known about the+ epoch, so the request must be refused (fail closed), but the session+ record is KEPT: only durable state that was actually read and does+ not vouch for the session ends it. Callers answer 503 (HTTP) or+ reject the socket action, and must NOT clear the cookie -- that would+ log out every active user on a transient fd-exhaustion spike.++ Deliberately not a ``RekeyOperationalError`` and not a falsy return:+ a caller that forgets to handle it gets an error response, never an+ authenticated one.+ """++ def __init__(self, username: str, error_number: int | None) -> None:+ self.username = username+ self.errno = error_number+ super().__init__(+ "session epoch temporarily unreadable "+ f"(errno={'unknown' if error_number is None else error_number})"+ )+++@dataclass(frozen=True, slots=True)+class ValidatedSession:+ """An identity proven against the durable rekey epoch it was bound to."""++ session_id: str+ username: str+ rekey_epoch: str++class SessionManager:"""Manages user sessions and database connection lifecycle.@@ -24,8 +103,17 @@ class SessionManager:worker restart drops all sessions it held. ``validate_session`` onlyever sees sessions created by *this* process — including for callerslike the socket connect-time session gate, which now relies on this- lookup. See ``security/account_lockout.py`` and- ``web/routers/rag.py`` for the same caveat elsewhere in the codebase.+ lookup — but every record is validated against the shared durable+ rekey epoch, so a password change in ANY worker fences the stale+ local identities of every other worker too. ``has_active_sessions_for``+ re-projects the epoch too (a stale or unverifiable record does not+ count as active). The idle sweeps (``cleanup_expired_sessions``,+ ``get_active_usernames``) are time-based only and do NOT re-project+ the epoch: a stale-epoch record is removed by its next validation or+ when it idles out, and until then ``get_active_usernames`` can keep+ its user counted as active by the idle-connection sweep. See+ ``security/account_lockout.py`` and ``web/routers/rag.py`` for the+ same multi-worker caveat elsewhere in the codebase."""def __init__(self):@@ -45,69 +133,171 @@ class SessionManager:self.session_timeout = datetime.timedelta(hours=session_hours)self.remember_me_timeout = datetime.timedelta(days=remember_days)- def create_session(self, username: str, remember_me: bool = False) -> str:+ def create_session(+ self,+ username: str,+ remember_me: bool = False,+ *,+ proven_epoch: str | None = None,+ ) -> str:"""Create a new session for a user.Args:username: The username to create a session for.remember_me: If True, use extended session timeout.+ proven_epoch: The durable epoch the caller's credential check+ proved (``LoginProof.epoch`` from the login's+ ``open_user_database``). When given, the session is created+ only if it is still the committed epoch.Returns:The session ID as a URL-safe string.++ Raises:+ StaleLoginProofError: ``proven_epoch`` is no longer committed+ (a rekey committed after the password was proved).+ RekeyOperationalError: the durable epoch cannot be projected+ (the login route answers 503)."""+ # Bind the committed durable epoch. With a proof, it must equal the+ # epoch the password was verified against: an old-password login+ # straddling a rekey committed in another worker would otherwise be+ # bound to the NEW epoch and survive the password change.+ rekey_epoch = read_user_authorization_epoch(username)+ if proven_epoch is not None and proven_epoch != rekey_epoch:+ raise StaleLoginProofError(username)session_id = secrets.token_urlsafe(32)+ now = datetime.datetime.now(UTC)with self._lock:self.sessions[session_id] = {"username": username,- "created_at": datetime.datetime.now(UTC),- "last_access": datetime.datetime.now(UTC),+ "created_at": now,+ "last_access": now,"remember_me": remember_me,+ "rekey_epoch": rekey_epoch,}logger.debug(f"Created session {session_id[:8]}... for user {username}")return session_id- def validate_session(self, session_id: str) -> Optional[str]:- """- Validate a session and return username if valid.- Updates last access time and disconnects sockets for an expired session.+ def validate_session_identity(+ self, session_id: str+ ) -> Optional[ValidatedSession]:+ """Validate one epoch-bound session and refresh its idle timestamp.++ The durable epoch is projected OUTSIDE ``_lock`` (it reads a file),+ then the record is re-checked under the lock so a concurrent+ destroy/recreate cannot be validated against a stale snapshot.++ Failure discipline (fail closed):++ * epoch mismatch — the database was rekeyed (password changed)+ somewhere; this session's credential can no longer open it, so+ the record is REMOVED;+ * the epoch files could not be READ at all+ (``RekeyEpochUnreadableError``: EMFILE, ENFILE, EAGAIN, EIO,+ ...) — nothing is known about the epoch, so this check+ fails closed by raising ``SessionValidationUnavailableError``,+ but the record is KEPT. Every caller that sees a ``None``+ clears the cookie, so deleting (or returning ``None``) here+ would log out every active user on a transient fd spike;+ * any other ``RekeyOperationalError`` — the files were read and+ the durable state (malformed token, a pending rekey in flight+ or awaiting recovery, pending without committed) cannot vouch+ for the identity; the record is REMOVED. A pending state is+ kept on this side on purpose: after a crashed rekey it stays+ until the next open recovers it (which always advances the+ epoch), and a kept record would answer 503 until it idled out+ instead of sending the user back to the login that recovers.++ Whenever the record is REMOVED (idle expiry, epoch mismatch, or+ unvouchable durable state), the live sockets that handshook on+ this session id are disconnected too, best-effort and after+ ``_lock`` is released (``_disconnect_expired_sockets``; it never+ raises). The removed record is invisible to every later sweep, so+ this is the only point that can still revoke them.++ Raises:+ SessionValidationUnavailableError: the epoch was unreadable."""- expired_username = None+ expired_username: Optional[str] = Nonewith self._lock:- if session_id not in self.sessions:+ session_data = self.sessions.get(session_id)+ if session_data is None:return None-- session_data = self.sessions[session_id]- now = datetime.datetime.now(UTC)-- # Check timeouttimeout = (self.remember_me_timeoutif session_data["remember_me"]else self.session_timeout)+ username = session_data["username"]+ rekey_epoch = session_data["rekey_epoch"]++ try:+ current_epoch = read_user_authorization_epoch(username)+ except RekeyEpochUnreadableError as error:+ # Transient I/O: refuse THIS check, keep the record.+ raise SessionValidationUnavailableError(+ username, error.errno+ ) from None+ except RekeyOperationalError:+ with self._lock:+ removed = self.sessions.get(session_id) is session_data+ if removed:+ del self.sessions[session_id]+ if removed:+ self._disconnect_expired_sockets([session_id])+ return None++ now = datetime.datetime.now(UTC)+ with self._lock:+ if self.sessions.get(session_id) is not session_data:+ return Noneif now - session_data["last_access"] > timeout:# Session expired — remove inline (already under lock).# Defer the log call until after the lock is released: see# the warning in __init__ about not logging while holding# _lock.- expired_username = session_data["username"]+ expired_username = username+ del self.sessions[session_id]+ elif current_epoch != rekey_epoch:+ # The database was rekeyed after this session was issued:+ # its credential is dead and so is the identity it+ # vouches for. Remove the record (other sessions of the+ # same user bound to the new epoch stay).del self.sessions[session_id]else:# Update last accesssession_data["last_access"] = now- return str(session_data["username"])-- # The scheduled sweep cannot find a session removed inline above.- # Disconnect only after releasing _lock: socket teardown can enter- # session code again, and this lock is not reentrant.+ return ValidatedSession(session_id, username, rekey_epoch)++ # The record was removed above (expired or bound to a superseded+ # epoch). Sever the sockets that handshook on it too, after the+ # lock is released and best-effort: once the record is gone no+ # sweep can find this session again, and a socket is authorised+ # once at handshake, so a left-over sid would keep receiving the+ # user's events (``settings_changed`` carries plaintext secrets).self._disconnect_expired_sockets([session_id])- logger.debug(- f"Session {session_id[:8]}... expired for {expired_username}"- )+ if expired_username is not None:+ logger.debug(+ f"Session {session_id[:8]}... expired for {expired_username}"+ )return None+ def validate_session(self, session_id: str) -> Optional[str]:+ """Validate a session and return username if valid.++ Updates last access time.++ Raises:+ SessionValidationUnavailableError: the epoch was unreadable;+ refuse the request but keep the session (see+ ``validate_session_identity``).+ """+ identity = self.validate_session_identity(session_id)+ return None if identity is None else identity.username+def touch_session(self, session_id: str) -> None:"""Refresh a session's ``last_access`` without enforcing the timeout.@@ -124,6 +314,41 @@ class SessionManager:if session_data is not None:session_data["last_access"] = datetime.datetime.now(UTC)+ def session_owner(self, session_id: str) -> Optional[str]:+ """The username the server-side record for ``session_id`` names.++ ``None`` when there is no record. NOT a validation, exactly like+ ``session_belongs_to``: no epoch read, no expiry check, no+ ``last_access`` refresh, so it must never gate access to anything.+ Its only caller is ``_revoke_server_session``, which scopes a+ teardown with it: an epoch-read fault must not stop the teardown+ of a cookie that is being cleared anyway.+ """+ if not session_id:+ return None+ with self._lock:+ session_data = self.sessions.get(session_id)+ return None if session_data is None else session_data["username"]++ def session_belongs_to(self, session_id: str, username: str) -> bool:+ """Whether a server-side record still binds ``session_id`` to+ ``username``.++ NOT a validation: no epoch read, no expiry check, no+ ``last_access`` refresh, so it must never gate access to anything.+ Its only caller is logout during an epoch-read fault, where it+ scopes what logout may destroy: a cookie whose record is already+ gone tears down nothing of that user's beyond its own cookie.+ """+ if not session_id:+ return False+ with self._lock:+ session_data = self.sessions.get(session_id)+ return (+ session_data is not None+ and session_data["username"] == username+ )+def destroy_session(self, session_id: str):"""Destroy a session and clean up."""username = None@@ -185,7 +410,11 @@ class SessionManager:@staticmethoddef _disconnect_expired_sockets(session_ids: list[str]) -> None:- """Best-effort disconnect of the sockets for expired session ids.+ """Best-effort disconnect of the sockets for removed session ids.++ Called by ``cleanup_expired_sessions`` and by+ ``validate_session_identity`` whenever it removes a record (idle+ expiry, superseded epoch, unvouchable durable state). Never raises.Lazy-imports the socket layer to avoid an import cycle and totolerate its absence in non-web contexts (CLI, tests). Each session is@@ -253,20 +482,43 @@ class SessionManager:}def has_active_sessions_for(self, username: str) -> bool:- """Check if a user has any non-expired sessions."""+ """Check if a user has any non-expired, epoch-current session.++ Uses the same epoch check as ``validate_session_identity``: a+ record bound to an epoch that is no longer committed (the password+ was changed, possibly in another worker) is NOT active, even+ before its next validation removes it. When the epoch cannot be+ read, raise ``SessionValidationUnavailableError`` without changing+ any records. Cleanup callers must defer shared teardown when the+ answer is unknown. Readable durable state that cannot vouch for an+ identity still counts as inactive.+ Read-only: records are neither refreshed nor removed here.+ """now = datetime.datetime.now(UTC)with self._lock:- for data in self.sessions.values():- if data["username"] != username:- continue- timeout = (+ bound_epochs = {+ data["rekey_epoch"]+ for data in self.sessions.values()+ if data["username"] == username+ and now - data["last_access"]+ <= (self.remember_me_timeoutif data["remember_me"]else self.session_timeout)- if now - data["last_access"] <= timeout:- return True+ }+ if not bound_epochs:+ return False+ # The epoch is a file read: outside _lock, like validation.+ try:+ current_epoch = read_user_authorization_epoch(username)+ except RekeyEpochUnreadableError as error:+ raise SessionValidationUnavailableError(+ username, error.errno+ ) from None+ except RekeyOperationalError:return False+ return current_epoch in bound_epochs# Module-level singletondiff --git a/src/local_deep_research/web/dependencies/auth.py b/src/local_deep_research/web/dependencies/auth.pyindex 72649e8202..6ec9a6da83 100644--- a/src/local_deep_research/web/dependencies/auth.py+++ b/src/local_deep_research/web/dependencies/auth.py@@ -22,7 +22,11 @@ from ...database.rekey_lock import RekeyOperationalErrorfrom ...database.session_context import get_user_db_sessionfrom ...database.session_passwords import session_password_storefrom ...utilities.db_utils import get_settings_manager-from ..auth.session_manager import session_manager+from ..auth.session_manager import (+ VALIDATED_SESSION_STATE_KEY,+ SessionValidationUnavailableError,+ session_manager,+)def get_session_username(request: Request) -> str | None:@@ -142,9 +146,17 @@ def _revoke_server_session(session_id: str, username: str) -> None:Only this session is affected. The user's other sessions keep working.A session id owned by a different user is left alone. Teardown isbest-effort so a failure here cannot turn the 401 into a 500.++ The owner is looked up with ``session_owner``, which reads no epoch.+ ``validate_session`` re-projects the durable epoch and raises+ ``SessionValidationUnavailableError`` while the epoch files cannot be+ read; looking the owner up through it skipped the teardown below while+ the cookie was still cleared, leaving exactly the orphaned record this+ function exists to remove. A stale-epoch or expired record of this+ user is destroyed too: the cookie bound to it is being cleared."""try:- owner = session_manager.validate_session(session_id)+ owner = session_manager.session_owner(session_id)if owner is not None and owner != username:returnif owner is not None:@@ -348,7 +360,18 @@ def require_auth(request: Request) -> str:# chokepoint. It also gives `session_timeout_hours` / `remember_me_days`# real effect, since `validate_session` enforces the timeouts and refreshes# last-access on use.- if not _server_session_valid(request, username):+ try:+ session_valid = _server_session_valid(request, username)+ except SessionValidationUnavailableError:+ # The durable epoch could not be read (e.g. EMFILE). Refuse THIS+ # request, but keep the session and its cookie: a transient fault+ # proves nothing about the session and must not log users out.+ raise HTTPException(+ status_code=503,+ detail="Service temporarily unavailable, please retry",+ headers={"Retry-After": "5"},+ ) from None+ if not session_valid:# Destroyed by logout or password change, expired, or never valid.# Clear the cookie so the client stops presenting a dead session.request.session.clear()@@ -391,11 +414,31 @@ def _server_session_valid(request: Request, username: str) -> bool:Only ever called after a username has been confirmed present, so"accept unconditionally" is exactly the pre-revocation contract.++ Per-request memo: ``DatabaseMiddleware`` validates the same session+ (``_enforce_session_revocation``) earlier in this request and records+ the ``(session_id, username)`` it proved in ``request.state``. When+ both still match the cookie, that result is reused instead of+ re-projecting the durable epoch a second time. Anything else (no+ memo, a different id or user) validates for real.++ Raises ``SessionValidationUnavailableError`` when the epoch cannot be+ read (``require_auth`` answers 503 and keeps the session)."""session_id = request.session.get("session_id")- valid_owner = bool(- session_id and session_manager.validate_session(session_id) == username- )+ if not session_id:+ return False+ scope = getattr(request, "scope", None)+ state = scope.get("state") if isinstance(scope, dict) else None+ if isinstance(state, dict) and state.get(VALIDATED_SESSION_STATE_KEY) == (+ session_id,+ username,+ ):+ valid_owner = True+ else:+ valid_owner = session_manager.validate_session(session_id) == username+ # The memo only caches the epoch check. A credential lost later in this+ # request must still retire the session, even with a shared DB open.return valid_owner and not clear_session_if_unrecoverable(request, username)diff --git a/src/local_deep_research/web/fastapi_app.py b/src/local_deep_research/web/fastapi_app.pyindex a0f13c0066..1bc3aff9da 100644--- a/src/local_deep_research/web/fastapi_app.py+++ b/src/local_deep_research/web/fastapi_app.py@@ -1961,7 +1961,7 @@ def _stamp_session_expiry(session) -> None:)-def _enforce_session_revocation(session) -> None:+def _enforce_session_revocation(session, state: dict | None = None) -> None:"""Clear `session` in place if its server-side record is gone.`require_auth` rejects a cookie whose `session_id` no longer resolves@@ -1986,8 +1986,11 @@ def _enforce_session_revocation(session) -> None:downstream, so no call site needs to know about revocation.Anonymous sessions return immediately, so the cost on the hot path is- one dict lookup; for authenticated ones it is `validate_session`'s- in-memory dict read under a lock. That call also refreshes+ one dict lookup; for authenticated ones it is `validate_session`: a+ dict read under a lock plus a re-projection of the user's durable+ rekey epoch (two small sidecar-file reads, `.rekey.pending` and+ `.rekey.epoch`), which is what lets a password change in another+ worker fence this session. That call also refreshes`last_access`, which is why this runs unconditionally rather thanbehind `_skip_prefixes` — same reasoning as the expiry check above.@@ -1996,18 +1999,47 @@ def _enforce_session_revocation(session) -> None:behaviour, not something this widens: it already applied to everyauthenticated route. The only change is that `/` and `/auth/check`now agree with the rest of the app about who is logged in.++ A validated ``(session_id, username)`` is recorded in ``state`` (the+ request's ``scope["state"]``) under ``VALIDATED_SESSION_STATE_KEY`` so+ ``require_auth`` can skip an identical second epoch projection in the+ same request.++ Raises ``SessionValidationUnavailableError`` (session left untouched)+ when the durable epoch cannot be read; the caller answers 503, except+ for ``POST /auth/logout``, which it lets through unvalidated."""username = session.get("username")if not username:return- from .auth.session_manager import session_manager+ from .auth.session_manager import (+ VALIDATED_SESSION_STATE_KEY,+ session_manager,+ )session_id = session.get("session_id")if not session_id or session_manager.validate_session(session_id) != (username):session.clear()+ elif state is not None:+ state[VALIDATED_SESSION_STATE_KEY] = (session_id, username)+++async def _send_session_validation_unavailable(scope, receive, send) -> None:+ """503 for a request whose session could not be validated right now.++ The session (and its cookie) is left alone: the epoch read failed+ transiently (e.g. EMFILE), which proves nothing about the session, so+ clearing it would log the user out. The request itself is refused.+ """+ response = JSONResponse(+ {"error": "Service temporarily unavailable, please retry"},+ status_code=503,+ headers={"Retry-After": "5"},+ )+ await response(scope, receive, send)class DatabaseMiddleware:@@ -2070,6 +2102,7 @@ class DatabaseMiddleware:# e.g. static-asset requests would let an idle-but-signature-valid# session's deadline silently stop advancing/enforcing whenever# the user's tab is only pulling static assets.+ unvalidated_logout = Falsesession = scope.get("session")if session is not None:_enforce_session_expiry(session)@@ -2079,13 +2112,51 @@ class DatabaseMiddleware:# but its signed cookie is still replayable". `require_auth`# covers the latter for the routes that use it; this covers# the ones that deliberately don't. See the docstring.- _enforce_session_revocation(session)+ from .auth.session_manager import (+ UNVALIDATED_LOGOUT_STATE_KEY,+ SessionValidationUnavailableError,+ )++ try:+ _enforce_session_revocation(+ session, scope.setdefault("state", {})+ )+ except SessionValidationUnavailableError as unavailable:+ # Transient epoch-read failure: fail THIS request closed+ # without touching the session (no logout). Only the+ # errno number is logged.+ unreadable_errno = unavailable.errno+ if method == "POST" and path == "/auth/logout":+ # Except logout: it only destroys (the caller's own+ # session, its credentials, its cookie), so refusing it+ # would only stop a user from revoking their session+ # during the fault. It runs unvalidated: the flag tells+ # the route to scope its teardown, and the database+ # open below is skipped, so nothing is opened, stored+ # or published for this cookie.+ logger.warning(+ "Letting logout through unvalidated: session "+ f"epoch unreadable (errno={unreadable_errno})"+ )+ scope["state"][UNVALIDATED_LOGOUT_STATE_KEY] = True+ unvalidated_logout = True+ else:+ logger.warning(+ "Refusing request with 503: session epoch "+ f"unreadable (errno={unreadable_errno})"+ )+ await _send_session_validation_unavailable(+ scope, receive, send+ )+ returnctx_tokens = None# Run the inner app first to let SessionMiddleware populate scope# Then ensure database is open- if method != "OPTIONS" and not any(- path.startswith(p) for p in self._skip_prefixes+ if (+ method != "OPTIONS"+ and not unvalidated_logout+ and not any(path.startswith(p) for p in self._skip_prefixes)):# Session data is in scope["session"] after SessionMiddlewaresession_data = scope.get("session", {})diff --git a/src/local_deep_research/web/routers/auth.py b/src/local_deep_research/web/routers/auth.pyindex 98ac00fd1e..f01efdf09e 100644--- a/src/local_deep_research/web/routers/auth.py+++ b/src/local_deep_research/web/routers/auth.py@@ -26,7 +26,12 @@ from datetime import datetime, timezone, UTCfrom loguru import loggerfrom ...database.auth_db import auth_db_session-from ...database.encrypted_db import DatabaseInitializationError, db_manager+from ...database.encrypted_db import (+ DatabaseInitializationError,+ MissingLoginProofError,+ capture_login_proof,+ db_manager,+)from ...database.rekey_lock import (RekeyIndeterminateError,RekeyLockContentionError,@@ -37,6 +42,8 @@ from ...database.models.auth import Userfrom ...database.thread_local_session import thread_cleanupfrom sqlalchemy.exc import IntegrityErrorfrom ..auth.session_manager import (+ UNVALIDATED_LOGOUT_STATE_KEY,+ StaleLoginProofError,session_manager,) # singleton from session_manager modulefrom ..server_config import load_server_config@@ -115,8 +122,14 @@ def _rollback_partial_session(request: Request, username: str) -> None:from ...database.session_passwords import session_password_storesession_password_store.clear_session(username, session_id)- except Exception:- logger.exception("Failed to roll back a partial user session")+ except Exception as error:+ # Type name only, no traceback: this helper runs inside login+ # and register, whose caller frames hold the plaintext password+ # (see _perform_post_login_tasks for when diagnose renders it).+ logger.warning(+ "Failed to roll back a partial user session ({})",+ type(error).__name__,+ )def _disconnect_user_sockets(username: str) -> None:@@ -282,8 +295,12 @@ def login(# the alembic_runner permission check) → 503, do NOT count toward# lockout. The user's password is correct; punishing them with a# lockout for a server-side configuration problem would be wrong.+ # capture_login_proof records the durable rekey epoch the password was+ # verified against, so the session below is bound to THAT epoch rather+ # than to whatever epoch is committed by the time it is created.try:- engine = db_manager.open_user_database(username, password)+ with capture_login_proof() as login_proof:+ engine = db_manager.open_user_database(username, password)except DatabaseInitializationError:logger.warning(f"Login refused for {redact_and_bound_for_log(username, 50)}: "@@ -349,8 +366,33 @@ def login(status_code=401,)- # Success- lockout_mgr.record_success(username)+ # The session is bound to the epoch this login PROVED. A successful+ # open always records a proof for this user, so a missing or foreign+ # proof is an internal inconsistency: refuse (fail closed) rather than+ # create a session with the straddle check skipped. Not a guessing+ # failure, so the lockout counter is left alone.+ try:+ proven_epoch = login_proof.proven_epoch_for(username)+ except MissingLoginProofError:+ logger.warning(+ f"Login refused for {redact_and_bound_for_log(username, 50)}: the database "+ "open recorded no epoch proof for this user"+ )+ flash(+ request,+ "Your account database is pending maintenance. "+ "Please try again shortly or contact the administrator.",+ "error",+ )+ return render_template(+ request,+ "auth/login.html",+ {+ "has_encryption": db_manager.has_encryption,+ "allow_registrations": config.get("allow_registrations", True),+ },+ status_code=503,+ )# Prevent session fixationrequest.session.clear()@@ -360,7 +402,9 @@ def login(# silent login (see _rollback_partial_session / #5006).try:# Create session- session_id = session_manager.create_session(username, remember_bool)+ session_id = session_manager.create_session(+ username, remember_bool, proven_epoch=proven_epoch+ )request.session["session_id"] = session_idrequest.session["username"] = username# RememberMeMiddleware reads this to decide whether to strip@@ -380,10 +424,65 @@ def login(session_password_store.store_session_password(username, session_id, password)+ except StaleLoginProofError:+ # The password was right when it was verified, but a rekey+ # (password change, possibly in another worker) committed before+ # the session could be created: it no longer opens the database.+ # Not a guessing failure, so the lockout counter is left alone:+ # neither incremented nor reset (record_success runs only after a+ # session is issued).+ _rollback_partial_session(request, username)+ logger.warning(+ f"Login refused for {redact_and_bound_for_log(username, 50)}: the password "+ "changed while the login was in flight"+ )+ flash(+ request,+ "Your password was changed while you were signing in. "+ "Please sign in with the current password.",+ "error",+ )+ return render_template(+ request,+ "auth/login.html",+ {+ "has_encryption": db_manager.has_encryption,+ "allow_registrations": config.get("allow_registrations", True),+ },+ status_code=401,+ )+ except RekeyOperationalError as error:+ # The durable epoch could not be projected, so the session cannot+ # be bound to it: same answer as the open path's rekey failures.+ _rollback_partial_session(request, username)+ logger.warning(+ f"Login refused for {redact_and_bound_for_log(username, 50)}: rekey epoch "+ f"unavailable at session creation ({type(error).__name__})"+ )+ flash(+ request,+ "Your account database is pending maintenance. "+ "Please try again shortly or contact the administrator.",+ "error",+ )+ return render_template(+ request,+ "auth/login.html",+ {+ "has_encryption": db_manager.has_encryption,+ "allow_registrations": config.get("allow_registrations", True),+ },+ status_code=503,+ )except Exception:_rollback_partial_session(request, username)raise+ # Success: only now, with a session actually issued, reset the+ # lockout counter. A login refused at session creation (stale proof,+ # unreadable epoch) must neither count as a failure nor clear the+ # failures already recorded.+ lockout_mgr.record_success(username)logger.info(f"User {_log_value_preview(username)} logged in successfully")# Defer non-critical post-login work to a background thread so the@@ -424,14 +523,27 @@ def _perform_post_login_tasks(An outer try/except wraps the whole body so any exception thatescapes the per-step handlers (for example a failure inside a- ``with`` context manager's __enter__ / __exit__) is logged loudly- with a traceback instead of dying silently in the daemon thread.+ ``with`` context manager's __enter__ / __exit__) is logged instead of+ dying silently in the daemon thread.++ Every handler in this worker and in ``register`` logs a WARNING with+ the exception TYPE only -- never ``logger.exception``, ``exc_info`` or+ ``logger.opt(exception=...)``: these frames hold the plaintext+ password, and a rendered traceback prints frame locals whenever a+ sink has loguru ``diagnose`` on. That is not reliably knowable from+ here: ``config_logger`` enables it on the stderr sink when debug is+ on (from the ``app.debug`` setting or the environment) AND+ ``LDR_LOGURU_DIAGNOSE`` is set, and a process that never ran+ ``config_logger`` keeps loguru's default handler, which has+ ``diagnose`` on."""try:_perform_post_login_tasks_body(username, password, session_id)- except Exception:- logger.exception(- f"Post-login background thread crashed for user {_log_value_preview(username)}"+ except Exception as err:+ logger.warning(+ "Post-login background thread crashed for user {} ({})",+ redact_and_bound_for_log(username, 50),+ type(err).__name__,)@@ -486,9 +598,10 @@ def _perform_post_login_tasks_body(f"Missing default settings loaded and version "f"updated for user {_log_value_preview(username)}")- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login settings migration failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 1 (settings version check)", step_start, username)@@ -507,9 +620,10 @@ def _perform_post_login_tasks_body(f"Library initialization issue for {_log_value_preview(username)}: "f"{init_results.get('error', 'Unknown error')}")- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login library init failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 2 (library init)", step_start, username)@@ -525,10 +639,11 @@ def _perform_post_login_tasks_body(DocumentDeletionService(username).reconcile_orphaned_document_chunks(password)- except Exception:- logger.exception(- "Post-login orphan chunk reconciliation failed for {}",- _log_value_preview(username),+ except Exception as err:+ # No traceback: this frame holds the plaintext password.+ logger.warning(+ f"Post-login orphan chunk reconciliation failed for "+ f"{_log_value_preview(username)} ({type(err).__name__})")_log_step_duration("step 2a (orphan chunk reconciliation)", step_start, username@@ -553,13 +668,17 @@ def _perform_post_login_tasks_body(logger.info(f"Updated scheduler with user info for {_log_value_preview(username)}")- except Exception:- logger.exception("Could not update scheduler on login")+ except Exception as scheduler_err:+ logger.warning(+ "Could not update scheduler on login"+ f" ({type(scheduler_err).__name__})"+ )auth_db.commit()- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login auth DB update failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 3 (auth DB + scheduler notify)", step_start, username@@ -589,9 +708,10 @@ def _perform_post_login_tasks_body(logger.info(f"Background backup scheduled for user {_log_value_preview(username)}")- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login backup scheduling failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 4 (schedule backup)", step_start, username)@@ -617,9 +737,10 @@ def _perform_post_login_tasks_body(queue_processor.reconcile_orphan_active_research(username, db_session)- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login research reconciliation failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 6 (reconcile orphan research)", step_start, username@@ -638,9 +759,10 @@ def _perform_post_login_tasks_body(from ..queue.processor_v2 import queue_processorqueue_processor.notify_user_activity(username, session_id)- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Post-login queue-resume registration failed for {_log_value_preview(username)}"+ f" ({type(err).__name__})")_log_step_duration("step 7 (resume queued research)", step_start, username)@@ -818,10 +940,11 @@ def register(},status_code=400,)- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Registration failed for {log_username} while creating the "f"auth record"+ f" ({type(err).__name__})")auth_db.rollback()flash(request, "Registration failed. Please try again.", "error")@@ -846,10 +969,11 @@ def register(# the user DB exists and the account is legitimate.try:db_manager.create_user_database(username, password)- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Registration failed for {log_username} while creating the "f"encrypted database"+ f" ({type(err).__name__})")# Delete the orphaned auth row by primary key in a fresh session# (the original auth_db session has already closed). Deleting by ID@@ -865,11 +989,12 @@ def register(logger.info(f"Cleaned up orphaned auth entry for {log_username}")- except Exception:+ except Exception as cleanup_err:# Surface a stuck orphan so it's diagnosable, but don't mask the# original registration failure returned to the user.- logger.exception(+ logger.warning(f"Failed to clean up orphaned auth entry for {log_username}"+ f" ({type(cleanup_err).__name__})")flash(request, "Registration failed. Please try again.", "error")return render_template(@@ -922,8 +1047,11 @@ def register(logger.info(f"Updated scheduler with new user info for {log_username}")- except Exception:- logger.exception("Could not update scheduler on registration")+ except Exception as err:+ logger.warning(+ "Could not update scheduler on registration"+ f" ({type(err).__name__})"+ )logger.info(f"New user registered: {log_username}")@@ -940,15 +1068,44 @@ def register(logger.warning(f"Library initialization issue for {log_username}: {init_results.get('error', 'Unknown error')}")- except Exception:- logger.exception(+ except Exception as err:+ logger.warning(f"Error initializing library for new user {log_username}"+ f" ({type(err).__name__})")# Don't block registration on library init failurereturn RedirectResponse(url="/", status_code=302)- except Exception:+ except RekeyOperationalError as error:+ # create_session could not project the durable epoch the session+ # must be bound to (a transient sidecar fault, or state that needs+ # the operator): the same retryable 503 the login route answers for+ # it, never a 500. The account and its database exist, so the auth+ # row stays and the user is sent to sign in once the fault clears;+ # the half-set session is rolled back exactly like the 500 below.+ _rollback_partial_session(request, username)+ logger.warning(+ f"Registration auto-login refused for {log_username}: rekey "+ f"epoch unavailable at session creation ({type(error).__name__})"+ )+ flash(+ request,+ "Your account was created, but signing in is temporarily "+ "unavailable. Please sign in again shortly.",+ "error",+ )+ return render_template(+ request,+ "auth/login.html",+ {+ "has_encryption": db_manager.has_encryption,+ "allow_registrations": config.get("allow_registrations", True),+ },+ status_code=503,+ )++ except Exception as err:# The encrypted DB already exists at this point, so this failure is in# post-creation setup (auto-login / session). The account is created# and loginable; do NOT clean up the auth row here (narrow scope).@@ -956,9 +1113,10 @@ def register(# the user silently logged in (SessionMiddleware would otherwise write# the partially-set cookie on this response — see #5006)._rollback_partial_session(request, username)- logger.exception(+ logger.warning(f"Registration failed for {log_username} after database creation "f"(post-creation session/login setup)"+ f" ({type(err).__name__})")flash(request, "Registration failed. Please try again.", "error")return render_template(@@ -982,6 +1140,21 @@ def logout(request: Request):username = request.session.get("username")session_id = request.session.get("session_id")+ state = request.scope.get("state")+ if (+ username+ and isinstance(state, dict)+ and state.get(UNVALIDATED_LOGOUT_STATE_KEY)+ and not session_manager.session_belongs_to(session_id, username)+ ):+ # DatabaseMiddleware let this logout through without validating the+ # cookie (the durable epoch was unreadable). The user-wide teardown+ # below runs only for a cookie whose server-side record still binds+ # it to this username; any other cookie just loses itself, which is+ # what the validating middleware would have done anyway.+ request.session.clear()+ return RedirectResponse(url="/auth/login", status_code=302)+if username:# LOGOUT CLEANUP ORDER (order matters):# 1. Unregister from news scheduler — removes the password from the@@ -1547,15 +1720,21 @@ def change_password("old-key backups remain.")except RekeyLockContentionError:- logger.exception(+ # Same rule as below: no traceback, the frame holds the+ # passwords.+ logger.warning(f"Backup refresh after password change for "f"{_log_value_preview(username)} could not take the rekey ""lock; backups under the old key may remain")- except Exception:- logger.exception(+ except Exception as error:+ # Type name only, no traceback: this frame holds+ # current_password, new_password and confirm_password in its+ # locals, and a rendered traceback (loguru diagnose) would+ # print them.+ logger.warning(f"Could not refresh backups after password change "- f"for {_log_value_preview(username)}"+ f"for {_log_value_preview(username)} ({type(error).__name__})")# 3. Revoke every old-password credential and session.diff --git a/src/local_deep_research/web/services/socketio_asgi.py b/src/local_deep_research/web/services/socketio_asgi.pyindex 5f12e37a1a..95151f92d9 100644--- a/src/local_deep_research/web/services/socketio_asgi.py+++ b/src/local_deep_research/web/services/socketio_asgi.py@@ -454,13 +454,26 @@ async def connect(sid, environ, auth=None):# had no successor here: DatabaseMiddleware's `_enforce_session_revocation`# cannot cover it, because that middleware returns early for non-HTTP# scopes and additionally skips the "/ws/" prefix.- from ..auth.session_manager import session_manager+ from ..auth.session_manager import (+ SessionValidationUnavailableError,+ session_manager,+ )handshake_session_id = session_data.get("session_id")- if (- not handshake_session_id- or session_manager.validate_session(handshake_session_id) != username- ):+ try:+ handshake_valid = bool(handshake_session_id) and (+ session_manager.validate_session(handshake_session_id) == username+ )+ except SessionValidationUnavailableError:+ # The durable epoch could not be read (transient I/O). Refuse this+ # handshake (fail closed) but leave the session intact; the client+ # reconnects on its own.+ logger.warning(+ f"Rejected WebSocket connection for {username} from {sid}: "+ "session could not be validated right now"+ )+ return False+ if not handshake_valid:logger.warning(f"Rejected WebSocket connection for {username} from {sid}: ""session is not valid server-side (logged out, expired, or "@@ -570,7 +583,15 @@ async def connect(sid, environ, auth=None):# enumerating sockets -- logout destroys the session first, then# disconnects -- so a socket that registers after invalidation is# guaranteed to observe the failed re-check.- if session_manager.validate_session(handshake_session_id) != username:+ try:+ still_valid = (+ session_manager.validate_session(handshake_session_id) == username+ )+ except SessionValidationUnavailableError:+ # Cannot prove the session is still alive: fail closed like an+ # invalidated one (the session record itself is kept).+ still_valid = False+ if not still_valid:async with _lock:_sid_users.pop(sid, None)_sid_sessions.pop(sid, None)@@ -708,12 +729,15 @@ async def _user_owns_research(username: str, research_id: str) -> bool:async def _socket_session_still_valid(sid: str, username: str-) -> tuple[str | None, bool]:+) -> tuple[str | None, bool | None]:"""Re-check that this socket's originating session is still alive.Returns ``(session_id, is_valid)``. The id comes back so the caller cantear down EVERY socket of a dead session, not just the one that asked --- see the rejection path in ``on_subscribe``.+ see the rejection path in ``on_subscribe``. ``is_valid`` is ``None``+ when the session could not be validated right now (the durable epoch+ was unreadable): the caller must refuse the action but NOT tear the+ session's sockets down, since nothing is known to be wrong with it.The connect-time gate is not sufficient on its own: identity is capturedinto ``_sid_users`` at handshake and then frozen for the socket's whole@@ -735,9 +759,18 @@ async def _socket_session_still_valid(if not session_id:return None, False- from ..auth.session_manager import session_manager+ from ..auth.session_manager import (+ SessionValidationUnavailableError,+ session_manager,+ )- return session_id, session_manager.validate_session(session_id) == username+ try:+ return (+ session_id,+ session_manager.validate_session(session_id) == username,+ )+ except SessionValidationUnavailableError:+ return session_id, None@sio.on("subscribe_to_research")@@ -772,6 +805,22 @@ async def on_subscribe(sid, data):stale_session_id, session_ok = await _socket_session_still_valid(sid, username)+ if session_ok is None:+ # Unverifiable right now (transient epoch-read failure): refuse+ # this subscribe, keep the socket and its session.+ logger.warning(+ f"Rejected subscribe from {sid} ({username}): session could not "+ "be validated right now"+ )+ await sio.emit(+ "subscribe_error",+ {+ "error": "Temporarily unavailable",+ "research_id": research_id,+ },+ room=sid,+ )+ returnif not session_ok:logger.warning(f"Rejected subscribe from {sid} ({username}): originating session "@@ -892,6 +941,13 @@ async def on_unsubscribe(sid, data):stale_session_id, session_ok = await _socket_session_still_valid(sid, username)+ if session_ok is None:+ # Unverifiable right now: refuse this unsubscribe only.+ logger.warning(+ f"Rejected unsubscribe from {sid} ({username}): session could "+ "not be validated right now"+ )+ returnif not session_ok:logger.warning(f"Rejected unsubscribe from {sid} ({username}): originating "diff --git a/src/local_deep_research/web/static/js/services/api.js b/src/local_deep_research/web/static/js/services/api.jsindex 27b47994b5..24539aa4ed 100644--- a/src/local_deep_research/web/static/js/services/api.js+++ b/src/local_deep_research/web/static/js/services/api.js@@ -64,8 +64,9 @@ async function fetchWithCsrfRecovery(url, options = {}, send = fetch) {if (getAuthContext() !== context) {throw new Error('Your sign-in changed. Reload this page before trying again.');}- // The endpoint returns 401/409 if the login expired or was replaced. Let- // the existing auth/error handling surface it; no original write is sent.+ // The refresh returns 401/409 if the login expired or was replaced, or+ // 503 if the session cannot be checked right now (it is kept). Let the+ // existing auth/error handling surface it; no original write is sent.if (!refresh.ok) return refresh;const fresh = await refresh.json();options.signal?.throwIfAborted();diff --git a/tests/conftest.py b/tests/conftest.pyindex f70fa6679a..2f37737bd4 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -652,6 +652,126 @@ def _legacy_bare_username_auth(request):yield+# Last epoch projected per username under the _durable_epoch_projection+# shim -- see the fixture for why stability across failures matters.+_LAST_EPOCH_PROJECTION: dict[str, str] = {}++[email protected](autouse=True)+def _durable_epoch_projection(request):+ """Keep fabricated test sessions working under durable-epoch binding.++ ``SessionManager.create_session`` binds the user's durable rekey epoch+ (read from disk) so a password change in ANY worker fences every+ stale session. In production a session is only ever created at login,+ right after ``open_user_database`` proved the credential against the+ epoch-fenced database, so the projection always resolves. The test+ fleet, however, fabricates sessions and logins against mocked+ databases for users with no on-disk state; the projection would raise+ ``RekeyRecoveryRequiredError`` for them.++ This autouse fixture relaxes the seam the same way+ ``_legacy_bare_username_auth`` relaxes the server-side-session gate:+ sessions fabricated under tests bind a deterministic per-username+ epoch. Tests that must prove the real fencing contract (binding,+ stale-epoch removal, transient-vs-ambiguous failure discipline) opt+ out with ``@pytest.mark.real_durable_epoch`` and run against real+ epoch files.+ """+ if request.node.get_closest_marker("real_durable_epoch"):+ # Fencing tests: leave the real epoch projection in place.+ yield+ return++ # A shared monkeypatch dependency would outlive singleton teardown and+ # leave unrelated tests' provider mocks active during that reset.+ with pytest.MonkeyPatch.context() as epoch_patch:+ _install_durable_epoch_projection(epoch_patch)+ yield+++def _install_durable_epoch_projection(monkeypatch):+ import local_deep_research.database.encrypted_db as _encrypted_db+ from local_deep_research.database.rekey_epoch import (+ read_user_authorization_epoch as _real_projection,+ )++ def _fabricated(username):+ return _LAST_EPOCH_PROJECTION.setdefault(+ username, f"test-epoch-{username}"+ )++ def _project_through_or_fabricate(username):+ # Sessions created against real databases (module-scoped client+ # fixtures register + log in real users) keep their REAL epochs,+ # and any failure projecting them propagates exactly as it would+ # in production. Only two cases get a deterministic stand-in:+ #+ # * ``db_manager`` itself is a test double (not a DatabaseManager,+ # or one whose path resolution is mocked), so there is nothing+ # durable to project;+ # * the resolved path holds NO durable epoch state at all (neither+ # ``.rekey.epoch`` nor ``.rekey.pending``): a fabricated login+ # against a mocked database, or a real user whose data dir a+ # module fixture un-pinned or deleted.+ #+ # STABILITY cache: a username's last real projection is remembered+ # and reused for the no-state case, so a create/validate pair that+ # straddles a data-dir pin/restore does not mismatch-destroy the+ # record.+ manager = _encrypted_db.db_manager+ if not isinstance(manager, _encrypted_db.DatabaseManager):+ return _fabricated(username)+ db_path = manager._get_user_db_path(username)+ if not isinstance(db_path, Path):+ return _fabricated(username)+ has_state = (+ db_path.with_name(f"{db_path.name}.rekey.epoch").exists()+ or db_path.with_name(f"{db_path.name}.rekey.pending").exists()+ )+ if not has_state:+ return _fabricated(username)+ projected = _real_projection(username)+ _LAST_EPOCH_PROJECTION[username] = projected+ return projected++ monkeypatch.setattr(+ "local_deep_research.web.auth.session_manager."+ "read_user_authorization_epoch",+ _project_through_or_fabricate,+ )++ # The login route refuses (503) when the open recorded no epoch proof+ # for the user. A test that swaps the login route's ``db_manager`` (or+ # its ``open_user_database``) for a double records none, so for that+ # case only the legacy "no proof, no straddle check" path is kept+ # (``create_session(proven_epoch=None)``). A real open that records no+ # proof still refuses exactly as in production.+ _real_proven_epoch_for = _encrypted_db.LoginProof.proven_epoch_for++ def _proven_epoch_for_or_double(self, username):+ try:+ return _real_proven_epoch_for(self, username)+ except _encrypted_db.MissingLoginProofError:+ router = sys.modules.get("local_deep_research.web.routers.auth")+ manager = getattr(router, "db_manager", None)+ real_open = _encrypted_db.DatabaseManager.open_user_database+ is_double = (+ not isinstance(manager, _encrypted_db.DatabaseManager)+ or "open_user_database" in vars(manager)+ or type(manager).open_user_database is not real_open+ )+ if is_double:+ return None+ raise++ monkeypatch.setattr(+ _encrypted_db.LoginProof,+ "proven_epoch_for",+ _proven_epoch_for_or_double,+ )++@pytest.fixturedef app(temp_data_dir, monkeypatch):"""Create an app configured for testing."""diff --git a/tests/database/test_login_epoch_proof.py b/tests/database/test_login_epoch_proof.pynew file mode 100644index 0000000000..008cea6bdc--- /dev/null+++ b/tests/database/test_login_epoch_proof.py@@ -0,0 +1,353 @@+"""What ``open_user_database`` proves, and when it refuses to prove it.++``capture_login_proof`` records the durable epoch a successful open+verified the password against under the rekey sidecar lock; the login+route binds its session to that epoch. These tests pin the recorded+epoch on each success path, that nothing is recorded outside a capture+block, that the outer open wins over the phase-2 re-entry, and two+refusals the proof relies on: a cold open whose epoch re-read does not+find the epoch it ran under (the committed epoch moved, went missing or+is pending, or the read fails) is discarded and fails closed instead of+reading as a wrong password, and+pending-rekey recovery verifies the candidate key before it finalizes+the epoch.+"""++from __future__ import annotations++from pathlib import Path+from unittest.mock import MagicMock, patch++import pytest++from local_deep_research.database import encrypted_db+from local_deep_research.database import rekey_lock as rekey_lock_module+from local_deep_research.database.encrypted_db import (+ DatabaseManager,+ LoginProof,+ MissingLoginProofError,+ _RekeyVerificationError,+ capture_login_proof,+)+from local_deep_research.database.rekey_lock import (+ RekeyEpochUnreadableError,+ RekeyRecoveryRequiredError,+ acquire_rekey_lock,+)+++def _new_manager(data_dir: Path) -> DatabaseManager:+ with patch.object(+ DatabaseManager, "_check_encryption_available", return_value=True+ ):+ manager = DatabaseManager()+ manager.data_dir = data_dir+ return manager+++def manager(tmp_path: Path) -> DatabaseManager:+ return _new_manager(tmp_path)+++def _existing_db(manager: DatabaseManager, username: str) -> Path:+ db_path = manager._get_user_db_path(username)+ db_path.touch()+ return db_path+++def _initialize_epoch(db_path: Path) -> str:+ with acquire_rekey_lock(db_path):+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)+++def test_cold_open_records_the_epoch_it_ran_under(+ manager: DatabaseManager,+) -> None:+ epoch = _initialize_epoch(_existing_db(manager, "alice"))+ engine = MagicMock()++ with (+ patch.object(manager, "_open_user_database_cold", return_value=engine),+ patch.object(manager, "_run_phase2_rekey"),+ capture_login_proof() as proof,+ ):+ opened = manager.open_user_database("alice", "password")++ assert opened is engine+ assert (proof.username, proof.epoch) == ("alice", epoch)+++def test_cache_claim_records_the_claimed_generation_epoch(+ manager: DatabaseManager,+) -> None:+ epoch = _initialize_epoch(_existing_db(manager, "alice"))+ engine = MagicMock()+ manager._cache_connection(+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)+ )++ with (+ patch.object(manager, "_open_user_database_cold") as cold_open,+ capture_login_proof() as proof,+ ):+ opened = manager.open_user_database("alice", "password")++ assert opened is engine+ cold_open.assert_not_called()+ assert (proof.username, proof.epoch) == ("alice", epoch)+++def test_nothing_is_recorded_outside_a_capture_block(+ manager: DatabaseManager,+) -> None:+ _initialize_epoch(_existing_db(manager, "alice"))++ with (+ patch.object(+ manager, "_open_user_database_cold", return_value=MagicMock()+ ),+ patch.object(manager, "_run_phase2_rekey"),+ ):+ manager.open_user_database("alice", "password")++ with capture_login_proof() as proof:+ pass+ assert (proof.username, proof.epoch) == (None, None)+++def test_outer_open_wins_over_the_phase2_reentry(+ manager: DatabaseManager,+) -> None:+ epoch = _initialize_epoch(_existing_db(manager, "alice"))++ def failed_reentry(username: str, _password: str) -> None:+ # Phase 2 re-enters open_user_database on this thread; a failed+ # re-entry records "no proof" before the outer call returns.+ encrypted_db._record_login_proof(username, None)++ with (+ patch.object(+ manager, "_open_user_database_cold", return_value=MagicMock()+ ),+ patch.object(manager, "_run_phase2_rekey", side_effect=failed_reentry),+ capture_login_proof() as proof,+ ):+ manager.open_user_database("alice", "password")++ assert proof.epoch == epoch+++def test_cold_open_is_discarded_when_the_epoch_moves_underneath_it(+ manager: DatabaseManager,+) -> None:+ db_path = _existing_db(manager, "alice")+ _initialize_epoch(db_path)+ engine = MagicMock()++ def cold_open_racing_a_publisher(*_args, **_kwargs) -> MagicMock:+ # The sidecar is held by this open; a publisher that bypasses it+ # (or a durable write this process cannot see coming) moves the+ # committed epoch while the engine is being built.+ rekey_lock_module.advance_rekey_epoch(db_path)+ return engine++ with (+ patch.object(+ manager,+ "_open_user_database_cold",+ side_effect=cold_open_racing_a_publisher,+ ),+ patch.object(manager, "_run_phase2_rekey") as phase2,+ capture_login_proof() as proof,+ ):+ # The password opened the database, so this is not a wrong+ # password: the open fails closed with the error the login route+ # answers with 503 (lockout counter untouched), never the None a+ # wrong password returns (401, counted toward lockout).+ with pytest.raises(RekeyRecoveryRequiredError):+ manager.open_user_database("alice", "password")++ # The engine built under the superseded epoch is never published.+ engine.dispose.assert_called_once_with()+ assert "alice" not in manager.connections+ phase2.assert_not_called()+ assert proof.epoch is None+++def _make_epoch_file_unreadable(db_path: Path) -> None:+ # A directory where the epoch file should be: read_text raises+ # IsADirectoryError, an OSError other than FileNotFoundError and not+ # a PermissionError, so the reader raises RekeyEpochUnreadableError+ # on the first attempt (the same outcome as EMFILE or EIO).+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")+ epoch_file.unlink()+ epoch_file.mkdir()+++def _remove_committed_epoch(db_path: Path) -> None:+ # No committed and no pending epoch: the state is read, and does not+ # vouch for the epoch the open ran under.+ db_path.with_name(f"{db_path.name}.rekey.epoch").unlink()+++ ("disturb", "expected_error"),+ [+ pytest.param(+ _make_epoch_file_unreadable,+ RekeyEpochUnreadableError,+ id="epoch-file-unreadable",+ ),+ pytest.param(+ rekey_lock_module.begin_rekey_epoch,+ RekeyRecoveryRequiredError,+ id="pending-epoch-appeared",+ ),+ pytest.param(+ _remove_committed_epoch,+ RekeyRecoveryRequiredError,+ id="committed-epoch-removed",+ ),+ ],+)+def test_post_open_epoch_recheck_that_cannot_vouch_fails_closed(+ manager: DatabaseManager, disturb, expected_error+) -> None:+ # The cold open proved the password. If the re-read of the epoch+ # right after it fails, or finds durable state that does not vouch,+ # nothing is known about the credential: the open must raise the+ # RekeyOperationalError the login route answers with 503 (lockout+ # counter untouched), never return the (None, ...) a wrong password+ # returns (401, counted toward lockout).+ db_path = _existing_db(manager, "alice")+ _initialize_epoch(db_path)+ engine = MagicMock()++ def cold_open_then_disturb(*_args, **_kwargs) -> MagicMock:+ disturb(db_path)+ return engine++ with (+ patch.object(+ manager,+ "_open_user_database_cold",+ side_effect=cold_open_then_disturb,+ ),+ patch.object(manager, "_run_phase2_rekey") as phase2,+ capture_login_proof() as proof,+ ):+ with pytest.raises(expected_error):+ manager.open_user_database("alice", "password")++ # The engine is still discarded unpublished.+ engine.dispose.assert_called_once_with()+ assert "alice" not in manager.connections+ phase2.assert_not_called()+ assert proof.epoch is None+++def test_pending_recovery_verifies_the_candidate_before_finalizing(+ manager: DatabaseManager,+) -> None:+ db_path = _existing_db(manager, "alice")+ committed = _initialize_epoch(db_path)+ with acquire_rekey_lock(db_path):+ pending = rekey_lock_module.begin_rekey_epoch(db_path)+ connection = MagicMock()++ with (+ patch.object(+ manager, "_make_sqlcipher_connection", return_value=connection+ ),+ patch.object(+ manager,+ "_verify_rekey_candidate",+ side_effect=_RekeyVerificationError("quick check"),+ ) as verify,+ patch.object(+ manager, "_open_user_database_cold", return_value=MagicMock()+ ) as cold_open,+ patch.object(manager, "_run_phase2_rekey"),+ ):+ # A password that opens the file but whose candidate key fails+ # verification must not finalize the pending epoch.+ with pytest.raises(RekeyRecoveryRequiredError):+ manager.open_user_database("alice", "password")++ verify.assert_called_once_with(connection, db_path)+ cold_open.assert_not_called()+ state = rekey_lock_module.read_rekey_epoch_state(db_path)+ assert (state.committed_epoch, state.pending_epoch) == (committed, pending)+++def test_concurrent_first_login_records_the_epoch_of_the_engine_it_joins(+ manager: DatabaseManager,+) -> None:+ # Two first logins race: this one's cache probe missed (nothing cached+ # yet), then it waited on the per-user init lock while the other login+ # cold-opened and cached the engine. Inside the lock it finds that+ # engine, verifies the password against it, and returns it -- the+ # proof must carry the epoch it verified under, like every other path.+ epoch = _initialize_epoch(_existing_db(manager, "alice"))+ engine = MagicMock()+ real_init_lock = manager._get_init_lock("alice")++ class _OtherLoginFinishesFirst:+ def acquire(self, *args, **kwargs):+ acquired = real_init_lock.acquire(*args, **kwargs)+ if acquired:+ manager._cache_connection(+ "alice",+ manager._make_connection_cache_entry(+ engine, "password", epoch+ ),+ )+ return acquired++ def release(self):+ real_init_lock.release()++ with (+ patch.object(+ manager, "_get_init_lock", return_value=_OtherLoginFinishesFirst()+ ),+ patch.object(manager, "_open_user_database_cold") as cold_open,+ patch.object(manager, "_run_phase2_rekey") as phase2,+ capture_login_proof() as proof,+ ):+ opened = manager.open_user_database("alice", "password")++ assert opened is engine+ cold_open.assert_not_called()+ phase2.assert_not_called()+ assert (proof.username, proof.epoch) == ("alice", epoch)+++# The autouse epoch shim relaxes proven_epoch_for for test doubles; these+# two pin the real method.[email protected]_durable_epoch+def test_proven_epoch_for_returns_the_epoch_proved_for_that_user() -> None:+ proof = LoginProof()+ proof.username = "alice"+ proof.epoch = "e1"++ assert proof.proven_epoch_for("alice") == "e1"++[email protected]_durable_epoch+ ("recorded_user", "recorded_epoch"),+ [(None, None), ("mallory", "e1"), ("alice", None)],+)+def test_proven_epoch_for_refuses_a_missing_or_foreign_proof(+ recorded_user: str | None, recorded_epoch: str | None+) -> None:+ # A proof for another user (or none at all) must refuse, not quietly+ # let the session skip the straddle check.+ proof = LoginProof()+ proof.username = recorded_user+ proof.epoch = recorded_epoch++ with pytest.raises(MissingLoginProofError):+ proof.proven_epoch_for("alice")diff --git a/tests/database/test_rekey_epoch.py b/tests/database/test_rekey_epoch.pyindex 5ad8ee35f9..a812676295 100644--- a/tests/database/test_rekey_epoch.py+++ b/tests/database/test_rekey_epoch.py@@ -17,6 +17,7 @@ from local_deep_research.database.rekey_epoch import (read_user_authorization_epoch,)from local_deep_research.database.rekey_lock import (+ RekeyEpochUnreadableError,RekeyRecoveryRequiredError,acquire_rekey_lock,read_rekey_epoch,@@ -162,14 +163,19 @@ def test_authorization_projection_does_not_acquire_sidecar(def test_user_authorization_epoch_resolves_hashed_database_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:- # Given Alice's metadata exists beneath the configured data root+ # Given Alice's metadata exists beneath the manager's data directory+ # (pinned explicitly: another module in this worker may have pinned+ # the singleton elsewhere, and the projection resolves through it)+ from local_deep_research.database.encrypted_db import db_manager+monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))database_dir = tmp_path / "encrypted_databases"database_dir.mkdir()+ monkeypatch.setattr(db_manager, "_data_dir_override", database_dir)db_path = database_dir / get_user_database_filename("alice")_write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)- # When authorization resolves Alice rather than importing db_manager+ # When authorization resolves Alice through db_managerepoch = read_user_authorization_epoch("alice")# Then it projects the identity from Alice's hashed database path@@ -219,6 +225,207 @@ def test_projection_cannot_tear_into_the_superseded_epoch(assert epoch == _EPOCH_TWO[email protected]("error_number", [errno.EMFILE, errno.EIO])+def test_epoch_read_os_error_is_typed_unreadable(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, error_number: int+) -> None:+ # An I/O failure on the read is distinguishable (for session+ # validation) but still a RekeyRecoveryRequiredError, so every+ # existing handler keeps failing closed.+ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ real_open = os.open++ def failing_open(path, *args, **kwargs):+ if Path(path).name.endswith(".rekey.epoch"):+ raise OSError(error_number, "injected")+ return real_open(path, *args, **kwargs)++ monkeypatch.setattr(os, "open", failing_open)++ with pytest.raises(RekeyEpochUnreadableError) as raised:+ read_authorization_epoch(db_path)+ assert isinstance(raised.value, RekeyRecoveryRequiredError)+ assert raised.value.errno == error_number+ assert read_rekey_epoch(db_path) is None+++def test_undecodable_epoch_is_not_typed_unreadable(tmp_path: Path) -> None:+ # Content that was read but cannot be decoded is durable state, not a+ # transient read fault.+ db_path = tmp_path / "user.db"+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_bytes(b"\xff\xfe")++ with pytest.raises(RekeyRecoveryRequiredError) as raised:+ read_authorization_epoch(db_path)+ assert not isinstance(raised.value, RekeyEpochUnreadableError)+++# --- Windows sharing violations between lock-free readers and writers ----+#+# CPython opens files on Windows without FILE_SHARE_DELETE. Session+# validation reads ``.rekey.epoch``/``.rekey.pending`` WITHOUT the sidecar+# lock on every authenticated request, so a reader can hold the file open+# while ``change_password`` replaces/unlinks it (the writer then fails with+# PermissionError, after the physical rekey), and a reader can open a file+# whose replace/unlink is in flight (the reader gets PermissionError). The+# window is microseconds; both sides retry a PermissionError a bounded+# number of times. A Linux CI runner cannot produce the violation, so it is+# injected at the real call sites.+++def _sharing_violations(+ monkeypatch: pytest.MonkeyPatch, target, name: str, failures: int+) -> list[object]:+ """Make ``target.name`` raise PermissionError ``failures`` times."""+ import local_deep_research.database.rekey_lock as rekey_lock_module++ monkeypatch.setattr(+ rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0, raising=False+ )+ real = getattr(target, name)+ calls: list[object] = []++ def flaky(*args, **kwargs):+ calls.append(args[0] if args else None)+ if len(calls) <= failures:+ raise PermissionError(errno.EACCES, "sharing violation")+ return real(*args, **kwargs)++ monkeypatch.setattr(target, name, flaky)+ return calls+++def test_transient_read_sharing_violation_is_retried(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ failures = 2+ calls = _sharing_violations(monkeypatch, os, "open", failures)++ # pending (missing) read: `failures` violations then FileNotFoundError;+ # the epoch read then succeeds on its first attempt.+ assert read_authorization_epoch(db_path) == _EPOCH_ONE+ assert len(calls) == failures + 2+++def test_persistent_read_sharing_violation_fails_closed(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ import local_deep_research.database.rekey_lock as rekey_lock_module++ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ attempts = rekey_lock_module._SHARING_VIOLATION_ATTEMPTS+ calls = _sharing_violations(monkeypatch, os, "open", 10**6)++ # Bounded: the reader gives up after `attempts` tries and NEVER yields+ # an epoch -- the typed unreadable error makes session validation+ # answer 503 for that request (record kept), and cache validation+ # treat the epoch as unknown.+ with pytest.raises(RekeyEpochUnreadableError) as raised:+ read_authorization_epoch(db_path)+ assert raised.value.errno == errno.EACCES+ assert len(calls) == attempts+ assert read_rekey_epoch(db_path) is None+++def test_transient_replace_sharing_violation_still_commits(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ import os++ import local_deep_research.database.rekey_lock as rekey_lock_module++ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ with acquire_rekey_lock(db_path):+ pending = rekey_lock_module.begin_rekey_epoch(db_path)+ calls = _sharing_violations(monkeypatch, os, "replace", 2)+ rekey_lock_module.commit_rekey_epoch(db_path, pending)++ assert len(calls) == 3+ assert rekey_lock_module.read_rekey_epoch_state(+ db_path+ ).committed_epoch == (pending)+ # No temporary token file is left behind by the failed attempts.+ assert sorted(p.name for p in tmp_path.iterdir()) == [+ "user.db.rekey.epoch",+ "user.db.rekey.lock",+ "user.db.rekey.pending",+ ]+++def test_transient_unlink_sharing_violation_still_clears_pending(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ import local_deep_research.database.rekey_lock as rekey_lock_module++ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ with acquire_rekey_lock(db_path):+ pending = rekey_lock_module.begin_rekey_epoch(db_path)+ rekey_lock_module.commit_rekey_epoch(db_path, pending)+ calls = _sharing_violations(monkeypatch, Path, "unlink", 2)+ rekey_lock_module.clear_pending_rekey_epoch(db_path, pending)++ assert len(calls) == 3+ assert read_authorization_epoch(db_path) == pending+ assert not db_path.with_name("user.db.rekey.pending").exists()+++def test_transient_unlink_sharing_violation_still_publishes_a_new_epoch(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # publish_new_database_epoch removes a stale pending token left beside+ # a just-created database; a reader holding it open on Windows must+ # not fail the creation (#7830). Only the pending token's unlink is+ # made flaky: the epoch writer's own temp-file cleanup is untouched.+ import local_deep_research.database.rekey_lock as rekey_lock_module++ monkeypatch.setattr(rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0)+ db_path = tmp_path / "user.db"+ pending_path = db_path.with_name("user.db.rekey.pending")+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=_EPOCH_TWO)+ real_unlink = Path.unlink+ pending_calls: list[Path] = []++ def flaky_unlink(self: Path, *args, **kwargs):+ if self == pending_path:+ pending_calls.append(self)+ if len(pending_calls) <= 2:+ raise PermissionError(errno.EACCES, "sharing violation")+ return real_unlink(self, *args, **kwargs)++ monkeypatch.setattr(Path, "unlink", flaky_unlink)+ with acquire_rekey_lock(db_path):+ epoch = rekey_lock_module.publish_new_database_epoch(db_path)++ assert len(pending_calls) == 3+ assert not pending_path.exists()+ assert epoch not in (_EPOCH_ONE, _EPOCH_TWO)+ assert read_authorization_epoch(db_path) == epoch+++def test_persistent_replace_permission_error_still_propagates(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ import os++ import local_deep_research.database.rekey_lock as rekey_lock_module++ db_path = tmp_path / "user.db"+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)+ with acquire_rekey_lock(db_path):+ calls = _sharing_violations(monkeypatch, os, "replace", 10**6)+ with pytest.raises(PermissionError):+ rekey_lock_module.begin_rekey_epoch(db_path)++ assert len(calls) == rekey_lock_module._SHARING_VIOLATION_ATTEMPTS+ assert not db_path.with_name("user.db.rekey.pending").exists()++def test_authorization_epoch_import_contract() -> None:name = "local_deep_research.database.authorization_epoch"assert importlib.util.find_spec(name) is not Nonediff --git a/tests/database/test_rekey_gate_registry.py b/tests/database/test_rekey_gate_registry.pyindex 9a10265ac6..47d939d78d 100644--- a/tests/database/test_rekey_gate_registry.py+++ b/tests/database/test_rekey_gate_registry.py@@ -38,7 +38,6 @@ def test_wrapper_cleanup(tmp_path, monkeypatch, failure):request = sut._SidecarLockRequest(db_path=tmp_path / "user.db",lock_path=path,- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,blocking=False,)references = []@@ -89,7 +88,6 @@ def test_waiter_keeps_gate_across_holder_exit_and_new_arrival(request = sut._SidecarLockRequest(db_path=tmp_path / "user.db",lock_path=path,- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,blocking=False,)looked_up = threading.Event()@@ -167,7 +165,6 @@ def test_blocking_waiter_and_independent_path_use_real_sidecars(return sut._SidecarLockRequest(db_path=tmp_path / "user.db",lock_path=lock_path,- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,blocking=False,gate_wait=gate_wait,)diff --git a/tests/database/test_rekey_lock.py b/tests/database/test_rekey_lock.pyindex 9b2f2be752..7f0df51865 100644--- a/tests/database/test_rekey_lock.py+++ b/tests/database/test_rekey_lock.py@@ -44,21 +44,6 @@ def _hold_rekey_lock(raise RuntimeError("parent did not release rekey-lock worker")-def _hold_authorization_lease(- db_path: str,- mode: str,- acquired: Event,- release: Event,-) -> None:- lease_mode = rekey_lock_module.AuthorizationLeaseMode(mode)- with rekey_lock_module.acquire_authorization_lease(- Path(db_path), lease_mode- ):- acquired.set()- if not release.wait(timeout=10):- raise _LockWorkerTimeoutError--def _assert_sidecar_io_error(error: rekey_lock_module.RekeySidecarIOError,db_path: Path,@@ -76,10 +61,6 @@ class _ProtectedBodyError(Exception):pass-class _LockWorkerTimeoutError(Exception):- pass--def test_rekey_lock_contends_across_processes(tmp_path: Path) -> None:# Given a separate process holding this database's sidecar lockdb_path = tmp_path / "user.db"@@ -288,136 +269,6 @@ def test_timeout_bounds_the_wait_for_an_in_process_holder(assert bound <= elapsed < bound + 5-def test_authorization_shared_leases_coexist_across_processes(- tmp_path: Path,-) -> None:- # Given another process holds a shared authorization lease- db_path = tmp_path / "user.db"- shared = rekey_lock_module.AuthorizationLeaseMode.SHARED- context = multiprocessing.get_context("spawn")- acquired = context.Event()- release = context.Event()- process = context.Process(- target=_hold_authorization_lease,- args=(str(db_path), shared.value, acquired, release),- )- parent_acquired = False- process.start()-- try:- assert acquired.wait(timeout=10), "child did not acquire shared lease"-- # When this process requests another nonblocking shared lease- with rekey_lock_module.acquire_authorization_lease(- db_path, shared, blocking=False- ):- parent_acquired = True- finally:- release.set()- process.join(timeout=10)- if process.is_alive():- process.terminate()- process.join(timeout=5)-- # Then both shared leases coexist and the child exits cleanly- assert parent_acquired is True- assert process.exitcode == 0---def test_authorization_exclusive_contends_with_shared_holder(- tmp_path: Path,-) -> None:- # Given another process holds a shared authorization lease- db_path = tmp_path / "user.db"- modes = rekey_lock_module.AuthorizationLeaseMode- context = multiprocessing.get_context("spawn")- acquired = context.Event()- release = context.Event()- process = context.Process(- target=_hold_authorization_lease,- args=(str(db_path), modes.SHARED.value, acquired, release),- )- process.start()-- try:- assert acquired.wait(timeout=10), "child did not acquire shared lease"-- # When this process requests a nonblocking exclusive lease- with pytest.raises(RekeyLockContentionError):- with rekey_lock_module.acquire_authorization_lease(- db_path, modes.EXCLUSIVE, blocking=False- ):- pytest.fail("exclusive lease bypassed shared holder")- finally:- release.set()- process.join(timeout=10)- if process.is_alive():- process.terminate()- process.join(timeout=5)-- # Then contention is distinct and the shared holder exits cleanly- assert process.exitcode == 0---def test_authorization_shared_contends_with_exclusive_holder(- tmp_path: Path,-) -> None:- # Given another process holds an exclusive authorization lease- db_path = tmp_path / "user.db"- modes = rekey_lock_module.AuthorizationLeaseMode- context = multiprocessing.get_context("spawn")- acquired = context.Event()- release = context.Event()- process = context.Process(- target=_hold_authorization_lease,- args=(str(db_path), modes.EXCLUSIVE.value, acquired, release),- )- process.start()-- try:- assert acquired.wait(timeout=10), (- "child did not acquire exclusive lease"- )-- # When this process requests a nonblocking shared lease- with pytest.raises(RekeyLockContentionError):- with rekey_lock_module.acquire_authorization_lease(- db_path, modes.SHARED, blocking=False- ):- pytest.fail("shared lease bypassed exclusive holder")- finally:- release.set()- process.join(timeout=10)- if process.is_alive():- process.terminate()- process.join(timeout=5)-- # Then contention is distinct and the exclusive holder exits cleanly- assert process.exitcode == 0---def test_authorization_sidecar_is_persistent_and_owner_only(- tmp_path: Path,-) -> None:- # Given a database path without an authorization sidecar- db_path = tmp_path / "user.db"- lock_path = tmp_path / "user.db.rekey.authorization.lock"- shared = rekey_lock_module.AuthorizationLeaseMode.SHARED-- # When the authorization lease is acquired and released twice- with rekey_lock_module.acquire_authorization_lease(db_path, shared):- assert lock_path.exists()- original_inode = lock_path.stat().st_ino- with rekey_lock_module.acquire_authorization_lease(db_path, shared):- pass-- # Then the same owner-only sidecar remains available- assert lock_path.exists()- assert lock_path.stat().st_ino == original_inode- if os.name != "nt":- assert stat.S_IMODE(lock_path.stat().st_mode) == 0o600--def test_rekey_lock_sidecar_is_persistent_and_owner_only(tmp_path: Path,) -> None:@@ -488,11 +339,8 @@ def test_symlinked_rekey_lock_sidecar_is_refused(tmp_path: Path) -> None:os.name == "nt" or not hasattr(os, "link"),reason="POSIX hard links required",)- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]-)def test_hard_linked_sidecar_is_locked_without_touching_its_mode(- tmp_path: Path, lock_suffix: str+ tmp_path: Path,) -> None:# Given a hard link at the sidecar name whose other name is outside the# sidecar's directory (a backup or snapshot tool that hard-links the@@ -505,17 +353,11 @@ def test_hard_linked_sidecar_is_locked_without_touching_its_mode(target = snapshot_dir / "unrelated.txt"target.write_text("unrelated\n", encoding="utf-8")os.chmod(target, 0o644)- os.link(target, db_path.with_name(f"user.db{lock_suffix}"))+ os.link(target, db_path.with_name("user.db.rekey.lock"))- # When the sidecar is opened for the rekey lock or authorization lease- if lock_suffix == ".rekey.lock":- acquire = acquire_rekey_lock(db_path)- else:- acquire = rekey_lock_module.acquire_authorization_lease(- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED- )+ # When the sidecar is opened for the rekey lockentered = False- with acquire:+ with acquire_rekey_lock(db_path):entered = True# Then the lock is taken (no lockout), and the linked file keeps its@@ -529,11 +371,8 @@ def test_hard_linked_sidecar_is_locked_without_touching_its_mode(os.name == "nt" or not hasattr(os, "link"),reason="POSIX hard links required",)- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]-)def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(- tmp_path: Path, monkeypatch: pytest.MonkeyPatch, lock_suffix: str+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:# Given an external hard link distinct from the real DB, WAL and shm,# so classification must reach the directory scan to rule out aliases.@@ -553,7 +392,7 @@ def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(original_bytes = b"unrelated\n"target.write_bytes(original_bytes)target.chmod(0o644)- sidecar = db_path.with_name(f"user.db{lock_suffix}")+ sidecar = db_path.with_name("user.db.rekey.lock")os.link(target, sidecar)original = target.stat()identity = (original.st_dev, original.st_ino)@@ -581,12 +420,9 @@ def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(open_attempts.append(os.fspath(path))return real_open(path, *args, **kwargs)- if lock_suffix == ".rekey.lock":- acquire = acquire_rekey_lock(db_path)- else:- acquire = rekey_lock_module.acquire_authorization_lease(- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED- )+ # The authorization-lease sidecar main also covered here was removed+ # with its unused API (#6750); the rekey lock is the only sidecar.+ acquire = acquire_rekey_lock(db_path)entered = Falserefusal = None# Only the exact lock parent is unlistable; all opens remain real.@@ -716,15 +552,12 @@ def test_sidecar_hard_linked_to_the_database_is_refused_unopened(os.name == "nt" or not hasattr(os, "mkfifo"),reason="POSIX FIFOs required",)- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]-)def test_non_regular_sidecar_is_refused_and_its_fd_closed(- tmp_path: Path, monkeypatch: pytest.MonkeyPatch, lock_suffix: str+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:# Given a FIFO planted at the sidecar namedb_path = tmp_path / "user.db"- sidecar = db_path.with_name(f"user.db{lock_suffix}")+ sidecar = db_path.with_name("user.db.rekey.lock")os.mkfifo(sidecar, 0o600)opened_fds: list[int] = []real_open = os.open@@ -737,15 +570,9 @@ def test_non_regular_sidecar_is_refused_and_its_fd_closed(monkeypatch.setattr(rekey_lock_module.os, "open", recording_open)- # When the sidecar is opened for the rekey lock or authorization lease- if lock_suffix == ".rekey.lock":- acquire = acquire_rekey_lock(db_path)- else:- acquire = rekey_lock_module.acquire_authorization_lease(- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED- )+ # When the sidecar is opened for the rekey lockwith pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:- with acquire:+ with acquire_rekey_lock(db_path):pytest.fail("sidecar lock acquired on a FIFO")monkeypatch.undo()@@ -1140,7 +967,6 @@ def test_sidecar_acquire_failure_is_typed(_fd: int,_lock_path: Path,*,- mode: rekey_lock_module.AuthorizationLeaseMode | None = None,blocking: bool = False,) -> None:raise OSError(errno.EIO, "sensitive raw OS text")@@ -1365,8 +1191,6 @@ def test_body_error_inside_a_callers_except_block_carries_the_release_note(class _FakeMsvcrt(ModuleType):LK_LOCK: ClassVar[int] = 0LK_NBLCK: ClassVar[int] = 1- LK_RLCK: ClassVar[int] = 2- LK_NBRLCK: ClassVar[int] = 3LK_UNLCK: ClassVar[int] = 4def __init__(self, acquire_error: OSError | None = None) -> None:@@ -1376,10 +1200,7 @@ class _FakeMsvcrt(ModuleType):def locking(self, fd: int, mode: int, size: int) -> None:self.calls.append((os.lseek(fd, 0, os.SEEK_CUR), mode, size))- if (- mode in {self.LK_NBLCK, self.LK_NBRLCK}- and self.acquire_error is not None- ):+ if mode == self.LK_NBLCK and self.acquire_error is not None:raise self.acquire_error@@ -1407,30 +1228,27 @@ def test_windows_backend_locks_one_byte_from_start(]-def test_windows_backend_selects_shared_and_exclusive_modes(+def test_windows_backend_selects_blocking_and_nonblocking_modes(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:# Given a persistent sidecar and a Windows locking backend- lock_path = tmp_path / "user.db.rekey.authorization.lock"+ lock_path = tmp_path / "user.db.rekey.lock"fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)backend = _FakeMsvcrt()monkeypatch.setitem(sys.modules, "msvcrt", backend)- modes = rekey_lock_module.AuthorizationLeaseModetry:- # When shared nonblocking and exclusive blocking leases are acquired- _acquire_windows_lock(fd, lock_path, mode=modes.SHARED)+ # When nonblocking and blocking exclusive locks are acquired+ _acquire_windows_lock(fd, lock_path)_release_windows_lock(fd)- _acquire_windows_lock(- fd, lock_path, mode=modes.EXCLUSIVE, blocking=True- )+ _acquire_windows_lock(fd, lock_path, blocking=True)_release_windows_lock(fd)finally:os.close(fd)- # Then Windows uses read-lock and write-lock modes respectively+ # Then Windows uses the non-blocking and blocking write-lock modesassert [call[1] for call in backend.calls] == [- backend.LK_NBRLCK,+ backend.LK_NBLCK,backend.LK_UNLCK,backend.LK_LOCK,backend.LK_UNLCK,diff --git a/tests/security/test_auth_router_credential_frames.py b/tests/security/test_auth_router_credential_frames.pynew file mode 100644index 0000000000..bb0d2a6983--- /dev/null+++ b/tests/security/test_auth_router_credential_frames.py@@ -0,0 +1,143 @@+"""No traceback may be rendered from an auth-router frame holding a password.++``login``, ``register``, ``change_password`` and the post-login worker+(``_perform_post_login_tasks*``) hold the plaintext password in their+locals; ``_rollback_partial_session`` runs inside ``login`` and+``register``, and loguru's ``backtrace`` extends a traceback rendered+from it up into those caller frames.+A traceback rendered with loguru ``diagnose`` on prints every frame local,+and whether any sink has ``diagnose`` on is not knowable at the call site+(``config_logger`` enables it on the stderr sink when debug is on, from+the ``app.debug`` setting or the environment, and ``LDR_LOGURU_DIAGNOSE``+is set; loguru's default handler has it on). So these functions log+exception TYPES only. This source sweep pins that:+no ``logger.exception``, no ``exc_info=``, no ``logger.opt(exception=...)``+other than a literal False, and no ``logger.catch`` inside them.+"""++import ast+from pathlib import Path++import local_deep_research.web.routers.auth as auth_module++CREDENTIAL_FRAMES = {+ "login",+ "register",+ "change_password",+ "_perform_post_login_tasks",+ "_perform_post_login_tasks_body",+ "_rollback_partial_session",+}+++def _traceback_emitters(func: ast.AST) -> list[str]:+ found = []+ for node in ast.walk(func):+ if not isinstance(node, ast.Call):+ continue+ for keyword in node.keywords:+ if keyword.arg == "exc_info":+ found.append(f"line {node.lineno}: exc_info=")+ target = node.func+ if not isinstance(target, ast.Attribute):+ continue+ if target.attr in {"exception", "catch"}:+ found.append(f"line {node.lineno}: .{target.attr}()")+ if target.attr == "opt":+ for keyword in node.keywords:+ if keyword.arg == "exception" and not (+ isinstance(keyword.value, ast.Constant)+ and keyword.value.value is False+ ):+ found.append(f"line {node.lineno}: opt(exception=...)")+ return found+++def test_credential_frames_render_no_tracebacks() -> None:+ source = Path(auth_module.__file__).read_text(encoding="utf-8")+ tree = ast.parse(source)+ functions = {+ node.name: node+ for node in ast.walk(tree)+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))+ and node.name in CREDENTIAL_FRAMES+ }++ # Non-vacuous: every listed frame is still there to be checked.+ assert set(functions) == CREDENTIAL_FRAMES++ offenders = {+ name: hits+ for name, func in functions.items()+ if (hits := _traceback_emitters(func))+ }+ assert offenders == {}, offenders+++def test_post_login_failure_with_diagnostics_does_not_render_credentials(+ monkeypatch, real_post_login_tasks+):+ """Exercise real Loguru rendering, including a sink with diagnose enabled."""+ import importlib+ import inspect+ from loguru._logger import Core, Logger++ session_module = importlib.import_module(+ "local_deep_research.web.auth.session_manager"+ )+ monkeypatch.setattr(+ session_module,+ "read_user_authorization_epoch",+ lambda username: "test-epoch",+ )+ manager = session_module.SessionManager()+ session_id = manager.create_session("alice")+ monkeypatch.setattr(auth_module, "session_manager", manager)++ isolated_logger = Logger(+ core=Core(),+ exception=None,+ depth=0,+ record=False,+ lazy=False,+ colors=False,+ raw=False,+ capture=True,+ patchers=[],+ extra={},+ )+ rendered = []+ records = []++ def capture(message):+ rendered.append(str(message))+ records.append(message.record)++ sink = isolated_logger.add(+ capture, format="{message}", diagnose=True, backtrace=True+ )+ password = "auth-password-opaque-canary"+ upstream_body = "opaque-upstream-credential-body"++ body_calls = []++ def fail_body(username, password, session_id):+ body_calls.append((username, password, session_id))+ raise RuntimeError(upstream_body)++ monkeypatch.setattr(auth_module, "logger", isolated_logger)+ monkeypatch.setattr(+ auth_module, "_perform_post_login_tasks_body", fail_body+ )+ try:+ worker = inspect.unwrap(real_post_login_tasks)+ assert worker("alice", password, session_id) is None+ finally:+ isolated_logger.remove(sink)++ assert body_calls == [("alice", password, session_id)]+ text = "".join(rendered)+ assert "alice" in text and "RuntimeError" in text+ assert password not in text+ assert upstream_body not in text+ assert records and all(record["exception"] is None for record in records)diff --git a/tests/web/auth/test_idle_cleanup_epoch_unavailable.py b/tests/web/auth/test_idle_cleanup_epoch_unavailable.pynew file mode 100644index 0000000000..2fa3ba1a9a--- /dev/null+++ b/tests/web/auth/test_idle_cleanup_epoch_unavailable.py@@ -0,0 +1,56 @@+"""Do not interpret an unreadable epoch as proof that a new login is idle."""++import errno+from pathlib import Path+from unittest.mock import Mock++from local_deep_research.database.rekey_lock import RekeyEpochUnreadableError+from local_deep_research.database.session_passwords import SessionPasswordStore+from local_deep_research.scheduler import background+from local_deep_research.web.auth import connection_cleanup+from local_deep_research.web.auth import session_manager as sm_module+++def test_unknown_activity_preserves_new_login_and_still_cleans_idle_users(+ monkeypatch,+):+ monkeypatch.setattr(+ sm_module, "read_user_authorization_epoch", lambda _u: "epoch"+ )+ manager = sm_module.SessionManager()+ store = SessionPasswordStore()+ store.store_session_password("idle", "retained", "idle-test-key")+ monkeypatch.setattr(connection_cleanup, "session_password_store", store)+ monkeypatch.setattr(+ connection_cleanup, "get_usernames_with_active_research", lambda: set()+ )+ monkeypatch.setattr(+ connection_cleanup, "_pop_per_user_locks", lambda _u: None+ )+ monkeypatch.setattr(+ background,+ "get_background_job_scheduler",+ lambda: Mock(is_running=False),+ )+ monkeypatch.setattr(connection_cleanup, "_last_dispose_time", float("inf"))++ def unreadable(_username):+ raise RekeyEpochUnreadableError(Path("review.db"), errno.EMFILE)++ def connected_after_snapshot():+ # A login completes after the sweep's initial activity snapshot.+ sid = manager.create_session("alice")+ store.store_session_password("alice", sid, "active-test-key")+ monkeypatch.setattr(+ sm_module, "read_user_authorization_epoch", unreadable+ )+ return {"alice", "idle"}++ db = Mock()+ db.get_connected_usernames.side_effect = connected_after_snapshot++ connection_cleanup.cleanup_idle_connections(manager, db)++ assert store.get_any_session_password("alice") == "active-test-key"+ assert store.get_any_session_password("idle") is None+ db.close_user_database.assert_called_once_with("idle")diff --git a/tests/web/auth/test_session_epoch.py b/tests/web/auth/test_session_epoch.pynew file mode 100644index 0000000000..8cbce77ae3--- /dev/null+++ b/tests/web/auth/test_session_epoch.py@@ -0,0 +1,647 @@+"""Durable-epoch fencing of server-side auth sessions.++A session is created at login, right after the password proved itself+against the epoch-fenced SQLCipher database (PR #6662). The session+record binds the durable authorization epoch that was live at that+moment. Every validation re-projects the epoch from disk:++* a session whose bound epoch no longer matches (the password was+ changed — rekeyed — in ANY worker) is destroyed: its credentials can+ no longer open the rekeyed database, so the identity it vouches for+ is dead;+* a durable state that was read but cannot vouch for the identity+ (recovery required, a rekey pending) fences the session the same way —+ fail closed, the record is removed;+* an epoch file that cannot be READ (EMFILE, EIO, ...) fails the check+ closed with ``SessionValidationUnavailableError`` but KEEPS the record:+ a transient fault must not log every active user out;+* a login whose password was proved against an epoch that is no longer+ committed by the time the session is created is refused+ (``StaleLoginProofError``): the proof, not the later projection, is+ what the session may be bound to.++``require_auth`` (HTTP), ``_enforce_session_revocation``, the socket+connect gate and the subscribe re-validation all funnel through+``validate_session``/``validate_session_identity``, so binding the epoch+here fences each of them. ``has_active_sessions_for`` applies the same+epoch check (read-only); the other idle sweeps are time-based only.+"""++import errno+import os+from pathlib import Path++import pytest++from local_deep_research.database.rekey_epoch import (+ read_user_authorization_epoch,+)+from local_deep_research.database.rekey_lock import (+ RekeyRecoveryRequiredError,+ acquire_rekey_lock,+ advance_rekey_epoch,+ read_or_initialize_rekey_epoch,+)+from local_deep_research.web.auth import session_manager as sm_module+from local_deep_research.web.auth.session_manager import (+ SessionManager,+ SessionValidationUnavailableError,+ StaleLoginProofError,+)++# Opt out of the autouse _durable_epoch_projection shim: these tests+# exercise the REAL epoch files and failure discipline.+pytestmark = pytest.mark.real_durable_epoch+++def data_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:+ """An isolated data directory whose user epochs are real files.++ ``db_manager.data_dir`` is pinned to the SAME directory: another+ module in this xdist worker may have pinned the singleton's override+ to its own tmp dir, and the epoch projection resolves through the+ manager -- env and manager must agree, or the projection reads the+ wrong directory and every assertion below fences against nothing.+ """+ from local_deep_research.database.encrypted_db import db_manager++ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))+ # Pin through the override attribute so monkeypatch restores the+ # ORIGINAL override (often None = "resolve from env"), instead of+ # re-pinning whatever path the getter happened to resolve.+ (tmp_path / "encrypted_databases").mkdir(parents=True, exist_ok=True)+ monkeypatch.setattr(+ db_manager, "_data_dir_override", tmp_path / "encrypted_databases"+ )+ return tmp_path+++def _user_db_path(data_dir: Path, username: str) -> Path:+ from local_deep_research.config.paths import get_user_database_filename++ return (+ data_dir / "encrypted_databases" / get_user_database_filename(username)+ )+++def _initialize_epoch(data_dir: Path, username: str) -> str:+ db_path = _user_db_path(data_dir, username)+ db_path.parent.mkdir(parents=True, exist_ok=True)+ with acquire_rekey_lock(db_path):+ return read_or_initialize_rekey_epoch(db_path)+++def _manager() -> SessionManager:+ manager = SessionManager()+ manager.sessions.clear()+ return manager+++def test_session_binds_durable_epoch_at_creation(+ data_dir: Path,+) -> None:+ epoch = _initialize_epoch(data_dir, "alice")+ manager = _manager()++ session_id = manager.create_session("alice")++ record = manager.sessions[session_id]+ assert record["rekey_epoch"] == epoch+++def test_stale_epoch_removes_exact_local_session(data_dir: Path) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ stale_session = manager.create_session("alice")++ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))+ fresh_session = manager.create_session("alice")++ # The stale-epoch session is destroyed, not merely denied...+ assert manager.validate_session(stale_session) is None+ assert stale_session not in manager.sessions+ # ...while the session bound to the current epoch survives.+ assert manager.validate_session(fresh_session) == "alice"+++def test_validate_session_identity_returns_the_bound_epoch(+ data_dir: Path,+) -> None:+ epoch = _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ identity = manager.validate_session_identity(session_id)++ assert identity is not None+ assert identity.username == "alice"+ assert identity.rekey_epoch == epoch+++def _fail_epoch_reads(+ monkeypatch: pytest.MonkeyPatch, error_number: int+) -> None:+ """Inject a real OSError at the epoch-file read itself."""+ real_open = os.open++ def failing_open(path, *args, **kwargs):+ if Path(path).name.endswith((".rekey.epoch", ".rekey.pending")):+ raise OSError(error_number, "injected epoch read failure")+ return real_open(path, *args, **kwargs)++ monkeypatch.setattr(os, "open", failing_open)+++ "error_number",+ [errno.EMFILE, errno.ENFILE, errno.EAGAIN, errno.EIO],+)+def test_unreadable_epoch_file_refuses_but_keeps_the_session(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, error_number: int+) -> None:+ # A transient I/O error proves nothing about the session: the check+ # fails closed (raises, never returns the username) but the record+ # survives, so an fd-exhaustion spike does not log everyone out.+ epoch = _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")+ before = dict(manager.sessions[session_id])++ with monkeypatch.context() as patched:+ _fail_epoch_reads(patched, error_number)+ with pytest.raises(SessionValidationUnavailableError) as raised:+ manager.validate_session(session_id)+ with pytest.raises(SessionValidationUnavailableError):+ manager.validate_session_identity(session_id)++ assert raised.value.errno == error_number+ # Kept, untouched (not even refreshed).+ assert manager.sessions[session_id] == before+ assert manager.sessions[session_id]["rekey_epoch"] == epoch+ # Once the fault clears, the same session validates again.+ assert manager.validate_session(session_id) == "alice"+++def test_epoch_advanced_during_a_read_fault_still_removes_after_it(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # Keeping the record through a fault is safe only because the next+ # successful read re-checks it: a password change committed during+ # the fault still ends the session.+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ with monkeypatch.context() as patched:+ _fail_epoch_reads(patched, errno.EMFILE)+ with pytest.raises(SessionValidationUnavailableError):+ manager.validate_session(session_id)+ assert session_id in manager.sessions++ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))++ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+++def test_malformed_epoch_file_fences_and_removes(data_dir: Path) -> None:+ # Read successfully, but the content cannot vouch for anything: that is+ # durable state, not a transient fault, so the record goes.+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")+ db_path = _user_db_path(data_dir, "alice")+ epoch_path = db_path.with_name(f"{db_path.name}.rekey.epoch")+ epoch_path.write_text("NOT-A-CANONICAL-UUID", encoding="ascii")++ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+++def test_recovery_required_state_fences_and_removes(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ def ambiguous_state(_username: str) -> str:+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))++ monkeypatch.setattr(+ sm_module,+ "read_user_authorization_epoch",+ ambiguous_state,+ )++ # Ambiguous durable state cannot vouch for ANY identity: fail closed+ # and remove the record.+ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+++def test_current_epoch_projection_matches_durable_file(+ data_dir: Path,+) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))++ assert (+ read_user_authorization_epoch("alice")+ != manager.sessions[session_id]["rekey_epoch"]+ )+ assert manager.validate_session(session_id) is None+++def test_create_session_refuses_a_proof_from_a_superseded_epoch(+ data_dir: Path,+) -> None:+ # The login proved its password against E1; a rekey committed E2+ # (another worker's password change) before the session was created.+ proven = _initialize_epoch(data_dir, "alice")+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))+ manager = _manager()++ with pytest.raises(StaleLoginProofError):+ manager.create_session("alice", proven_epoch=proven)++ assert manager.sessions == {}+++def test_create_session_binds_a_current_proof(data_dir: Path) -> None:+ proven = _initialize_epoch(data_dir, "alice")+ manager = _manager()++ session_id = manager.create_session("alice", proven_epoch=proven)++ assert manager.sessions[session_id]["rekey_epoch"] == proven+ assert manager.validate_session(session_id) == "alice"+++def test_projection_resolves_through_the_managers_pinned_data_dir(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # The env data dir and the manager's pinned override diverge; the+ # epoch lives only beside the manager's databases, where every writer+ # puts it. An env-only resolution reads the other directory and fails+ # closed on a perfectly healthy identity.+ from local_deep_research.config.paths import get_user_database_filename+ from local_deep_research.database.encrypted_db import db_manager++ env_dir = tmp_path / "env"+ pinned_dir = tmp_path / "pinned" / "encrypted_databases"+ (env_dir / "encrypted_databases").mkdir(parents=True)+ pinned_dir.mkdir(parents=True)+ monkeypatch.setenv("LDR_DATA_DIR", str(env_dir))+ monkeypatch.setattr(db_manager, "_data_dir_override", pinned_dir)+ db_path = pinned_dir / get_user_database_filename("alice")+ with acquire_rekey_lock(db_path):+ epoch = read_or_initialize_rekey_epoch(db_path)++ assert read_user_authorization_epoch("alice") == epoch+++def test_has_active_sessions_for_ignores_a_stale_epoch_session(+ data_dir: Path,+) -> None:+ # A password change in another worker leaves this worker's record in+ # place until its next validation; it must not count as active.+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ stale_session = manager.create_session("alice")+ assert manager.has_active_sessions_for("alice") is True++ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))++ assert manager.has_active_sessions_for("alice") is False+ # Read-only: the stale record is left for validation to remove.+ assert stale_session in manager.sessions++ # A session bound to the new epoch does count.+ manager.create_session("alice")+ assert manager.has_active_sessions_for("alice") is True+++def test_has_active_sessions_for_is_unavailable_when_the_epoch_is_unreadable(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # An unavailable check refuses authorization but does not prove that+ # shared resources belong to no remaining session.+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ with monkeypatch.context() as patched:+ _fail_epoch_reads(patched, errno.EMFILE)+ with pytest.raises(sm_module.SessionValidationUnavailableError):+ manager.has_active_sessions_for("alice")++ assert session_id in manager.sessions+ assert manager.has_active_sessions_for("alice") is True+++def test_has_active_sessions_for_is_false_on_recovery_required_state(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ manager.create_session("alice")++ def ambiguous_state(_username: str) -> str:+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))++ monkeypatch.setattr(+ sm_module, "read_user_authorization_epoch", ambiguous_state+ )++ assert manager.has_active_sessions_for("alice") is False+++# --- socket teardown when validation removes the record --------------------+#+# A socket is authorised once, at handshake, against a session id recorded+# in ``socketio_asgi._sid_sessions``; ``emit_to_user`` fans out to every sid+# in ``_sid_users`` without re-checking the session. When validation+# REMOVES a record (epoch mismatch after a password change in another+# worker, unvouchable durable state, inline idle expiry) the record is gone+# from every later sweep, so validation itself must sever the sockets that+# handshook on it -- otherwise a socket opened with a stolen cookie keeps+# receiving the user's events (``settings_changed`` carries plaintext+# secrets) after the password change that was meant to fence it.+++_SOCKET_MODULE = "local_deep_research.web.services.socketio_asgi"+++def severed(monkeypatch: pytest.MonkeyPatch) -> list[str]:+ """Record every session id handed to ``disconnect_session``."""+ calls: list[str] = []+ monkeypatch.setattr(+ f"{_SOCKET_MODULE}.disconnect_session",+ lambda session_id: calls.append(session_id),+ )+ return calls+++def test_epoch_mismatch_severs_that_sessions_sockets(+ data_dir: Path, severed: list[str]+) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ stale_session = manager.create_session("alice")+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))+ fresh_session = manager.create_session("alice")++ assert manager.validate_session(stale_session) is None++ # Only the removed session's sockets go; the new-epoch session (another+ # tab that logged in after the change) keeps its own.+ assert severed == [stale_session]+ assert manager.validate_session(fresh_session) == "alice"+ assert severed == [stale_session]+++def test_unvouchable_durable_state_severs_that_sessions_sockets(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]+) -> None:+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ def ambiguous_state(_username: str) -> str:+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))++ monkeypatch.setattr(+ sm_module, "read_user_authorization_epoch", ambiguous_state+ )++ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+ assert severed == [session_id]+++def test_inline_idle_expiry_severs_that_sessions_sockets(+ data_dir: Path, severed: list[str]+) -> None:+ import datetime++ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice", remember_me=False)+ manager.sessions[session_id]["last_access"] -= (+ manager.session_timeout + datetime.timedelta(minutes=1)+ )++ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+ assert severed == [session_id]+++def test_kept_or_unknown_records_sever_nothing(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]+) -> None:+ # Negative controls: a valid session, a transient read fault (record+ # KEPT, 503) and an unknown id must not tear down any socket.+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")++ assert manager.validate_session(session_id) == "alice"+ with monkeypatch.context() as patched:+ _fail_epoch_reads(patched, errno.EMFILE)+ with pytest.raises(SessionValidationUnavailableError):+ manager.validate_session(session_id)+ assert manager.validate_session("no-such-session") is None++ assert severed == []+ assert session_id in manager.sessions+++class _CookieRequest:+ """``end_rejected_session`` only touches ``request.session``."""++ def __init__(self, session: dict) -> None:+ self.session = dict(session)+++def test_rejected_session_teardown_survives_an_epoch_read_fault(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]+) -> None:+ # The cookie is cleared whatever happens, so the record and the sockets+ # behind it must go too, even while the epoch files cannot be read+ # (#7830): otherwise a replayed copy of the cleared cookie keeps a live+ # record that nothing can reach to destroy.+ from local_deep_research.web.dependencies import auth as auth_dep++ _initialize_epoch(data_dir, "alice")+ _initialize_epoch(data_dir, "bob")+ manager = _manager()+ session_id = manager.create_session("alice")+ other_tab = manager.create_session("alice")+ bobs_session = manager.create_session("bob")+ monkeypatch.setattr(auth_dep, "session_manager", manager)+ request = _CookieRequest({"username": "alice", "session_id": session_id})+ foreign = _CookieRequest({"username": "alice", "session_id": bobs_session})++ with monkeypatch.context() as patched:+ _fail_epoch_reads(patched, errno.EMFILE)+ # The fault is real: validation cannot answer for this record.+ with pytest.raises(SessionValidationUnavailableError):+ manager.validate_session(session_id)+ auth_dep.end_rejected_session(request)+ auth_dep.end_rejected_session(foreign)++ assert request.session == {}+ assert session_id not in manager.sessions+ assert severed == [session_id]+ # Scoped to that one record: the user's other tab and another user's+ # session named by a mismatched cookie are left alone.+ assert other_tab in manager.sessions+ assert bobs_session in manager.sessions+++def test_socket_teardown_failure_never_breaks_validation(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # Best-effort: a socket layer that raises must not turn the removal+ # into an exception at the HTTP middleware.+ def broken(_session_id: str) -> None:+ raise RuntimeError("socket layer down")++ monkeypatch.setattr(f"{_SOCKET_MODULE}.disconnect_session", broken)+ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))++ assert manager.validate_session(session_id) is None+ assert session_id not in manager.sessions+++def test_epoch_mismatch_clears_the_sid_maps_on_a_live_loop(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ """End to end through the real socket layer.++ ``disconnect_session`` -> ``_disconnect_matching`` runs on a live event+ loop; the fake ``sio.disconnect`` fires the real ``disconnect`` event+ handler, as python-socketio does, so the sid maps are cleaned by the+ same code path a logout uses. The other session's socket stays.+ """+ import asyncio+ import threading+ from unittest.mock import Mock++ from local_deep_research.web.services import socketio_asgi as sio_mod++ _initialize_epoch(data_dir, "alice")+ manager = _manager()+ stale_session = manager.create_session("alice")+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))+ fresh_session = manager.create_session("alice")++ monkeypatch.setattr(sio_mod, "_lock", asyncio.Lock())+ monkeypatch.setattr(sio_mod, "_subscriptions", {})+ monkeypatch.setattr(+ sio_mod, "_sid_users", {"sid-stolen": "alice", "sid-fresh": "alice"}+ )+ monkeypatch.setattr(+ sio_mod,+ "_sid_sessions",+ {"sid-stolen": stale_session, "sid-fresh": fresh_session},+ )++ loop = asyncio.new_event_loop()+ thread = threading.Thread(target=loop.run_forever, daemon=True)+ thread.start()+ disconnected: list[str] = []++ async def fake_disconnect(sid: str) -> None:+ disconnected.append(sid)+ await sio_mod.disconnect(sid)++ fake_sio = Mock()+ fake_sio.disconnect = fake_disconnect+ futures = []+ real_schedule = asyncio.run_coroutine_threadsafe++ def capture(coro, target_loop):+ future = real_schedule(coro, target_loop)+ futures.append(future)+ return future++ try:+ with monkeypatch.context() as patched:+ patched.setattr(sio_mod, "_get_main_loop", lambda: loop)+ patched.setattr(sio_mod, "sio", fake_sio)+ patched.setattr(asyncio, "run_coroutine_threadsafe", capture)++ assert manager.validate_session(stale_session) is None+ assert futures, "validation scheduled no socket teardown"+ for future in futures:+ future.result(timeout=5)+ finally:+ loop.call_soon_threadsafe(loop.stop)+ thread.join(timeout=5)+ loop.close()++ assert disconnected == ["sid-stolen"]+ assert sio_mod._sid_users == {"sid-fresh": "alice"}+ assert sio_mod._sid_sessions == {"sid-fresh": fresh_session}+++def test_epoch_read_sharing_violation_is_retried_then_fails_closed(+ data_dir: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ """A Windows sharing violation on the lock-free epoch read.++ A writer's ``os.replace``/``unlink`` in flight makes the reader's open+ fail with PermissionError for a few microseconds. A transient one is+ retried and the session validates; a persistent one fails closed+ (503 for this request, the record KEPT, never a validated identity).+ """+ from local_deep_research.database import rekey_lock as rekey_lock_module++ epoch = _initialize_epoch(data_dir, "alice")+ manager = _manager()+ session_id = manager.create_session("alice")+ monkeypatch.setattr(+ rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0, raising=False+ )+ real_open = os.open+ budget = {"failures": 2}++ def sharing_violation(path, *args, **kwargs):+ if (+ Path(path).name.endswith((".rekey.epoch", ".rekey.pending"))+ and budget["failures"] > 0+ ):+ budget["failures"] -= 1+ raise PermissionError(errno.EACCES, "sharing violation")+ return real_open(path, *args, **kwargs)++ monkeypatch.setattr(os, "open", sharing_violation)++ identity = manager.validate_session_identity(session_id)+ assert identity is not None and identity.rekey_epoch == epoch+ assert budget["failures"] == 0++ budget["failures"] = 10**6+ before = dict(manager.sessions[session_id])+ with pytest.raises(SessionValidationUnavailableError) as raised:+ manager.validate_session_identity(session_id)+ assert raised.value.errno == errno.EACCES+ assert manager.sessions[session_id] == beforediff --git a/tests/web/auth/test_session_validation_unavailable.py b/tests/web/auth/test_session_validation_unavailable.pynew file mode 100644index 0000000000..93e1be659e--- /dev/null+++ b/tests/web/auth/test_session_validation_unavailable.py@@ -0,0 +1,524 @@+"""A transient epoch-read failure refuses the request but keeps the session.++``validate_session`` raises ``SessionValidationUnavailableError`` when the+durable epoch files cannot be read at all (EMFILE, EIO, ...). Every caller+must fail THIS request closed without clearing the session or its cookie:+clearing it would log out every active user during an fd-exhaustion spike,+including on static-file requests (``DatabaseMiddleware`` validates on every+request). Only durable state that was read and does not vouch for the+session ends it; that is covered in ``test_session_epoch.py``.++Logout is the one exception: it only destroys, so ``POST /auth/logout``+goes through unvalidated (no database open), and the route tears down the+user's credentials only for a cookie whose record still binds it to that+user.++Also pins the per-request memo: ``DatabaseMiddleware`` records the session+it validated in ``scope["state"]`` and ``require_auth`` reuses it instead of+re-projecting the epoch a second time in the same request.++Marked ``real_session_check`` so the autouse ``_legacy_bare_username_auth``+shim does not replace ``_server_session_valid``.+"""++import asyncio+from unittest.mock import AsyncMock, MagicMock++import pytest+from fastapi import HTTPException+from starlette.requests import Request++from local_deep_research.web.auth.session_manager import (+ UNVALIDATED_LOGOUT_STATE_KEY,+ VALIDATED_SESSION_STATE_KEY,+ SessionManager,+ SessionValidationUnavailableError,+)+from local_deep_research.web.dependencies import auth as auth_dep+from local_deep_research.web.services import socketio_asgi as sio_mod++pytestmark = pytest.mark.real_session_check++USER = "transient_user"+SID = "session-id-1"+++def _unavailable(*_args, **_kwargs):+ raise SessionValidationUnavailableError(USER, 24) # EMFILE+++def fake_session_manager(monkeypatch):+ """Replace the singleton's validate_session where each caller looks it up.++ Only ``validate_session`` is replaced: the singleton's ``sessions`` dict+ is left as it is, and no record for ``SID`` is seeded. A test that+ reaches the cookie-clearing teardown (``_revoke_server_session``) must+ seed one itself (``_seed_server_record``): the teardown looks the owner+ up with ``session_owner``, which reads the record directly, not through+ ``validate_session``, and tears down nothing for an id it cannot find.+ """+ from local_deep_research.web.auth import session_manager as sm_module++ manager = MagicMock()+ monkeypatch.setattr(+ sm_module.session_manager, "validate_session", manager.validate_session+ )+ return manager+++def _seed_server_record(monkeypatch, username: str = USER) -> None:+ """Give the real singleton a live record for SID (restored afterwards)."""+ import datetime++ from local_deep_research.web.auth import session_manager as sm_module++ now = datetime.datetime.now(datetime.UTC)+ monkeypatch.setitem(+ sm_module.session_manager.sessions,+ SID,+ {+ "username": username,+ "created_at": now,+ "last_access": now,+ "remember_me": False,+ "rekey_epoch": "test-epoch",+ },+ )+++def _request(session: dict, state: dict | None = None) -> Request:+ scope = {+ "type": "http",+ "method": "GET",+ "path": "/api/whatever",+ "headers": [],+ "session": session,+ }+ if state is not None:+ scope["state"] = state+ return Request(scope)+++# --- _enforce_session_revocation / DatabaseMiddleware ----------------------+++def test_revocation_check_raises_and_leaves_the_session(fake_session_manager):+ from local_deep_research.web.fastapi_app import (+ _enforce_session_revocation,+ )++ fake_session_manager.validate_session.side_effect = _unavailable+ session = {"username": USER, "session_id": SID}+ state: dict = {}++ with pytest.raises(SessionValidationUnavailableError):+ _enforce_session_revocation(session, state)++ assert session == {"username": USER, "session_id": SID}+ assert VALIDATED_SESSION_STATE_KEY not in state+++def test_revocation_check_records_the_validated_session(fake_session_manager):+ from local_deep_research.web.fastapi_app import (+ _enforce_session_revocation,+ )++ fake_session_manager.validate_session.return_value = USER+ state: dict = {}++ _enforce_session_revocation({"username": USER, "session_id": SID}, state)++ assert state[VALIDATED_SESSION_STATE_KEY] == (SID, USER)+++def test_revoked_session_is_cleared_and_not_recorded(fake_session_manager):+ from local_deep_research.web.fastapi_app import (+ _enforce_session_revocation,+ )++ fake_session_manager.validate_session.return_value = None+ session = {"username": USER, "session_id": SID}+ state: dict = {}++ _enforce_session_revocation(session, state)++ assert session == {}+ assert VALIDATED_SESSION_STATE_KEY not in state++[email protected]("path", ["/static/css/app.css", "/", "/auth/check"])+def test_middleware_answers_503_without_touching_the_session(+ fake_session_manager, path+):+ from local_deep_research.web.fastapi_app import DatabaseMiddleware++ fake_session_manager.validate_session.side_effect = _unavailable+ inner = AsyncMock()+ middleware = DatabaseMiddleware(inner)+ session = {"username": USER, "session_id": SID}+ scope = {+ "type": "http",+ "method": "GET",+ "path": path,+ "headers": [],+ "session": session,+ }+ sent: list[dict] = []++ async def send(message):+ sent.append(message)++ async def receive():+ return {"type": "http.request", "body": b"", "more_body": False}++ asyncio.run(middleware(scope, receive, send))++ inner.assert_not_called()+ assert sent[0]["type"] == "http.response.start"+ assert sent[0]["status"] == 503+ # Not a logout: the session is exactly as it came in.+ assert session == {"username": USER, "session_id": SID}+++def _run_middleware(fake_session_manager, monkeypatch, method, path):+ from local_deep_research.web import fastapi_app+ from local_deep_research.web.dependencies import auth as auth_module++ fake_session_manager.validate_session.side_effect = _unavailable+ ensure = MagicMock()+ monkeypatch.setattr(auth_module, "ensure_user_database", ensure)+ inner = AsyncMock()+ middleware = fastapi_app.DatabaseMiddleware(inner)+ session = {"username": USER, "session_id": SID}+ scope = {+ "type": "http",+ "method": method,+ "path": path,+ "headers": [],+ "session": session,+ }+ sent: list[dict] = []++ async def send(message):+ sent.append(message)++ async def receive():+ return {"type": "http.request", "body": b"", "more_body": False}++ asyncio.run(middleware(scope, receive, send))+ return inner, ensure, scope, session, sent+++def test_middleware_lets_logout_through_unvalidated(+ fake_session_manager, monkeypatch+):+ inner, ensure, scope, session, sent = _run_middleware(+ fake_session_manager, monkeypatch, "POST", "/auth/logout"+ )++ inner.assert_called_once()+ assert sent == []+ # Nothing is opened (or stored) for the unvalidated cookie, and it is+ # not recorded as validated for require_auth.+ ensure.assert_not_called()+ assert scope["state"][UNVALIDATED_LOGOUT_STATE_KEY] is True+ assert VALIDATED_SESSION_STATE_KEY not in scope["state"]+ assert session == {"username": USER, "session_id": SID}+++ ("method", "path"),+ [+ ("GET", "/auth/logout"),+ ("POST", "/auth/logout/"),+ ("POST", "/auth/logoutx"),+ ("POST", "/auth/change-password"),+ ],+)+def test_only_post_logout_bypasses_the_503(+ fake_session_manager, monkeypatch, method, path+):+ inner, ensure, scope, session, sent = _run_middleware(+ fake_session_manager, monkeypatch, method, path+ )++ inner.assert_not_called()+ ensure.assert_not_called()+ assert sent[0]["status"] == 503+ assert UNVALIDATED_LOGOUT_STATE_KEY not in scope["state"]+++# --- logout during a fault ------------------------------------------------+++def test_session_belongs_to_only_reads_the_record():+ manager = SessionManager()+ manager.sessions[SID] = {"username": USER}++ assert manager.session_belongs_to(SID, USER) is True+ assert manager.session_belongs_to(SID, "other-user") is False+ assert manager.session_belongs_to("other-session", USER) is False+ assert manager.session_belongs_to(None, USER) is False+ assert manager.session_belongs_to("", USER) is False+++def logout_env(monkeypatch):+ from local_deep_research.database import session_passwords+ from local_deep_research.database import thread_local_session+ from local_deep_research.scheduler import background+ from local_deep_research.web.routers import auth as auth_router++ manager = SessionManager()+ monkeypatch.setattr(auth_router, "session_manager", manager)+ env = MagicMock()+ env.manager = manager+ monkeypatch.setattr(auth_router, "db_manager", env.db_manager)+ monkeypatch.setattr(+ auth_router, "_disconnect_session_sockets", env.disconnect_sockets+ )+ monkeypatch.setattr(+ session_passwords.session_password_store,+ "clear_all_for_user",+ env.clear_passwords,+ )+ monkeypatch.setattr(+ thread_local_session, "clear_user_credentials", env.clear_credentials+ )+ monkeypatch.setattr(+ background, "get_background_job_scheduler", env.get_scheduler+ )+ env.logout = auth_router.logout+ return env+++def _logout_request(session: dict) -> Request:+ return Request(+ {+ "type": "http",+ "method": "POST",+ "path": "/auth/logout",+ "headers": [],+ "session": session,+ "state": {UNVALIDATED_LOGOUT_STATE_KEY: True},+ }+ )+++def test_unvalidated_logout_revokes_the_callers_own_session(logout_env):+ logout_env.manager.sessions[SID] = {"username": USER}+ other = "session-id-2"+ logout_env.manager.sessions[other] = {"username": "someone-else"}+ request = _logout_request({"username": USER, "session_id": SID})++ response = logout_env.logout(request)++ assert response.status_code == 302+ assert SID not in logout_env.manager.sessions+ assert other in logout_env.manager.sessions+ logout_env.clear_passwords.assert_called_once_with(USER)+ logout_env.clear_credentials.assert_called_once_with(USER)+ logout_env.disconnect_sockets.assert_called_once_with(SID)+ assert "username" not in request.session+ assert "session_id" not in request.session+++ "record", [None, {"username": "someone-else"}], ids=["gone", "foreign"]+)+def test_unvalidated_logout_with_an_unbound_cookie_only_drops_the_cookie(+ logout_env, record+):+ # A cookie whose record is already gone (or not this user's) must not+ # reach the user-wide teardown while nothing validated it: it only+ # loses itself, as the validating middleware would have done.+ if record is not None:+ logout_env.manager.sessions[SID] = record+ request = _logout_request({"username": USER, "session_id": SID})++ response = logout_env.logout(request)++ assert response.status_code == 302+ assert request.session == {}+ if record is not None:+ assert logout_env.manager.sessions[SID] == record+ logout_env.clear_passwords.assert_not_called()+ logout_env.clear_credentials.assert_not_called()+ logout_env.disconnect_sockets.assert_not_called()+ logout_env.get_scheduler.assert_not_called()+ logout_env.db_manager.close_user_database.assert_not_called()+++# --- require_auth ---------------------------------------------------------+++def connected(monkeypatch):+ db = MagicMock()+ db.is_user_connected.return_value = True+ monkeypatch.setattr(auth_dep, "db_manager", db)+ return db+++def test_require_auth_answers_503_and_keeps_the_session(+ fake_session_manager, connected+):+ fake_session_manager.validate_session.side_effect = _unavailable+ request = _request({"username": USER, "session_id": SID})++ with pytest.raises(HTTPException) as raised:+ auth_dep.require_auth(request)++ assert raised.value.status_code == 503+ assert request.session == {"username": USER, "session_id": SID}+++def test_require_auth_still_rejects_an_invalid_session(+ fake_session_manager, connected+):+ fake_session_manager.validate_session.return_value = None+ request = _request({"username": USER, "session_id": SID})++ with pytest.raises(HTTPException) as raised:+ auth_dep.require_auth(request)++ assert raised.value.status_code == 401+ assert request.session == {}+++def test_require_auth_reuses_the_middleware_validation(+ fake_session_manager, connected, monkeypatch+):+ monkeypatch.setattr(+ auth_dep.session_password_store,+ "get_session_password",+ lambda username, session_id: "available-credential",+ )+ request = _request(+ {"username": USER, "session_id": SID},+ state={VALIDATED_SESSION_STATE_KEY: (SID, USER)},+ )++ assert auth_dep.require_auth(request) == USER+ fake_session_manager.validate_session.assert_not_called()+++def test_validation_memo_does_not_preserve_a_lost_credential(+ fake_session_manager, connected, monkeypatch+):+ from local_deep_research.database.session_passwords import (+ SessionPasswordStore,+ )++ store = SessionPasswordStore()+ store.store_session_password(USER, SID, "available-credential")+ monkeypatch.setattr(auth_dep, "session_password_store", store)+ connected.has_encryption = True+ fake_session_manager.validate_session.return_value = USER+ _seed_server_record(monkeypatch)+ destroy = MagicMock()+ disconnect = MagicMock()+ monkeypatch.setattr(auth_dep.session_manager, "destroy_session", destroy)+ monkeypatch.setattr(sio_mod, "disconnect_session", disconnect)+ request = _request(+ {"username": USER, "session_id": SID},+ state={VALIDATED_SESSION_STATE_KEY: (SID, USER)},+ )++ assert auth_dep.require_auth(request) == USER+ fake_session_manager.validate_session.assert_not_called()++ # Losing the credential after middleware validation must still revoke+ # this session, even though its user's database remains connected.+ store.clear_session(USER, SID)+ with pytest.raises(HTTPException) as raised:+ auth_dep.require_auth(request)++ assert raised.value.status_code == 401+ assert request.session == {}+ destroy.assert_called_once_with(SID)+ disconnect.assert_called_once_with(SID)+++ "memo", [("other-session", USER), (SID, "other-user"), None]+)+def test_require_auth_validates_when_the_memo_does_not_match(+ fake_session_manager, connected, memo+):+ fake_session_manager.validate_session.return_value = None+ state = {} if memo is None else {VALIDATED_SESSION_STATE_KEY: memo}+ request = _request({"username": USER, "session_id": SID}, state=state)++ with pytest.raises(HTTPException) as raised:+ auth_dep.require_auth(request)++ assert raised.value.status_code == 401+ fake_session_manager.validate_session.assert_called_once_with(SID)+++# --- socket gates ---------------------------------------------------------+++def socket_state(monkeypatch):+ monkeypatch.setattr(sio_mod, "_sid_users", {"sock-1": USER})+ monkeypatch.setattr(sio_mod, "_sid_sessions", {"sock-1": SID})+ monkeypatch.setattr(sio_mod, "_subscriptions", {})+ monkeypatch.setattr(sio_mod, "_main_loop", sio_mod._main_loop)+ monkeypatch.setattr(sio_mod, "_lock", None)+ sio_mod.init_lock()+++def test_socket_revalidation_reports_unknown_not_invalid(+ fake_session_manager, socket_state+):+ fake_session_manager.validate_session.side_effect = _unavailable++ result = asyncio.run(sio_mod._socket_session_still_valid("sock-1", USER))++ assert result == (SID, None)+++def test_subscribe_is_refused_without_severing_the_session(+ fake_session_manager, socket_state, monkeypatch+):+ fake_session_manager.validate_session.side_effect = _unavailable+ emit = AsyncMock()+ disconnect = AsyncMock()+ monkeypatch.setattr(sio_mod.sio, "emit", emit)+ monkeypatch.setattr(sio_mod.sio, "disconnect", disconnect)+ severed = MagicMock()+ monkeypatch.setattr(sio_mod, "disconnect_session", severed)+ owns = AsyncMock(return_value=True)+ monkeypatch.setattr(sio_mod, "_user_owns_research", owns)++ asyncio.run(sio_mod.on_subscribe("sock-1", {"research_id": "r-1"}))++ severed.assert_not_called()+ disconnect.assert_not_called()+ owns.assert_not_called()+ assert sio_mod._subscriptions == {}+ event, payload = emit.call_args.args[:2]+ assert event == "subscribe_error"+ assert payload["error"] == "Temporarily unavailable"+ assert sio_mod._sid_sessions == {"sock-1": SID}+++def test_connect_is_refused_when_the_session_cannot_be_validated(+ fake_session_manager, socket_state, monkeypatch+):+ fake_session_manager.validate_session.side_effect = _unavailable+ monkeypatch.setattr(+ sio_mod,+ "_decode_session_cookie",+ lambda _cookie: {"username": USER, "session_id": SID},+ )++ accepted = asyncio.run(+ sio_mod.connect("sock-2", {"HTTP_COOKIE": "session=whatever"})+ )++ assert accepted is False+ assert "sock-2" not in sio_mod._sid_usersdiff --git a/tests/web/dependencies/test_dependency_resolution_contracts.py b/tests/web/dependencies/test_dependency_resolution_contracts.pyindex 8294b32d85..71efb29f61 100644--- a/tests/web/dependencies/test_dependency_resolution_contracts.py+++ b/tests/web/dependencies/test_dependency_resolution_contracts.py@@ -1049,6 +1049,10 @@ class _DeadSessionManager:def validate_session(session_id): # noqa: ARG004return None+ @staticmethod+ def session_owner(session_id): # noqa: ARG004+ return None+@pytest.mark.real_session_checkclass TestUsernameReadersDiverge:diff --git a/tests/web/dependencies/test_session_revocation.py b/tests/web/dependencies/test_session_revocation.pyindex 6f5205eb93..bdd3a097d8 100644--- a/tests/web/dependencies/test_session_revocation.py+++ b/tests/web/dependencies/test_session_revocation.py@@ -65,6 +65,9 @@ class _FakeSessionManager:def validate_session(self, session_id):return self._sessions.get(session_id)+ def session_owner(self, session_id):+ return self._sessions.get(session_id)+@pytest.fixturedef connected_db(monkeypatch):diff --git a/tests/web/dependencies/test_stale_session_cleanup.py b/tests/web/dependencies/test_stale_session_cleanup.pyindex 189e458e35..997b61a9c2 100644--- a/tests/web/dependencies/test_stale_session_cleanup.py+++ b/tests/web/dependencies/test_stale_session_cleanup.py@@ -83,6 +83,9 @@ class _FakeSessionManager:return Nonereturn {"s1": "alice", "s-bob": "bob"}.get(session_id)+ def session_owner(self, session_id):+ return self.validate_session(session_id)+def destroy_session(self, session_id):self.destroyed.append(session_id)diff --git a/tests/web/routers/test_login_rekey_straddle.py b/tests/web/routers/test_login_rekey_straddle.pynew file mode 100644index 0000000000..a7f70db5d8--- /dev/null+++ b/tests/web/routers/test_login_rekey_straddle.py@@ -0,0 +1,384 @@+"""Session creation at login is fenced by the epoch the login proved.++Worker A's login proves the OLD password against the durable epoch E1+(``open_user_database`` returns an engine). Before A creates the session,+worker B's ``change_password`` rekeys the database and commits E2. The+session must be bound to what A PROVED (E1), not to whatever epoch is+committed by the time ``create_session`` runs (E2): bound to E2 it would+survive the password change it straddled, and the old password would+keep a live session.++The interleaving is forced deterministically: the real+``open_user_database`` runs, then a second ``DatabaseManager`` on the+same data directory (worker B) performs a real SQLCipher rekey before+the route continues to ``create_session``.++A login whose epoch cannot be read at session creation or right after+its cold open verified the password, whose committed epoch moved under+that cold open, or whose open recorded no epoch proof for the user,+answers 503 rather than issuing a+session it cannot fence. None of these refusals touches the lockout+counter: it is not incremented (not a guessing failure) and+not reset (no session was issued).+"""++import uuid++import pytest+from fastapi.testclient import TestClient++pytest.importorskip("sqlcipher3", reason="requires SQLCipher (encrypted mode)")++# Real durable epochs end to end: the autouse projection shim is off.+pytestmark = pytest.mark.real_durable_epoch+++def app(tmp_path, monkeypatch):+ """The real app on a throwaway data directory with test KDF rounds."""+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))+ # Production PBKDF2 rounds (256000) would dominate wall-clock: every+ # test here derives several keys. sqlcipher_utils allows low rounds+ # under pytest (MIN_KDF_ITERATIONS_TESTING).+ monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")++ from local_deep_research.database.auth_db import init_auth_database+ from local_deep_research.database.encrypted_db import db_manager+ from local_deep_research.web.fastapi_app import app as fastapi_app++ databases = tmp_path / "encrypted_databases"+ databases.mkdir(parents=True, exist_ok=True)+ # Pinned through monkeypatch so the ORIGINAL override is restored.+ monkeypatch.setattr(db_manager, "_data_dir_override", databases)+ init_auth_database()+ monkeypatch.setattr(+ "local_deep_research.web.routers.auth._perform_post_login_tasks",+ lambda *_a, **_k: None,+ )+ try:+ yield fastapi_app+ finally:+ db_manager.close_all_databases()+++def _client(app) -> TestClient:+ client = TestClient(app, raise_server_exceptions=False)+ ip = f"10.{uuid.uuid4().int % 254 + 1}.{uuid.uuid4().int % 254 + 1}.9"+ client.headers.update({"X-Forwarded-For": ip})+ return client+++def _csrf(client: TestClient) -> str:+ client.get("/auth/login")+ return client.get("/auth/csrf-token").json().get("csrf_token", "")+++def _register_and_logout(app, user: str, pw: str) -> None:+ client = _client(app)+ resp = client.post(+ "/auth/register",+ data={+ "username": user,+ "password": pw,+ "confirm_password": pw,+ "acknowledge": "true",+ "csrf_token": _csrf(client),+ },+ follow_redirects=False,+ )+ if resp.status_code != 302:+ pytest.fail(f"registration returned {resp.status_code}")+ token = client.get("/auth/csrf-token").json().get("csrf_token", "")+ client.post(+ "/auth/logout", headers={"X-CSRFToken": token}, follow_redirects=False+ )+++def _login(app, user: str, pw: str):+ client = _client(app)+ return client.post(+ "/auth/login",+ data={"username": user, "password": pw, "csrf_token": _csrf(client)},+ follow_redirects=False,+ )+++def _user_sessions(username: str) -> list[str]:+ from local_deep_research.web.auth.session_manager import session_manager++ with session_manager._lock:+ return [+ sid+ for sid, data in session_manager.sessions.items()+ if data["username"] == username+ ]+++def _failure_count(username: str) -> int:+ from local_deep_research.security.account_lockout import (+ get_account_lockout_manager,+ )++ return (+ get_account_lockout_manager()._state.get(username, {}).get("count", 0)+ )+++def _one_prior_failure(username: str) -> None:+ from local_deep_research.security.account_lockout import (+ get_account_lockout_manager,+ )++ get_account_lockout_manager().record_failure(username)+ assert _failure_count(username) == 1+++def test_old_password_login_straddling_a_rekey_gets_no_session(app):+ from local_deep_research.database.encrypted_db import (+ DatabaseManager,+ db_manager,+ )++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"straddle_{uuid.uuid4().hex[:8]}"+ old_pw = "StraddleOld-Password1!" # noqa: S105+ new_pw = "StraddleNew-Password2!" # noqa: S105+ _register_and_logout(app, user, old_pw)+ _one_prior_failure(user)++ # Worker B: a separate manager over the same data directory.+ worker_b = DatabaseManager()+ worker_b.data_dir = db_manager.data_dir+ real_open = db_manager.open_user_database+ rekeyed = []++ def open_then_rekey_elsewhere(username, password):+ engine = real_open(username, password)+ if engine is not None and not rekeyed:+ rekeyed.append(worker_b.change_password(username, old_pw, new_pw))+ return engine++ try:+ with pytest.MonkeyPatch.context() as mp:+ mp.setattr(+ db_manager, "open_user_database", open_then_rekey_elsewhere+ )+ resp = _login(app, user, old_pw)+ finally:+ worker_b.close_all_databases()++ assert rekeyed == [True], "worker B's rekey did not commit"+ # The old password proved E1; E2 is committed now. No session.+ assert resp.status_code == 401, (+ f"old-password login straddling a rekey answered {resp.status_code}"+ )+ assert _user_sessions(user) == []+ # Refused before any session was issued: the earlier failure is+ # neither cleared (record_success) nor added to.+ assert _failure_count(user) == 1++ # The new password still logs in normally, and that clears the count.+ assert _login(app, user, new_pw).status_code == 302+ assert _failure_count(user) == 0+++def test_unreadable_epoch_at_session_creation_answers_503(app, monkeypatch):+ from local_deep_research.database.encrypted_db import db_manager+ from local_deep_research.database.rekey_lock import (+ RekeyRecoveryRequiredError,+ )+ from local_deep_research.web.auth import session_manager as sm_module++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"epoch503_{uuid.uuid4().hex[:8]}"+ pw = "Epoch503-Password1!" # noqa: S105+ _register_and_logout(app, user, pw)+ _one_prior_failure(user)+ db_path = db_manager._get_user_db_path(user)++ reads = []++ def unreadable(username: str) -> str:+ reads.append(username)+ raise RekeyRecoveryRequiredError(db_path)++ # The open proves the password; only the session-creation read fails.+ monkeypatch.setattr(sm_module, "read_user_authorization_epoch", unreadable)+ resp = _login(app, user, pw)++ # Exactly one epoch read, and it was create_session's: the 503 comes+ # from the session-creation path, not from some other reader.+ assert reads == [user]+ assert resp.status_code == 503+ assert _user_sessions(user) == []+ assert _failure_count(user) == 1+++def test_login_without_a_proof_for_the_user_answers_503(app, monkeypatch):+ # Every engine-returning open records a proof today; if one ever does+ # not (or records it for another username), the login must refuse+ # rather than create a session with the straddle check skipped.+ from local_deep_research.database import encrypted_db+ from local_deep_research.database.encrypted_db import db_manager++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"noproof_{uuid.uuid4().hex[:8]}"+ pw = "NoProof-Password1!" # noqa: S105+ _register_and_logout(app, user, pw)+ _one_prior_failure(user)+ real_open = db_manager.open_user_database++ def open_recording_a_foreign_proof(username, password):+ engine = real_open(username, password)+ encrypted_db._record_login_proof("someone-else", "not-this-epoch")+ return engine++ monkeypatch.setattr(+ db_manager, "open_user_database", open_recording_a_foreign_proof+ )+ resp = _login(app, user, pw)++ assert resp.status_code == 503+ assert _user_sessions(user) == []+ # Not a guessing failure, and no session was issued.+ assert _failure_count(user) == 1+++def test_unreadable_epoch_after_a_correct_cold_open_answers_503(+ app, monkeypatch+):+ # The cold open verified the CORRECT password; the epoch re-read right+ # after it then fails (EISDIR here; EMFILE/EIO behave the same). That+ # proves nothing about the credential, so the login answers 503 and+ # leaves the lockout counter alone instead of reporting "invalid+ # credentials" (401) and counting a failure.+ from local_deep_research.database.encrypted_db import db_manager++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"recheck_{uuid.uuid4().hex[:8]}"+ pw = "Recheck-Password1!" # noqa: S105+ _register_and_logout(app, user, pw)+ _one_prior_failure(user)+ # Force the cold-open path (no cached engine to claim).+ db_manager.close_user_database(user)+ db_path = db_manager._get_user_db_path(user)+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")+ real_cold_open = db_manager._open_user_database_cold+ cold_opens = []++ def cold_open_then_epoch_unreadable(*args, **kwargs):+ engine = real_cold_open(*args, **kwargs)+ cold_opens.append(engine is not None)+ epoch_file.unlink()+ epoch_file.mkdir()+ return engine++ monkeypatch.setattr(+ db_manager, "_open_user_database_cold", cold_open_then_epoch_unreadable+ )+ resp = _login(app, user, pw)++ assert cold_opens == [True], "the correct password did not cold-open"+ assert resp.status_code == 503+ assert _user_sessions(user) == []+ assert _failure_count(user) == 1+ assert user not in db_manager.connections+++def test_epoch_moved_under_a_correct_cold_open_answers_503(app, monkeypatch):+ # The cold open verified the CORRECT password; the committed epoch is+ # then found to have moved (only a writer that bypassed the rekey+ # sidecar this open holds can do that). The password opened the+ # database, so it is not a guessing failure: the login answers 503 and+ # leaves the lockout counter alone instead of reporting "invalid+ # credentials" (401) and counting a failure.+ from local_deep_research.database import rekey_lock+ from local_deep_research.database.encrypted_db import db_manager++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"moved_{uuid.uuid4().hex[:8]}"+ pw = "Moved-Password1!" # noqa: S105+ _register_and_logout(app, user, pw)+ _one_prior_failure(user)+ # Force the cold-open path (no cached engine to claim).+ db_manager.close_user_database(user)+ db_path = db_manager._get_user_db_path(user)+ real_cold_open = db_manager._open_user_database_cold+ cold_opens = []++ def cold_open_then_epoch_moves(*args, **kwargs):+ engine = real_cold_open(*args, **kwargs)+ cold_opens.append(engine is not None)+ rekey_lock.advance_rekey_epoch(db_path)+ return engine++ monkeypatch.setattr(+ db_manager, "_open_user_database_cold", cold_open_then_epoch_moves+ )+ resp = _login(app, user, pw)++ assert cold_opens == [True], "the correct password did not cold-open"+ assert resp.status_code == 503+ assert _user_sessions(user) == []+ assert _failure_count(user) == 1+ assert user not in db_manager.connections+++def test_unreadable_epoch_at_registration_auto_login_answers_503(+ app, monkeypatch+):+ # Registration auto-logs in through the same create_session, so an+ # epoch that cannot be read there gets login's retryable 503, not a+ # 500 (#7830). The account it just created stays: once the fault+ # clears, the same credentials sign in.+ import errno++ from local_deep_research.database.encrypted_db import db_manager+ from local_deep_research.database.rekey_lock import (+ RekeyEpochUnreadableError,+ )+ from local_deep_research.web.auth import session_manager as sm_module++ if not db_manager.has_encryption:+ pytest.skip("requires SQLCipher (encrypted mode)")++ user = f"reg503_{uuid.uuid4().hex[:8]}"+ pw = "Register503-Password1!" # noqa: S105+ reads = []++ def unreadable(username: str) -> str:+ reads.append(username)+ raise RekeyEpochUnreadableError(+ db_manager._get_user_db_path(username), errno.EMFILE+ )++ client = _client(app)+ with pytest.MonkeyPatch.context() as mp:+ mp.setattr(sm_module, "read_user_authorization_epoch", unreadable)+ resp = client.post(+ "/auth/register",+ data={+ "username": user,+ "password": pw,+ "confirm_password": pw,+ "acknowledge": "true",+ "csrf_token": _csrf(client),+ },+ follow_redirects=False,+ )++ # The refusal came from create_session's epoch read.+ assert reads == [user]+ assert resp.status_code == 503+ assert _user_sessions(user) == []+ assert _login(app, user, pw).status_code == 302diff --git a/tests/web/test_route_table_parity.py b/tests/web/test_route_table_parity.pyindex e83c5002db..cf582b5433 100644--- a/tests/web/test_route_table_parity.py+++ b/tests/web/test_route_table_parity.py@@ -1730,6 +1730,10 @@ EXPECTED_STATUS_CODES_GAINED = {# revokes every session and redirects to login like a success instead.# web/routers/auth.py:change_password.("POST", "/auth/change-password"): [503],+ # Registration's auto-login answers 503 when the new account's session+ # epoch cannot be read (RekeyOperationalError from create_session), as+ # login does; the account itself stays created. auth.py:register.+ ("POST", "/auth/register"): [503],# Intentional: an unimplemented endpoint answers 501 rather than letting# a bare `except Exception` turn NotImplementedException into a 500.("GET", "/news/api/categories"): [501],diff --git a/tests/web/test_thread_excepthook.py b/tests/web/test_thread_excepthook.pyindex eed6d103cf..e39b9319d2 100644--- a/tests/web/test_thread_excepthook.py+++ b/tests/web/test_thread_excepthook.py@@ -138,6 +138,14 @@ def test_perform_post_login_tasks_catches_outer_exceptions((for example from inside a `with` context manager's __enter__) —not inside a per-step try/except — the wrapper must log theexception and not let the daemon thread die silently.++ The crash log is a loguru brace-format call with positional+ arguments (``"... user {} ({})", sanitize_for_log(username),+ type(err).__name__``), not an f-string, so ``call_args[0][0]`` is+ only the template. The assertions render the template with its+ arguments the way loguru does (``str.format``) instead of reading+ the bare first argument, which no longer carries the username or+ the exception type."""from local_deep_research.web.routers import auth as routes@@ -152,10 +160,20 @@ def test_perform_post_login_tasks_catches_outer_exceptions(# Must not raise.real_post_login_tasks("alice", "pw", "sess-1")- assert mock_logger.exception.called- msg = mock_logger.exception.call_args[0][0]+ # Logged as a WARNING with the exception TYPE only: this frame+ # holds the plaintext password, so no traceback may be rendered+ # from it (no logger.exception, no exc_info, no opt(exception=)).+ mock_logger.exception.assert_not_called()+ mock_logger.opt.assert_not_called()+ assert mock_logger.warning.called+ args = mock_logger.warning.call_args.args+ assert not mock_logger.warning.call_args.kwargs+ msg = args[0].format(*args[1:])assert "alice" in msgassert "crashed" in msg.lower() or "post-login" in msg.lower()+ assert "RuntimeError" in msg+ assert "outer failure" not in msg+ assert all("outer failure" not in str(arg) for arg in args)def test_perform_post_login_tasks_body_runs_when_no_outer_error(@@ -176,3 +194,4 @@ def test_perform_post_login_tasks_body_runs_when_no_outer_error(body.assert_called_once_with("bob", "pw", "sess-1")mock_logger.exception.assert_not_called()+ mock_logger.warning.assert_not_called()