Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

feat(auth): fence server-side sessions with durable rekey epochs (#6750)

LearningCircuita95385be4c24
diff --git a/changelog.d/+post-login-credential-logging.security.md b/changelog.d/+post-login-credential-logging.security.md
new file mode 100644
index 0000000000..ed3bda0efe
--- /dev/null
+++ b/changelog.d/+post-login-credential-logging.security.md
@@ -0,0 +1 @@
+Post-login background-task and registration failure handlers no longer render a traceback, whose frame locals include the plaintext password, when loguru diagnostic logging is enabled: each now logs a warning naming the exception type instead, and the background-thread crash log uses the sanitized username.
diff --git a/changelog.d/6662.security.md b/changelog.d/6662.security.md
index e88591cd27..cd6c1d632c 100644
--- a/changelog.d/6662.security.md
+++ b/changelog.d/6662.security.md
@@ -76,10 +76,10 @@ spending a full key derivation (PBKDF2) on a cold open that needs the same
sidecar. The rekey epoch state is read
pending-first, so a reader that does not hold the rekey sidecar lock cannot
report an epoch that a concurrent password-change commit superseded as
-stable. In this release that covers only the classification of a
-password-change failure raised after the lock was released: cache probes,
-logins and password changes read the epoch under the lock, and session
-validation does not read it yet.
+stable. In this release the lock-free readers include session and socket
+identity validation and the classification of a password-change failure
+raised after the lock was released. Cache probes, logins and password
+changes read the epoch under the lock.
These waits for a user's rekey sidecar give up after 5 seconds: the cache
claim behind every authenticated request (and behind `is_user_connected`,
diff --git a/changelog.d/6750.security.md b/changelog.d/6750.security.md
new file mode 100644
index 0000000000..460bb71bb8
--- /dev/null
+++ b/changelog.d/6750.security.md
@@ -0,0 +1,62 @@
+Server-side login sessions are now bound to the database's durable rekey
+epoch. A password change in any worker process ends every session issued
+before it on its next check. A login that proved the old password just as
+another worker committed a password change is refused (401) instead of
+receiving a session. If the epoch cannot be read, or a password change is
+pending, when the session is created (at login, or at the automatic
+sign-in after registration) or when the login's database open re-reads
+the epoch after verifying the password, or that re-read finds a different
+committed epoch (possible only when something changed the epoch file
+without taking the rekey lock), or that open recorded no epoch proof for
+the user, the response is 503 rather than a session the server cannot
+fence. A registration refused this way keeps the account it created, so
+the user can sign in once the fault clears. None of these refusals
+changes the account-lockout counter.
+
+A session checked while a password change is pending (its new epoch
+written but not yet committed) is removed, because the durable state
+cannot vouch for it. This holds even when that change later fails with a
+retryable error and the old password keeps working: the sessions removed
+in the meantime stay removed, and their users sign in again.
+
+If the epoch file cannot be read while a request is being authorized (for
+example file-descriptor exhaustion), every request whose session cookie
+names a signed-in user is refused with 503, including static files and
+pages that need no login, except `POST /auth/logout`. The session is
+kept, so a transient fault no longer logs every active user out. The
+logout endpoint itself stays available during such a fault: it ends the
+caller's own session and clears its cookie without opening the database.
+The page's logout button first checks the session by fetching a fresh
+CSRF token, which is refused like any other request, so during such a
+fault it shows a message asking the user to try again. Clearing a cookie
+whose database can no longer be reopened also destroys its server-side
+session and disconnects its sockets during such a fault.
+
+`SessionManager.has_active_sessions_for` no longer counts a session whose
+epoch has been superseded. Only that check changed: the idle-connection
+sweep's snapshot of active users (`get_active_usernames`) is still
+time-based and keeps counting such a session until it idles out or its
+next check removes it.
+
+When a check removes a session (its epoch was superseded, the durable
+epoch state cannot vouch for it, or it idled out), the live Socket.IO
+connections opened with that session are disconnected too. A password
+change ends every session of that user held by the worker process that
+handled it and disconnects all of that user's sockets in that process at
+once. When the password is changed in another worker, a socket opened with
+a stolen cookie is disconnected only when its session is next checked (an
+HTTP request, a socket connect, or a subscribe/unsubscribe carrying that
+cookie); a socket that sends nothing keeps receiving the user's events in
+its own worker until that session idles out, and progress events delivered
+to a socket subscribed to a research keep the session from idling out. On
+Windows, a reader holding an epoch file open briefly blocks a password
+change or a new database from replacing or removing it (and the reverse);
+both sides now retry that sharing violation a few times before failing,
+and a read that still fails is refused with 503 as above.
+
+The unused authorization-lease API (`acquire_authorization_lease`,
+`AuthorizationLeaseMode`) and its `.rekey.authorization.lock` sidecar are
+removed; the rekey lock is the only sidecar lock. When the backup refresh
+after a password change cannot take the rekey lock (#7755), that is now
+logged as a warning without a traceback, because the frame holds the old
+and new passwords.
diff --git a/docs/decisions/0015-rekey-lock-architecture.md b/docs/decisions/0015-rekey-lock-architecture.md
index 9979b84c74..8e35706d01 100644
--- a/docs/decisions/0015-rekey-lock-architecture.md
+++ b/docs/decisions/0015-rekey-lock-architecture.md
@@ -1,6 +1,6 @@
# ADR-0015: Rekey locks and connection generation publication
-**Date:** 2026-10-20 (Asia/Seoul)
+**Date:** 2026-10-20 (Asia/Seoul; updated 2026-10-24)
**Status:** Descriptive record of existing implementation
## Context
@@ -14,6 +14,12 @@ policy or assert maintainer approval of a new design. The
reference. [ADR-0004](0004-nullpool-for-sqlcipher.md) describes engine pooling;
[ADR-0009](0009-per-user-isolation-boundary.md) describes user isolation.
+> **Updated 2026-10-24:** the `_sidecar_gates` row describes the weak gate
+> registry from #7705, and the epoch-projection paragraph at the end of
+> *Password change and epoch publication* describes the web session
+> fencing added by #6750, which also removed the unused authorization-lease
+> API. The rest of the record was written against the commit above.
+
## Decision
Record the following path-specific ownership and publication rules. Each
@@ -36,7 +42,7 @@ not every application lock. The declarations and lookup helpers are in
| `_rekey_locks` | Username; one `Lock` dictionary per manager | Lifecycle exclusion for cached-generation claims and disposal/reset. Entries are lazy and retained across close/reset. The name does not mean every password change takes this lock. |
| `_password_change_locks` | Username; a separate `Lock` dictionary per manager | Nonblocking outer gate for concurrent `change_password` calls. Entries are lazy and retained; releasing a call's gate does not remove its entry. |
| `_connections_lock` | One manager-wide `RLock`, created with the manager | Protects lock-registry lookup and short engine/verifier/epoch snapshots, identity claims, detach and publication. It is not a fourth per-user dictionary. |
-| `_sidecar_gates` | Actual sidecar `Path`; module-global dictionary of `Lock` objects | Serializes threads using the same sidecar, including different managers. Current storage is a strong dictionary with no entry removal. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release. |
+| `_sidecar_gates` | Actual sidecar `Path`; module-global `WeakValueDictionary` of `Lock` objects | Serializes threads using the same sidecar, including different managers. `_get_sidecar_gate` looks up or creates the entry under `_sidecar_gates_guard`. An entry is reclaimed once no holder or waiter references its gate, and the next caller creates a fresh one. `_acquire_sidecar_lock` keeps a strong local gate reference through its wait and release, so a gate that is waited on or held cannot be reclaimed and replaced. |
`_sidecar_gates_guard` only protects brief registry lookup/creation. It is
released before waiting for the gate or OS lock. Manager lookup/snapshot
@@ -197,10 +203,21 @@ made safe by an epoch token.
`read_rekey_epoch_state` reads pending first, then committed, including for
exception classification and read-only projections that do not hold a
-sidecar. The unused application authorization projection in
-[rekey_epoch.py](../../src/local_deep_research/database/rekey_epoch.py) is not
-evidence of implemented web session fencing. Not every reader repairs
-state or owns a lock.
+sidecar. Web session fencing is built on one such projection,
+`read_user_authorization_epoch` in
+[authorization_epoch.py](../../src/local_deep_research/database/authorization_epoch.py):
+[SessionManager](../../src/local_deep_research/web/auth/session_manager.py)
+binds each session to the committed epoch when it is created (refusing a
+login whose proven epoch is no longer committed) and re-projects the epoch
+each time it validates the session (HTTP requests, Socket.IO connects,
+subscribes and unsubscribes). A superseded epoch, a pending
+rekey or state that cannot vouch for the session removes it; an unreadable
+epoch file refuses that request (503 over HTTP) but keeps the session. The
+projection takes no lock, so fencing applies per check, not across a
+request already admitted. The former authorization-lease API
+(`AuthorizationLeaseMode`, `acquire_authorization_lease`) and its
+`.rekey.authorization.lock` sidecar were removed; `.rekey.lock` is the only
+sidecar lock. Not every reader repairs state or owns a lock.
## Consequences
@@ -217,5 +234,6 @@ state or owns a lock.
successful epoch publication does not prove all WAL pages reached the
main database file.
- This documentation changes no runtime behavior and supplies no concurrency
- safety proof. Separate pending gate-reclamation, rollback-retention and
- authentication/session proposals are not implemented by this record.
+ safety proof. Gate reclamation (#7705) and web session fencing (#6750)
+ have since been implemented and are described above; the separate
+ rollback-retention proposal is not implemented by this record.
diff --git a/pyproject.toml b/pyproject.toml
index d711800858..d698c148c3 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -299,6 +299,7 @@ markers = [
"fd_canary: heavy file-descriptor-leak regression tests (#3816). CPU-bound real-event-loop spins that exceeded the legacy 60s timeout under coverage + '-n auto' contention, and whose FD-count assertions are noisy under parallel execution; run in a dedicated serial, no-coverage CI step instead of the parallel suite.",
"lifespan: enters the app's real lifespan (startup/shutdown). At most ONE such test may run per process: socketio_asgi.init_lock() only assigns when _lock is None, so a second cycle would leave the asyncio.Lock bound to a dead loop. Do not add a second one or convert it into a fixture.",
"real_session_check: opt out of the autouse _legacy_bare_username_auth shim so the real server-side session-id check is exercised. For tests that must prove a destroyed/invalid session is rejected.",
+ "real_durable_epoch: opt out of the autouse _durable_epoch_projection shim so the real durable rekey-epoch projection is exercised. For tests that must prove epoch-bound sessions fence on password change.",
"serial: tests that mutate PROCESS-GLOBAL state (os.umask, os.chdir, signal handlers) for the duration of an assertion. Correct on their own, but under '-n auto' any other file creation happening concurrently in the same worker process inherits the hostile global -- so they run in a dedicated serial CI step instead of the parallel suite. This marker says NOTHING about privilege by itself: if a serial test's assertions also require a genuinely unprivileged process (e.g. it self-skips with `if os.geteuid() == 0: pytest.skip(...)`), it MUST also carry the 'nonroot' marker explicitly -- do not rely on a CI step happening to invoke pytest under an unprivileged user (docker-tests.yml's serial step uses '--user ldruser', but that is CI wiring, not a property of the marker, and a lane that ran '-m serial' as root would silently no-op every assertion in such a test). docker-tests.yml runs 'serial' and 'nonroot' together in one step selecting '-m serial or nonroot' under '--user ldruser', so a module carrying both markers runs exactly once.",
"connected: marks the connected pytest suite (real per-user encrypted SQLCipher DBs; heavier scenarios run out of the default unit suite).",
"nonroot: tests whose assertions require a genuinely unprivileged process -- they start with `if os.geteuid() == 0: pytest.skip(...)` because permission bits (chmod/O_CREAT denial, read-only dirs, etc.) are not enforced for root. A test may carry 'nonroot' alone (does not mutate process-global state, e.g. tests/security/test_db_connection_create_hostile_dir.py, tests/security/test_db_connection_readonly_hostile_fs.py) or 'nonroot' together with 'serial' (also mutates process-global state, e.g. tests/security/test_db_connection_umask_perms.py). docker-tests.yml runs both markers together in one step, '-m serial or nonroot' under '--user ldruser', so a module carrying both runs exactly once regardless of which marker it also carries. The distinction matters because docker-tests.yml's default (main) container runs as root (it needs write access to the bind-mounted checkout for coverage/junit output), so nonroot-marked tests would silently no-op there unless excluded from the main step ('not serial and not nonroot') and run in that dedicated '--user ldruser' step instead. No exclusion is needed in security-tests.yml, which already executes as the non-root Actions runner user for every step (main and serial alike), so a serial+nonroot module's non-root requirement is satisfied there without a dedicated nonroot step or selector.",
diff --git a/src/local_deep_research/database/authorization_epoch.py b/src/local_deep_research/database/authorization_epoch.py
index 2ae4c7241a..9ee9a91303 100644
--- a/src/local_deep_research/database/authorization_epoch.py
+++ b/src/local_deep_research/database/authorization_epoch.py
@@ -1,5 +1,8 @@
"""Read-only authorization identity projection for encrypted user databases.
+Session and socket identity validation use these projections to fence
+identities issued before a password change.
+
The durable token, pending/committed epoch, and fencing state machine lives
in ``rekey_lock``. These projections read without taking its sidecar lock;
its pending-first reads prevent a concurrent commit and pending cleanup
diff --git a/src/local_deep_research/database/encrypted_db.py b/src/local_deep_research/database/encrypted_db.py
index 93e357c8d2..c0d1ef8fc1 100644
--- a/src/local_deep_research/database/encrypted_db.py
+++ b/src/local_deep_research/database/encrypted_db.py
@@ -139,6 +139,81 @@ type _ConnectionGenerationClaim = (
)
+class LoginProof:
+ """The durable rekey epoch a successful ``open_user_database`` proved.
+
+ Filled only inside :func:`capture_login_proof`. ``epoch`` is the
+ committed epoch the password was verified against UNDER the rekey
+ sidecar lock (the claimed cache generation's epoch, or the epoch the
+ cold open ran under). A login binds its session to this value, not
+ to whatever epoch is live by the time the session is created: a rekey
+ in another worker can commit in between, and an old-password login
+ straddling it must not come away with a new-epoch session.
+ """
+
+ __slots__ = ("username", "epoch")
+
+ def __init__(self) -> None:
+ self.username: str | None = None
+ self.epoch: str | None = None
+
+ def proven_epoch_for(self, username: str) -> str:
+ """The epoch proved for ``username``, or refuse (fail closed).
+
+ A successful open always records a proof for the user it opened,
+ so a missing proof, one recorded for a different username, or one
+ without an epoch means the caller cannot show which epoch the
+ password was verified against; minting a session anyway would
+ skip the straddle check silently.
+
+ Raises:
+ MissingLoginProofError: no usable proof for ``username``.
+ """
+ if self.username != username or self.epoch is None:
+ raise MissingLoginProofError(username)
+ return self.epoch
+
+
+class MissingLoginProofError(Exception):
+ """A successful login open left no epoch proof for the logged-in user."""
+
+ def __init__(self, username: str) -> None:
+ self.username = username
+ super().__init__("no login epoch proof was recorded for this login")
+
+
+# Per-thread proof collector. Only set while a caller is inside
+# capture_login_proof(), so opens made by every other caller (request
+# middleware, sockets, background threads) record nothing and nothing
+# lingers on a pooled worker thread between requests.
+_login_proof_collector = threading.local()
+
+
+@contextmanager
+def capture_login_proof() -> Generator[LoginProof, None, None]:
+ """Collect the epoch proved by ``open_user_database`` on this thread.
+
+ The OUTERMOST successful open inside the block wins: the phase-2 RAG
+ rekey re-enters ``open_user_database`` on the same thread before the
+ outer call returns, and the outer call records last.
+ """
+ previous = getattr(_login_proof_collector, "proof", None)
+ proof = LoginProof()
+ _login_proof_collector.proof = proof
+ try:
+ yield proof
+ finally:
+ _login_proof_collector.proof = previous
+
+
+def _record_login_proof(username: str, epoch: str | None) -> None:
+ proof = getattr(_login_proof_collector, "proof", None)
+ if proof is None:
+ return
+ proof.username = username
+ proof.epoch = epoch
+
+
@dataclass(frozen=True, slots=True)
class _ConnectionGenerationUnverifiable:
"""A generation is cached, but its epoch could not be checked.
@@ -1965,17 +2040,11 @@ class DatabaseManager:
) -> Optional[Engine]:
"""Open a user's database, verifying the password only with SQLCipher.
- With encryption enabled, a cold open validates the derived key
- against the database; a cached open requires a matching in-memory
- password verifier. An already-open database alone is not proof of
- the caller's password.
-
- In the plaintext development fallback, neither cold nor cached
- opens verify the account password. The nonblank-input check below
- still applies, but a different password can open the same database.
- A returned engine in that mode is not evidence of authentication.
- Callers must not use this fallback for shared accounts that rely on
- LDR password verification without adding independent authentication.
+ With encryption enabled, cold opens validate the derived key and
+ cached opens require a matching password verifier. The plaintext
+ development fallback does not verify account passwords: its returned
+ engine and epoch are not evidence of authentication. Shared plaintext
+ accounts require independent authentication.
When an engine is cached but its rekey sidecar cannot be opened or
locked, or its epoch cannot be read, the typed fault
@@ -1988,6 +2057,39 @@ class DatabaseManager:
raises it after the derivation (without waiting, in a request whose earlier
wait ran out: ``sidecar_wait_budget``). None stays reserved for a
credential or missing-database verdict.
+
+ Inside :func:`capture_login_proof` the epoch the password was
+ proved against is recorded for the caller (the login route binds
+ the new session to it).
+ """
+ engine, proven_epoch, cold_opened = self._open_user_database_proving(
+ username, password
+ )
+ # Phase 2 of the RAG vector-store cutover runs OUTSIDE init_lock (only
+ # on the path that actually did the cold-open — the early returns
+ # belong to a thread that already ran it). It re-enters
+ # open_user_database() via get_user_db_session on a session-cache miss;
+ # doing it while still holding the non-reentrant init_lock would
+ # self-deadlock if a concurrent close_user_database() evicted the
+ # freshly-cached connection between the cache write and that re-entry.
+ if cold_opened:
+ self._run_phase2_rekey(username, password)
+ # Recorded AFTER phase 2 so this (outer) call's proof wins over any
+ # proof the phase-2 re-entry recorded on this thread.
+ _record_login_proof(
+ username, proven_epoch if engine is not None else None
+ )
+ return engine
+
+ def _open_user_database_proving(
+ self, username: str, password: str
+ ) -> tuple[Optional[Engine], Optional[str], bool]:
+ """Open (or claim) the user's engine and report what was proved.
+
+ Returns ``(engine, proven_epoch, cold_opened)``. ``proven_epoch`` is
+ the committed durable epoch the password was verified against while
+ the rekey sidecar lock was held; ``cold_opened`` is True only when
+ this call built and cached a new engine (phase 2 must then run).
"""
open_start = time.perf_counter()
@@ -2040,7 +2142,7 @@ class DatabaseManager:
generation, password
)
):
- return generation.engine
+ return generation.engine, generation.epoch, False
# Serialize the cold-open (engine build + migration) per user so two
# concurrent first-opens never run alembic against the same database
@@ -2059,10 +2161,10 @@ class DatabaseManager:
password, db_path=db_path
).hex()
if _db_path_is_symlinked(db_path, username):
- return None
+ return None, None, False
if not db_path.exists():
logger.error(f"No database found for user {username}")
- return None
+ return None, None, False
# The stable sidecar serializes cold opens with cross-process rekeys.
# Epoch repair and publication happen only while this inode is locked.
@@ -2078,10 +2180,10 @@ class DatabaseManager:
# Re-checked under the lock (cheap now): the pre-gate above
# ran without any exclusion.
if _db_path_is_symlinked(db_path, username):
- return None
+ return None, None, False
if not db_path.exists():
logger.error(f"No database found for user {username}")
- return None
+ return None, None, False
cleanup_stale_rekey_temps(db_path)
rekey_state = read_rekey_epoch_state(db_path)
if rekey_state.pending_epoch is not None:
@@ -2095,7 +2197,7 @@ class DatabaseManager:
if not self._recover_pending_rekey(
db_path, password, rekey_state
):
- return None
+ return None, None, False
recovered_pending_rekey = True
try:
current_epoch = read_or_initialize_rekey_epoch(db_path)
@@ -2114,7 +2216,7 @@ class DatabaseManager:
if cached is not None and self._cached_engine_trusted(
username, password, current_epoch
):
- return cached
+ return cached, current_epoch, False
if (
cached is not None
and not self._connection_epoch_matches(
@@ -2130,7 +2232,7 @@ class DatabaseManager:
username, password, open_start, prederived_hex_key
)
if engine is None:
- return None
+ return None, None, False
# Re-read the durable state the password was just proved
# under. No outcome other than the same stable committed
@@ -2185,20 +2287,11 @@ class DatabaseManager:
replaced = self._cache_connection(username, entry)
self._dispose_detached_connection(username, replaced)
- # Phase 2 of the RAG vector-store cutover runs OUTSIDE init_lock (only
- # on the path that actually did the cold-open — the early returns above
- # belong to a thread that already ran it). It re-enters
- # open_user_database() via get_user_db_session on a session-cache miss;
- # doing it while still holding the non-reentrant init_lock would
- # self-deadlock if a concurrent close_user_database() evicted the
- # freshly-cached connection between the cache write and that re-entry.
- if engine is not None:
- if recovered_pending_rekey:
- self._purge_superseded_key_backups(
- username, password, expected_epoch=current_epoch
- )
- self._run_phase2_rekey(username, password)
- return engine
+ if engine is not None and recovered_pending_rekey:
+ self._purge_superseded_key_backups(
+ username, password, expected_epoch=current_epoch
+ )
+ return engine, current_epoch, True
def _purge_superseded_key_backups(
self, username: str, password: str, *, expected_epoch: str
diff --git a/src/local_deep_research/database/rekey_lock.py b/src/local_deep_research/database/rekey_lock.py
index 40ad123a31..f9dcb9dc04 100644
--- a/src/local_deep_research/database/rekey_lock.py
+++ b/src/local_deep_research/database/rekey_lock.py
@@ -9,12 +9,12 @@ import tempfile
import threading
import time
import weakref
-from collections.abc import Generator
+from collections.abc import Callable, Generator
from contextlib import contextmanager
from dataclasses import dataclass
from enum import StrEnum, unique
from pathlib import Path
-from typing import assert_never
+from typing import TypeVar
from uuid import UUID, uuid4
@@ -22,14 +22,6 @@ class RekeyOperationalError(Exception):
"""Base for explicit operational failures during password rekey."""
-@unique
-class AuthorizationLeaseMode(StrEnum):
- """Cross-process authorization lease compatibility modes."""
-
- SHARED = "shared"
- EXCLUSIVE = "exclusive"
-
-
@unique
class RekeySidecarOperation(StrEnum):
"""Safe lifecycle stages for sidecar I/O diagnostics."""
@@ -172,9 +164,12 @@ class RekeyEpochUnreadableError(RekeyRecoveryRequiredError):
Raised for every ``OSError`` other than ``FileNotFoundError`` while
reading ``.rekey.epoch`` / ``.rekey.pending`` (EMFILE, ENFILE, EAGAIN,
EIO, EACCES, ...; EINTR never gets here, Python retries it itself per
- PEP 475), and when the reader refuses a planted file without reading
- it: a symlink (ELOOP) or anything that is not a regular file, such as
- a FIFO or a directory (EINVAL). It is a ``RekeyRecoveryRequiredError``, so
+ PEP 475; a PermissionError is first retried a bounded number of times
+ as a possible Windows sharing violation, see
+ ``_retry_sharing_violation``), and when the reader refuses a planted
+ file without reading it: a symlink (ELOOP) or anything that is not a
+ regular file, such as a FIFO or directory (EINVAL). It is a
+ ``RekeyRecoveryRequiredError``, so
every existing handler keeps failing closed exactly as before; the
subclass only lets a caller that can retry later (session validation)
tell "nothing could be read" apart from durable state that was read
@@ -222,7 +217,6 @@ class RekeyEpochState:
class _SidecarLockRequest:
db_path: Path
lock_path: Path
- mode: AuthorizationLeaseMode
blocking: bool
gate_wait: float | None = None
contention_retry: float = 0.0
@@ -268,18 +262,11 @@ def _acquire_posix_lock(
fd: int,
lock_path: Path,
*,
- mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,
blocking: bool = False,
) -> None:
import fcntl
- match mode:
- case AuthorizationLeaseMode.SHARED:
- operation = fcntl.LOCK_SH
- case AuthorizationLeaseMode.EXCLUSIVE:
- operation = fcntl.LOCK_EX
- case _ as unreachable:
- assert_never(unreachable)
+ operation = fcntl.LOCK_EX
if not blocking:
operation |= fcntl.LOCK_NB
try:
@@ -304,7 +291,6 @@ def _acquire_windows_lock(
fd: int,
lock_path: Path,
*,
- mode: AuthorizationLeaseMode = AuthorizationLeaseMode.EXCLUSIVE,
blocking: bool = False,
) -> None:
import msvcrt
@@ -314,13 +300,7 @@ def _acquire_windows_lock(
_ = os.write(fd, b"\0")
_ = os.lseek(fd, 0, os.SEEK_SET)
try:
- match mode:
- case AuthorizationLeaseMode.SHARED:
- lock_mode = msvcrt.LK_RLCK if blocking else msvcrt.LK_NBRLCK
- case AuthorizationLeaseMode.EXCLUSIVE:
- lock_mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCK
- case _ as unreachable:
- assert_never(unreachable)
+ lock_mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCK
msvcrt.locking(fd, lock_mode, 1)
except OSError as error:
if error.errno in {
@@ -343,10 +323,6 @@ def _release_windows_lock(fd: int) -> None:
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
-def _authorization_lock_path(db_path: Path) -> Path:
- return db_path.with_name(f"{db_path.name}.rekey.authorization.lock")
-
-
def _sidecar_io_error(
lock_request: _SidecarLockRequest,
operation: RekeySidecarOperation,
@@ -541,14 +517,12 @@ def _acquire_os_lock_with_retry(
_acquire_windows_lock(
fd,
lock_request.lock_path,
- mode=lock_request.mode,
blocking=lock_request.blocking,
)
else:
_acquire_posix_lock(
fd,
lock_request.lock_path,
- mode=lock_request.mode,
blocking=lock_request.blocking,
)
except RekeyLockContentionError:
@@ -655,7 +629,6 @@ def acquire_rekey_lock(
lock_request = _SidecarLockRequest(
db_path=db_path,
lock_path=db_path.with_name(f"{db_path.name}.rekey.lock"),
- mode=AuthorizationLeaseMode.EXCLUSIVE,
blocking=blocking and timeout is None,
gate_wait=local_wait,
contention_retry=contention_retry,
@@ -665,23 +638,6 @@ def acquire_rekey_lock(
yield
-@contextmanager
-def acquire_authorization_lease(
- db_path: Path,
- mode: AuthorizationLeaseMode,
- blocking: bool = True,
-) -> Generator[None, None, None]:
- """Hold a persistent shared or exclusive authorization sidecar lease."""
- lock_request = _SidecarLockRequest(
- db_path=db_path,
- lock_path=_authorization_lock_path(db_path),
- mode=mode,
- blocking=blocking,
- )
- with _acquire_sidecar_lock(lock_request):
- yield
-
-
def _epoch_path(db_path: Path) -> Path:
return db_path.with_name(f"{db_path.name}.rekey.epoch")
@@ -690,6 +646,37 @@ def _pending_epoch_path(db_path: Path) -> Path:
return db_path.with_name(f"{db_path.name}.rekey.pending")
+# Windows sharing violations between the lock-free epoch readers and the
+# epoch writers. CPython opens files on Windows without FILE_SHARE_DELETE,
+# so while a reader (session validation, on every authenticated request)
+# holds ``.rekey.epoch`` / ``.rekey.pending`` open, the writer's
+# ``os.replace`` onto it or ``unlink`` of it fails with PermissionError;
+# and a reader that opens a file whose replace/unlink is in flight
+# (delete-pending) gets PermissionError too. Both windows are a few
+# microseconds, so each side retries a PermissionError a bounded number of
+# times before giving up. Giving up keeps the pre-existing outcome: the
+# reader raises ``RekeyEpochUnreadableError`` (fail closed: no epoch is
+# ever returned from a failed read) and the writer re-raises. On POSIX a
+# PermissionError here is a genuine permission problem; the retry then
+# costs at most ``(attempts - 1) * delay`` before the same outcome.
+_SHARING_VIOLATION_ATTEMPTS = 5
+_SHARING_VIOLATION_DELAY = 0.01
+
+_T = TypeVar("_T")
+
+
+def _retry_sharing_violation(operation: Callable[[], _T]) -> _T:
+ """Run ``operation``, retrying a transient PermissionError (bounded)."""
+ for attempt in range(1, _SHARING_VIOLATION_ATTEMPTS + 1):
+ try:
+ return operation()
+ except PermissionError:
+ if attempt >= _SHARING_VIOLATION_ATTEMPTS:
+ raise
+ time.sleep(_SHARING_VIOLATION_DELAY)
+ raise AssertionError("unreachable") # pragma: no cover
+
+
# Every token this module writes is a canonical UUID string: 36 ASCII bytes.
_EPOCH_TOKEN_MAX_BYTES = len(str(UUID(int=0)))
@@ -732,7 +719,7 @@ def _read_epoch_token(path: Path, db_path: Path) -> str | None:
| getattr(os, "O_BINARY", 0)
)
try:
- fd = os.open(path, flags)
+ fd = _retry_sharing_violation(lambda: os.open(path, flags))
except FileNotFoundError:
return None
except OSError as error:
@@ -773,8 +760,8 @@ def read_rekey_epoch_state(db_path: Path) -> RekeyEpochState:
fails after that lock has been released (the read only picks which
error to report). The read order matters for that reader and for the
ones that never take the lock: ``_record_password_verifier`` (only
- tests call it) and the ``rekey_epoch`` authorization projection
- (nothing in the application calls it yet). Such a reader may run while
+ tests call it) and the ``authorization_epoch`` projection
+ used by session and socket identity validation. Such a reader may run while
a rekey is publishing, so the two files are read in the order that
cannot tear into a stale-but-stable answer. Writers publish ``.pending`` before
they change ``.epoch`` and remove it only after ``.epoch`` is
@@ -914,8 +901,11 @@ def _write_epoch_token(path: Path, epoch: str, db_path: Path) -> None:
stream.flush()
os.fsync(stream.fileno())
+ # A lock-free reader holding ``path`` open makes this fail with a
+ # sharing violation on Windows: retry it (bounded), see
+ # ``_retry_sharing_violation``.
# nosemgrep: semgrep.rules.path-traversal-risk, reason: path is a fixed suffix on the sha256-named user database path and temporary_path is mkstemp output in its directory
- os.replace(temporary_path, path)
+ _retry_sharing_violation(lambda: os.replace(temporary_path, path))
_sync_parent_directory(path)
finally:
if fd >= 0:
@@ -955,7 +945,8 @@ def clear_pending_rekey_epoch(db_path: Path, pending_epoch: str) -> None:
if state.committed_epoch != candidate or state.pending_epoch != candidate:
raise RekeyRecoveryRequiredError(db_path)
pending_path = _pending_epoch_path(db_path)
- pending_path.unlink()
+ # Same Windows sharing violation as the replace in _write_epoch_token.
+ _retry_sharing_violation(pending_path.unlink)
_sync_parent_directory(pending_path)
@@ -993,7 +984,10 @@ def publish_new_database_epoch(db_path: Path) -> str:
_write_epoch_token(_epoch_path(db_path), epoch, db_path)
pending_path = _pending_epoch_path(db_path)
try:
- pending_path.unlink()
+ # Same Windows sharing violation as clear_pending_rekey_epoch's
+ # unlink: a lock-free reader holding the stale pending token open
+ # must not fail the creation. FileNotFoundError is not retried.
+ _retry_sharing_violation(pending_path.unlink)
except FileNotFoundError:
return epoch
_sync_parent_directory(pending_path)
diff --git a/src/local_deep_research/web/auth/connection_cleanup.py b/src/local_deep_research/web/auth/connection_cleanup.py
index 94b3cd0715..d6af925d1a 100644
--- a/src/local_deep_research/web/auth/connection_cleanup.py
+++ b/src/local_deep_research/web/auth/connection_cleanup.py
@@ -176,6 +176,8 @@ def _count_open_fds() -> int:
def cleanup_idle_connections(session_manager, db_manager):
"""Close db connections for users with no active sessions and no active research."""
+ from .session_manager import SessionValidationUnavailableError
+
# 1. Purge expired sessions first
session_manager.cleanup_expired_sessions()
@@ -193,7 +195,16 @@ def cleanup_idle_connections(session_manager, db_manager):
# 5. Double-check before closing (narrows race window)
closed = 0
for username in candidates:
- if session_manager.has_active_sessions_for(username):
+ try:
+ has_active_sessions = session_manager.has_active_sessions_for(
+ username
+ )
+ except SessionValidationUnavailableError:
+ # Unknown activity is insufficient evidence for shared teardown.
+ # Request authentication still refuses an unreadable epoch.
+ logger.warning("Deferred idle cleanup: session state unavailable")
+ continue
+ if has_active_sessions:
logger.debug(
f"Skipped {username} (active session appeared since snapshot)"
)
diff --git a/src/local_deep_research/web/auth/session_manager.py b/src/local_deep_research/web/auth/session_manager.py
index 03d4528459..bd17c998a0 100644
--- a/src/local_deep_research/web/auth/session_manager.py
+++ b/src/local_deep_research/web/auth/session_manager.py
@@ -6,15 +6,94 @@ Handles session creation, validation, and cleanup.
import datetime
import secrets
import threading
+from dataclasses import dataclass
from datetime import UTC
-from typing import Dict, Optional, Set
+from typing import Dict, Optional, Set, TypedDict
from loguru import logger
+from ...database.authorization_epoch import read_user_authorization_epoch
+from ...database.rekey_lock import (
+ RekeyEpochUnreadableError,
+ RekeyOperationalError,
+)
from ...security import get_security_default
from ...utilities.formatting import short_id
+class _SessionData(TypedDict):
+ username: str
+ created_at: datetime.datetime
+ last_access: datetime.datetime
+ remember_me: bool
+ rekey_epoch: str
+
+
+class StaleLoginProofError(Exception):
+ """The login proved its password against a no-longer-current epoch.
+
+ Raised by ``create_session`` when the durable epoch the login's
+ ``open_user_database`` proved differs from the committed epoch at
+ session creation: a rekey (password change, possibly in another
+ worker) committed in between, so the password this login presented
+ no longer opens the database and must not mint a session.
+ """
+
+ def __init__(self, username: str) -> None:
+ self.username = username
+ super().__init__("login proof is stale: the password changed")
+
+
+# Key under the ASGI ``scope["state"]`` (``request.state``) where
+# ``DatabaseMiddleware`` (``_enforce_session_revocation``) records the
+# ``(session_id, username)`` it validated for THIS request, so
+# ``require_auth`` does not re-project the durable epoch a second time.
+# ``scope["state"]`` is per request (Starlette/uvicorn copy it for each
+# request) and is written only by server code after a successful
+# validation; nothing client-controlled reaches it.
+VALIDATED_SESSION_STATE_KEY = "ldr_validated_session"
+
+# Set in ``scope["state"]`` by ``DatabaseMiddleware`` when it let a
+# ``POST /auth/logout`` through WITHOUT validating the session, because the
+# durable epoch could not be read. Only the logout route reads it.
+UNVALIDATED_LOGOUT_STATE_KEY = "ldr_unvalidated_logout"
+
+
+class SessionValidationUnavailableError(Exception):
+ """The session's durable epoch could not be read for THIS check.
+
+ Raised by ``validate_session_identity`` / ``validate_session`` when
+ the epoch file read itself failed (``RekeyEpochUnreadableError``:
+ EMFILE, ENFILE, EAGAIN, EIO, ...). Nothing is known about the
+ epoch, so the request must be refused (fail closed), but the session
+ record is KEPT: only durable state that was actually read and does
+ not vouch for the session ends it. Callers answer 503 (HTTP) or
+ reject the socket action, and must NOT clear the cookie -- that would
+ log out every active user on a transient fd-exhaustion spike.
+
+ Deliberately not a ``RekeyOperationalError`` and not a falsy return:
+ a caller that forgets to handle it gets an error response, never an
+ authenticated one.
+ """
+
+ def __init__(self, username: str, error_number: int | None) -> None:
+ self.username = username
+ self.errno = error_number
+ super().__init__(
+ "session epoch temporarily unreadable "
+ f"(errno={'unknown' if error_number is None else error_number})"
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class ValidatedSession:
+ """An identity proven against the durable rekey epoch it was bound to."""
+
+ session_id: str
+ username: str
+ rekey_epoch: str
+
+
class SessionManager:
"""Manages user sessions and database connection lifecycle.
@@ -24,8 +103,17 @@ class SessionManager:
worker restart drops all sessions it held. ``validate_session`` only
ever sees sessions created by *this* process — including for callers
like the socket connect-time session gate, which now relies on this
- lookup. See ``security/account_lockout.py`` and
- ``web/routers/rag.py`` for the same caveat elsewhere in the codebase.
+ lookup — but every record is validated against the shared durable
+ rekey epoch, so a password change in ANY worker fences the stale
+ local identities of every other worker too. ``has_active_sessions_for``
+ re-projects the epoch too (a stale or unverifiable record does not
+ count as active). The idle sweeps (``cleanup_expired_sessions``,
+ ``get_active_usernames``) are time-based only and do NOT re-project
+ the epoch: a stale-epoch record is removed by its next validation or
+ when it idles out, and until then ``get_active_usernames`` can keep
+ its user counted as active by the idle-connection sweep. See
+ ``security/account_lockout.py`` and ``web/routers/rag.py`` for the
+ same multi-worker caveat elsewhere in the codebase.
"""
def __init__(self):
@@ -45,69 +133,171 @@ class SessionManager:
self.session_timeout = datetime.timedelta(hours=session_hours)
self.remember_me_timeout = datetime.timedelta(days=remember_days)
- def create_session(self, username: str, remember_me: bool = False) -> str:
+ def create_session(
+ self,
+ username: str,
+ remember_me: bool = False,
+ *,
+ proven_epoch: str | None = None,
+ ) -> str:
"""Create a new session for a user.
Args:
username: The username to create a session for.
remember_me: If True, use extended session timeout.
+ proven_epoch: The durable epoch the caller's credential check
+ proved (``LoginProof.epoch`` from the login's
+ ``open_user_database``). When given, the session is created
+ only if it is still the committed epoch.
Returns:
The session ID as a URL-safe string.
+
+ Raises:
+ StaleLoginProofError: ``proven_epoch`` is no longer committed
+ (a rekey committed after the password was proved).
+ RekeyOperationalError: the durable epoch cannot be projected
+ (the login route answers 503).
"""
+ # Bind the committed durable epoch. With a proof, it must equal the
+ # epoch the password was verified against: an old-password login
+ # straddling a rekey committed in another worker would otherwise be
+ # bound to the NEW epoch and survive the password change.
+ rekey_epoch = read_user_authorization_epoch(username)
+ if proven_epoch is not None and proven_epoch != rekey_epoch:
+ raise StaleLoginProofError(username)
session_id = secrets.token_urlsafe(32)
+ now = datetime.datetime.now(UTC)
with self._lock:
self.sessions[session_id] = {
"username": username,
- "created_at": datetime.datetime.now(UTC),
- "last_access": datetime.datetime.now(UTC),
+ "created_at": now,
+ "last_access": now,
"remember_me": remember_me,
+ "rekey_epoch": rekey_epoch,
}
logger.debug(f"Created session {session_id[:8]}... for user {username}")
return session_id
- def validate_session(self, session_id: str) -> Optional[str]:
- """
- Validate a session and return username if valid.
- Updates last access time and disconnects sockets for an expired session.
+ def validate_session_identity(
+ self, session_id: str
+ ) -> Optional[ValidatedSession]:
+ """Validate one epoch-bound session and refresh its idle timestamp.
+
+ The durable epoch is projected OUTSIDE ``_lock`` (it reads a file),
+ then the record is re-checked under the lock so a concurrent
+ destroy/recreate cannot be validated against a stale snapshot.
+
+ Failure discipline (fail closed):
+
+ * epoch mismatch — the database was rekeyed (password changed)
+ somewhere; this session's credential can no longer open it, so
+ the record is REMOVED;
+ * the epoch files could not be READ at all
+ (``RekeyEpochUnreadableError``: EMFILE, ENFILE, EAGAIN, EIO,
+ ...) — nothing is known about the epoch, so this check
+ fails closed by raising ``SessionValidationUnavailableError``,
+ but the record is KEPT. Every caller that sees a ``None``
+ clears the cookie, so deleting (or returning ``None``) here
+ would log out every active user on a transient fd spike;
+ * any other ``RekeyOperationalError`` — the files were read and
+ the durable state (malformed token, a pending rekey in flight
+ or awaiting recovery, pending without committed) cannot vouch
+ for the identity; the record is REMOVED. A pending state is
+ kept on this side on purpose: after a crashed rekey it stays
+ until the next open recovers it (which always advances the
+ epoch), and a kept record would answer 503 until it idled out
+ instead of sending the user back to the login that recovers.
+
+ Whenever the record is REMOVED (idle expiry, epoch mismatch, or
+ unvouchable durable state), the live sockets that handshook on
+ this session id are disconnected too, best-effort and after
+ ``_lock`` is released (``_disconnect_expired_sockets``; it never
+ raises). The removed record is invisible to every later sweep, so
+ this is the only point that can still revoke them.
+
+ Raises:
+ SessionValidationUnavailableError: the epoch was unreadable.
"""
- expired_username = None
+ expired_username: Optional[str] = None
with self._lock:
- if session_id not in self.sessions:
+ session_data = self.sessions.get(session_id)
+ if session_data is None:
return None
-
- session_data = self.sessions[session_id]
- now = datetime.datetime.now(UTC)
-
- # Check timeout
timeout = (
self.remember_me_timeout
if session_data["remember_me"]
else self.session_timeout
)
+ username = session_data["username"]
+ rekey_epoch = session_data["rekey_epoch"]
+
+ try:
+ current_epoch = read_user_authorization_epoch(username)
+ except RekeyEpochUnreadableError as error:
+ # Transient I/O: refuse THIS check, keep the record.
+ raise SessionValidationUnavailableError(
+ username, error.errno
+ ) from None
+ except RekeyOperationalError:
+ with self._lock:
+ removed = self.sessions.get(session_id) is session_data
+ if removed:
+ del self.sessions[session_id]
+ if removed:
+ self._disconnect_expired_sockets([session_id])
+ return None
+
+ now = datetime.datetime.now(UTC)
+ with self._lock:
+ if self.sessions.get(session_id) is not session_data:
+ return None
if now - session_data["last_access"] > timeout:
# Session expired — remove inline (already under lock).
# Defer the log call until after the lock is released: see
# the warning in __init__ about not logging while holding
# _lock.
- expired_username = session_data["username"]
+ expired_username = username
+ del self.sessions[session_id]
+ elif current_epoch != rekey_epoch:
+ # The database was rekeyed after this session was issued:
+ # its credential is dead and so is the identity it
+ # vouches for. Remove the record (other sessions of the
+ # same user bound to the new epoch stay).
del self.sessions[session_id]
else:
# Update last access
session_data["last_access"] = now
- return str(session_data["username"])
-
- # The scheduled sweep cannot find a session removed inline above.
- # Disconnect only after releasing _lock: socket teardown can enter
- # session code again, and this lock is not reentrant.
+ return ValidatedSession(session_id, username, rekey_epoch)
+
+ # The record was removed above (expired or bound to a superseded
+ # epoch). Sever the sockets that handshook on it too, after the
+ # lock is released and best-effort: once the record is gone no
+ # sweep can find this session again, and a socket is authorised
+ # once at handshake, so a left-over sid would keep receiving the
+ # user's events (``settings_changed`` carries plaintext secrets).
self._disconnect_expired_sockets([session_id])
- logger.debug(
- f"Session {session_id[:8]}... expired for {expired_username}"
- )
+ if expired_username is not None:
+ logger.debug(
+ f"Session {session_id[:8]}... expired for {expired_username}"
+ )
return None
+ def validate_session(self, session_id: str) -> Optional[str]:
+ """Validate a session and return username if valid.
+
+ Updates last access time.
+
+ Raises:
+ SessionValidationUnavailableError: the epoch was unreadable;
+ refuse the request but keep the session (see
+ ``validate_session_identity``).
+ """
+ identity = self.validate_session_identity(session_id)
+ return None if identity is None else identity.username
+
def touch_session(self, session_id: str) -> None:
"""Refresh a session's ``last_access`` without enforcing the timeout.
@@ -124,6 +314,41 @@ class SessionManager:
if session_data is not None:
session_data["last_access"] = datetime.datetime.now(UTC)
+ def session_owner(self, session_id: str) -> Optional[str]:
+ """The username the server-side record for ``session_id`` names.
+
+ ``None`` when there is no record. NOT a validation, exactly like
+ ``session_belongs_to``: no epoch read, no expiry check, no
+ ``last_access`` refresh, so it must never gate access to anything.
+ Its only caller is ``_revoke_server_session``, which scopes a
+ teardown with it: an epoch-read fault must not stop the teardown
+ of a cookie that is being cleared anyway.
+ """
+ if not session_id:
+ return None
+ with self._lock:
+ session_data = self.sessions.get(session_id)
+ return None if session_data is None else session_data["username"]
+
+ def session_belongs_to(self, session_id: str, username: str) -> bool:
+ """Whether a server-side record still binds ``session_id`` to
+ ``username``.
+
+ NOT a validation: no epoch read, no expiry check, no
+ ``last_access`` refresh, so it must never gate access to anything.
+ Its only caller is logout during an epoch-read fault, where it
+ scopes what logout may destroy: a cookie whose record is already
+ gone tears down nothing of that user's beyond its own cookie.
+ """
+ if not session_id:
+ return False
+ with self._lock:
+ session_data = self.sessions.get(session_id)
+ return (
+ session_data is not None
+ and session_data["username"] == username
+ )
+
def destroy_session(self, session_id: str):
"""Destroy a session and clean up."""
username = None
@@ -185,7 +410,11 @@ class SessionManager:
@staticmethod
def _disconnect_expired_sockets(session_ids: list[str]) -> None:
- """Best-effort disconnect of the sockets for expired session ids.
+ """Best-effort disconnect of the sockets for removed session ids.
+
+ Called by ``cleanup_expired_sessions`` and by
+ ``validate_session_identity`` whenever it removes a record (idle
+ expiry, superseded epoch, unvouchable durable state). Never raises.
Lazy-imports the socket layer to avoid an import cycle and to
tolerate its absence in non-web contexts (CLI, tests). Each session is
@@ -253,20 +482,43 @@ class SessionManager:
}
def has_active_sessions_for(self, username: str) -> bool:
- """Check if a user has any non-expired sessions."""
+ """Check if a user has any non-expired, epoch-current session.
+
+ Uses the same epoch check as ``validate_session_identity``: a
+ record bound to an epoch that is no longer committed (the password
+ was changed, possibly in another worker) is NOT active, even
+ before its next validation removes it. When the epoch cannot be
+ read, raise ``SessionValidationUnavailableError`` without changing
+ any records. Cleanup callers must defer shared teardown when the
+ answer is unknown. Readable durable state that cannot vouch for an
+ identity still counts as inactive.
+ Read-only: records are neither refreshed nor removed here.
+ """
now = datetime.datetime.now(UTC)
with self._lock:
- for data in self.sessions.values():
- if data["username"] != username:
- continue
- timeout = (
+ bound_epochs = {
+ data["rekey_epoch"]
+ for data in self.sessions.values()
+ if data["username"] == username
+ and now - data["last_access"]
+ <= (
self.remember_me_timeout
if data["remember_me"]
else self.session_timeout
)
- if now - data["last_access"] <= timeout:
- return True
+ }
+ if not bound_epochs:
+ return False
+ # The epoch is a file read: outside _lock, like validation.
+ try:
+ current_epoch = read_user_authorization_epoch(username)
+ except RekeyEpochUnreadableError as error:
+ raise SessionValidationUnavailableError(
+ username, error.errno
+ ) from None
+ except RekeyOperationalError:
return False
+ return current_epoch in bound_epochs
# Module-level singleton
diff --git a/src/local_deep_research/web/dependencies/auth.py b/src/local_deep_research/web/dependencies/auth.py
index 72649e8202..6ec9a6da83 100644
--- a/src/local_deep_research/web/dependencies/auth.py
+++ b/src/local_deep_research/web/dependencies/auth.py
@@ -22,7 +22,11 @@ from ...database.rekey_lock import RekeyOperationalError
from ...database.session_context import get_user_db_session
from ...database.session_passwords import session_password_store
from ...utilities.db_utils import get_settings_manager
-from ..auth.session_manager import session_manager
+from ..auth.session_manager import (
+ VALIDATED_SESSION_STATE_KEY,
+ SessionValidationUnavailableError,
+ session_manager,
+)
def get_session_username(request: Request) -> str | None:
@@ -142,9 +146,17 @@ def _revoke_server_session(session_id: str, username: str) -> None:
Only this session is affected. The user's other sessions keep working.
A session id owned by a different user is left alone. Teardown is
best-effort so a failure here cannot turn the 401 into a 500.
+
+ The owner is looked up with ``session_owner``, which reads no epoch.
+ ``validate_session`` re-projects the durable epoch and raises
+ ``SessionValidationUnavailableError`` while the epoch files cannot be
+ read; looking the owner up through it skipped the teardown below while
+ the cookie was still cleared, leaving exactly the orphaned record this
+ function exists to remove. A stale-epoch or expired record of this
+ user is destroyed too: the cookie bound to it is being cleared.
"""
try:
- owner = session_manager.validate_session(session_id)
+ owner = session_manager.session_owner(session_id)
if owner is not None and owner != username:
return
if owner is not None:
@@ -348,7 +360,18 @@ def require_auth(request: Request) -> str:
# chokepoint. It also gives `session_timeout_hours` / `remember_me_days`
# real effect, since `validate_session` enforces the timeouts and refreshes
# last-access on use.
- if not _server_session_valid(request, username):
+ try:
+ session_valid = _server_session_valid(request, username)
+ except SessionValidationUnavailableError:
+ # The durable epoch could not be read (e.g. EMFILE). Refuse THIS
+ # request, but keep the session and its cookie: a transient fault
+ # proves nothing about the session and must not log users out.
+ raise HTTPException(
+ status_code=503,
+ detail="Service temporarily unavailable, please retry",
+ headers={"Retry-After": "5"},
+ ) from None
+ if not session_valid:
# Destroyed by logout or password change, expired, or never valid.
# Clear the cookie so the client stops presenting a dead session.
request.session.clear()
@@ -391,11 +414,31 @@ def _server_session_valid(request: Request, username: str) -> bool:
Only ever called after a username has been confirmed present, so
"accept unconditionally" is exactly the pre-revocation contract.
+
+ Per-request memo: ``DatabaseMiddleware`` validates the same session
+ (``_enforce_session_revocation``) earlier in this request and records
+ the ``(session_id, username)`` it proved in ``request.state``. When
+ both still match the cookie, that result is reused instead of
+ re-projecting the durable epoch a second time. Anything else (no
+ memo, a different id or user) validates for real.
+
+ Raises ``SessionValidationUnavailableError`` when the epoch cannot be
+ read (``require_auth`` answers 503 and keeps the session).
"""
session_id = request.session.get("session_id")
- valid_owner = bool(
- session_id and session_manager.validate_session(session_id) == username
- )
+ if not session_id:
+ return False
+ scope = getattr(request, "scope", None)
+ state = scope.get("state") if isinstance(scope, dict) else None
+ if isinstance(state, dict) and state.get(VALIDATED_SESSION_STATE_KEY) == (
+ session_id,
+ username,
+ ):
+ valid_owner = True
+ else:
+ valid_owner = session_manager.validate_session(session_id) == username
+ # The memo only caches the epoch check. A credential lost later in this
+ # request must still retire the session, even with a shared DB open.
return valid_owner and not clear_session_if_unrecoverable(request, username)
diff --git a/src/local_deep_research/web/fastapi_app.py b/src/local_deep_research/web/fastapi_app.py
index a0f13c0066..1bc3aff9da 100644
--- a/src/local_deep_research/web/fastapi_app.py
+++ b/src/local_deep_research/web/fastapi_app.py
@@ -1961,7 +1961,7 @@ def _stamp_session_expiry(session) -> None:
)
-def _enforce_session_revocation(session) -> None:
+def _enforce_session_revocation(session, state: dict | None = None) -> None:
"""Clear `session` in place if its server-side record is gone.
`require_auth` rejects a cookie whose `session_id` no longer resolves
@@ -1986,8 +1986,11 @@ def _enforce_session_revocation(session) -> None:
downstream, so no call site needs to know about revocation.
Anonymous sessions return immediately, so the cost on the hot path is
- one dict lookup; for authenticated ones it is `validate_session`'s
- in-memory dict read under a lock. That call also refreshes
+ one dict lookup; for authenticated ones it is `validate_session`: a
+ dict read under a lock plus a re-projection of the user's durable
+ rekey epoch (two small sidecar-file reads, `.rekey.pending` and
+ `.rekey.epoch`), which is what lets a password change in another
+ worker fence this session. That call also refreshes
`last_access`, which is why this runs unconditionally rather than
behind `_skip_prefixes` — same reasoning as the expiry check above.
@@ -1996,18 +1999,47 @@ def _enforce_session_revocation(session) -> None:
behaviour, not something this widens: it already applied to every
authenticated route. The only change is that `/` and `/auth/check`
now agree with the rest of the app about who is logged in.
+
+ A validated ``(session_id, username)`` is recorded in ``state`` (the
+ request's ``scope["state"]``) under ``VALIDATED_SESSION_STATE_KEY`` so
+ ``require_auth`` can skip an identical second epoch projection in the
+ same request.
+
+ Raises ``SessionValidationUnavailableError`` (session left untouched)
+ when the durable epoch cannot be read; the caller answers 503, except
+ for ``POST /auth/logout``, which it lets through unvalidated.
"""
username = session.get("username")
if not username:
return
- from .auth.session_manager import session_manager
+ from .auth.session_manager import (
+ VALIDATED_SESSION_STATE_KEY,
+ session_manager,
+ )
session_id = session.get("session_id")
if not session_id or session_manager.validate_session(session_id) != (
username
):
session.clear()
+ elif state is not None:
+ state[VALIDATED_SESSION_STATE_KEY] = (session_id, username)
+
+
+async def _send_session_validation_unavailable(scope, receive, send) -> None:
+ """503 for a request whose session could not be validated right now.
+
+ The session (and its cookie) is left alone: the epoch read failed
+ transiently (e.g. EMFILE), which proves nothing about the session, so
+ clearing it would log the user out. The request itself is refused.
+ """
+ response = JSONResponse(
+ {"error": "Service temporarily unavailable, please retry"},
+ status_code=503,
+ headers={"Retry-After": "5"},
+ )
+ await response(scope, receive, send)
class DatabaseMiddleware:
@@ -2070,6 +2102,7 @@ class DatabaseMiddleware:
# e.g. static-asset requests would let an idle-but-signature-valid
# session's deadline silently stop advancing/enforcing whenever
# the user's tab is only pulling static assets.
+ unvalidated_logout = False
session = scope.get("session")
if session is not None:
_enforce_session_expiry(session)
@@ -2079,13 +2112,51 @@ class DatabaseMiddleware:
# but its signed cookie is still replayable". `require_auth`
# covers the latter for the routes that use it; this covers
# the ones that deliberately don't. See the docstring.
- _enforce_session_revocation(session)
+ from .auth.session_manager import (
+ UNVALIDATED_LOGOUT_STATE_KEY,
+ SessionValidationUnavailableError,
+ )
+
+ try:
+ _enforce_session_revocation(
+ session, scope.setdefault("state", {})
+ )
+ except SessionValidationUnavailableError as unavailable:
+ # Transient epoch-read failure: fail THIS request closed
+ # without touching the session (no logout). Only the
+ # errno number is logged.
+ unreadable_errno = unavailable.errno
+ if method == "POST" and path == "/auth/logout":
+ # Except logout: it only destroys (the caller's own
+ # session, its credentials, its cookie), so refusing it
+ # would only stop a user from revoking their session
+ # during the fault. It runs unvalidated: the flag tells
+ # the route to scope its teardown, and the database
+ # open below is skipped, so nothing is opened, stored
+ # or published for this cookie.
+ logger.warning(
+ "Letting logout through unvalidated: session "
+ f"epoch unreadable (errno={unreadable_errno})"
+ )
+ scope["state"][UNVALIDATED_LOGOUT_STATE_KEY] = True
+ unvalidated_logout = True
+ else:
+ logger.warning(
+ "Refusing request with 503: session epoch "
+ f"unreadable (errno={unreadable_errno})"
+ )
+ await _send_session_validation_unavailable(
+ scope, receive, send
+ )
+ return
ctx_tokens = None
# Run the inner app first to let SessionMiddleware populate scope
# Then ensure database is open
- if method != "OPTIONS" and not any(
- path.startswith(p) for p in self._skip_prefixes
+ if (
+ method != "OPTIONS"
+ and not unvalidated_logout
+ and not any(path.startswith(p) for p in self._skip_prefixes)
):
# Session data is in scope["session"] after SessionMiddleware
session_data = scope.get("session", {})
diff --git a/src/local_deep_research/web/routers/auth.py b/src/local_deep_research/web/routers/auth.py
index 98ac00fd1e..f01efdf09e 100644
--- a/src/local_deep_research/web/routers/auth.py
+++ b/src/local_deep_research/web/routers/auth.py
@@ -26,7 +26,12 @@ from datetime import datetime, timezone, UTC
from loguru import logger
from ...database.auth_db import auth_db_session
-from ...database.encrypted_db import DatabaseInitializationError, db_manager
+from ...database.encrypted_db import (
+ DatabaseInitializationError,
+ MissingLoginProofError,
+ capture_login_proof,
+ db_manager,
+)
from ...database.rekey_lock import (
RekeyIndeterminateError,
RekeyLockContentionError,
@@ -37,6 +42,8 @@ from ...database.models.auth import User
from ...database.thread_local_session import thread_cleanup
from sqlalchemy.exc import IntegrityError
from ..auth.session_manager import (
+ UNVALIDATED_LOGOUT_STATE_KEY,
+ StaleLoginProofError,
session_manager,
) # singleton from session_manager module
from ..server_config import load_server_config
@@ -115,8 +122,14 @@ def _rollback_partial_session(request: Request, username: str) -> None:
from ...database.session_passwords import session_password_store
session_password_store.clear_session(username, session_id)
- except Exception:
- logger.exception("Failed to roll back a partial user session")
+ except Exception as error:
+ # Type name only, no traceback: this helper runs inside login
+ # and register, whose caller frames hold the plaintext password
+ # (see _perform_post_login_tasks for when diagnose renders it).
+ logger.warning(
+ "Failed to roll back a partial user session ({})",
+ type(error).__name__,
+ )
def _disconnect_user_sockets(username: str) -> None:
@@ -282,8 +295,12 @@ def login(
# the alembic_runner permission check) → 503, do NOT count toward
# lockout. The user's password is correct; punishing them with a
# lockout for a server-side configuration problem would be wrong.
+ # capture_login_proof records the durable rekey epoch the password was
+ # verified against, so the session below is bound to THAT epoch rather
+ # than to whatever epoch is committed by the time it is created.
try:
- engine = db_manager.open_user_database(username, password)
+ with capture_login_proof() as login_proof:
+ engine = db_manager.open_user_database(username, password)
except DatabaseInitializationError:
logger.warning(
f"Login refused for {redact_and_bound_for_log(username, 50)}: "
@@ -349,8 +366,33 @@ def login(
status_code=401,
)
- # Success
- lockout_mgr.record_success(username)
+ # The session is bound to the epoch this login PROVED. A successful
+ # open always records a proof for this user, so a missing or foreign
+ # proof is an internal inconsistency: refuse (fail closed) rather than
+ # create a session with the straddle check skipped. Not a guessing
+ # failure, so the lockout counter is left alone.
+ try:
+ proven_epoch = login_proof.proven_epoch_for(username)
+ except MissingLoginProofError:
+ logger.warning(
+ f"Login refused for {redact_and_bound_for_log(username, 50)}: the database "
+ "open recorded no epoch proof for this user"
+ )
+ flash(
+ request,
+ "Your account database is pending maintenance. "
+ "Please try again shortly or contact the administrator.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/login.html",
+ {
+ "has_encryption": db_manager.has_encryption,
+ "allow_registrations": config.get("allow_registrations", True),
+ },
+ status_code=503,
+ )
# Prevent session fixation
request.session.clear()
@@ -360,7 +402,9 @@ def login(
# silent login (see _rollback_partial_session / #5006).
try:
# Create session
- session_id = session_manager.create_session(username, remember_bool)
+ session_id = session_manager.create_session(
+ username, remember_bool, proven_epoch=proven_epoch
+ )
request.session["session_id"] = session_id
request.session["username"] = username
# RememberMeMiddleware reads this to decide whether to strip
@@ -380,10 +424,65 @@ def login(
session_password_store.store_session_password(
username, session_id, password
)
+ except StaleLoginProofError:
+ # The password was right when it was verified, but a rekey
+ # (password change, possibly in another worker) committed before
+ # the session could be created: it no longer opens the database.
+ # Not a guessing failure, so the lockout counter is left alone:
+ # neither incremented nor reset (record_success runs only after a
+ # session is issued).
+ _rollback_partial_session(request, username)
+ logger.warning(
+ f"Login refused for {redact_and_bound_for_log(username, 50)}: the password "
+ "changed while the login was in flight"
+ )
+ flash(
+ request,
+ "Your password was changed while you were signing in. "
+ "Please sign in with the current password.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/login.html",
+ {
+ "has_encryption": db_manager.has_encryption,
+ "allow_registrations": config.get("allow_registrations", True),
+ },
+ status_code=401,
+ )
+ except RekeyOperationalError as error:
+ # The durable epoch could not be projected, so the session cannot
+ # be bound to it: same answer as the open path's rekey failures.
+ _rollback_partial_session(request, username)
+ logger.warning(
+ f"Login refused for {redact_and_bound_for_log(username, 50)}: rekey epoch "
+ f"unavailable at session creation ({type(error).__name__})"
+ )
+ flash(
+ request,
+ "Your account database is pending maintenance. "
+ "Please try again shortly or contact the administrator.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/login.html",
+ {
+ "has_encryption": db_manager.has_encryption,
+ "allow_registrations": config.get("allow_registrations", True),
+ },
+ status_code=503,
+ )
except Exception:
_rollback_partial_session(request, username)
raise
+ # Success: only now, with a session actually issued, reset the
+ # lockout counter. A login refused at session creation (stale proof,
+ # unreadable epoch) must neither count as a failure nor clear the
+ # failures already recorded.
+ lockout_mgr.record_success(username)
logger.info(f"User {_log_value_preview(username)} logged in successfully")
# Defer non-critical post-login work to a background thread so the
@@ -424,14 +523,27 @@ def _perform_post_login_tasks(
An outer try/except wraps the whole body so any exception that
escapes the per-step handlers (for example a failure inside a
- ``with`` context manager's __enter__ / __exit__) is logged loudly
- with a traceback instead of dying silently in the daemon thread.
+ ``with`` context manager's __enter__ / __exit__) is logged instead of
+ dying silently in the daemon thread.
+
+ Every handler in this worker and in ``register`` logs a WARNING with
+ the exception TYPE only -- never ``logger.exception``, ``exc_info`` or
+ ``logger.opt(exception=...)``: these frames hold the plaintext
+ password, and a rendered traceback prints frame locals whenever a
+ sink has loguru ``diagnose`` on. That is not reliably knowable from
+ here: ``config_logger`` enables it on the stderr sink when debug is
+ on (from the ``app.debug`` setting or the environment) AND
+ ``LDR_LOGURU_DIAGNOSE`` is set, and a process that never ran
+ ``config_logger`` keeps loguru's default handler, which has
+ ``diagnose`` on.
"""
try:
_perform_post_login_tasks_body(username, password, session_id)
- except Exception:
- logger.exception(
- f"Post-login background thread crashed for user {_log_value_preview(username)}"
+ except Exception as err:
+ logger.warning(
+ "Post-login background thread crashed for user {} ({})",
+ redact_and_bound_for_log(username, 50),
+ type(err).__name__,
)
@@ -486,9 +598,10 @@ def _perform_post_login_tasks_body(
f"Missing default settings loaded and version "
f"updated for user {_log_value_preview(username)}"
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login settings migration failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration("step 1 (settings version check)", step_start, username)
@@ -507,9 +620,10 @@ def _perform_post_login_tasks_body(
f"Library initialization issue for {_log_value_preview(username)}: "
f"{init_results.get('error', 'Unknown error')}"
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login library init failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration("step 2 (library init)", step_start, username)
@@ -525,10 +639,11 @@ def _perform_post_login_tasks_body(
DocumentDeletionService(username).reconcile_orphaned_document_chunks(
password
)
- except Exception:
- logger.exception(
- "Post-login orphan chunk reconciliation failed for {}",
- _log_value_preview(username),
+ except Exception as err:
+ # No traceback: this frame holds the plaintext password.
+ logger.warning(
+ f"Post-login orphan chunk reconciliation failed for "
+ f"{_log_value_preview(username)} ({type(err).__name__})"
)
_log_step_duration(
"step 2a (orphan chunk reconciliation)", step_start, username
@@ -553,13 +668,17 @@ def _perform_post_login_tasks_body(
logger.info(
f"Updated scheduler with user info for {_log_value_preview(username)}"
)
- except Exception:
- logger.exception("Could not update scheduler on login")
+ except Exception as scheduler_err:
+ logger.warning(
+ "Could not update scheduler on login"
+ f" ({type(scheduler_err).__name__})"
+ )
auth_db.commit()
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login auth DB update failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration(
"step 3 (auth DB + scheduler notify)", step_start, username
@@ -589,9 +708,10 @@ def _perform_post_login_tasks_body(
logger.info(
f"Background backup scheduled for user {_log_value_preview(username)}"
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login backup scheduling failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration("step 4 (schedule backup)", step_start, username)
@@ -617,9 +737,10 @@ def _perform_post_login_tasks_body(
queue_processor.reconcile_orphan_active_research(
username, db_session
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login research reconciliation failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration(
"step 6 (reconcile orphan research)", step_start, username
@@ -638,9 +759,10 @@ def _perform_post_login_tasks_body(
from ..queue.processor_v2 import queue_processor
queue_processor.notify_user_activity(username, session_id)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Post-login queue-resume registration failed for {_log_value_preview(username)}"
+ f" ({type(err).__name__})"
)
_log_step_duration("step 7 (resume queued research)", step_start, username)
@@ -818,10 +940,11 @@ def register(
},
status_code=400,
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Registration failed for {log_username} while creating the "
f"auth record"
+ f" ({type(err).__name__})"
)
auth_db.rollback()
flash(request, "Registration failed. Please try again.", "error")
@@ -846,10 +969,11 @@ def register(
# the user DB exists and the account is legitimate.
try:
db_manager.create_user_database(username, password)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Registration failed for {log_username} while creating the "
f"encrypted database"
+ f" ({type(err).__name__})"
)
# Delete the orphaned auth row by primary key in a fresh session
# (the original auth_db session has already closed). Deleting by ID
@@ -865,11 +989,12 @@ def register(
logger.info(
f"Cleaned up orphaned auth entry for {log_username}"
)
- except Exception:
+ except Exception as cleanup_err:
# Surface a stuck orphan so it's diagnosable, but don't mask the
# original registration failure returned to the user.
- logger.exception(
+ logger.warning(
f"Failed to clean up orphaned auth entry for {log_username}"
+ f" ({type(cleanup_err).__name__})"
)
flash(request, "Registration failed. Please try again.", "error")
return render_template(
@@ -922,8 +1047,11 @@ def register(
logger.info(
f"Updated scheduler with new user info for {log_username}"
)
- except Exception:
- logger.exception("Could not update scheduler on registration")
+ except Exception as err:
+ logger.warning(
+ "Could not update scheduler on registration"
+ f" ({type(err).__name__})"
+ )
logger.info(f"New user registered: {log_username}")
@@ -940,15 +1068,44 @@ def register(
logger.warning(
f"Library initialization issue for {log_username}: {init_results.get('error', 'Unknown error')}"
)
- except Exception:
- logger.exception(
+ except Exception as err:
+ logger.warning(
f"Error initializing library for new user {log_username}"
+ f" ({type(err).__name__})"
)
# Don't block registration on library init failure
return RedirectResponse(url="/", status_code=302)
- except Exception:
+ except RekeyOperationalError as error:
+ # create_session could not project the durable epoch the session
+ # must be bound to (a transient sidecar fault, or state that needs
+ # the operator): the same retryable 503 the login route answers for
+ # it, never a 500. The account and its database exist, so the auth
+ # row stays and the user is sent to sign in once the fault clears;
+ # the half-set session is rolled back exactly like the 500 below.
+ _rollback_partial_session(request, username)
+ logger.warning(
+ f"Registration auto-login refused for {log_username}: rekey "
+ f"epoch unavailable at session creation ({type(error).__name__})"
+ )
+ flash(
+ request,
+ "Your account was created, but signing in is temporarily "
+ "unavailable. Please sign in again shortly.",
+ "error",
+ )
+ return render_template(
+ request,
+ "auth/login.html",
+ {
+ "has_encryption": db_manager.has_encryption,
+ "allow_registrations": config.get("allow_registrations", True),
+ },
+ status_code=503,
+ )
+
+ except Exception as err:
# The encrypted DB already exists at this point, so this failure is in
# post-creation setup (auto-login / session). The account is created
# and loginable; do NOT clean up the auth row here (narrow scope).
@@ -956,9 +1113,10 @@ def register(
# the user silently logged in (SessionMiddleware would otherwise write
# the partially-set cookie on this response — see #5006).
_rollback_partial_session(request, username)
- logger.exception(
+ logger.warning(
f"Registration failed for {log_username} after database creation "
f"(post-creation session/login setup)"
+ f" ({type(err).__name__})"
)
flash(request, "Registration failed. Please try again.", "error")
return render_template(
@@ -982,6 +1140,21 @@ def logout(request: Request):
username = request.session.get("username")
session_id = request.session.get("session_id")
+ state = request.scope.get("state")
+ if (
+ username
+ and isinstance(state, dict)
+ and state.get(UNVALIDATED_LOGOUT_STATE_KEY)
+ and not session_manager.session_belongs_to(session_id, username)
+ ):
+ # DatabaseMiddleware let this logout through without validating the
+ # cookie (the durable epoch was unreadable). The user-wide teardown
+ # below runs only for a cookie whose server-side record still binds
+ # it to this username; any other cookie just loses itself, which is
+ # what the validating middleware would have done anyway.
+ request.session.clear()
+ return RedirectResponse(url="/auth/login", status_code=302)
+
if username:
# LOGOUT CLEANUP ORDER (order matters):
# 1. Unregister from news scheduler — removes the password from the
@@ -1547,15 +1720,21 @@ def change_password(
"old-key backups remain."
)
except RekeyLockContentionError:
- logger.exception(
+ # Same rule as below: no traceback, the frame holds the
+ # passwords.
+ logger.warning(
f"Backup refresh after password change for "
f"{_log_value_preview(username)} could not take the rekey "
"lock; backups under the old key may remain"
)
- except Exception:
- logger.exception(
+ except Exception as error:
+ # Type name only, no traceback: this frame holds
+ # current_password, new_password and confirm_password in its
+ # locals, and a rendered traceback (loguru diagnose) would
+ # print them.
+ logger.warning(
f"Could not refresh backups after password change "
- f"for {_log_value_preview(username)}"
+ f"for {_log_value_preview(username)} ({type(error).__name__})"
)
# 3. Revoke every old-password credential and session.
diff --git a/src/local_deep_research/web/services/socketio_asgi.py b/src/local_deep_research/web/services/socketio_asgi.py
index 5f12e37a1a..95151f92d9 100644
--- a/src/local_deep_research/web/services/socketio_asgi.py
+++ b/src/local_deep_research/web/services/socketio_asgi.py
@@ -454,13 +454,26 @@ async def connect(sid, environ, auth=None):
# had no successor here: DatabaseMiddleware's `_enforce_session_revocation`
# cannot cover it, because that middleware returns early for non-HTTP
# scopes and additionally skips the "/ws/" prefix.
- from ..auth.session_manager import session_manager
+ from ..auth.session_manager import (
+ SessionValidationUnavailableError,
+ session_manager,
+ )
handshake_session_id = session_data.get("session_id")
- if (
- not handshake_session_id
- or session_manager.validate_session(handshake_session_id) != username
- ):
+ try:
+ handshake_valid = bool(handshake_session_id) and (
+ session_manager.validate_session(handshake_session_id) == username
+ )
+ except SessionValidationUnavailableError:
+ # The durable epoch could not be read (transient I/O). Refuse this
+ # handshake (fail closed) but leave the session intact; the client
+ # reconnects on its own.
+ logger.warning(
+ f"Rejected WebSocket connection for {username} from {sid}: "
+ "session could not be validated right now"
+ )
+ return False
+ if not handshake_valid:
logger.warning(
f"Rejected WebSocket connection for {username} from {sid}: "
"session is not valid server-side (logged out, expired, or "
@@ -570,7 +583,15 @@ async def connect(sid, environ, auth=None):
# enumerating sockets -- logout destroys the session first, then
# disconnects -- so a socket that registers after invalidation is
# guaranteed to observe the failed re-check.
- if session_manager.validate_session(handshake_session_id) != username:
+ try:
+ still_valid = (
+ session_manager.validate_session(handshake_session_id) == username
+ )
+ except SessionValidationUnavailableError:
+ # Cannot prove the session is still alive: fail closed like an
+ # invalidated one (the session record itself is kept).
+ still_valid = False
+ if not still_valid:
async with _lock:
_sid_users.pop(sid, None)
_sid_sessions.pop(sid, None)
@@ -708,12 +729,15 @@ async def _user_owns_research(username: str, research_id: str) -> bool:
async def _socket_session_still_valid(
sid: str, username: str
-) -> tuple[str | None, bool]:
+) -> tuple[str | None, bool | None]:
"""Re-check that this socket's originating session is still alive.
Returns ``(session_id, is_valid)``. The id comes back so the caller can
tear down EVERY socket of a dead session, not just the one that asked --
- see the rejection path in ``on_subscribe``.
+ see the rejection path in ``on_subscribe``. ``is_valid`` is ``None``
+ when the session could not be validated right now (the durable epoch
+ was unreadable): the caller must refuse the action but NOT tear the
+ session's sockets down, since nothing is known to be wrong with it.
The connect-time gate is not sufficient on its own: identity is captured
into ``_sid_users`` at handshake and then frozen for the socket's whole
@@ -735,9 +759,18 @@ async def _socket_session_still_valid(
if not session_id:
return None, False
- from ..auth.session_manager import session_manager
+ from ..auth.session_manager import (
+ SessionValidationUnavailableError,
+ session_manager,
+ )
- return session_id, session_manager.validate_session(session_id) == username
+ try:
+ return (
+ session_id,
+ session_manager.validate_session(session_id) == username,
+ )
+ except SessionValidationUnavailableError:
+ return session_id, None
@sio.on("subscribe_to_research")
@@ -772,6 +805,22 @@ async def on_subscribe(sid, data):
stale_session_id, session_ok = await _socket_session_still_valid(
sid, username
)
+ if session_ok is None:
+ # Unverifiable right now (transient epoch-read failure): refuse
+ # this subscribe, keep the socket and its session.
+ logger.warning(
+ f"Rejected subscribe from {sid} ({username}): session could not "
+ "be validated right now"
+ )
+ await sio.emit(
+ "subscribe_error",
+ {
+ "error": "Temporarily unavailable",
+ "research_id": research_id,
+ },
+ room=sid,
+ )
+ return
if not session_ok:
logger.warning(
f"Rejected subscribe from {sid} ({username}): originating session "
@@ -892,6 +941,13 @@ async def on_unsubscribe(sid, data):
stale_session_id, session_ok = await _socket_session_still_valid(
sid, username
)
+ if session_ok is None:
+ # Unverifiable right now: refuse this unsubscribe only.
+ logger.warning(
+ f"Rejected unsubscribe from {sid} ({username}): session could "
+ "not be validated right now"
+ )
+ return
if not session_ok:
logger.warning(
f"Rejected unsubscribe from {sid} ({username}): originating "
diff --git a/src/local_deep_research/web/static/js/services/api.js b/src/local_deep_research/web/static/js/services/api.js
index 27b47994b5..24539aa4ed 100644
--- a/src/local_deep_research/web/static/js/services/api.js
+++ b/src/local_deep_research/web/static/js/services/api.js
@@ -64,8 +64,9 @@ async function fetchWithCsrfRecovery(url, options = {}, send = fetch) {
if (getAuthContext() !== context) {
throw new Error('Your sign-in changed. Reload this page before trying again.');
}
- // The endpoint returns 401/409 if the login expired or was replaced. Let
- // the existing auth/error handling surface it; no original write is sent.
+ // The refresh returns 401/409 if the login expired or was replaced, or
+ // 503 if the session cannot be checked right now (it is kept). Let the
+ // existing auth/error handling surface it; no original write is sent.
if (!refresh.ok) return refresh;
const fresh = await refresh.json();
options.signal?.throwIfAborted();
diff --git a/tests/conftest.py b/tests/conftest.py
index f70fa6679a..2f37737bd4 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -652,6 +652,126 @@ def _legacy_bare_username_auth(request):
yield
+# Last epoch projected per username under the _durable_epoch_projection
+# shim -- see the fixture for why stability across failures matters.
+_LAST_EPOCH_PROJECTION: dict[str, str] = {}
+
+
[email protected](autouse=True)
+def _durable_epoch_projection(request):
+ """Keep fabricated test sessions working under durable-epoch binding.
+
+ ``SessionManager.create_session`` binds the user's durable rekey epoch
+ (read from disk) so a password change in ANY worker fences every
+ stale session. In production a session is only ever created at login,
+ right after ``open_user_database`` proved the credential against the
+ epoch-fenced database, so the projection always resolves. The test
+ fleet, however, fabricates sessions and logins against mocked
+ databases for users with no on-disk state; the projection would raise
+ ``RekeyRecoveryRequiredError`` for them.
+
+ This autouse fixture relaxes the seam the same way
+ ``_legacy_bare_username_auth`` relaxes the server-side-session gate:
+ sessions fabricated under tests bind a deterministic per-username
+ epoch. Tests that must prove the real fencing contract (binding,
+ stale-epoch removal, transient-vs-ambiguous failure discipline) opt
+ out with ``@pytest.mark.real_durable_epoch`` and run against real
+ epoch files.
+ """
+ if request.node.get_closest_marker("real_durable_epoch"):
+ # Fencing tests: leave the real epoch projection in place.
+ yield
+ return
+
+ # A shared monkeypatch dependency would outlive singleton teardown and
+ # leave unrelated tests' provider mocks active during that reset.
+ with pytest.MonkeyPatch.context() as epoch_patch:
+ _install_durable_epoch_projection(epoch_patch)
+ yield
+
+
+def _install_durable_epoch_projection(monkeypatch):
+ import local_deep_research.database.encrypted_db as _encrypted_db
+ from local_deep_research.database.rekey_epoch import (
+ read_user_authorization_epoch as _real_projection,
+ )
+
+ def _fabricated(username):
+ return _LAST_EPOCH_PROJECTION.setdefault(
+ username, f"test-epoch-{username}"
+ )
+
+ def _project_through_or_fabricate(username):
+ # Sessions created against real databases (module-scoped client
+ # fixtures register + log in real users) keep their REAL epochs,
+ # and any failure projecting them propagates exactly as it would
+ # in production. Only two cases get a deterministic stand-in:
+ #
+ # * ``db_manager`` itself is a test double (not a DatabaseManager,
+ # or one whose path resolution is mocked), so there is nothing
+ # durable to project;
+ # * the resolved path holds NO durable epoch state at all (neither
+ # ``.rekey.epoch`` nor ``.rekey.pending``): a fabricated login
+ # against a mocked database, or a real user whose data dir a
+ # module fixture un-pinned or deleted.
+ #
+ # STABILITY cache: a username's last real projection is remembered
+ # and reused for the no-state case, so a create/validate pair that
+ # straddles a data-dir pin/restore does not mismatch-destroy the
+ # record.
+ manager = _encrypted_db.db_manager
+ if not isinstance(manager, _encrypted_db.DatabaseManager):
+ return _fabricated(username)
+ db_path = manager._get_user_db_path(username)
+ if not isinstance(db_path, Path):
+ return _fabricated(username)
+ has_state = (
+ db_path.with_name(f"{db_path.name}.rekey.epoch").exists()
+ or db_path.with_name(f"{db_path.name}.rekey.pending").exists()
+ )
+ if not has_state:
+ return _fabricated(username)
+ projected = _real_projection(username)
+ _LAST_EPOCH_PROJECTION[username] = projected
+ return projected
+
+ monkeypatch.setattr(
+ "local_deep_research.web.auth.session_manager."
+ "read_user_authorization_epoch",
+ _project_through_or_fabricate,
+ )
+
+ # The login route refuses (503) when the open recorded no epoch proof
+ # for the user. A test that swaps the login route's ``db_manager`` (or
+ # its ``open_user_database``) for a double records none, so for that
+ # case only the legacy "no proof, no straddle check" path is kept
+ # (``create_session(proven_epoch=None)``). A real open that records no
+ # proof still refuses exactly as in production.
+ _real_proven_epoch_for = _encrypted_db.LoginProof.proven_epoch_for
+
+ def _proven_epoch_for_or_double(self, username):
+ try:
+ return _real_proven_epoch_for(self, username)
+ except _encrypted_db.MissingLoginProofError:
+ router = sys.modules.get("local_deep_research.web.routers.auth")
+ manager = getattr(router, "db_manager", None)
+ real_open = _encrypted_db.DatabaseManager.open_user_database
+ is_double = (
+ not isinstance(manager, _encrypted_db.DatabaseManager)
+ or "open_user_database" in vars(manager)
+ or type(manager).open_user_database is not real_open
+ )
+ if is_double:
+ return None
+ raise
+
+ monkeypatch.setattr(
+ _encrypted_db.LoginProof,
+ "proven_epoch_for",
+ _proven_epoch_for_or_double,
+ )
+
+
@pytest.fixture
def app(temp_data_dir, monkeypatch):
"""Create an app configured for testing."""
diff --git a/tests/database/test_login_epoch_proof.py b/tests/database/test_login_epoch_proof.py
new file mode 100644
index 0000000000..008cea6bdc
--- /dev/null
+++ b/tests/database/test_login_epoch_proof.py
@@ -0,0 +1,353 @@
+"""What ``open_user_database`` proves, and when it refuses to prove it.
+
+``capture_login_proof`` records the durable epoch a successful open
+verified the password against under the rekey sidecar lock; the login
+route binds its session to that epoch. These tests pin the recorded
+epoch on each success path, that nothing is recorded outside a capture
+block, that the outer open wins over the phase-2 re-entry, and two
+refusals the proof relies on: a cold open whose epoch re-read does not
+find the epoch it ran under (the committed epoch moved, went missing or
+is pending, or the read fails) is discarded and fails closed instead of
+reading as a wrong password, and
+pending-rekey recovery verifies the candidate key before it finalizes
+the epoch.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+import pytest
+
+from local_deep_research.database import encrypted_db
+from local_deep_research.database import rekey_lock as rekey_lock_module
+from local_deep_research.database.encrypted_db import (
+ DatabaseManager,
+ LoginProof,
+ MissingLoginProofError,
+ _RekeyVerificationError,
+ capture_login_proof,
+)
+from local_deep_research.database.rekey_lock import (
+ RekeyEpochUnreadableError,
+ RekeyRecoveryRequiredError,
+ acquire_rekey_lock,
+)
+
+
+def _new_manager(data_dir: Path) -> DatabaseManager:
+ with patch.object(
+ DatabaseManager, "_check_encryption_available", return_value=True
+ ):
+ manager = DatabaseManager()
+ manager.data_dir = data_dir
+ return manager
+
+
+def manager(tmp_path: Path) -> DatabaseManager:
+ return _new_manager(tmp_path)
+
+
+def _existing_db(manager: DatabaseManager, username: str) -> Path:
+ db_path = manager._get_user_db_path(username)
+ db_path.touch()
+ return db_path
+
+
+def _initialize_epoch(db_path: Path) -> str:
+ with acquire_rekey_lock(db_path):
+ return rekey_lock_module.read_or_initialize_rekey_epoch(db_path)
+
+
+def test_cold_open_records_the_epoch_it_ran_under(
+ manager: DatabaseManager,
+) -> None:
+ epoch = _initialize_epoch(_existing_db(manager, "alice"))
+ engine = MagicMock()
+
+ with (
+ patch.object(manager, "_open_user_database_cold", return_value=engine),
+ patch.object(manager, "_run_phase2_rekey"),
+ capture_login_proof() as proof,
+ ):
+ opened = manager.open_user_database("alice", "password")
+
+ assert opened is engine
+ assert (proof.username, proof.epoch) == ("alice", epoch)
+
+
+def test_cache_claim_records_the_claimed_generation_epoch(
+ manager: DatabaseManager,
+) -> None:
+ epoch = _initialize_epoch(_existing_db(manager, "alice"))
+ engine = MagicMock()
+ manager._cache_connection(
+ "alice", manager._make_connection_cache_entry(engine, "password", epoch)
+ )
+
+ with (
+ patch.object(manager, "_open_user_database_cold") as cold_open,
+ capture_login_proof() as proof,
+ ):
+ opened = manager.open_user_database("alice", "password")
+
+ assert opened is engine
+ cold_open.assert_not_called()
+ assert (proof.username, proof.epoch) == ("alice", epoch)
+
+
+def test_nothing_is_recorded_outside_a_capture_block(
+ manager: DatabaseManager,
+) -> None:
+ _initialize_epoch(_existing_db(manager, "alice"))
+
+ with (
+ patch.object(
+ manager, "_open_user_database_cold", return_value=MagicMock()
+ ),
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ manager.open_user_database("alice", "password")
+
+ with capture_login_proof() as proof:
+ pass
+ assert (proof.username, proof.epoch) == (None, None)
+
+
+def test_outer_open_wins_over_the_phase2_reentry(
+ manager: DatabaseManager,
+) -> None:
+ epoch = _initialize_epoch(_existing_db(manager, "alice"))
+
+ def failed_reentry(username: str, _password: str) -> None:
+ # Phase 2 re-enters open_user_database on this thread; a failed
+ # re-entry records "no proof" before the outer call returns.
+ encrypted_db._record_login_proof(username, None)
+
+ with (
+ patch.object(
+ manager, "_open_user_database_cold", return_value=MagicMock()
+ ),
+ patch.object(manager, "_run_phase2_rekey", side_effect=failed_reentry),
+ capture_login_proof() as proof,
+ ):
+ manager.open_user_database("alice", "password")
+
+ assert proof.epoch == epoch
+
+
+def test_cold_open_is_discarded_when_the_epoch_moves_underneath_it(
+ manager: DatabaseManager,
+) -> None:
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ engine = MagicMock()
+
+ def cold_open_racing_a_publisher(*_args, **_kwargs) -> MagicMock:
+ # The sidecar is held by this open; a publisher that bypasses it
+ # (or a durable write this process cannot see coming) moves the
+ # committed epoch while the engine is being built.
+ rekey_lock_module.advance_rekey_epoch(db_path)
+ return engine
+
+ with (
+ patch.object(
+ manager,
+ "_open_user_database_cold",
+ side_effect=cold_open_racing_a_publisher,
+ ),
+ patch.object(manager, "_run_phase2_rekey") as phase2,
+ capture_login_proof() as proof,
+ ):
+ # The password opened the database, so this is not a wrong
+ # password: the open fails closed with the error the login route
+ # answers with 503 (lockout counter untouched), never the None a
+ # wrong password returns (401, counted toward lockout).
+ with pytest.raises(RekeyRecoveryRequiredError):
+ manager.open_user_database("alice", "password")
+
+ # The engine built under the superseded epoch is never published.
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+ phase2.assert_not_called()
+ assert proof.epoch is None
+
+
+def _make_epoch_file_unreadable(db_path: Path) -> None:
+ # A directory where the epoch file should be: read_text raises
+ # IsADirectoryError, an OSError other than FileNotFoundError and not
+ # a PermissionError, so the reader raises RekeyEpochUnreadableError
+ # on the first attempt (the same outcome as EMFILE or EIO).
+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ epoch_file.unlink()
+ epoch_file.mkdir()
+
+
+def _remove_committed_epoch(db_path: Path) -> None:
+ # No committed and no pending epoch: the state is read, and does not
+ # vouch for the epoch the open ran under.
+ db_path.with_name(f"{db_path.name}.rekey.epoch").unlink()
+
+
+ ("disturb", "expected_error"),
+ [
+ pytest.param(
+ _make_epoch_file_unreadable,
+ RekeyEpochUnreadableError,
+ id="epoch-file-unreadable",
+ ),
+ pytest.param(
+ rekey_lock_module.begin_rekey_epoch,
+ RekeyRecoveryRequiredError,
+ id="pending-epoch-appeared",
+ ),
+ pytest.param(
+ _remove_committed_epoch,
+ RekeyRecoveryRequiredError,
+ id="committed-epoch-removed",
+ ),
+ ],
+)
+def test_post_open_epoch_recheck_that_cannot_vouch_fails_closed(
+ manager: DatabaseManager, disturb, expected_error
+) -> None:
+ # The cold open proved the password. If the re-read of the epoch
+ # right after it fails, or finds durable state that does not vouch,
+ # nothing is known about the credential: the open must raise the
+ # RekeyOperationalError the login route answers with 503 (lockout
+ # counter untouched), never return the (None, ...) a wrong password
+ # returns (401, counted toward lockout).
+ db_path = _existing_db(manager, "alice")
+ _initialize_epoch(db_path)
+ engine = MagicMock()
+
+ def cold_open_then_disturb(*_args, **_kwargs) -> MagicMock:
+ disturb(db_path)
+ return engine
+
+ with (
+ patch.object(
+ manager,
+ "_open_user_database_cold",
+ side_effect=cold_open_then_disturb,
+ ),
+ patch.object(manager, "_run_phase2_rekey") as phase2,
+ capture_login_proof() as proof,
+ ):
+ with pytest.raises(expected_error):
+ manager.open_user_database("alice", "password")
+
+ # The engine is still discarded unpublished.
+ engine.dispose.assert_called_once_with()
+ assert "alice" not in manager.connections
+ phase2.assert_not_called()
+ assert proof.epoch is None
+
+
+def test_pending_recovery_verifies_the_candidate_before_finalizing(
+ manager: DatabaseManager,
+) -> None:
+ db_path = _existing_db(manager, "alice")
+ committed = _initialize_epoch(db_path)
+ with acquire_rekey_lock(db_path):
+ pending = rekey_lock_module.begin_rekey_epoch(db_path)
+ connection = MagicMock()
+
+ with (
+ patch.object(
+ manager, "_make_sqlcipher_connection", return_value=connection
+ ),
+ patch.object(
+ manager,
+ "_verify_rekey_candidate",
+ side_effect=_RekeyVerificationError("quick check"),
+ ) as verify,
+ patch.object(
+ manager, "_open_user_database_cold", return_value=MagicMock()
+ ) as cold_open,
+ patch.object(manager, "_run_phase2_rekey"),
+ ):
+ # A password that opens the file but whose candidate key fails
+ # verification must not finalize the pending epoch.
+ with pytest.raises(RekeyRecoveryRequiredError):
+ manager.open_user_database("alice", "password")
+
+ verify.assert_called_once_with(connection, db_path)
+ cold_open.assert_not_called()
+ state = rekey_lock_module.read_rekey_epoch_state(db_path)
+ assert (state.committed_epoch, state.pending_epoch) == (committed, pending)
+
+
+def test_concurrent_first_login_records_the_epoch_of_the_engine_it_joins(
+ manager: DatabaseManager,
+) -> None:
+ # Two first logins race: this one's cache probe missed (nothing cached
+ # yet), then it waited on the per-user init lock while the other login
+ # cold-opened and cached the engine. Inside the lock it finds that
+ # engine, verifies the password against it, and returns it -- the
+ # proof must carry the epoch it verified under, like every other path.
+ epoch = _initialize_epoch(_existing_db(manager, "alice"))
+ engine = MagicMock()
+ real_init_lock = manager._get_init_lock("alice")
+
+ class _OtherLoginFinishesFirst:
+ def acquire(self, *args, **kwargs):
+ acquired = real_init_lock.acquire(*args, **kwargs)
+ if acquired:
+ manager._cache_connection(
+ "alice",
+ manager._make_connection_cache_entry(
+ engine, "password", epoch
+ ),
+ )
+ return acquired
+
+ def release(self):
+ real_init_lock.release()
+
+ with (
+ patch.object(
+ manager, "_get_init_lock", return_value=_OtherLoginFinishesFirst()
+ ),
+ patch.object(manager, "_open_user_database_cold") as cold_open,
+ patch.object(manager, "_run_phase2_rekey") as phase2,
+ capture_login_proof() as proof,
+ ):
+ opened = manager.open_user_database("alice", "password")
+
+ assert opened is engine
+ cold_open.assert_not_called()
+ phase2.assert_not_called()
+ assert (proof.username, proof.epoch) == ("alice", epoch)
+
+
+# The autouse epoch shim relaxes proven_epoch_for for test doubles; these
+# two pin the real method.
[email protected]_durable_epoch
+def test_proven_epoch_for_returns_the_epoch_proved_for_that_user() -> None:
+ proof = LoginProof()
+ proof.username = "alice"
+ proof.epoch = "e1"
+
+ assert proof.proven_epoch_for("alice") == "e1"
+
+
[email protected]_durable_epoch
+ ("recorded_user", "recorded_epoch"),
+ [(None, None), ("mallory", "e1"), ("alice", None)],
+)
+def test_proven_epoch_for_refuses_a_missing_or_foreign_proof(
+ recorded_user: str | None, recorded_epoch: str | None
+) -> None:
+ # A proof for another user (or none at all) must refuse, not quietly
+ # let the session skip the straddle check.
+ proof = LoginProof()
+ proof.username = recorded_user
+ proof.epoch = recorded_epoch
+
+ with pytest.raises(MissingLoginProofError):
+ proof.proven_epoch_for("alice")
diff --git a/tests/database/test_rekey_epoch.py b/tests/database/test_rekey_epoch.py
index 5ad8ee35f9..a812676295 100644
--- a/tests/database/test_rekey_epoch.py
+++ b/tests/database/test_rekey_epoch.py
@@ -17,6 +17,7 @@ from local_deep_research.database.rekey_epoch import (
read_user_authorization_epoch,
)
from local_deep_research.database.rekey_lock import (
+ RekeyEpochUnreadableError,
RekeyRecoveryRequiredError,
acquire_rekey_lock,
read_rekey_epoch,
@@ -162,14 +163,19 @@ def test_authorization_projection_does_not_acquire_sidecar(
def test_user_authorization_epoch_resolves_hashed_database_path(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
- # Given Alice's metadata exists beneath the configured data root
+ # Given Alice's metadata exists beneath the manager's data directory
+ # (pinned explicitly: another module in this worker may have pinned
+ # the singleton elsewhere, and the projection resolves through it)
+ from local_deep_research.database.encrypted_db import db_manager
+
monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
database_dir = tmp_path / "encrypted_databases"
database_dir.mkdir()
+ monkeypatch.setattr(db_manager, "_data_dir_override", database_dir)
db_path = database_dir / get_user_database_filename("alice")
_write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
- # When authorization resolves Alice rather than importing db_manager
+ # When authorization resolves Alice through db_manager
epoch = read_user_authorization_epoch("alice")
# Then it projects the identity from Alice's hashed database path
@@ -219,6 +225,207 @@ def test_projection_cannot_tear_into_the_superseded_epoch(
assert epoch == _EPOCH_TWO
[email protected]("error_number", [errno.EMFILE, errno.EIO])
+def test_epoch_read_os_error_is_typed_unreadable(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ # An I/O failure on the read is distinguishable (for session
+ # validation) but still a RekeyRecoveryRequiredError, so every
+ # existing handler keeps failing closed.
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ real_open = os.open
+
+ def failing_open(path, *args, **kwargs):
+ if Path(path).name.endswith(".rekey.epoch"):
+ raise OSError(error_number, "injected")
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(os, "open", failing_open)
+
+ with pytest.raises(RekeyEpochUnreadableError) as raised:
+ read_authorization_epoch(db_path)
+ assert isinstance(raised.value, RekeyRecoveryRequiredError)
+ assert raised.value.errno == error_number
+ assert read_rekey_epoch(db_path) is None
+
+
+def test_undecodable_epoch_is_not_typed_unreadable(tmp_path: Path) -> None:
+ # Content that was read but cannot be decoded is durable state, not a
+ # transient read fault.
+ db_path = tmp_path / "user.db"
+ db_path.with_name(f"{db_path.name}.rekey.epoch").write_bytes(b"\xff\xfe")
+
+ with pytest.raises(RekeyRecoveryRequiredError) as raised:
+ read_authorization_epoch(db_path)
+ assert not isinstance(raised.value, RekeyEpochUnreadableError)
+
+
+# --- Windows sharing violations between lock-free readers and writers ----
+#
+# CPython opens files on Windows without FILE_SHARE_DELETE. Session
+# validation reads ``.rekey.epoch``/``.rekey.pending`` WITHOUT the sidecar
+# lock on every authenticated request, so a reader can hold the file open
+# while ``change_password`` replaces/unlinks it (the writer then fails with
+# PermissionError, after the physical rekey), and a reader can open a file
+# whose replace/unlink is in flight (the reader gets PermissionError). The
+# window is microseconds; both sides retry a PermissionError a bounded
+# number of times. A Linux CI runner cannot produce the violation, so it is
+# injected at the real call sites.
+
+
+def _sharing_violations(
+ monkeypatch: pytest.MonkeyPatch, target, name: str, failures: int
+) -> list[object]:
+ """Make ``target.name`` raise PermissionError ``failures`` times."""
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ monkeypatch.setattr(
+ rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0, raising=False
+ )
+ real = getattr(target, name)
+ calls: list[object] = []
+
+ def flaky(*args, **kwargs):
+ calls.append(args[0] if args else None)
+ if len(calls) <= failures:
+ raise PermissionError(errno.EACCES, "sharing violation")
+ return real(*args, **kwargs)
+
+ monkeypatch.setattr(target, name, flaky)
+ return calls
+
+
+def test_transient_read_sharing_violation_is_retried(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ failures = 2
+ calls = _sharing_violations(monkeypatch, os, "open", failures)
+
+ # pending (missing) read: `failures` violations then FileNotFoundError;
+ # the epoch read then succeeds on its first attempt.
+ assert read_authorization_epoch(db_path) == _EPOCH_ONE
+ assert len(calls) == failures + 2
+
+
+def test_persistent_read_sharing_violation_fails_closed(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ attempts = rekey_lock_module._SHARING_VIOLATION_ATTEMPTS
+ calls = _sharing_violations(monkeypatch, os, "open", 10**6)
+
+ # Bounded: the reader gives up after `attempts` tries and NEVER yields
+ # an epoch -- the typed unreadable error makes session validation
+ # answer 503 for that request (record kept), and cache validation
+ # treat the epoch as unknown.
+ with pytest.raises(RekeyEpochUnreadableError) as raised:
+ read_authorization_epoch(db_path)
+ assert raised.value.errno == errno.EACCES
+ assert len(calls) == attempts
+ assert read_rekey_epoch(db_path) is None
+
+
+def test_transient_replace_sharing_violation_still_commits(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import os
+
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ with acquire_rekey_lock(db_path):
+ pending = rekey_lock_module.begin_rekey_epoch(db_path)
+ calls = _sharing_violations(monkeypatch, os, "replace", 2)
+ rekey_lock_module.commit_rekey_epoch(db_path, pending)
+
+ assert len(calls) == 3
+ assert rekey_lock_module.read_rekey_epoch_state(
+ db_path
+ ).committed_epoch == (pending)
+ # No temporary token file is left behind by the failed attempts.
+ assert sorted(p.name for p in tmp_path.iterdir()) == [
+ "user.db.rekey.epoch",
+ "user.db.rekey.lock",
+ "user.db.rekey.pending",
+ ]
+
+
+def test_transient_unlink_sharing_violation_still_clears_pending(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ with acquire_rekey_lock(db_path):
+ pending = rekey_lock_module.begin_rekey_epoch(db_path)
+ rekey_lock_module.commit_rekey_epoch(db_path, pending)
+ calls = _sharing_violations(monkeypatch, Path, "unlink", 2)
+ rekey_lock_module.clear_pending_rekey_epoch(db_path, pending)
+
+ assert len(calls) == 3
+ assert read_authorization_epoch(db_path) == pending
+ assert not db_path.with_name("user.db.rekey.pending").exists()
+
+
+def test_transient_unlink_sharing_violation_still_publishes_a_new_epoch(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # publish_new_database_epoch removes a stale pending token left beside
+ # a just-created database; a reader holding it open on Windows must
+ # not fail the creation (#7830). Only the pending token's unlink is
+ # made flaky: the epoch writer's own temp-file cleanup is untouched.
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ monkeypatch.setattr(rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0)
+ db_path = tmp_path / "user.db"
+ pending_path = db_path.with_name("user.db.rekey.pending")
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=_EPOCH_TWO)
+ real_unlink = Path.unlink
+ pending_calls: list[Path] = []
+
+ def flaky_unlink(self: Path, *args, **kwargs):
+ if self == pending_path:
+ pending_calls.append(self)
+ if len(pending_calls) <= 2:
+ raise PermissionError(errno.EACCES, "sharing violation")
+ return real_unlink(self, *args, **kwargs)
+
+ monkeypatch.setattr(Path, "unlink", flaky_unlink)
+ with acquire_rekey_lock(db_path):
+ epoch = rekey_lock_module.publish_new_database_epoch(db_path)
+
+ assert len(pending_calls) == 3
+ assert not pending_path.exists()
+ assert epoch not in (_EPOCH_ONE, _EPOCH_TWO)
+ assert read_authorization_epoch(db_path) == epoch
+
+
+def test_persistent_replace_permission_error_still_propagates(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ import os
+
+ import local_deep_research.database.rekey_lock as rekey_lock_module
+
+ db_path = tmp_path / "user.db"
+ _write_epoch_state(db_path, committed=_EPOCH_ONE, pending=None)
+ with acquire_rekey_lock(db_path):
+ calls = _sharing_violations(monkeypatch, os, "replace", 10**6)
+ with pytest.raises(PermissionError):
+ rekey_lock_module.begin_rekey_epoch(db_path)
+
+ assert len(calls) == rekey_lock_module._SHARING_VIOLATION_ATTEMPTS
+ assert not db_path.with_name("user.db.rekey.pending").exists()
+
+
def test_authorization_epoch_import_contract() -> None:
name = "local_deep_research.database.authorization_epoch"
assert importlib.util.find_spec(name) is not None
diff --git a/tests/database/test_rekey_gate_registry.py b/tests/database/test_rekey_gate_registry.py
index 9a10265ac6..47d939d78d 100644
--- a/tests/database/test_rekey_gate_registry.py
+++ b/tests/database/test_rekey_gate_registry.py
@@ -38,7 +38,6 @@ def test_wrapper_cleanup(tmp_path, monkeypatch, failure):
request = sut._SidecarLockRequest(
db_path=tmp_path / "user.db",
lock_path=path,
- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,
blocking=False,
)
references = []
@@ -89,7 +88,6 @@ def test_waiter_keeps_gate_across_holder_exit_and_new_arrival(
request = sut._SidecarLockRequest(
db_path=tmp_path / "user.db",
lock_path=path,
- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,
blocking=False,
)
looked_up = threading.Event()
@@ -167,7 +165,6 @@ def test_blocking_waiter_and_independent_path_use_real_sidecars(
return sut._SidecarLockRequest(
db_path=tmp_path / "user.db",
lock_path=lock_path,
- mode=sut.AuthorizationLeaseMode.EXCLUSIVE,
blocking=False,
gate_wait=gate_wait,
)
diff --git a/tests/database/test_rekey_lock.py b/tests/database/test_rekey_lock.py
index 9b2f2be752..7f0df51865 100644
--- a/tests/database/test_rekey_lock.py
+++ b/tests/database/test_rekey_lock.py
@@ -44,21 +44,6 @@ def _hold_rekey_lock(
raise RuntimeError("parent did not release rekey-lock worker")
-def _hold_authorization_lease(
- db_path: str,
- mode: str,
- acquired: Event,
- release: Event,
-) -> None:
- lease_mode = rekey_lock_module.AuthorizationLeaseMode(mode)
- with rekey_lock_module.acquire_authorization_lease(
- Path(db_path), lease_mode
- ):
- acquired.set()
- if not release.wait(timeout=10):
- raise _LockWorkerTimeoutError
-
-
def _assert_sidecar_io_error(
error: rekey_lock_module.RekeySidecarIOError,
db_path: Path,
@@ -76,10 +61,6 @@ class _ProtectedBodyError(Exception):
pass
-class _LockWorkerTimeoutError(Exception):
- pass
-
-
def test_rekey_lock_contends_across_processes(tmp_path: Path) -> None:
# Given a separate process holding this database's sidecar lock
db_path = tmp_path / "user.db"
@@ -288,136 +269,6 @@ def test_timeout_bounds_the_wait_for_an_in_process_holder(
assert bound <= elapsed < bound + 5
-def test_authorization_shared_leases_coexist_across_processes(
- tmp_path: Path,
-) -> None:
- # Given another process holds a shared authorization lease
- db_path = tmp_path / "user.db"
- shared = rekey_lock_module.AuthorizationLeaseMode.SHARED
- context = multiprocessing.get_context("spawn")
- acquired = context.Event()
- release = context.Event()
- process = context.Process(
- target=_hold_authorization_lease,
- args=(str(db_path), shared.value, acquired, release),
- )
- parent_acquired = False
- process.start()
-
- try:
- assert acquired.wait(timeout=10), "child did not acquire shared lease"
-
- # When this process requests another nonblocking shared lease
- with rekey_lock_module.acquire_authorization_lease(
- db_path, shared, blocking=False
- ):
- parent_acquired = True
- finally:
- release.set()
- process.join(timeout=10)
- if process.is_alive():
- process.terminate()
- process.join(timeout=5)
-
- # Then both shared leases coexist and the child exits cleanly
- assert parent_acquired is True
- assert process.exitcode == 0
-
-
-def test_authorization_exclusive_contends_with_shared_holder(
- tmp_path: Path,
-) -> None:
- # Given another process holds a shared authorization lease
- db_path = tmp_path / "user.db"
- modes = rekey_lock_module.AuthorizationLeaseMode
- context = multiprocessing.get_context("spawn")
- acquired = context.Event()
- release = context.Event()
- process = context.Process(
- target=_hold_authorization_lease,
- args=(str(db_path), modes.SHARED.value, acquired, release),
- )
- process.start()
-
- try:
- assert acquired.wait(timeout=10), "child did not acquire shared lease"
-
- # When this process requests a nonblocking exclusive lease
- with pytest.raises(RekeyLockContentionError):
- with rekey_lock_module.acquire_authorization_lease(
- db_path, modes.EXCLUSIVE, blocking=False
- ):
- pytest.fail("exclusive lease bypassed shared holder")
- finally:
- release.set()
- process.join(timeout=10)
- if process.is_alive():
- process.terminate()
- process.join(timeout=5)
-
- # Then contention is distinct and the shared holder exits cleanly
- assert process.exitcode == 0
-
-
-def test_authorization_shared_contends_with_exclusive_holder(
- tmp_path: Path,
-) -> None:
- # Given another process holds an exclusive authorization lease
- db_path = tmp_path / "user.db"
- modes = rekey_lock_module.AuthorizationLeaseMode
- context = multiprocessing.get_context("spawn")
- acquired = context.Event()
- release = context.Event()
- process = context.Process(
- target=_hold_authorization_lease,
- args=(str(db_path), modes.EXCLUSIVE.value, acquired, release),
- )
- process.start()
-
- try:
- assert acquired.wait(timeout=10), (
- "child did not acquire exclusive lease"
- )
-
- # When this process requests a nonblocking shared lease
- with pytest.raises(RekeyLockContentionError):
- with rekey_lock_module.acquire_authorization_lease(
- db_path, modes.SHARED, blocking=False
- ):
- pytest.fail("shared lease bypassed exclusive holder")
- finally:
- release.set()
- process.join(timeout=10)
- if process.is_alive():
- process.terminate()
- process.join(timeout=5)
-
- # Then contention is distinct and the exclusive holder exits cleanly
- assert process.exitcode == 0
-
-
-def test_authorization_sidecar_is_persistent_and_owner_only(
- tmp_path: Path,
-) -> None:
- # Given a database path without an authorization sidecar
- db_path = tmp_path / "user.db"
- lock_path = tmp_path / "user.db.rekey.authorization.lock"
- shared = rekey_lock_module.AuthorizationLeaseMode.SHARED
-
- # When the authorization lease is acquired and released twice
- with rekey_lock_module.acquire_authorization_lease(db_path, shared):
- assert lock_path.exists()
- original_inode = lock_path.stat().st_ino
- with rekey_lock_module.acquire_authorization_lease(db_path, shared):
- pass
-
- # Then the same owner-only sidecar remains available
- assert lock_path.exists()
- assert lock_path.stat().st_ino == original_inode
- if os.name != "nt":
- assert stat.S_IMODE(lock_path.stat().st_mode) == 0o600
-
-
def test_rekey_lock_sidecar_is_persistent_and_owner_only(
tmp_path: Path,
) -> None:
@@ -488,11 +339,8 @@ def test_symlinked_rekey_lock_sidecar_is_refused(tmp_path: Path) -> None:
os.name == "nt" or not hasattr(os, "link"),
reason="POSIX hard links required",
)
- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]
-)
def test_hard_linked_sidecar_is_locked_without_touching_its_mode(
- tmp_path: Path, lock_suffix: str
+ tmp_path: Path,
) -> None:
# Given a hard link at the sidecar name whose other name is outside the
# sidecar's directory (a backup or snapshot tool that hard-links the
@@ -505,17 +353,11 @@ def test_hard_linked_sidecar_is_locked_without_touching_its_mode(
target = snapshot_dir / "unrelated.txt"
target.write_text("unrelated\n", encoding="utf-8")
os.chmod(target, 0o644)
- os.link(target, db_path.with_name(f"user.db{lock_suffix}"))
+ os.link(target, db_path.with_name("user.db.rekey.lock"))
- # When the sidecar is opened for the rekey lock or authorization lease
- if lock_suffix == ".rekey.lock":
- acquire = acquire_rekey_lock(db_path)
- else:
- acquire = rekey_lock_module.acquire_authorization_lease(
- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED
- )
+ # When the sidecar is opened for the rekey lock
entered = False
- with acquire:
+ with acquire_rekey_lock(db_path):
entered = True
# Then the lock is taken (no lockout), and the linked file keeps its
@@ -529,11 +371,8 @@ def test_hard_linked_sidecar_is_locked_without_touching_its_mode(
os.name == "nt" or not hasattr(os, "link"),
reason="POSIX hard links required",
)
- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]
-)
def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(
- tmp_path: Path, monkeypatch: pytest.MonkeyPatch, lock_suffix: str
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# Given an external hard link distinct from the real DB, WAL and shm,
# so classification must reach the directory scan to rule out aliases.
@@ -553,7 +392,7 @@ def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(
original_bytes = b"unrelated\n"
target.write_bytes(original_bytes)
target.chmod(0o644)
- sidecar = db_path.with_name(f"user.db{lock_suffix}")
+ sidecar = db_path.with_name("user.db.rekey.lock")
os.link(target, sidecar)
original = target.stat()
identity = (original.st_dev, original.st_ino)
@@ -581,12 +420,9 @@ def test_hard_linked_sidecar_is_refused_unopened_when_directory_scan_fails(
open_attempts.append(os.fspath(path))
return real_open(path, *args, **kwargs)
- if lock_suffix == ".rekey.lock":
- acquire = acquire_rekey_lock(db_path)
- else:
- acquire = rekey_lock_module.acquire_authorization_lease(
- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED
- )
+ # The authorization-lease sidecar main also covered here was removed
+ # with its unused API (#6750); the rekey lock is the only sidecar.
+ acquire = acquire_rekey_lock(db_path)
entered = False
refusal = None
# Only the exact lock parent is unlistable; all opens remain real.
@@ -716,15 +552,12 @@ def test_sidecar_hard_linked_to_the_database_is_refused_unopened(
os.name == "nt" or not hasattr(os, "mkfifo"),
reason="POSIX FIFOs required",
)
- "lock_suffix", [".rekey.lock", ".rekey.authorization.lock"]
-)
def test_non_regular_sidecar_is_refused_and_its_fd_closed(
- tmp_path: Path, monkeypatch: pytest.MonkeyPatch, lock_suffix: str
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# Given a FIFO planted at the sidecar name
db_path = tmp_path / "user.db"
- sidecar = db_path.with_name(f"user.db{lock_suffix}")
+ sidecar = db_path.with_name("user.db.rekey.lock")
os.mkfifo(sidecar, 0o600)
opened_fds: list[int] = []
real_open = os.open
@@ -737,15 +570,9 @@ def test_non_regular_sidecar_is_refused_and_its_fd_closed(
monkeypatch.setattr(rekey_lock_module.os, "open", recording_open)
- # When the sidecar is opened for the rekey lock or authorization lease
- if lock_suffix == ".rekey.lock":
- acquire = acquire_rekey_lock(db_path)
- else:
- acquire = rekey_lock_module.acquire_authorization_lease(
- db_path, rekey_lock_module.AuthorizationLeaseMode.SHARED
- )
+ # When the sidecar is opened for the rekey lock
with pytest.raises(rekey_lock_module.RekeySidecarIOError) as raised:
- with acquire:
+ with acquire_rekey_lock(db_path):
pytest.fail("sidecar lock acquired on a FIFO")
monkeypatch.undo()
@@ -1140,7 +967,6 @@ def test_sidecar_acquire_failure_is_typed(
_fd: int,
_lock_path: Path,
*,
- mode: rekey_lock_module.AuthorizationLeaseMode | None = None,
blocking: bool = False,
) -> None:
raise OSError(errno.EIO, "sensitive raw OS text")
@@ -1365,8 +1191,6 @@ def test_body_error_inside_a_callers_except_block_carries_the_release_note(
class _FakeMsvcrt(ModuleType):
LK_LOCK: ClassVar[int] = 0
LK_NBLCK: ClassVar[int] = 1
- LK_RLCK: ClassVar[int] = 2
- LK_NBRLCK: ClassVar[int] = 3
LK_UNLCK: ClassVar[int] = 4
def __init__(self, acquire_error: OSError | None = None) -> None:
@@ -1376,10 +1200,7 @@ class _FakeMsvcrt(ModuleType):
def locking(self, fd: int, mode: int, size: int) -> None:
self.calls.append((os.lseek(fd, 0, os.SEEK_CUR), mode, size))
- if (
- mode in {self.LK_NBLCK, self.LK_NBRLCK}
- and self.acquire_error is not None
- ):
+ if mode == self.LK_NBLCK and self.acquire_error is not None:
raise self.acquire_error
@@ -1407,30 +1228,27 @@ def test_windows_backend_locks_one_byte_from_start(
]
-def test_windows_backend_selects_shared_and_exclusive_modes(
+def test_windows_backend_selects_blocking_and_nonblocking_modes(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
# Given a persistent sidecar and a Windows locking backend
- lock_path = tmp_path / "user.db.rekey.authorization.lock"
+ lock_path = tmp_path / "user.db.rekey.lock"
fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
backend = _FakeMsvcrt()
monkeypatch.setitem(sys.modules, "msvcrt", backend)
- modes = rekey_lock_module.AuthorizationLeaseMode
try:
- # When shared nonblocking and exclusive blocking leases are acquired
- _acquire_windows_lock(fd, lock_path, mode=modes.SHARED)
+ # When nonblocking and blocking exclusive locks are acquired
+ _acquire_windows_lock(fd, lock_path)
_release_windows_lock(fd)
- _acquire_windows_lock(
- fd, lock_path, mode=modes.EXCLUSIVE, blocking=True
- )
+ _acquire_windows_lock(fd, lock_path, blocking=True)
_release_windows_lock(fd)
finally:
os.close(fd)
- # Then Windows uses read-lock and write-lock modes respectively
+ # Then Windows uses the non-blocking and blocking write-lock modes
assert [call[1] for call in backend.calls] == [
- backend.LK_NBRLCK,
+ backend.LK_NBLCK,
backend.LK_UNLCK,
backend.LK_LOCK,
backend.LK_UNLCK,
diff --git a/tests/security/test_auth_router_credential_frames.py b/tests/security/test_auth_router_credential_frames.py
new file mode 100644
index 0000000000..bb0d2a6983
--- /dev/null
+++ b/tests/security/test_auth_router_credential_frames.py
@@ -0,0 +1,143 @@
+"""No traceback may be rendered from an auth-router frame holding a password.
+
+``login``, ``register``, ``change_password`` and the post-login worker
+(``_perform_post_login_tasks*``) hold the plaintext password in their
+locals; ``_rollback_partial_session`` runs inside ``login`` and
+``register``, and loguru's ``backtrace`` extends a traceback rendered
+from it up into those caller frames.
+A traceback rendered with loguru ``diagnose`` on prints every frame local,
+and whether any sink has ``diagnose`` on is not knowable at the call site
+(``config_logger`` enables it on the stderr sink when debug is on, from
+the ``app.debug`` setting or the environment, and ``LDR_LOGURU_DIAGNOSE``
+is set; loguru's default handler has it on). So these functions log
+exception TYPES only. This source sweep pins that:
+no ``logger.exception``, no ``exc_info=``, no ``logger.opt(exception=...)``
+other than a literal False, and no ``logger.catch`` inside them.
+"""
+
+import ast
+from pathlib import Path
+
+import local_deep_research.web.routers.auth as auth_module
+
+CREDENTIAL_FRAMES = {
+ "login",
+ "register",
+ "change_password",
+ "_perform_post_login_tasks",
+ "_perform_post_login_tasks_body",
+ "_rollback_partial_session",
+}
+
+
+def _traceback_emitters(func: ast.AST) -> list[str]:
+ found = []
+ for node in ast.walk(func):
+ if not isinstance(node, ast.Call):
+ continue
+ for keyword in node.keywords:
+ if keyword.arg == "exc_info":
+ found.append(f"line {node.lineno}: exc_info=")
+ target = node.func
+ if not isinstance(target, ast.Attribute):
+ continue
+ if target.attr in {"exception", "catch"}:
+ found.append(f"line {node.lineno}: .{target.attr}()")
+ if target.attr == "opt":
+ for keyword in node.keywords:
+ if keyword.arg == "exception" and not (
+ isinstance(keyword.value, ast.Constant)
+ and keyword.value.value is False
+ ):
+ found.append(f"line {node.lineno}: opt(exception=...)")
+ return found
+
+
+def test_credential_frames_render_no_tracebacks() -> None:
+ source = Path(auth_module.__file__).read_text(encoding="utf-8")
+ tree = ast.parse(source)
+ functions = {
+ node.name: node
+ for node in ast.walk(tree)
+ if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
+ and node.name in CREDENTIAL_FRAMES
+ }
+
+ # Non-vacuous: every listed frame is still there to be checked.
+ assert set(functions) == CREDENTIAL_FRAMES
+
+ offenders = {
+ name: hits
+ for name, func in functions.items()
+ if (hits := _traceback_emitters(func))
+ }
+ assert offenders == {}, offenders
+
+
+def test_post_login_failure_with_diagnostics_does_not_render_credentials(
+ monkeypatch, real_post_login_tasks
+):
+ """Exercise real Loguru rendering, including a sink with diagnose enabled."""
+ import importlib
+ import inspect
+ from loguru._logger import Core, Logger
+
+ session_module = importlib.import_module(
+ "local_deep_research.web.auth.session_manager"
+ )
+ monkeypatch.setattr(
+ session_module,
+ "read_user_authorization_epoch",
+ lambda username: "test-epoch",
+ )
+ manager = session_module.SessionManager()
+ session_id = manager.create_session("alice")
+ monkeypatch.setattr(auth_module, "session_manager", manager)
+
+ isolated_logger = Logger(
+ core=Core(),
+ exception=None,
+ depth=0,
+ record=False,
+ lazy=False,
+ colors=False,
+ raw=False,
+ capture=True,
+ patchers=[],
+ extra={},
+ )
+ rendered = []
+ records = []
+
+ def capture(message):
+ rendered.append(str(message))
+ records.append(message.record)
+
+ sink = isolated_logger.add(
+ capture, format="{message}", diagnose=True, backtrace=True
+ )
+ password = "auth-password-opaque-canary"
+ upstream_body = "opaque-upstream-credential-body"
+
+ body_calls = []
+
+ def fail_body(username, password, session_id):
+ body_calls.append((username, password, session_id))
+ raise RuntimeError(upstream_body)
+
+ monkeypatch.setattr(auth_module, "logger", isolated_logger)
+ monkeypatch.setattr(
+ auth_module, "_perform_post_login_tasks_body", fail_body
+ )
+ try:
+ worker = inspect.unwrap(real_post_login_tasks)
+ assert worker("alice", password, session_id) is None
+ finally:
+ isolated_logger.remove(sink)
+
+ assert body_calls == [("alice", password, session_id)]
+ text = "".join(rendered)
+ assert "alice" in text and "RuntimeError" in text
+ assert password not in text
+ assert upstream_body not in text
+ assert records and all(record["exception"] is None for record in records)
diff --git a/tests/web/auth/test_idle_cleanup_epoch_unavailable.py b/tests/web/auth/test_idle_cleanup_epoch_unavailable.py
new file mode 100644
index 0000000000..2fa3ba1a9a
--- /dev/null
+++ b/tests/web/auth/test_idle_cleanup_epoch_unavailable.py
@@ -0,0 +1,56 @@
+"""Do not interpret an unreadable epoch as proof that a new login is idle."""
+
+import errno
+from pathlib import Path
+from unittest.mock import Mock
+
+from local_deep_research.database.rekey_lock import RekeyEpochUnreadableError
+from local_deep_research.database.session_passwords import SessionPasswordStore
+from local_deep_research.scheduler import background
+from local_deep_research.web.auth import connection_cleanup
+from local_deep_research.web.auth import session_manager as sm_module
+
+
+def test_unknown_activity_preserves_new_login_and_still_cleans_idle_users(
+ monkeypatch,
+):
+ monkeypatch.setattr(
+ sm_module, "read_user_authorization_epoch", lambda _u: "epoch"
+ )
+ manager = sm_module.SessionManager()
+ store = SessionPasswordStore()
+ store.store_session_password("idle", "retained", "idle-test-key")
+ monkeypatch.setattr(connection_cleanup, "session_password_store", store)
+ monkeypatch.setattr(
+ connection_cleanup, "get_usernames_with_active_research", lambda: set()
+ )
+ monkeypatch.setattr(
+ connection_cleanup, "_pop_per_user_locks", lambda _u: None
+ )
+ monkeypatch.setattr(
+ background,
+ "get_background_job_scheduler",
+ lambda: Mock(is_running=False),
+ )
+ monkeypatch.setattr(connection_cleanup, "_last_dispose_time", float("inf"))
+
+ def unreadable(_username):
+ raise RekeyEpochUnreadableError(Path("review.db"), errno.EMFILE)
+
+ def connected_after_snapshot():
+ # A login completes after the sweep's initial activity snapshot.
+ sid = manager.create_session("alice")
+ store.store_session_password("alice", sid, "active-test-key")
+ monkeypatch.setattr(
+ sm_module, "read_user_authorization_epoch", unreadable
+ )
+ return {"alice", "idle"}
+
+ db = Mock()
+ db.get_connected_usernames.side_effect = connected_after_snapshot
+
+ connection_cleanup.cleanup_idle_connections(manager, db)
+
+ assert store.get_any_session_password("alice") == "active-test-key"
+ assert store.get_any_session_password("idle") is None
+ db.close_user_database.assert_called_once_with("idle")
diff --git a/tests/web/auth/test_session_epoch.py b/tests/web/auth/test_session_epoch.py
new file mode 100644
index 0000000000..8cbce77ae3
--- /dev/null
+++ b/tests/web/auth/test_session_epoch.py
@@ -0,0 +1,647 @@
+"""Durable-epoch fencing of server-side auth sessions.
+
+A session is created at login, right after the password proved itself
+against the epoch-fenced SQLCipher database (PR #6662). The session
+record binds the durable authorization epoch that was live at that
+moment. Every validation re-projects the epoch from disk:
+
+* a session whose bound epoch no longer matches (the password was
+ changed — rekeyed — in ANY worker) is destroyed: its credentials can
+ no longer open the rekeyed database, so the identity it vouches for
+ is dead;
+* a durable state that was read but cannot vouch for the identity
+ (recovery required, a rekey pending) fences the session the same way —
+ fail closed, the record is removed;
+* an epoch file that cannot be READ (EMFILE, EIO, ...) fails the check
+ closed with ``SessionValidationUnavailableError`` but KEEPS the record:
+ a transient fault must not log every active user out;
+* a login whose password was proved against an epoch that is no longer
+ committed by the time the session is created is refused
+ (``StaleLoginProofError``): the proof, not the later projection, is
+ what the session may be bound to.
+
+``require_auth`` (HTTP), ``_enforce_session_revocation``, the socket
+connect gate and the subscribe re-validation all funnel through
+``validate_session``/``validate_session_identity``, so binding the epoch
+here fences each of them. ``has_active_sessions_for`` applies the same
+epoch check (read-only); the other idle sweeps are time-based only.
+"""
+
+import errno
+import os
+from pathlib import Path
+
+import pytest
+
+from local_deep_research.database.rekey_epoch import (
+ read_user_authorization_epoch,
+)
+from local_deep_research.database.rekey_lock import (
+ RekeyRecoveryRequiredError,
+ acquire_rekey_lock,
+ advance_rekey_epoch,
+ read_or_initialize_rekey_epoch,
+)
+from local_deep_research.web.auth import session_manager as sm_module
+from local_deep_research.web.auth.session_manager import (
+ SessionManager,
+ SessionValidationUnavailableError,
+ StaleLoginProofError,
+)
+
+# Opt out of the autouse _durable_epoch_projection shim: these tests
+# exercise the REAL epoch files and failure discipline.
+pytestmark = pytest.mark.real_durable_epoch
+
+
+def data_dir(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+ """An isolated data directory whose user epochs are real files.
+
+ ``db_manager.data_dir`` is pinned to the SAME directory: another
+ module in this xdist worker may have pinned the singleton's override
+ to its own tmp dir, and the epoch projection resolves through the
+ manager -- env and manager must agree, or the projection reads the
+ wrong directory and every assertion below fences against nothing.
+ """
+ from local_deep_research.database.encrypted_db import db_manager
+
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ # Pin through the override attribute so monkeypatch restores the
+ # ORIGINAL override (often None = "resolve from env"), instead of
+ # re-pinning whatever path the getter happened to resolve.
+ (tmp_path / "encrypted_databases").mkdir(parents=True, exist_ok=True)
+ monkeypatch.setattr(
+ db_manager, "_data_dir_override", tmp_path / "encrypted_databases"
+ )
+ return tmp_path
+
+
+def _user_db_path(data_dir: Path, username: str) -> Path:
+ from local_deep_research.config.paths import get_user_database_filename
+
+ return (
+ data_dir / "encrypted_databases" / get_user_database_filename(username)
+ )
+
+
+def _initialize_epoch(data_dir: Path, username: str) -> str:
+ db_path = _user_db_path(data_dir, username)
+ db_path.parent.mkdir(parents=True, exist_ok=True)
+ with acquire_rekey_lock(db_path):
+ return read_or_initialize_rekey_epoch(db_path)
+
+
+def _manager() -> SessionManager:
+ manager = SessionManager()
+ manager.sessions.clear()
+ return manager
+
+
+def test_session_binds_durable_epoch_at_creation(
+ data_dir: Path,
+) -> None:
+ epoch = _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+
+ session_id = manager.create_session("alice")
+
+ record = manager.sessions[session_id]
+ assert record["rekey_epoch"] == epoch
+
+
+def test_stale_epoch_removes_exact_local_session(data_dir: Path) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ stale_session = manager.create_session("alice")
+
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+ fresh_session = manager.create_session("alice")
+
+ # The stale-epoch session is destroyed, not merely denied...
+ assert manager.validate_session(stale_session) is None
+ assert stale_session not in manager.sessions
+ # ...while the session bound to the current epoch survives.
+ assert manager.validate_session(fresh_session) == "alice"
+
+
+def test_validate_session_identity_returns_the_bound_epoch(
+ data_dir: Path,
+) -> None:
+ epoch = _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ identity = manager.validate_session_identity(session_id)
+
+ assert identity is not None
+ assert identity.username == "alice"
+ assert identity.rekey_epoch == epoch
+
+
+def _fail_epoch_reads(
+ monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ """Inject a real OSError at the epoch-file read itself."""
+ real_open = os.open
+
+ def failing_open(path, *args, **kwargs):
+ if Path(path).name.endswith((".rekey.epoch", ".rekey.pending")):
+ raise OSError(error_number, "injected epoch read failure")
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(os, "open", failing_open)
+
+
+ "error_number",
+ [errno.EMFILE, errno.ENFILE, errno.EAGAIN, errno.EIO],
+)
+def test_unreadable_epoch_file_refuses_but_keeps_the_session(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, error_number: int
+) -> None:
+ # A transient I/O error proves nothing about the session: the check
+ # fails closed (raises, never returns the username) but the record
+ # survives, so an fd-exhaustion spike does not log everyone out.
+ epoch = _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+ before = dict(manager.sessions[session_id])
+
+ with monkeypatch.context() as patched:
+ _fail_epoch_reads(patched, error_number)
+ with pytest.raises(SessionValidationUnavailableError) as raised:
+ manager.validate_session(session_id)
+ with pytest.raises(SessionValidationUnavailableError):
+ manager.validate_session_identity(session_id)
+
+ assert raised.value.errno == error_number
+ # Kept, untouched (not even refreshed).
+ assert manager.sessions[session_id] == before
+ assert manager.sessions[session_id]["rekey_epoch"] == epoch
+ # Once the fault clears, the same session validates again.
+ assert manager.validate_session(session_id) == "alice"
+
+
+def test_epoch_advanced_during_a_read_fault_still_removes_after_it(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Keeping the record through a fault is safe only because the next
+ # successful read re-checks it: a password change committed during
+ # the fault still ends the session.
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ with monkeypatch.context() as patched:
+ _fail_epoch_reads(patched, errno.EMFILE)
+ with pytest.raises(SessionValidationUnavailableError):
+ manager.validate_session(session_id)
+ assert session_id in manager.sessions
+
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+
+
+def test_malformed_epoch_file_fences_and_removes(data_dir: Path) -> None:
+ # Read successfully, but the content cannot vouch for anything: that is
+ # durable state, not a transient fault, so the record goes.
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+ db_path = _user_db_path(data_dir, "alice")
+ epoch_path = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ epoch_path.write_text("NOT-A-CANONICAL-UUID", encoding="ascii")
+
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+
+
+def test_recovery_required_state_fences_and_removes(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ def ambiguous_state(_username: str) -> str:
+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))
+
+ monkeypatch.setattr(
+ sm_module,
+ "read_user_authorization_epoch",
+ ambiguous_state,
+ )
+
+ # Ambiguous durable state cannot vouch for ANY identity: fail closed
+ # and remove the record.
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+
+
+def test_current_epoch_projection_matches_durable_file(
+ data_dir: Path,
+) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+
+ assert (
+ read_user_authorization_epoch("alice")
+ != manager.sessions[session_id]["rekey_epoch"]
+ )
+ assert manager.validate_session(session_id) is None
+
+
+def test_create_session_refuses_a_proof_from_a_superseded_epoch(
+ data_dir: Path,
+) -> None:
+ # The login proved its password against E1; a rekey committed E2
+ # (another worker's password change) before the session was created.
+ proven = _initialize_epoch(data_dir, "alice")
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+ manager = _manager()
+
+ with pytest.raises(StaleLoginProofError):
+ manager.create_session("alice", proven_epoch=proven)
+
+ assert manager.sessions == {}
+
+
+def test_create_session_binds_a_current_proof(data_dir: Path) -> None:
+ proven = _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+
+ session_id = manager.create_session("alice", proven_epoch=proven)
+
+ assert manager.sessions[session_id]["rekey_epoch"] == proven
+ assert manager.validate_session(session_id) == "alice"
+
+
+def test_projection_resolves_through_the_managers_pinned_data_dir(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # The env data dir and the manager's pinned override diverge; the
+ # epoch lives only beside the manager's databases, where every writer
+ # puts it. An env-only resolution reads the other directory and fails
+ # closed on a perfectly healthy identity.
+ from local_deep_research.config.paths import get_user_database_filename
+ from local_deep_research.database.encrypted_db import db_manager
+
+ env_dir = tmp_path / "env"
+ pinned_dir = tmp_path / "pinned" / "encrypted_databases"
+ (env_dir / "encrypted_databases").mkdir(parents=True)
+ pinned_dir.mkdir(parents=True)
+ monkeypatch.setenv("LDR_DATA_DIR", str(env_dir))
+ monkeypatch.setattr(db_manager, "_data_dir_override", pinned_dir)
+ db_path = pinned_dir / get_user_database_filename("alice")
+ with acquire_rekey_lock(db_path):
+ epoch = read_or_initialize_rekey_epoch(db_path)
+
+ assert read_user_authorization_epoch("alice") == epoch
+
+
+def test_has_active_sessions_for_ignores_a_stale_epoch_session(
+ data_dir: Path,
+) -> None:
+ # A password change in another worker leaves this worker's record in
+ # place until its next validation; it must not count as active.
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ stale_session = manager.create_session("alice")
+ assert manager.has_active_sessions_for("alice") is True
+
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+
+ assert manager.has_active_sessions_for("alice") is False
+ # Read-only: the stale record is left for validation to remove.
+ assert stale_session in manager.sessions
+
+ # A session bound to the new epoch does count.
+ manager.create_session("alice")
+ assert manager.has_active_sessions_for("alice") is True
+
+
+def test_has_active_sessions_for_is_unavailable_when_the_epoch_is_unreadable(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # An unavailable check refuses authorization but does not prove that
+ # shared resources belong to no remaining session.
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ with monkeypatch.context() as patched:
+ _fail_epoch_reads(patched, errno.EMFILE)
+ with pytest.raises(sm_module.SessionValidationUnavailableError):
+ manager.has_active_sessions_for("alice")
+
+ assert session_id in manager.sessions
+ assert manager.has_active_sessions_for("alice") is True
+
+
+def test_has_active_sessions_for_is_false_on_recovery_required_state(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ manager.create_session("alice")
+
+ def ambiguous_state(_username: str) -> str:
+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))
+
+ monkeypatch.setattr(
+ sm_module, "read_user_authorization_epoch", ambiguous_state
+ )
+
+ assert manager.has_active_sessions_for("alice") is False
+
+
+# --- socket teardown when validation removes the record --------------------
+#
+# A socket is authorised once, at handshake, against a session id recorded
+# in ``socketio_asgi._sid_sessions``; ``emit_to_user`` fans out to every sid
+# in ``_sid_users`` without re-checking the session. When validation
+# REMOVES a record (epoch mismatch after a password change in another
+# worker, unvouchable durable state, inline idle expiry) the record is gone
+# from every later sweep, so validation itself must sever the sockets that
+# handshook on it -- otherwise a socket opened with a stolen cookie keeps
+# receiving the user's events (``settings_changed`` carries plaintext
+# secrets) after the password change that was meant to fence it.
+
+
+_SOCKET_MODULE = "local_deep_research.web.services.socketio_asgi"
+
+
+def severed(monkeypatch: pytest.MonkeyPatch) -> list[str]:
+ """Record every session id handed to ``disconnect_session``."""
+ calls: list[str] = []
+ monkeypatch.setattr(
+ f"{_SOCKET_MODULE}.disconnect_session",
+ lambda session_id: calls.append(session_id),
+ )
+ return calls
+
+
+def test_epoch_mismatch_severs_that_sessions_sockets(
+ data_dir: Path, severed: list[str]
+) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ stale_session = manager.create_session("alice")
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+ fresh_session = manager.create_session("alice")
+
+ assert manager.validate_session(stale_session) is None
+
+ # Only the removed session's sockets go; the new-epoch session (another
+ # tab that logged in after the change) keeps its own.
+ assert severed == [stale_session]
+ assert manager.validate_session(fresh_session) == "alice"
+ assert severed == [stale_session]
+
+
+def test_unvouchable_durable_state_severs_that_sessions_sockets(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]
+) -> None:
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ def ambiguous_state(_username: str) -> str:
+ raise RekeyRecoveryRequiredError(_user_db_path(data_dir, "alice"))
+
+ monkeypatch.setattr(
+ sm_module, "read_user_authorization_epoch", ambiguous_state
+ )
+
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+ assert severed == [session_id]
+
+
+def test_inline_idle_expiry_severs_that_sessions_sockets(
+ data_dir: Path, severed: list[str]
+) -> None:
+ import datetime
+
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice", remember_me=False)
+ manager.sessions[session_id]["last_access"] -= (
+ manager.session_timeout + datetime.timedelta(minutes=1)
+ )
+
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+ assert severed == [session_id]
+
+
+def test_kept_or_unknown_records_sever_nothing(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]
+) -> None:
+ # Negative controls: a valid session, a transient read fault (record
+ # KEPT, 503) and an unknown id must not tear down any socket.
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+
+ assert manager.validate_session(session_id) == "alice"
+ with monkeypatch.context() as patched:
+ _fail_epoch_reads(patched, errno.EMFILE)
+ with pytest.raises(SessionValidationUnavailableError):
+ manager.validate_session(session_id)
+ assert manager.validate_session("no-such-session") is None
+
+ assert severed == []
+ assert session_id in manager.sessions
+
+
+class _CookieRequest:
+ """``end_rejected_session`` only touches ``request.session``."""
+
+ def __init__(self, session: dict) -> None:
+ self.session = dict(session)
+
+
+def test_rejected_session_teardown_survives_an_epoch_read_fault(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch, severed: list[str]
+) -> None:
+ # The cookie is cleared whatever happens, so the record and the sockets
+ # behind it must go too, even while the epoch files cannot be read
+ # (#7830): otherwise a replayed copy of the cleared cookie keeps a live
+ # record that nothing can reach to destroy.
+ from local_deep_research.web.dependencies import auth as auth_dep
+
+ _initialize_epoch(data_dir, "alice")
+ _initialize_epoch(data_dir, "bob")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+ other_tab = manager.create_session("alice")
+ bobs_session = manager.create_session("bob")
+ monkeypatch.setattr(auth_dep, "session_manager", manager)
+ request = _CookieRequest({"username": "alice", "session_id": session_id})
+ foreign = _CookieRequest({"username": "alice", "session_id": bobs_session})
+
+ with monkeypatch.context() as patched:
+ _fail_epoch_reads(patched, errno.EMFILE)
+ # The fault is real: validation cannot answer for this record.
+ with pytest.raises(SessionValidationUnavailableError):
+ manager.validate_session(session_id)
+ auth_dep.end_rejected_session(request)
+ auth_dep.end_rejected_session(foreign)
+
+ assert request.session == {}
+ assert session_id not in manager.sessions
+ assert severed == [session_id]
+ # Scoped to that one record: the user's other tab and another user's
+ # session named by a mismatched cookie are left alone.
+ assert other_tab in manager.sessions
+ assert bobs_session in manager.sessions
+
+
+def test_socket_teardown_failure_never_breaks_validation(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # Best-effort: a socket layer that raises must not turn the removal
+ # into an exception at the HTTP middleware.
+ def broken(_session_id: str) -> None:
+ raise RuntimeError("socket layer down")
+
+ monkeypatch.setattr(f"{_SOCKET_MODULE}.disconnect_session", broken)
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+
+ assert manager.validate_session(session_id) is None
+ assert session_id not in manager.sessions
+
+
+def test_epoch_mismatch_clears_the_sid_maps_on_a_live_loop(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """End to end through the real socket layer.
+
+ ``disconnect_session`` -> ``_disconnect_matching`` runs on a live event
+ loop; the fake ``sio.disconnect`` fires the real ``disconnect`` event
+ handler, as python-socketio does, so the sid maps are cleaned by the
+ same code path a logout uses. The other session's socket stays.
+ """
+ import asyncio
+ import threading
+ from unittest.mock import Mock
+
+ from local_deep_research.web.services import socketio_asgi as sio_mod
+
+ _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ stale_session = manager.create_session("alice")
+ with acquire_rekey_lock(_user_db_path(data_dir, "alice")):
+ advance_rekey_epoch(_user_db_path(data_dir, "alice"))
+ fresh_session = manager.create_session("alice")
+
+ monkeypatch.setattr(sio_mod, "_lock", asyncio.Lock())
+ monkeypatch.setattr(sio_mod, "_subscriptions", {})
+ monkeypatch.setattr(
+ sio_mod, "_sid_users", {"sid-stolen": "alice", "sid-fresh": "alice"}
+ )
+ monkeypatch.setattr(
+ sio_mod,
+ "_sid_sessions",
+ {"sid-stolen": stale_session, "sid-fresh": fresh_session},
+ )
+
+ loop = asyncio.new_event_loop()
+ thread = threading.Thread(target=loop.run_forever, daemon=True)
+ thread.start()
+ disconnected: list[str] = []
+
+ async def fake_disconnect(sid: str) -> None:
+ disconnected.append(sid)
+ await sio_mod.disconnect(sid)
+
+ fake_sio = Mock()
+ fake_sio.disconnect = fake_disconnect
+ futures = []
+ real_schedule = asyncio.run_coroutine_threadsafe
+
+ def capture(coro, target_loop):
+ future = real_schedule(coro, target_loop)
+ futures.append(future)
+ return future
+
+ try:
+ with monkeypatch.context() as patched:
+ patched.setattr(sio_mod, "_get_main_loop", lambda: loop)
+ patched.setattr(sio_mod, "sio", fake_sio)
+ patched.setattr(asyncio, "run_coroutine_threadsafe", capture)
+
+ assert manager.validate_session(stale_session) is None
+ assert futures, "validation scheduled no socket teardown"
+ for future in futures:
+ future.result(timeout=5)
+ finally:
+ loop.call_soon_threadsafe(loop.stop)
+ thread.join(timeout=5)
+ loop.close()
+
+ assert disconnected == ["sid-stolen"]
+ assert sio_mod._sid_users == {"sid-fresh": "alice"}
+ assert sio_mod._sid_sessions == {"sid-fresh": fresh_session}
+
+
+def test_epoch_read_sharing_violation_is_retried_then_fails_closed(
+ data_dir: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ """A Windows sharing violation on the lock-free epoch read.
+
+ A writer's ``os.replace``/``unlink`` in flight makes the reader's open
+ fail with PermissionError for a few microseconds. A transient one is
+ retried and the session validates; a persistent one fails closed
+ (503 for this request, the record KEPT, never a validated identity).
+ """
+ from local_deep_research.database import rekey_lock as rekey_lock_module
+
+ epoch = _initialize_epoch(data_dir, "alice")
+ manager = _manager()
+ session_id = manager.create_session("alice")
+ monkeypatch.setattr(
+ rekey_lock_module, "_SHARING_VIOLATION_DELAY", 0, raising=False
+ )
+ real_open = os.open
+ budget = {"failures": 2}
+
+ def sharing_violation(path, *args, **kwargs):
+ if (
+ Path(path).name.endswith((".rekey.epoch", ".rekey.pending"))
+ and budget["failures"] > 0
+ ):
+ budget["failures"] -= 1
+ raise PermissionError(errno.EACCES, "sharing violation")
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(os, "open", sharing_violation)
+
+ identity = manager.validate_session_identity(session_id)
+ assert identity is not None and identity.rekey_epoch == epoch
+ assert budget["failures"] == 0
+
+ budget["failures"] = 10**6
+ before = dict(manager.sessions[session_id])
+ with pytest.raises(SessionValidationUnavailableError) as raised:
+ manager.validate_session_identity(session_id)
+ assert raised.value.errno == errno.EACCES
+ assert manager.sessions[session_id] == before
diff --git a/tests/web/auth/test_session_validation_unavailable.py b/tests/web/auth/test_session_validation_unavailable.py
new file mode 100644
index 0000000000..93e1be659e
--- /dev/null
+++ b/tests/web/auth/test_session_validation_unavailable.py
@@ -0,0 +1,524 @@
+"""A transient epoch-read failure refuses the request but keeps the session.
+
+``validate_session`` raises ``SessionValidationUnavailableError`` when the
+durable epoch files cannot be read at all (EMFILE, EIO, ...). Every caller
+must fail THIS request closed without clearing the session or its cookie:
+clearing it would log out every active user during an fd-exhaustion spike,
+including on static-file requests (``DatabaseMiddleware`` validates on every
+request). Only durable state that was read and does not vouch for the
+session ends it; that is covered in ``test_session_epoch.py``.
+
+Logout is the one exception: it only destroys, so ``POST /auth/logout``
+goes through unvalidated (no database open), and the route tears down the
+user's credentials only for a cookie whose record still binds it to that
+user.
+
+Also pins the per-request memo: ``DatabaseMiddleware`` records the session
+it validated in ``scope["state"]`` and ``require_auth`` reuses it instead of
+re-projecting the epoch a second time in the same request.
+
+Marked ``real_session_check`` so the autouse ``_legacy_bare_username_auth``
+shim does not replace ``_server_session_valid``.
+"""
+
+import asyncio
+from unittest.mock import AsyncMock, MagicMock
+
+import pytest
+from fastapi import HTTPException
+from starlette.requests import Request
+
+from local_deep_research.web.auth.session_manager import (
+ UNVALIDATED_LOGOUT_STATE_KEY,
+ VALIDATED_SESSION_STATE_KEY,
+ SessionManager,
+ SessionValidationUnavailableError,
+)
+from local_deep_research.web.dependencies import auth as auth_dep
+from local_deep_research.web.services import socketio_asgi as sio_mod
+
+pytestmark = pytest.mark.real_session_check
+
+USER = "transient_user"
+SID = "session-id-1"
+
+
+def _unavailable(*_args, **_kwargs):
+ raise SessionValidationUnavailableError(USER, 24) # EMFILE
+
+
+def fake_session_manager(monkeypatch):
+ """Replace the singleton's validate_session where each caller looks it up.
+
+ Only ``validate_session`` is replaced: the singleton's ``sessions`` dict
+ is left as it is, and no record for ``SID`` is seeded. A test that
+ reaches the cookie-clearing teardown (``_revoke_server_session``) must
+ seed one itself (``_seed_server_record``): the teardown looks the owner
+ up with ``session_owner``, which reads the record directly, not through
+ ``validate_session``, and tears down nothing for an id it cannot find.
+ """
+ from local_deep_research.web.auth import session_manager as sm_module
+
+ manager = MagicMock()
+ monkeypatch.setattr(
+ sm_module.session_manager, "validate_session", manager.validate_session
+ )
+ return manager
+
+
+def _seed_server_record(monkeypatch, username: str = USER) -> None:
+ """Give the real singleton a live record for SID (restored afterwards)."""
+ import datetime
+
+ from local_deep_research.web.auth import session_manager as sm_module
+
+ now = datetime.datetime.now(datetime.UTC)
+ monkeypatch.setitem(
+ sm_module.session_manager.sessions,
+ SID,
+ {
+ "username": username,
+ "created_at": now,
+ "last_access": now,
+ "remember_me": False,
+ "rekey_epoch": "test-epoch",
+ },
+ )
+
+
+def _request(session: dict, state: dict | None = None) -> Request:
+ scope = {
+ "type": "http",
+ "method": "GET",
+ "path": "/api/whatever",
+ "headers": [],
+ "session": session,
+ }
+ if state is not None:
+ scope["state"] = state
+ return Request(scope)
+
+
+# --- _enforce_session_revocation / DatabaseMiddleware ----------------------
+
+
+def test_revocation_check_raises_and_leaves_the_session(fake_session_manager):
+ from local_deep_research.web.fastapi_app import (
+ _enforce_session_revocation,
+ )
+
+ fake_session_manager.validate_session.side_effect = _unavailable
+ session = {"username": USER, "session_id": SID}
+ state: dict = {}
+
+ with pytest.raises(SessionValidationUnavailableError):
+ _enforce_session_revocation(session, state)
+
+ assert session == {"username": USER, "session_id": SID}
+ assert VALIDATED_SESSION_STATE_KEY not in state
+
+
+def test_revocation_check_records_the_validated_session(fake_session_manager):
+ from local_deep_research.web.fastapi_app import (
+ _enforce_session_revocation,
+ )
+
+ fake_session_manager.validate_session.return_value = USER
+ state: dict = {}
+
+ _enforce_session_revocation({"username": USER, "session_id": SID}, state)
+
+ assert state[VALIDATED_SESSION_STATE_KEY] == (SID, USER)
+
+
+def test_revoked_session_is_cleared_and_not_recorded(fake_session_manager):
+ from local_deep_research.web.fastapi_app import (
+ _enforce_session_revocation,
+ )
+
+ fake_session_manager.validate_session.return_value = None
+ session = {"username": USER, "session_id": SID}
+ state: dict = {}
+
+ _enforce_session_revocation(session, state)
+
+ assert session == {}
+ assert VALIDATED_SESSION_STATE_KEY not in state
+
+
[email protected]("path", ["/static/css/app.css", "/", "/auth/check"])
+def test_middleware_answers_503_without_touching_the_session(
+ fake_session_manager, path
+):
+ from local_deep_research.web.fastapi_app import DatabaseMiddleware
+
+ fake_session_manager.validate_session.side_effect = _unavailable
+ inner = AsyncMock()
+ middleware = DatabaseMiddleware(inner)
+ session = {"username": USER, "session_id": SID}
+ scope = {
+ "type": "http",
+ "method": "GET",
+ "path": path,
+ "headers": [],
+ "session": session,
+ }
+ sent: list[dict] = []
+
+ async def send(message):
+ sent.append(message)
+
+ async def receive():
+ return {"type": "http.request", "body": b"", "more_body": False}
+
+ asyncio.run(middleware(scope, receive, send))
+
+ inner.assert_not_called()
+ assert sent[0]["type"] == "http.response.start"
+ assert sent[0]["status"] == 503
+ # Not a logout: the session is exactly as it came in.
+ assert session == {"username": USER, "session_id": SID}
+
+
+def _run_middleware(fake_session_manager, monkeypatch, method, path):
+ from local_deep_research.web import fastapi_app
+ from local_deep_research.web.dependencies import auth as auth_module
+
+ fake_session_manager.validate_session.side_effect = _unavailable
+ ensure = MagicMock()
+ monkeypatch.setattr(auth_module, "ensure_user_database", ensure)
+ inner = AsyncMock()
+ middleware = fastapi_app.DatabaseMiddleware(inner)
+ session = {"username": USER, "session_id": SID}
+ scope = {
+ "type": "http",
+ "method": method,
+ "path": path,
+ "headers": [],
+ "session": session,
+ }
+ sent: list[dict] = []
+
+ async def send(message):
+ sent.append(message)
+
+ async def receive():
+ return {"type": "http.request", "body": b"", "more_body": False}
+
+ asyncio.run(middleware(scope, receive, send))
+ return inner, ensure, scope, session, sent
+
+
+def test_middleware_lets_logout_through_unvalidated(
+ fake_session_manager, monkeypatch
+):
+ inner, ensure, scope, session, sent = _run_middleware(
+ fake_session_manager, monkeypatch, "POST", "/auth/logout"
+ )
+
+ inner.assert_called_once()
+ assert sent == []
+ # Nothing is opened (or stored) for the unvalidated cookie, and it is
+ # not recorded as validated for require_auth.
+ ensure.assert_not_called()
+ assert scope["state"][UNVALIDATED_LOGOUT_STATE_KEY] is True
+ assert VALIDATED_SESSION_STATE_KEY not in scope["state"]
+ assert session == {"username": USER, "session_id": SID}
+
+
+ ("method", "path"),
+ [
+ ("GET", "/auth/logout"),
+ ("POST", "/auth/logout/"),
+ ("POST", "/auth/logoutx"),
+ ("POST", "/auth/change-password"),
+ ],
+)
+def test_only_post_logout_bypasses_the_503(
+ fake_session_manager, monkeypatch, method, path
+):
+ inner, ensure, scope, session, sent = _run_middleware(
+ fake_session_manager, monkeypatch, method, path
+ )
+
+ inner.assert_not_called()
+ ensure.assert_not_called()
+ assert sent[0]["status"] == 503
+ assert UNVALIDATED_LOGOUT_STATE_KEY not in scope["state"]
+
+
+# --- logout during a fault ------------------------------------------------
+
+
+def test_session_belongs_to_only_reads_the_record():
+ manager = SessionManager()
+ manager.sessions[SID] = {"username": USER}
+
+ assert manager.session_belongs_to(SID, USER) is True
+ assert manager.session_belongs_to(SID, "other-user") is False
+ assert manager.session_belongs_to("other-session", USER) is False
+ assert manager.session_belongs_to(None, USER) is False
+ assert manager.session_belongs_to("", USER) is False
+
+
+def logout_env(monkeypatch):
+ from local_deep_research.database import session_passwords
+ from local_deep_research.database import thread_local_session
+ from local_deep_research.scheduler import background
+ from local_deep_research.web.routers import auth as auth_router
+
+ manager = SessionManager()
+ monkeypatch.setattr(auth_router, "session_manager", manager)
+ env = MagicMock()
+ env.manager = manager
+ monkeypatch.setattr(auth_router, "db_manager", env.db_manager)
+ monkeypatch.setattr(
+ auth_router, "_disconnect_session_sockets", env.disconnect_sockets
+ )
+ monkeypatch.setattr(
+ session_passwords.session_password_store,
+ "clear_all_for_user",
+ env.clear_passwords,
+ )
+ monkeypatch.setattr(
+ thread_local_session, "clear_user_credentials", env.clear_credentials
+ )
+ monkeypatch.setattr(
+ background, "get_background_job_scheduler", env.get_scheduler
+ )
+ env.logout = auth_router.logout
+ return env
+
+
+def _logout_request(session: dict) -> Request:
+ return Request(
+ {
+ "type": "http",
+ "method": "POST",
+ "path": "/auth/logout",
+ "headers": [],
+ "session": session,
+ "state": {UNVALIDATED_LOGOUT_STATE_KEY: True},
+ }
+ )
+
+
+def test_unvalidated_logout_revokes_the_callers_own_session(logout_env):
+ logout_env.manager.sessions[SID] = {"username": USER}
+ other = "session-id-2"
+ logout_env.manager.sessions[other] = {"username": "someone-else"}
+ request = _logout_request({"username": USER, "session_id": SID})
+
+ response = logout_env.logout(request)
+
+ assert response.status_code == 302
+ assert SID not in logout_env.manager.sessions
+ assert other in logout_env.manager.sessions
+ logout_env.clear_passwords.assert_called_once_with(USER)
+ logout_env.clear_credentials.assert_called_once_with(USER)
+ logout_env.disconnect_sockets.assert_called_once_with(SID)
+ assert "username" not in request.session
+ assert "session_id" not in request.session
+
+
+ "record", [None, {"username": "someone-else"}], ids=["gone", "foreign"]
+)
+def test_unvalidated_logout_with_an_unbound_cookie_only_drops_the_cookie(
+ logout_env, record
+):
+ # A cookie whose record is already gone (or not this user's) must not
+ # reach the user-wide teardown while nothing validated it: it only
+ # loses itself, as the validating middleware would have done.
+ if record is not None:
+ logout_env.manager.sessions[SID] = record
+ request = _logout_request({"username": USER, "session_id": SID})
+
+ response = logout_env.logout(request)
+
+ assert response.status_code == 302
+ assert request.session == {}
+ if record is not None:
+ assert logout_env.manager.sessions[SID] == record
+ logout_env.clear_passwords.assert_not_called()
+ logout_env.clear_credentials.assert_not_called()
+ logout_env.disconnect_sockets.assert_not_called()
+ logout_env.get_scheduler.assert_not_called()
+ logout_env.db_manager.close_user_database.assert_not_called()
+
+
+# --- require_auth ---------------------------------------------------------
+
+
+def connected(monkeypatch):
+ db = MagicMock()
+ db.is_user_connected.return_value = True
+ monkeypatch.setattr(auth_dep, "db_manager", db)
+ return db
+
+
+def test_require_auth_answers_503_and_keeps_the_session(
+ fake_session_manager, connected
+):
+ fake_session_manager.validate_session.side_effect = _unavailable
+ request = _request({"username": USER, "session_id": SID})
+
+ with pytest.raises(HTTPException) as raised:
+ auth_dep.require_auth(request)
+
+ assert raised.value.status_code == 503
+ assert request.session == {"username": USER, "session_id": SID}
+
+
+def test_require_auth_still_rejects_an_invalid_session(
+ fake_session_manager, connected
+):
+ fake_session_manager.validate_session.return_value = None
+ request = _request({"username": USER, "session_id": SID})
+
+ with pytest.raises(HTTPException) as raised:
+ auth_dep.require_auth(request)
+
+ assert raised.value.status_code == 401
+ assert request.session == {}
+
+
+def test_require_auth_reuses_the_middleware_validation(
+ fake_session_manager, connected, monkeypatch
+):
+ monkeypatch.setattr(
+ auth_dep.session_password_store,
+ "get_session_password",
+ lambda username, session_id: "available-credential",
+ )
+ request = _request(
+ {"username": USER, "session_id": SID},
+ state={VALIDATED_SESSION_STATE_KEY: (SID, USER)},
+ )
+
+ assert auth_dep.require_auth(request) == USER
+ fake_session_manager.validate_session.assert_not_called()
+
+
+def test_validation_memo_does_not_preserve_a_lost_credential(
+ fake_session_manager, connected, monkeypatch
+):
+ from local_deep_research.database.session_passwords import (
+ SessionPasswordStore,
+ )
+
+ store = SessionPasswordStore()
+ store.store_session_password(USER, SID, "available-credential")
+ monkeypatch.setattr(auth_dep, "session_password_store", store)
+ connected.has_encryption = True
+ fake_session_manager.validate_session.return_value = USER
+ _seed_server_record(monkeypatch)
+ destroy = MagicMock()
+ disconnect = MagicMock()
+ monkeypatch.setattr(auth_dep.session_manager, "destroy_session", destroy)
+ monkeypatch.setattr(sio_mod, "disconnect_session", disconnect)
+ request = _request(
+ {"username": USER, "session_id": SID},
+ state={VALIDATED_SESSION_STATE_KEY: (SID, USER)},
+ )
+
+ assert auth_dep.require_auth(request) == USER
+ fake_session_manager.validate_session.assert_not_called()
+
+ # Losing the credential after middleware validation must still revoke
+ # this session, even though its user's database remains connected.
+ store.clear_session(USER, SID)
+ with pytest.raises(HTTPException) as raised:
+ auth_dep.require_auth(request)
+
+ assert raised.value.status_code == 401
+ assert request.session == {}
+ destroy.assert_called_once_with(SID)
+ disconnect.assert_called_once_with(SID)
+
+
+ "memo", [("other-session", USER), (SID, "other-user"), None]
+)
+def test_require_auth_validates_when_the_memo_does_not_match(
+ fake_session_manager, connected, memo
+):
+ fake_session_manager.validate_session.return_value = None
+ state = {} if memo is None else {VALIDATED_SESSION_STATE_KEY: memo}
+ request = _request({"username": USER, "session_id": SID}, state=state)
+
+ with pytest.raises(HTTPException) as raised:
+ auth_dep.require_auth(request)
+
+ assert raised.value.status_code == 401
+ fake_session_manager.validate_session.assert_called_once_with(SID)
+
+
+# --- socket gates ---------------------------------------------------------
+
+
+def socket_state(monkeypatch):
+ monkeypatch.setattr(sio_mod, "_sid_users", {"sock-1": USER})
+ monkeypatch.setattr(sio_mod, "_sid_sessions", {"sock-1": SID})
+ monkeypatch.setattr(sio_mod, "_subscriptions", {})
+ monkeypatch.setattr(sio_mod, "_main_loop", sio_mod._main_loop)
+ monkeypatch.setattr(sio_mod, "_lock", None)
+ sio_mod.init_lock()
+
+
+def test_socket_revalidation_reports_unknown_not_invalid(
+ fake_session_manager, socket_state
+):
+ fake_session_manager.validate_session.side_effect = _unavailable
+
+ result = asyncio.run(sio_mod._socket_session_still_valid("sock-1", USER))
+
+ assert result == (SID, None)
+
+
+def test_subscribe_is_refused_without_severing_the_session(
+ fake_session_manager, socket_state, monkeypatch
+):
+ fake_session_manager.validate_session.side_effect = _unavailable
+ emit = AsyncMock()
+ disconnect = AsyncMock()
+ monkeypatch.setattr(sio_mod.sio, "emit", emit)
+ monkeypatch.setattr(sio_mod.sio, "disconnect", disconnect)
+ severed = MagicMock()
+ monkeypatch.setattr(sio_mod, "disconnect_session", severed)
+ owns = AsyncMock(return_value=True)
+ monkeypatch.setattr(sio_mod, "_user_owns_research", owns)
+
+ asyncio.run(sio_mod.on_subscribe("sock-1", {"research_id": "r-1"}))
+
+ severed.assert_not_called()
+ disconnect.assert_not_called()
+ owns.assert_not_called()
+ assert sio_mod._subscriptions == {}
+ event, payload = emit.call_args.args[:2]
+ assert event == "subscribe_error"
+ assert payload["error"] == "Temporarily unavailable"
+ assert sio_mod._sid_sessions == {"sock-1": SID}
+
+
+def test_connect_is_refused_when_the_session_cannot_be_validated(
+ fake_session_manager, socket_state, monkeypatch
+):
+ fake_session_manager.validate_session.side_effect = _unavailable
+ monkeypatch.setattr(
+ sio_mod,
+ "_decode_session_cookie",
+ lambda _cookie: {"username": USER, "session_id": SID},
+ )
+
+ accepted = asyncio.run(
+ sio_mod.connect("sock-2", {"HTTP_COOKIE": "session=whatever"})
+ )
+
+ assert accepted is False
+ assert "sock-2" not in sio_mod._sid_users
diff --git a/tests/web/dependencies/test_dependency_resolution_contracts.py b/tests/web/dependencies/test_dependency_resolution_contracts.py
index 8294b32d85..71efb29f61 100644
--- a/tests/web/dependencies/test_dependency_resolution_contracts.py
+++ b/tests/web/dependencies/test_dependency_resolution_contracts.py
@@ -1049,6 +1049,10 @@ class _DeadSessionManager:
def validate_session(session_id): # noqa: ARG004
return None
+ @staticmethod
+ def session_owner(session_id): # noqa: ARG004
+ return None
+
@pytest.mark.real_session_check
class TestUsernameReadersDiverge:
diff --git a/tests/web/dependencies/test_session_revocation.py b/tests/web/dependencies/test_session_revocation.py
index 6f5205eb93..bdd3a097d8 100644
--- a/tests/web/dependencies/test_session_revocation.py
+++ b/tests/web/dependencies/test_session_revocation.py
@@ -65,6 +65,9 @@ class _FakeSessionManager:
def validate_session(self, session_id):
return self._sessions.get(session_id)
+ def session_owner(self, session_id):
+ return self._sessions.get(session_id)
+
@pytest.fixture
def connected_db(monkeypatch):
diff --git a/tests/web/dependencies/test_stale_session_cleanup.py b/tests/web/dependencies/test_stale_session_cleanup.py
index 189e458e35..997b61a9c2 100644
--- a/tests/web/dependencies/test_stale_session_cleanup.py
+++ b/tests/web/dependencies/test_stale_session_cleanup.py
@@ -83,6 +83,9 @@ class _FakeSessionManager:
return None
return {"s1": "alice", "s-bob": "bob"}.get(session_id)
+ def session_owner(self, session_id):
+ return self.validate_session(session_id)
+
def destroy_session(self, session_id):
self.destroyed.append(session_id)
diff --git a/tests/web/routers/test_login_rekey_straddle.py b/tests/web/routers/test_login_rekey_straddle.py
new file mode 100644
index 0000000000..a7f70db5d8
--- /dev/null
+++ b/tests/web/routers/test_login_rekey_straddle.py
@@ -0,0 +1,384 @@
+"""Session creation at login is fenced by the epoch the login proved.
+
+Worker A's login proves the OLD password against the durable epoch E1
+(``open_user_database`` returns an engine). Before A creates the session,
+worker B's ``change_password`` rekeys the database and commits E2. The
+session must be bound to what A PROVED (E1), not to whatever epoch is
+committed by the time ``create_session`` runs (E2): bound to E2 it would
+survive the password change it straddled, and the old password would
+keep a live session.
+
+The interleaving is forced deterministically: the real
+``open_user_database`` runs, then a second ``DatabaseManager`` on the
+same data directory (worker B) performs a real SQLCipher rekey before
+the route continues to ``create_session``.
+
+A login whose epoch cannot be read at session creation or right after
+its cold open verified the password, whose committed epoch moved under
+that cold open, or whose open recorded no epoch proof for the user,
+answers 503 rather than issuing a
+session it cannot fence. None of these refusals touches the lockout
+counter: it is not incremented (not a guessing failure) and
+not reset (no session was issued).
+"""
+
+import uuid
+
+import pytest
+from fastapi.testclient import TestClient
+
+pytest.importorskip("sqlcipher3", reason="requires SQLCipher (encrypted mode)")
+
+# Real durable epochs end to end: the autouse projection shim is off.
+pytestmark = pytest.mark.real_durable_epoch
+
+
+def app(tmp_path, monkeypatch):
+ """The real app on a throwaway data directory with test KDF rounds."""
+ monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
+ # Production PBKDF2 rounds (256000) would dominate wall-clock: every
+ # test here derives several keys. sqlcipher_utils allows low rounds
+ # under pytest (MIN_KDF_ITERATIONS_TESTING).
+ monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")
+
+ from local_deep_research.database.auth_db import init_auth_database
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.web.fastapi_app import app as fastapi_app
+
+ databases = tmp_path / "encrypted_databases"
+ databases.mkdir(parents=True, exist_ok=True)
+ # Pinned through monkeypatch so the ORIGINAL override is restored.
+ monkeypatch.setattr(db_manager, "_data_dir_override", databases)
+ init_auth_database()
+ monkeypatch.setattr(
+ "local_deep_research.web.routers.auth._perform_post_login_tasks",
+ lambda *_a, **_k: None,
+ )
+ try:
+ yield fastapi_app
+ finally:
+ db_manager.close_all_databases()
+
+
+def _client(app) -> TestClient:
+ client = TestClient(app, raise_server_exceptions=False)
+ ip = f"10.{uuid.uuid4().int % 254 + 1}.{uuid.uuid4().int % 254 + 1}.9"
+ client.headers.update({"X-Forwarded-For": ip})
+ return client
+
+
+def _csrf(client: TestClient) -> str:
+ client.get("/auth/login")
+ return client.get("/auth/csrf-token").json().get("csrf_token", "")
+
+
+def _register_and_logout(app, user: str, pw: str) -> None:
+ client = _client(app)
+ resp = client.post(
+ "/auth/register",
+ data={
+ "username": user,
+ "password": pw,
+ "confirm_password": pw,
+ "acknowledge": "true",
+ "csrf_token": _csrf(client),
+ },
+ follow_redirects=False,
+ )
+ if resp.status_code != 302:
+ pytest.fail(f"registration returned {resp.status_code}")
+ token = client.get("/auth/csrf-token").json().get("csrf_token", "")
+ client.post(
+ "/auth/logout", headers={"X-CSRFToken": token}, follow_redirects=False
+ )
+
+
+def _login(app, user: str, pw: str):
+ client = _client(app)
+ return client.post(
+ "/auth/login",
+ data={"username": user, "password": pw, "csrf_token": _csrf(client)},
+ follow_redirects=False,
+ )
+
+
+def _user_sessions(username: str) -> list[str]:
+ from local_deep_research.web.auth.session_manager import session_manager
+
+ with session_manager._lock:
+ return [
+ sid
+ for sid, data in session_manager.sessions.items()
+ if data["username"] == username
+ ]
+
+
+def _failure_count(username: str) -> int:
+ from local_deep_research.security.account_lockout import (
+ get_account_lockout_manager,
+ )
+
+ return (
+ get_account_lockout_manager()._state.get(username, {}).get("count", 0)
+ )
+
+
+def _one_prior_failure(username: str) -> None:
+ from local_deep_research.security.account_lockout import (
+ get_account_lockout_manager,
+ )
+
+ get_account_lockout_manager().record_failure(username)
+ assert _failure_count(username) == 1
+
+
+def test_old_password_login_straddling_a_rekey_gets_no_session(app):
+ from local_deep_research.database.encrypted_db import (
+ DatabaseManager,
+ db_manager,
+ )
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"straddle_{uuid.uuid4().hex[:8]}"
+ old_pw = "StraddleOld-Password1!" # noqa: S105
+ new_pw = "StraddleNew-Password2!" # noqa: S105
+ _register_and_logout(app, user, old_pw)
+ _one_prior_failure(user)
+
+ # Worker B: a separate manager over the same data directory.
+ worker_b = DatabaseManager()
+ worker_b.data_dir = db_manager.data_dir
+ real_open = db_manager.open_user_database
+ rekeyed = []
+
+ def open_then_rekey_elsewhere(username, password):
+ engine = real_open(username, password)
+ if engine is not None and not rekeyed:
+ rekeyed.append(worker_b.change_password(username, old_pw, new_pw))
+ return engine
+
+ try:
+ with pytest.MonkeyPatch.context() as mp:
+ mp.setattr(
+ db_manager, "open_user_database", open_then_rekey_elsewhere
+ )
+ resp = _login(app, user, old_pw)
+ finally:
+ worker_b.close_all_databases()
+
+ assert rekeyed == [True], "worker B's rekey did not commit"
+ # The old password proved E1; E2 is committed now. No session.
+ assert resp.status_code == 401, (
+ f"old-password login straddling a rekey answered {resp.status_code}"
+ )
+ assert _user_sessions(user) == []
+ # Refused before any session was issued: the earlier failure is
+ # neither cleared (record_success) nor added to.
+ assert _failure_count(user) == 1
+
+ # The new password still logs in normally, and that clears the count.
+ assert _login(app, user, new_pw).status_code == 302
+ assert _failure_count(user) == 0
+
+
+def test_unreadable_epoch_at_session_creation_answers_503(app, monkeypatch):
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.rekey_lock import (
+ RekeyRecoveryRequiredError,
+ )
+ from local_deep_research.web.auth import session_manager as sm_module
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"epoch503_{uuid.uuid4().hex[:8]}"
+ pw = "Epoch503-Password1!" # noqa: S105
+ _register_and_logout(app, user, pw)
+ _one_prior_failure(user)
+ db_path = db_manager._get_user_db_path(user)
+
+ reads = []
+
+ def unreadable(username: str) -> str:
+ reads.append(username)
+ raise RekeyRecoveryRequiredError(db_path)
+
+ # The open proves the password; only the session-creation read fails.
+ monkeypatch.setattr(sm_module, "read_user_authorization_epoch", unreadable)
+ resp = _login(app, user, pw)
+
+ # Exactly one epoch read, and it was create_session's: the 503 comes
+ # from the session-creation path, not from some other reader.
+ assert reads == [user]
+ assert resp.status_code == 503
+ assert _user_sessions(user) == []
+ assert _failure_count(user) == 1
+
+
+def test_login_without_a_proof_for_the_user_answers_503(app, monkeypatch):
+ # Every engine-returning open records a proof today; if one ever does
+ # not (or records it for another username), the login must refuse
+ # rather than create a session with the straddle check skipped.
+ from local_deep_research.database import encrypted_db
+ from local_deep_research.database.encrypted_db import db_manager
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"noproof_{uuid.uuid4().hex[:8]}"
+ pw = "NoProof-Password1!" # noqa: S105
+ _register_and_logout(app, user, pw)
+ _one_prior_failure(user)
+ real_open = db_manager.open_user_database
+
+ def open_recording_a_foreign_proof(username, password):
+ engine = real_open(username, password)
+ encrypted_db._record_login_proof("someone-else", "not-this-epoch")
+ return engine
+
+ monkeypatch.setattr(
+ db_manager, "open_user_database", open_recording_a_foreign_proof
+ )
+ resp = _login(app, user, pw)
+
+ assert resp.status_code == 503
+ assert _user_sessions(user) == []
+ # Not a guessing failure, and no session was issued.
+ assert _failure_count(user) == 1
+
+
+def test_unreadable_epoch_after_a_correct_cold_open_answers_503(
+ app, monkeypatch
+):
+ # The cold open verified the CORRECT password; the epoch re-read right
+ # after it then fails (EISDIR here; EMFILE/EIO behave the same). That
+ # proves nothing about the credential, so the login answers 503 and
+ # leaves the lockout counter alone instead of reporting "invalid
+ # credentials" (401) and counting a failure.
+ from local_deep_research.database.encrypted_db import db_manager
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"recheck_{uuid.uuid4().hex[:8]}"
+ pw = "Recheck-Password1!" # noqa: S105
+ _register_and_logout(app, user, pw)
+ _one_prior_failure(user)
+ # Force the cold-open path (no cached engine to claim).
+ db_manager.close_user_database(user)
+ db_path = db_manager._get_user_db_path(user)
+ epoch_file = db_path.with_name(f"{db_path.name}.rekey.epoch")
+ real_cold_open = db_manager._open_user_database_cold
+ cold_opens = []
+
+ def cold_open_then_epoch_unreadable(*args, **kwargs):
+ engine = real_cold_open(*args, **kwargs)
+ cold_opens.append(engine is not None)
+ epoch_file.unlink()
+ epoch_file.mkdir()
+ return engine
+
+ monkeypatch.setattr(
+ db_manager, "_open_user_database_cold", cold_open_then_epoch_unreadable
+ )
+ resp = _login(app, user, pw)
+
+ assert cold_opens == [True], "the correct password did not cold-open"
+ assert resp.status_code == 503
+ assert _user_sessions(user) == []
+ assert _failure_count(user) == 1
+ assert user not in db_manager.connections
+
+
+def test_epoch_moved_under_a_correct_cold_open_answers_503(app, monkeypatch):
+ # The cold open verified the CORRECT password; the committed epoch is
+ # then found to have moved (only a writer that bypassed the rekey
+ # sidecar this open holds can do that). The password opened the
+ # database, so it is not a guessing failure: the login answers 503 and
+ # leaves the lockout counter alone instead of reporting "invalid
+ # credentials" (401) and counting a failure.
+ from local_deep_research.database import rekey_lock
+ from local_deep_research.database.encrypted_db import db_manager
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"moved_{uuid.uuid4().hex[:8]}"
+ pw = "Moved-Password1!" # noqa: S105
+ _register_and_logout(app, user, pw)
+ _one_prior_failure(user)
+ # Force the cold-open path (no cached engine to claim).
+ db_manager.close_user_database(user)
+ db_path = db_manager._get_user_db_path(user)
+ real_cold_open = db_manager._open_user_database_cold
+ cold_opens = []
+
+ def cold_open_then_epoch_moves(*args, **kwargs):
+ engine = real_cold_open(*args, **kwargs)
+ cold_opens.append(engine is not None)
+ rekey_lock.advance_rekey_epoch(db_path)
+ return engine
+
+ monkeypatch.setattr(
+ db_manager, "_open_user_database_cold", cold_open_then_epoch_moves
+ )
+ resp = _login(app, user, pw)
+
+ assert cold_opens == [True], "the correct password did not cold-open"
+ assert resp.status_code == 503
+ assert _user_sessions(user) == []
+ assert _failure_count(user) == 1
+ assert user not in db_manager.connections
+
+
+def test_unreadable_epoch_at_registration_auto_login_answers_503(
+ app, monkeypatch
+):
+ # Registration auto-logs in through the same create_session, so an
+ # epoch that cannot be read there gets login's retryable 503, not a
+ # 500 (#7830). The account it just created stays: once the fault
+ # clears, the same credentials sign in.
+ import errno
+
+ from local_deep_research.database.encrypted_db import db_manager
+ from local_deep_research.database.rekey_lock import (
+ RekeyEpochUnreadableError,
+ )
+ from local_deep_research.web.auth import session_manager as sm_module
+
+ if not db_manager.has_encryption:
+ pytest.skip("requires SQLCipher (encrypted mode)")
+
+ user = f"reg503_{uuid.uuid4().hex[:8]}"
+ pw = "Register503-Password1!" # noqa: S105
+ reads = []
+
+ def unreadable(username: str) -> str:
+ reads.append(username)
+ raise RekeyEpochUnreadableError(
+ db_manager._get_user_db_path(username), errno.EMFILE
+ )
+
+ client = _client(app)
+ with pytest.MonkeyPatch.context() as mp:
+ mp.setattr(sm_module, "read_user_authorization_epoch", unreadable)
+ resp = client.post(
+ "/auth/register",
+ data={
+ "username": user,
+ "password": pw,
+ "confirm_password": pw,
+ "acknowledge": "true",
+ "csrf_token": _csrf(client),
+ },
+ follow_redirects=False,
+ )
+
+ # The refusal came from create_session's epoch read.
+ assert reads == [user]
+ assert resp.status_code == 503
+ assert _user_sessions(user) == []
+ assert _login(app, user, pw).status_code == 302
diff --git a/tests/web/test_route_table_parity.py b/tests/web/test_route_table_parity.py
index e83c5002db..cf582b5433 100644
--- a/tests/web/test_route_table_parity.py
+++ b/tests/web/test_route_table_parity.py
@@ -1730,6 +1730,10 @@ EXPECTED_STATUS_CODES_GAINED = {
# revokes every session and redirects to login like a success instead.
# web/routers/auth.py:change_password.
("POST", "/auth/change-password"): [503],
+ # Registration's auto-login answers 503 when the new account's session
+ # epoch cannot be read (RekeyOperationalError from create_session), as
+ # login does; the account itself stays created. auth.py:register.
+ ("POST", "/auth/register"): [503],
# Intentional: an unimplemented endpoint answers 501 rather than letting
# a bare `except Exception` turn NotImplementedException into a 500.
("GET", "/news/api/categories"): [501],
diff --git a/tests/web/test_thread_excepthook.py b/tests/web/test_thread_excepthook.py
index eed6d103cf..e39b9319d2 100644
--- a/tests/web/test_thread_excepthook.py
+++ b/tests/web/test_thread_excepthook.py
@@ -138,6 +138,14 @@ def test_perform_post_login_tasks_catches_outer_exceptions(
(for example from inside a `with` context manager's __enter__) —
not inside a per-step try/except — the wrapper must log the
exception and not let the daemon thread die silently.
+
+ The crash log is a loguru brace-format call with positional
+ arguments (``"... user {} ({})", sanitize_for_log(username),
+ type(err).__name__``), not an f-string, so ``call_args[0][0]`` is
+ only the template. The assertions render the template with its
+ arguments the way loguru does (``str.format``) instead of reading
+ the bare first argument, which no longer carries the username or
+ the exception type.
"""
from local_deep_research.web.routers import auth as routes
@@ -152,10 +160,20 @@ def test_perform_post_login_tasks_catches_outer_exceptions(
# Must not raise.
real_post_login_tasks("alice", "pw", "sess-1")
- assert mock_logger.exception.called
- msg = mock_logger.exception.call_args[0][0]
+ # Logged as a WARNING with the exception TYPE only: this frame
+ # holds the plaintext password, so no traceback may be rendered
+ # from it (no logger.exception, no exc_info, no opt(exception=)).
+ mock_logger.exception.assert_not_called()
+ mock_logger.opt.assert_not_called()
+ assert mock_logger.warning.called
+ args = mock_logger.warning.call_args.args
+ assert not mock_logger.warning.call_args.kwargs
+ msg = args[0].format(*args[1:])
assert "alice" in msg
assert "crashed" in msg.lower() or "post-login" in msg.lower()
+ assert "RuntimeError" in msg
+ assert "outer failure" not in msg
+ assert all("outer failure" not in str(arg) for arg in args)
def test_perform_post_login_tasks_body_runs_when_no_outer_error(
@@ -176,3 +194,4 @@ def test_perform_post_login_tasks_body_runs_when_no_outer_error(
body.assert_called_once_with("bob", "pw", "sess-1")
mock_logger.exception.assert_not_called()
+ mock_logger.warning.assert_not_called()