~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Warm-cache wrong-password storm racing close_user_database + reopen.
Incident context (PR #5596): the cached-connection bypass existed because a
cache hit returned the engine with no credential check. The fix's moving
parts that THIS file stresses together, at real-SQLCipher level:
* the verifier's fail-closed liveness under a wrong-password storm (every
attempt must fall through to a key-validating cold open and fail);
* the atomicity of engine+verifier publication (``_cache_connection``) and
eviction (``close_user_database``) while another thread continuously
closes and reopens with the CORRECT password;
* that a close storm never turns into an availability hole or a trust hole
for the legitimate credential.
``test_login_cached_connection_password_extra.py`` pins the single-threaded
shapes (wrong password leaves the legit cache intact; concurrent FIRST open
with wrong/right). ``test_login_cached_connection_password.py`` pins the
warm-cache rejection itself. Neither drives the interleaving where the
cache is being torn down and rebuilt WHILE the attacker hammers it -- the
state in which a non-atomic publication or a verifier/engine mismatch
would be observable.
The invariant sampled throughout (under ``_connections_lock``, so each
snapshot is atomic): the set of usernames in ``connections`` always equals
the set in ``_password_verifiers``. A cached engine must never exist
without its verifier (fail-closed would save it, but it is still a broken
invariant) and a verifier must never outlive its engine (a stale verifier
is exactly what the change_password fix's eviction closes).
"""
import threading
import time
import uuid
import warnings
import pytest
from sqlalchemy import text
from local_deep_research.database.encrypted_db import DatabaseManager
# Module-level gate. Without a working SQLCipher build every test below
# is a no-op, and the dozens of in-body ``pytest.skip`` calls let a CI
# lane report this whole battery green having executed nothing. Skipping
# at import time makes the missing dependency one visible, uniform signal
# per module instead.
pytest.importorskip("sqlcipher3", reason="requires SQLCipher (encrypted mode)")
# Bounded concurrency: 2 explicit threads, explicit iteration caps, one
# barrier, wall-clock budget comfortably under 30s.
HAMMER_ITERATIONS = 12
JOIN_TIMEOUT_S = 60
BARRIER_TIMEOUT_S = 30
# Backoff before the single retry of a LEGITIMATE operation. A transient
# SQLITE_BUSY under storm contention is not the property under test, and
# reddening on it turns a loaded runner into a security-gate failure. A
# repeatable refusal still fails on the retry. Never applied to the
# attacker's wrong-password opens -- those must fail first time, every time.
RETRY_BACKOFF_S = 0.1
def _read_canary(engine):
"""Read the canary through ``engine``; return ``(note, error)``."""
try:
with engine.connect() as conn:
return (
conn.execute(
text("SELECT note FROM gap_c_canary WHERE id = 1")
).scalar(),
None,
)
except Exception as exc: # noqa: BLE001 - retried once, then reported
return None, exc
@pytest.fixture
def manager(tmp_path, monkeypatch):
"""A DatabaseManager writing to an isolated data directory.
Same self-contained pattern as the incident tests; KDF iterations are
lowered via the sanctioned test-mode knob (see tests/conftest.py's
``app`` fixture) because each wrong-password attempt performs a full
key-derivation + verify cycle.
"""
monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")
mgr = DatabaseManager()
mgr.data_dir = tmp_path
yield mgr
mgr.close_all_databases()
def test_wrong_password_storm_racing_close_and_reopen(manager):
"""T1 (attacker) hammers open(wrong) against whatever cache state exists;
T2 (owner) hammers close_user_database + open(correct) + SELECT.
Invariants:
* T1 NEVER receives an engine -- not on the warm cache, not mid-close,
not mid-reopen, not on the cold path.
* T2's retry-once availability policy holds: a working engine whose
SELECT reads the canary row is available within one retry (verifier
liveness + atomic publication); a repeatable refusal still fails.
* Every lock-consistent snapshot during the storm shows engines and
verifiers exactly paired for this user.
* Quiesced: closing leaves NEITHER an engine NOR a verifier; the next
wrong open still fails and the next correct open still works.
"""
if not manager.has_encryption:
pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")
username = f"storm_{uuid.uuid4().hex[:8]}"
password = "StormCorrectHorse1!" # noqa: S105
engine = manager.create_user_database(username, password)
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE gap_c_canary (id INTEGER PRIMARY KEY, note TEXT)"
)
)
conn.execute(
text("INSERT INTO gap_c_canary (id, note) VALUES (1, :note)"),
{"note": "gap-C"},
)
barrier = threading.Barrier(2)
errors = {}
attacker_violations = [] # every case where open(wrong) returned something
owner_failures = [] # every case where open(correct) failed or mis-read
pairing_violations = [] # engine/verifier snapshots that disagree
owner_retries = [] # every iteration where the retry-once path fired
def attacker():
try:
barrier.wait(timeout=BARRIER_TIMEOUT_S)
for i in range(HAMMER_ITERATIONS):
engine = manager.open_user_database(
username, f"attacker-guess-{i}-wrong"
)
if engine is not None:
attacker_violations.append(
f"iteration {i}: open(wrong) returned {engine!r}"
)
except Exception as exc: # noqa: BLE001 - surfaced via `errors` below
errors["attacker"] = exc
def owner():
try:
barrier.wait(timeout=BARRIER_TIMEOUT_S)
for i in range(HAMMER_ITERATIONS):
engine = manager.open_user_database(username, password)
if engine is None:
# Retry once (see RETRY_BACKOFF_S).
owner_retries.append(f"iteration {i}: open(correct) retry")
time.sleep(RETRY_BACKOFF_S)
engine = manager.open_user_database(username, password)
if engine is None:
owner_failures.append(
f"iteration {i}: open(correct) returned None twice"
)
continue
note, read_error = _read_canary(engine)
if read_error is not None:
# Retry once (see RETRY_BACKOFF_S).
owner_retries.append(f"iteration {i}: canary read retry")
time.sleep(RETRY_BACKOFF_S)
note, read_error = _read_canary(engine)
if read_error is not None:
owner_failures.append(
f"iteration {i}: SELECT on returned engine failed "
f"twice: {read_error!r}"
)
elif note != "gap-C":
owner_failures.append(
f"iteration {i}: canary read {note!r}"
)
manager.close_user_database(username)
# Lock-consistent snapshot: engines and verifiers must be
# exactly paired at every observable instant.
with manager._connections_lock:
engines = set(manager.connections)
verifiers = set(manager._password_verifiers)
if engines != verifiers:
pairing_violations.append(
f"iteration {i}: engines={engines} "
f"verifiers={verifiers}"
)
except Exception as exc: # noqa: BLE001 - surfaced via `errors` below
errors["owner"] = exc
threads = [
threading.Thread(target=attacker, name="gap-c-attacker"),
threading.Thread(target=owner, name="gap-c-owner"),
]
for t in threads:
t.start()
for t in threads:
t.join(timeout=JOIN_TIMEOUT_S)
assert not any(t.is_alive() for t in threads), "storm deadlocked"
assert not errors, f"a storm thread raised: {errors}"
assert not attacker_violations, (
"wrong-password opens succeeded during the close/reopen storm: "
f"{attacker_violations}"
)
assert not owner_failures, (
f"legitimate opens failed during the storm: {owner_failures}"
)
assert not pairing_violations, (
f"engine/verifier pairing broke during the storm: {pairing_violations}"
)
# Attempt witness: a run where the owner never needed its retry proves
# the retry-once policy was present but never exercised under this
# storm -- not that it works. Record it as a warning that survives
# into the run report rather than passing silently.
if not owner_retries:
warnings.warn(
"wrong-password storm retry-once policy not witnessed: the "
"owner thread never needed its single retry against the "
f"attacker's {HAMMER_ITERATIONS} concurrent wrong-password "
"attempts, so this run exercised no availability contention.",
stacklevel=2,
)
# Quiesced state: the owner's last action was a close, so neither an
# engine nor a verifier may remain (no verifier without an engine, no
# engine without a verifier).
with manager._connections_lock:
assert username not in manager.connections, (
"engine survived the final close"
)
assert username not in manager._password_verifiers, (
"verifier survived the final close -- a stale verifier must "
"never outlive the engine it described"
)
# Recovery canary: the storm left the manager healthy.
assert manager.open_user_database(username, "still-wrong") is None
recovered = manager.open_user_database(username, password)
assert recovered is not None
with recovered.connect() as conn:
assert (
conn.execute(
text("SELECT note FROM gap_c_canary WHERE id = 1")
).scalar()
== "gap-C"
)
assert manager._password_matches_cached(username, password) is True