Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

10.5 KB
Raw Blame
"""Warm-cache wrong-password storm racing close_user_database + reopen.

Incident context (PR #5596): the cached-connection bypass existed because a
cache hit returned the engine with no credential check. The fix's moving
parts that THIS file stresses together, at real-SQLCipher level:

* the verifier's fail-closed liveness under a wrong-password storm (every
  attempt must fall through to a key-validating cold open and fail);
* the atomicity of engine+verifier publication (``_cache_connection``) and
  eviction (``close_user_database``) while another thread continuously
  closes and reopens with the CORRECT password;
* that a close storm never turns into an availability hole or a trust hole
  for the legitimate credential.

``test_login_cached_connection_password_extra.py`` pins the single-threaded
shapes (wrong password leaves the legit cache intact; concurrent FIRST open
with wrong/right). ``test_login_cached_connection_password.py`` pins the
warm-cache rejection itself. Neither drives the interleaving where the
cache is being torn down and rebuilt WHILE the attacker hammers it -- the
state in which a non-atomic publication or a verifier/engine mismatch
would be observable.

The invariant sampled throughout (under ``_connections_lock``, so each
snapshot is atomic): the set of usernames in ``connections`` always equals
the set in ``_password_verifiers``. A cached engine must never exist
without its verifier (fail-closed would save it, but it is still a broken
invariant) and a verifier must never outlive its engine (a stale verifier
is exactly what the change_password fix's eviction closes).
"""

import threading
import time
import uuid
import warnings

import pytest
from sqlalchemy import text

from local_deep_research.database.encrypted_db import DatabaseManager

# Module-level gate. Without a working SQLCipher build every test below
# is a no-op, and the dozens of in-body ``pytest.skip`` calls let a CI
# lane report this whole battery green having executed nothing. Skipping
# at import time makes the missing dependency one visible, uniform signal
# per module instead.
pytest.importorskip("sqlcipher3", reason="requires SQLCipher (encrypted mode)")

# Bounded concurrency: 2 explicit threads, explicit iteration caps, one
# barrier, wall-clock budget comfortably under 30s.
HAMMER_ITERATIONS = 12
JOIN_TIMEOUT_S = 60
BARRIER_TIMEOUT_S = 30
# Backoff before the single retry of a LEGITIMATE operation. A transient
# SQLITE_BUSY under storm contention is not the property under test, and
# reddening on it turns a loaded runner into a security-gate failure. A
# repeatable refusal still fails on the retry. Never applied to the
# attacker's wrong-password opens -- those must fail first time, every time.
RETRY_BACKOFF_S = 0.1


def _read_canary(engine):
    """Read the canary through ``engine``; return ``(note, error)``."""
    try:
        with engine.connect() as conn:
            return (
                conn.execute(
                    text("SELECT note FROM gap_c_canary WHERE id = 1")
                ).scalar(),
                None,
            )
    except Exception as exc:  # noqa: BLE001 - retried once, then reported
        return None, exc


@pytest.fixture
def manager(tmp_path, monkeypatch):
    """A DatabaseManager writing to an isolated data directory.

    Same self-contained pattern as the incident tests; KDF iterations are
    lowered via the sanctioned test-mode knob (see tests/conftest.py's
    ``app`` fixture) because each wrong-password attempt performs a full
    key-derivation + verify cycle.
    """
    monkeypatch.setenv("LDR_DATA_DIR", str(tmp_path))
    monkeypatch.setenv("LDR_DB_CONFIG_KDF_ITERATIONS", "1000")
    mgr = DatabaseManager()
    mgr.data_dir = tmp_path
    yield mgr
    mgr.close_all_databases()


def test_wrong_password_storm_racing_close_and_reopen(manager):
    """T1 (attacker) hammers open(wrong) against whatever cache state exists;
    T2 (owner) hammers close_user_database + open(correct) + SELECT.

    Invariants:
    * T1 NEVER receives an engine -- not on the warm cache, not mid-close,
      not mid-reopen, not on the cold path.
    * T2's retry-once availability policy holds: a working engine whose
      SELECT reads the canary row is available within one retry (verifier
      liveness + atomic publication); a repeatable refusal still fails.
    * Every lock-consistent snapshot during the storm shows engines and
      verifiers exactly paired for this user.
    * Quiesced: closing leaves NEITHER an engine NOR a verifier; the next
      wrong open still fails and the next correct open still works.
    """
    if not manager.has_encryption:
        pytest.skip("requires SQLCipher (encrypted mode) to be meaningful")

    username = f"storm_{uuid.uuid4().hex[:8]}"
    password = "StormCorrectHorse1!"  # noqa: S105
    engine = manager.create_user_database(username, password)
    with engine.begin() as conn:
        conn.execute(
            text(
                "CREATE TABLE gap_c_canary (id INTEGER PRIMARY KEY, note TEXT)"
            )
        )
        conn.execute(
            text("INSERT INTO gap_c_canary (id, note) VALUES (1, :note)"),
            {"note": "gap-C"},
        )

    barrier = threading.Barrier(2)
    errors = {}
    attacker_violations = []  # every case where open(wrong) returned something
    owner_failures = []  # every case where open(correct) failed or mis-read
    pairing_violations = []  # engine/verifier snapshots that disagree
    owner_retries = []  # every iteration where the retry-once path fired

    def attacker():
        try:
            barrier.wait(timeout=BARRIER_TIMEOUT_S)
            for i in range(HAMMER_ITERATIONS):
                engine = manager.open_user_database(
                    username, f"attacker-guess-{i}-wrong"
                )
                if engine is not None:
                    attacker_violations.append(
                        f"iteration {i}: open(wrong) returned {engine!r}"
                    )
        except Exception as exc:  # noqa: BLE001 - surfaced via `errors` below
            errors["attacker"] = exc

    def owner():
        try:
            barrier.wait(timeout=BARRIER_TIMEOUT_S)
            for i in range(HAMMER_ITERATIONS):
                engine = manager.open_user_database(username, password)
                if engine is None:
                    # Retry once (see RETRY_BACKOFF_S).
                    owner_retries.append(f"iteration {i}: open(correct) retry")
                    time.sleep(RETRY_BACKOFF_S)
                    engine = manager.open_user_database(username, password)
                if engine is None:
                    owner_failures.append(
                        f"iteration {i}: open(correct) returned None twice"
                    )
                    continue
                note, read_error = _read_canary(engine)
                if read_error is not None:
                    # Retry once (see RETRY_BACKOFF_S).
                    owner_retries.append(f"iteration {i}: canary read retry")
                    time.sleep(RETRY_BACKOFF_S)
                    note, read_error = _read_canary(engine)
                if read_error is not None:
                    owner_failures.append(
                        f"iteration {i}: SELECT on returned engine failed "
                        f"twice: {read_error!r}"
                    )
                elif note != "gap-C":
                    owner_failures.append(
                        f"iteration {i}: canary read {note!r}"
                    )
                manager.close_user_database(username)
                # Lock-consistent snapshot: engines and verifiers must be
                # exactly paired at every observable instant.
                with manager._connections_lock:
                    engines = set(manager.connections)
                    verifiers = set(manager._password_verifiers)
                if engines != verifiers:
                    pairing_violations.append(
                        f"iteration {i}: engines={engines} "
                        f"verifiers={verifiers}"
                    )
        except Exception as exc:  # noqa: BLE001 - surfaced via `errors` below
            errors["owner"] = exc

    threads = [
        threading.Thread(target=attacker, name="gap-c-attacker"),
        threading.Thread(target=owner, name="gap-c-owner"),
    ]
    for t in threads:
        t.start()
    for t in threads:
        t.join(timeout=JOIN_TIMEOUT_S)
    assert not any(t.is_alive() for t in threads), "storm deadlocked"
    assert not errors, f"a storm thread raised: {errors}"

    assert not attacker_violations, (
        "wrong-password opens succeeded during the close/reopen storm: "
        f"{attacker_violations}"
    )
    assert not owner_failures, (
        f"legitimate opens failed during the storm: {owner_failures}"
    )
    assert not pairing_violations, (
        f"engine/verifier pairing broke during the storm: {pairing_violations}"
    )

    # Attempt witness: a run where the owner never needed its retry proves
    # the retry-once policy was present but never exercised under this
    # storm -- not that it works. Record it as a warning that survives
    # into the run report rather than passing silently.
    if not owner_retries:
        warnings.warn(
            "wrong-password storm retry-once policy not witnessed: the "
            "owner thread never needed its single retry against the "
            f"attacker's {HAMMER_ITERATIONS} concurrent wrong-password "
            "attempts, so this run exercised no availability contention.",
            stacklevel=2,
        )

    # Quiesced state: the owner's last action was a close, so neither an
    # engine nor a verifier may remain (no verifier without an engine, no
    # engine without a verifier).
    with manager._connections_lock:
        assert username not in manager.connections, (
            "engine survived the final close"
        )
        assert username not in manager._password_verifiers, (
            "verifier survived the final close -- a stale verifier must "
            "never outlive the engine it described"
        )

    # Recovery canary: the storm left the manager healthy.
    assert manager.open_user_database(username, "still-wrong") is None
    recovered = manager.open_user_database(username, password)
    assert recovered is not None
    with recovered.connect() as conn:
        assert (
            conn.execute(
                text("SELECT note FROM gap_c_canary WHERE id = 1")
            ).scalar()
            == "gap-C"
        )
    assert manager._password_matches_cached(username, password) is True