Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

5.7 KB
Raw Blame
import threading
import uuid
from typing import Any, Callable, Hashable, Tuple

from cachetools import cached, keys

g_thread_local_store = threading.local()


def thread_specific_cache(*args: Any, **kwargs: Any) -> Callable:
    """
    A version of `cached()` that is local to a single thread. In other words,
    cache entries will only be valid in the thread where they were created.

    WHY THIS IS STILL THREAD-KEYED UNDER FastAPI, AND NOT A `ContextVar`.

    This survived the Flask -> FastAPI migration on purpose; it is not a
    leftover. The two Flask helpers that used to live in this module
    (`thread_with_app_context`, `thread_context`) were deleted in that
    migration because they wrapped `flask.g` / `AppContext`, and identity and
    research-context propagation genuinely did move to `ContextVar` — see
    `utilities/request_context.py` and `utilities/thread_context.py`. This
    decorator is a different case, because what it caches is not context but a
    THREAD-AFFINE RESOURCE.

    Its only production caller is `db_utils._get_cached_user_session`, which
    caches a SQLAlchemy `Session`. The post-auth session accessor on
    `db_manager` builds a fresh `sessionmaker(bind=engine)()` on every call and
    does nothing to scope it
    per thread, so this key is the only thing keeping one cached `Session` from
    being handed to two anyio threadpool workers running concurrently for the
    same user. A `Session` is documented as not thread-safe: concurrent
    identity-map mutation and autoflush on one instance is the failure this
    prevents.

    A `ContextVar`-keyed cache would NOT be equivalent, and would be unsafe
    here. A single request context can straddle two worker threads: Starlette
    drives sync-generator dependencies through `contextmanager_in_threadpool`,
    which dispatches `__enter__` and `__exit__` as two separate
    `anyio.to_thread.run_sync` calls, and anyio selects a worker via
    `idle_workers.pop()` with no task affinity. (This hazard is written up at
    `web/dependencies/auth.py::get_db_session_dep`, and handled deliberately by
    the streaming generators in `web/routers/library.py`.) Keying on context
    would therefore return ONE `Session` to a context spread across two
    threads — reintroducing exactly the cross-thread sharing that keying on
    the thread prevents. `ContextVar` is the right primitive for propagating
    identity; `threading.local()` is the right primitive for a resource whose
    safety is defined per OS thread. Do not swap one for the other without
    also giving `Session` acquisition a different lifetime model.

    WHAT THE THREAD KEY DOES *NOT* DO. It provides no cross-user isolation.
    That comes from `username` being part of `keys.hashkey(*args_, **kwargs_)`
    below, i.e. from applying this decorator BELOW username resolution
    (`_get_cached_user_session(username, _namespace)`) rather than above it.
    See the note on the lock further down, and
    `tests/security/test_cross_user_isolation_invariants.py`, which pins both
    properties separately. Conflating the two is how the key-completeness bug
    described below got misdiagnosed for weeks.

    Args:
        *args: Will be forwarded to `cached()`.
        **kwargs: Will be forwarded to `cached()`.

    Returns:
        The wrapped function.

    """

    def _key_func(*args_: Any, **kwargs_: Any) -> Tuple[Hashable, ...]:
        base_hash = keys.hashkey(*args_, **kwargs_)

        if hasattr(g_thread_local_store, "thread_id"):
            # We already gave this thread a unique ID. Use that.
            thread_id = g_thread_local_store.thread_id
        else:
            # Give this thread a new unique ID.
            thread_id = uuid.uuid4().hex
            g_thread_local_store.thread_id = thread_id

        return (thread_id,) + base_hash

    # cachetools' `cached()` is NOT thread-safe unless a lock is supplied.
    # The cache object is shared by every calling thread, and concurrent
    # __setitem__/eviction can desync an LRUCache's internal ordering from
    # its data dict, after which `popitem()` raises KeyError. With the
    # per-thread key below the key space is (thread x args), so a small
    # maxsize under many worker threads means near-constant eviction —
    # precisely the condition that provokes it. Measured: without this lock,
    # ~432k KeyErrors across 480k concurrent calls; with it, zero.
    #
    # NOTE ON WHAT THIS LOCK DOES *NOT* FIX. An earlier version of this
    # comment claimed the unlocked cache could return "the entry stored
    # under a DIFFERENT key", and credited the lock with fixing the
    # cross-user settings leak this branch chased for weeks. That mechanism
    # is not possible: `cachetools.Cache.__getitem__` is `return
    # self.__data[key]`, a plain dict lookup on a tuple-of-str key, so a hit
    # always returns that key's own value. The failure mode here is loud
    # (KeyError), never a silently wrong value.
    #
    # The cross-user leak was a KEY-COMPLETENESS bug, fixed separately in
    # `utilities/db_utils.py` by moving the cache below username resolution
    # (`_get_cached_user_session(username, _namespace)`). Before that the key
    # was effectively `(thread_uuid,)` for web callers, so every user served
    # by one reused worker thread shared a single entry. Measured on the two
    # shapes: key-above-resolution leaks ~472k wrong-user sessions per 480k
    # calls with or without this lock; key-below-resolution leaks zero.
    #
    # Both changes are worth keeping — this one prevents crashes, that one
    # prevents the leak. They are not interchangeable, and this is the wrong
    # place to look for the leak fix.
    kwargs.setdefault("lock", threading.RLock())
    return cached(*args, **kwargs, key=_key_func)