~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
import threading
import uuid
from typing import Any, Callable, Hashable, Tuple
from cachetools import cached, keys
g_thread_local_store = threading.local()
def thread_specific_cache(*args: Any, **kwargs: Any) -> Callable:
"""
A version of `cached()` that is local to a single thread. In other words,
cache entries will only be valid in the thread where they were created.
WHY THIS IS STILL THREAD-KEYED UNDER FastAPI, AND NOT A `ContextVar`.
This survived the Flask -> FastAPI migration on purpose; it is not a
leftover. The two Flask helpers that used to live in this module
(`thread_with_app_context`, `thread_context`) were deleted in that
migration because they wrapped `flask.g` / `AppContext`, and identity and
research-context propagation genuinely did move to `ContextVar` — see
`utilities/request_context.py` and `utilities/thread_context.py`. This
decorator is a different case, because what it caches is not context but a
THREAD-AFFINE RESOURCE.
Its only production caller is `db_utils._get_cached_user_session`, which
caches a SQLAlchemy `Session`. The post-auth session accessor on
`db_manager` builds a fresh `sessionmaker(bind=engine)()` on every call and
does nothing to scope it
per thread, so this key is the only thing keeping one cached `Session` from
being handed to two anyio threadpool workers running concurrently for the
same user. A `Session` is documented as not thread-safe: concurrent
identity-map mutation and autoflush on one instance is the failure this
prevents.
A `ContextVar`-keyed cache would NOT be equivalent, and would be unsafe
here. A single request context can straddle two worker threads: Starlette
drives sync-generator dependencies through `contextmanager_in_threadpool`,
which dispatches `__enter__` and `__exit__` as two separate
`anyio.to_thread.run_sync` calls, and anyio selects a worker via
`idle_workers.pop()` with no task affinity. (This hazard is written up at
`web/dependencies/auth.py::get_db_session_dep`, and handled deliberately by
the streaming generators in `web/routers/library.py`.) Keying on context
would therefore return ONE `Session` to a context spread across two
threads — reintroducing exactly the cross-thread sharing that keying on
the thread prevents. `ContextVar` is the right primitive for propagating
identity; `threading.local()` is the right primitive for a resource whose
safety is defined per OS thread. Do not swap one for the other without
also giving `Session` acquisition a different lifetime model.
WHAT THE THREAD KEY DOES *NOT* DO. It provides no cross-user isolation.
That comes from `username` being part of `keys.hashkey(*args_, **kwargs_)`
below, i.e. from applying this decorator BELOW username resolution
(`_get_cached_user_session(username, _namespace)`) rather than above it.
See the note on the lock further down, and
`tests/security/test_cross_user_isolation_invariants.py`, which pins both
properties separately. Conflating the two is how the key-completeness bug
described below got misdiagnosed for weeks.
Args:
*args: Will be forwarded to `cached()`.
**kwargs: Will be forwarded to `cached()`.
Returns:
The wrapped function.
"""
def _key_func(*args_: Any, **kwargs_: Any) -> Tuple[Hashable, ...]:
base_hash = keys.hashkey(*args_, **kwargs_)
if hasattr(g_thread_local_store, "thread_id"):
# We already gave this thread a unique ID. Use that.
thread_id = g_thread_local_store.thread_id
else:
# Give this thread a new unique ID.
thread_id = uuid.uuid4().hex
g_thread_local_store.thread_id = thread_id
return (thread_id,) + base_hash
# cachetools' `cached()` is NOT thread-safe unless a lock is supplied.
# The cache object is shared by every calling thread, and concurrent
# __setitem__/eviction can desync an LRUCache's internal ordering from
# its data dict, after which `popitem()` raises KeyError. With the
# per-thread key below the key space is (thread x args), so a small
# maxsize under many worker threads means near-constant eviction —
# precisely the condition that provokes it. Measured: without this lock,
# ~432k KeyErrors across 480k concurrent calls; with it, zero.
#
# NOTE ON WHAT THIS LOCK DOES *NOT* FIX. An earlier version of this
# comment claimed the unlocked cache could return "the entry stored
# under a DIFFERENT key", and credited the lock with fixing the
# cross-user settings leak this branch chased for weeks. That mechanism
# is not possible: `cachetools.Cache.__getitem__` is `return
# self.__data[key]`, a plain dict lookup on a tuple-of-str key, so a hit
# always returns that key's own value. The failure mode here is loud
# (KeyError), never a silently wrong value.
#
# The cross-user leak was a KEY-COMPLETENESS bug, fixed separately in
# `utilities/db_utils.py` by moving the cache below username resolution
# (`_get_cached_user_session(username, _namespace)`). Before that the key
# was effectively `(thread_uuid,)` for web callers, so every user served
# by one reused worker thread shared a single entry. Measured on the two
# shapes: key-above-resolution leaks ~472k wrong-user sessions per 480k
# calls with or without this lock; key-below-resolution leaks zero.
#
# Both changes are worth keeping — this one prevents crashes, that one
# prevents the leak. They are not interchangeable, and this is the wrong
# place to look for the leak fix.
kwargs.setdefault("lock", threading.RLock())
return cached(*args, **kwargs, key=_key_func)