~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Concurrency regression for the sidecar gate registry's creation guard.
``_get_sidecar_gate`` looks up and, on a miss, inserts a gate into a
``WeakValueDictionary`` under ``_sidecar_gates_guard``. Without the guard
two threads can both miss and each install their own ``threading.Lock``,
so two in-process holders of "the" gate for one sidecar run concurrently.
Gates are reclaimed as soon as the last holder drops them, so the miss
path is taken again every time the registry entry dies.
The natural race window (between the dictionary miss and the insert) is
a few bytecodes wide, so a pure timing hammer only catches it some of the
time. The test therefore widens the window deterministically: the
registry is swapped for a ``WeakValueDictionary`` whose miss path sleeps
briefly, releasing the GIL. With the guard, every other caller is parked
on the guard during that sleep; without it, they all miss too and each
creates its own gate. Small switch intervals and ``gc.collect()`` keep the
surrounding create/release churn realistic.
"""
import gc
import sys
import threading
import time
import weakref
from local_deep_research.database import rekey_lock as sut
_THREADS = 8
_ROUNDS = 40
_GC_EVERY = 8
# Long enough for every runnable thread to reach the registry lookup.
_MISS_DELAY = 0.002
# Long enough for a second holder of a split gate to enter concurrently.
_HOLD_DELAY = 0.0005
class _SlowMissRegistry(weakref.WeakValueDictionary):
"""Registry whose miss path yields, widening the check-then-insert gap."""
def get(self, key, default=None):
value = super().get(key, default)
if value is None:
time.sleep(_MISS_DELAY)
return value
def _hammer_gate(lock_path):
"""Return (overlaps, split_rounds) from bounded concurrent gate use.
Each round starts with the registry entry for ``lock_path`` reclaimed.
All threads are released together, fetch the gate, enter it, and keep
a strong reference until the round ends, so every thread in a round
must have received the same object: the first one stays alive for the
whole round. ``split_rounds`` counts rounds in which they did not, and
``overlaps`` counts critical-section entries made while another thread
was already inside.
"""
stats_lock = threading.Lock()
active = 0
overlaps = 0
split_rounds = 0
rounds_done = 0
round_gates = []
errors = []
def end_round():
nonlocal split_rounds, rounds_done
if len({id(gate) for gate in round_gates}) != 1:
split_rounds += 1
round_gates.clear()
rounds_done += 1
if rounds_done % _GC_EVERY == 0:
gc.collect()
start = threading.Barrier(_THREADS)
finish = threading.Barrier(_THREADS, action=end_round)
def worker():
nonlocal active, overlaps
try:
for _ in range(_ROUNDS):
start.wait()
gate = sut._get_sidecar_gate(lock_path)
with gate:
with stats_lock:
active += 1
if active > 1:
overlaps += 1
round_gates.append(gate)
time.sleep(_HOLD_DELAY)
with stats_lock:
active -= 1
finish.wait()
# Drop the strong reference so the next round's first
# lookup misses and creates a fresh gate.
del gate
except BaseException as error: # pragma: no cover - surfaced below
errors.append(error)
start.abort()
finish.abort()
previous_interval = sys.getswitchinterval()
sys.setswitchinterval(1e-6)
try:
threads = [threading.Thread(target=worker) for _ in range(_THREADS)]
for thread in threads:
thread.start()
for thread in threads:
thread.join(timeout=30)
finally:
sys.setswitchinterval(previous_interval)
assert not any(thread.is_alive() for thread in threads), "worker hung"
assert not errors, errors
return overlaps, split_rounds
def test_concurrent_gate_creation_yields_one_gate(tmp_path, monkeypatch):
monkeypatch.setattr(sut, "_sidecar_gates", _SlowMissRegistry())
lock_path = tmp_path / "user.db.rekey.lock"
overlaps, split_rounds = _hammer_gate(lock_path)
assert split_rounds == 0, (
f"{split_rounds}/{_ROUNDS} rounds handed concurrent callers "
"different gates for one sidecar"
)
assert overlaps == 0, f"{overlaps} gate critical sections overlapped"
gc.collect()
with sut._sidecar_gates_guard:
assert lock_path not in sut._sidecar_gates