~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs: flag the fresh-user UI-test FD artifact (chat shards aren't a DB leak) (#4777)
LearningCircuitffa46f29cdb6
diff --git a/docs/developing/resource-cleanup.md b/docs/developing/resource-cleanup.mdindex a4cd90a75a..a3f53aa5bb 100644--- a/docs/developing/resource-cleanup.md+++ b/docs/developing/resource-cleanup.md@@ -537,6 +537,37 @@ re-derive it from the symptom.- Container health turns `unhealthy` because the healthcheck `urlopen`hangs on a process that no longer has FDs to accept connections.+### 0a. Rule out first — local UI-test "fresh-user churn" false positive++Before treating climbing FDs as a leak, confirm you are measuring the+**single-CI-user** condition. A very convincing *false* FD leak appears+when reproducing UI tests locally:++- The Puppeteer harness (`tests/ui_tests/auth_helper.js` →+ `ensureAuthenticated`) logs in as the shared CI user `test_admin` when+ `CI=true`. If that login fails, it **falls back to registering a fresh+ `testuser_<timestamp>` per test**. The usual local trigger is+ `test_admin` getting *failed-login lockout-locked* after a few+ iterations.+- Each fresh user opens its own per-user encrypted DB + engine. Those are+ disposed only on logout or the ~300s connection-cleanup sweep, so within+ one sub-300s shard run they accumulate and the server's FD count to+ `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per+ shard run, 0→533 over six runs). It looks identical to a real per-user+ connection leak.+- It is **not** a server bug. In real CI the one working `test_admin` is+ reused → one engine → FDs bounded by the pool cap (pool_size 20 ++ max_overflow 40 = 60). Confirm by grepping the server log for many+ distinct `testuser_<ts>` engine opens, or by checking the username the+ leaked FDs' DB files belong to.++Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing+in CI were investigated as a per-user DB FD leak and traced *twice* to+this artifact. Both shards pass locally in faithful CI mode with bounded+FDs; their CI failures are runner **contention** (60s navigation timeouts+on a heavily-loaded Docker runner), not a connection leak. Cross-verify+the user identity before committing to a leak hypothesis.+### 1. Capture diagnostic state BEFORE restartingThe single most important rule: **the snapshot does not survive adiff --git a/tests/ui_tests/auth_helper.js b/tests/ui_tests/auth_helper.jsindex 2d60c7ea61..39317bee0d 100644--- a/tests/ui_tests/auth_helper.js+++ b/tests/ui_tests/auth_helper.js@@ -667,6 +667,34 @@ class AuthHelper {}}+ // ---------------------------------------------------------------+ // GOTCHA — fresh-user fallback can masquerade as a server FD/DB leak.+ //+ // When the CI test-user login above fails, every test falls back to+ // registering a BRAND-NEW user (generateRandomUsername() below). The+ // most common cause locally is the shared `test_admin` getting+ // failed-login *lockout-locked* after several iterations — once that+ // happens, each test in a shard registers its own `testuser_<ts>`.+ //+ // Each fresh user gets its own per-user encrypted DB + SQLAlchemy+ // engine on the server. Those engines are only disposed by logout or+ // the periodic connection-cleanup sweep (~300s; see+ // docs/developing/resource-cleanup.md and ADR-0004), so within a+ // single sub-300s shard run they accumulate. The server's open+ // file-descriptor count to encrypted_databases/*.db(-wal/-shm) then+ // climbs ~linearly across tests — which looks EXACTLY like a per-user+ // DB connection leak but is purely this test artifact.+ //+ // In real CI a single working `test_admin` is reused, so there is one+ // engine and FDs stay bounded by the pool cap (pool_size 20 ++ // max_overflow 40 = 60). Before chasing a "chat shards leak / hang",+ // confirm you are NOT in this fallback: grep the server log for many+ // distinct `testuser_<ts>` opens. The chat-shard CI failures+ // themselves are runner contention (60s navigation timeouts under a+ // heavily-loaded Docker runner), not a server-side connection leak —+ // both chat shards pass locally in faithful CI mode with bounded FDs.+ // ---------------------------------------------------------------+// Generate random username if not providedif (!username) {username = generateRandomUsername();