Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

docs: flag the fresh-user UI-test FD artifact (chat shards aren't a DB leak) (#4777)

LearningCircuitffa46f29cdb6
diff --git a/docs/developing/resource-cleanup.md b/docs/developing/resource-cleanup.md
index a4cd90a75a..a3f53aa5bb 100644
--- a/docs/developing/resource-cleanup.md
+++ b/docs/developing/resource-cleanup.md
@@ -537,6 +537,37 @@ re-derive it from the symptom.
- Container health turns `unhealthy` because the healthcheck `urlopen`
hangs on a process that no longer has FDs to accept connections.
+### 0a. Rule out first — local UI-test "fresh-user churn" false positive
+
+Before treating climbing FDs as a leak, confirm you are measuring the
+**single-CI-user** condition. A very convincing *false* FD leak appears
+when reproducing UI tests locally:
+
+- The Puppeteer harness (`tests/ui_tests/auth_helper.js` →
+ `ensureAuthenticated`) logs in as the shared CI user `test_admin` when
+ `CI=true`. If that login fails, it **falls back to registering a fresh
+ `testuser_<timestamp>` per test**. The usual local trigger is
+ `test_admin` getting *failed-login lockout-locked* after a few
+ iterations.
+- Each fresh user opens its own per-user encrypted DB + engine. Those are
+ disposed only on logout or the ~300s connection-cleanup sweep, so within
+ one sub-300s shard run they accumulate and the server's FD count to
+ `encrypted_databases/*.db(-wal/-shm)` climbs ~linearly (e.g. 0→90 per
+ shard run, 0→533 over six runs). It looks identical to a real per-user
+ connection leak.
+- It is **not** a server bug. In real CI the one working `test_admin` is
+ reused → one engine → FDs bounded by the pool cap (pool_size 20 +
+ max_overflow 40 = 60). Confirm by grepping the server log for many
+ distinct `testuser_<ts>` engine opens, or by checking the username the
+ leaked FDs' DB files belong to.
+
+Concretely: the **chat UI shards** (`chat-core`, `chat-lifecycle`) failing
+in CI were investigated as a per-user DB FD leak and traced *twice* to
+this artifact. Both shards pass locally in faithful CI mode with bounded
+FDs; their CI failures are runner **contention** (60s navigation timeouts
+on a heavily-loaded Docker runner), not a connection leak. Cross-verify
+the user identity before committing to a leak hypothesis.
+
### 1. Capture diagnostic state BEFORE restarting
The single most important rule: **the snapshot does not survive a
diff --git a/tests/ui_tests/auth_helper.js b/tests/ui_tests/auth_helper.js
index 2d60c7ea61..39317bee0d 100644
--- a/tests/ui_tests/auth_helper.js
+++ b/tests/ui_tests/auth_helper.js
@@ -667,6 +667,34 @@ class AuthHelper {
}
}
+ // ---------------------------------------------------------------
+ // GOTCHA — fresh-user fallback can masquerade as a server FD/DB leak.
+ //
+ // When the CI test-user login above fails, every test falls back to
+ // registering a BRAND-NEW user (generateRandomUsername() below). The
+ // most common cause locally is the shared `test_admin` getting
+ // failed-login *lockout-locked* after several iterations — once that
+ // happens, each test in a shard registers its own `testuser_<ts>`.
+ //
+ // Each fresh user gets its own per-user encrypted DB + SQLAlchemy
+ // engine on the server. Those engines are only disposed by logout or
+ // the periodic connection-cleanup sweep (~300s; see
+ // docs/developing/resource-cleanup.md and ADR-0004), so within a
+ // single sub-300s shard run they accumulate. The server's open
+ // file-descriptor count to encrypted_databases/*.db(-wal/-shm) then
+ // climbs ~linearly across tests — which looks EXACTLY like a per-user
+ // DB connection leak but is purely this test artifact.
+ //
+ // In real CI a single working `test_admin` is reused, so there is one
+ // engine and FDs stay bounded by the pool cap (pool_size 20 +
+ // max_overflow 40 = 60). Before chasing a "chat shards leak / hang",
+ // confirm you are NOT in this fallback: grep the server log for many
+ // distinct `testuser_<ts>` opens. The chat-shard CI failures
+ // themselves are runner contention (60s navigation timeouts under a
+ // heavily-loaded Docker runner), not a server-side connection leak —
+ // both chat shards pass locally in faithful CI mode with bounded FDs.
+ // ---------------------------------------------------------------
+
// Generate random username if not provided
if (!username) {
username = generateRandomUsername();