~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
docs(developing): add resource-cleanup.md capturing the FD-leak campaign (#3856)
LearningCircuit5ede95d3b4ed
diff --git a/docs/developing/resource-cleanup.md b/docs/developing/resource-cleanup.mdnew file mode 100644index 0000000000..35f48b4ea8--- /dev/null+++ b/docs/developing/resource-cleanup.md@@ -0,0 +1,280 @@+# Resource cleanup in LDR++This document captures how LDR manages process-level resources (DB+connections, HTTP clients, file descriptors, threads) and the reasoning+trail behind the current model. It exists because file-descriptor+exhaustion has been a recurring class of bug in LDR, and the *journey*+of fixing it — what's been tried, what worked, what was ruled out — is+not reconstructable from `git log` alone.++If you're contributing code that holds a network connection, a database+session, an LLM client, or a thread, read this before adding `__del__`,+`weakref.finalize`, or a context manager.++---++## Current model++### Database connections++- **One shared per-user `QueuePool`.** No per-thread engines. Pool+ sizing: `pool_size=20`, `max_overflow=40`, with periodic `dispose()`+ every 30 minutes.+- **SQLCipher is decrypted once per connection-open.** `PRAGMA key`+ takes ~0.2 ms; pool reuse keeps that off the hot path.+- Engines are created at login, closed at logout (or process exit via+ the registered `atexit` shutdown).+- Background threads (research workers, metric writers, news scheduler+ jobs) use the same per-user pool — they no longer maintain a separate+ thread-engine system.++See [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md) for the+QueuePool-vs-NullPool decision and PR #3441 for the per-thread-engine+removal.++### LLM wrappers++LDR wraps every LLM in `ProcessingLLMWrapper` (and optionally+`RateLimitedLLMWrapper`) so that callers see a uniform interface and+the project owns the close path:++```+caller -> ProcessingLLMWrapper.close()+ -> _close_base_llm(base_llm) in utilities/llm_utils.py+ -> for ChatOllama:+ sync httpx client (ollama.Client._client) .close()+ async httpx client (ollama.AsyncClient._client) .aclose()+ -> for ChatOpenAI / ChatAnthropic:+ no close (those use @lru_cache'd shared httpx clients)+```++Key invariants:++- `ChatOllama` is the *only* provider where `_close_base_llm()` actually+ closes anything. ChatOpenAI and ChatAnthropic share LRU-cached httpx+ clients across instances; closing them would break other live LLMs.+- Both `_client` (sync) **and** `_async_client` (async) are released —+ the async side is exercised by every `ainvoke()` call (langgraph+ agents, modular strategies). Closing only the sync side leaks the+ async transport per call (root cause of #3816).+- The function is idempotent via an `_ldr_closed` sentinel on the inner+ httpx clients.+- The async close uses `asyncio.run(client.aclose())` only when no+ event loop is currently running. When called from inside async code+ it skips and leaves the close to the loop's owner.++### Search engines++- `BaseSearchEngine.close()` is the single entry point and **cascades**+ into `_preview_filters` and `_content_filters`. That cascade is what+ releases per-engine LLMs (e.g., `JournalReputationFilter.model`),+ SearXNG sessions, and other filter-held resources.+- Search-engine cleanup happens at the per-research finally block in+ `web/services/research_service.py:run_research_process()` and at the+ programmatic API entry points in `api/research_functions.py`.+- The `_owns_llm` flag pattern (introduced in #2712) tracks whether a+ filter or engine constructed its own LLM (and thus owns it) versus+ borrowed one from a caller (and must not close it).++### Thread lifecycle++- `@thread_cleanup` (decorator on `run_research_process` and similar+ workers) ensures thread-local DB sessions are released even on+ abnormal exits.+- `cleanup_current_thread()` is called from Flask teardown, the queue+ processor, the auth flow, and the RAG routes — six tier-1 paths in+ total.+- Background threads are daemon threads; the process exit handles any+ thread that did not clean up gracefully.++### Conventions++- **Use `safe_close(resource, "human name")`** from+ `utilities/resource_utils.py` for every cleanup. Never bare `.close()`+ in a `finally` (it can mask the original exception).+- **Prefer `try/finally` over `__del__`**. Python doesn't guarantee+ finalization order at interpreter exit; `__del__` interacts subtly+ with reference cycles and `weakref`.+- **Track ownership explicitly with `_owns_llm` (or analogous flag)**+ when a class accepts an injected resource that may or may not be its+ own.+- **News fragments (`changelog.d/<id>.bugfix.md`) are required for any+ user-visible cleanup behavior change** — see `changelog.d/README.md`.++---++## How to close X correctly++| You're holding | Do this |+| --- | --- |+| A `ChatOllama` (raw or wrapped) | Call `wrapper.close()` in a `finally`, or pass to `safe_close(wrapper, "...")`. The wrapper chain handles both sync and async httpx clients. |+| A search engine you constructed | `safe_close(engine, "...")` in `finally`. The engine's `close()` cascades into preview/content filters. |+| A holder class with an LLM | Add a `close()` method, gate the LLM close on `self._owns_llm`, document who calls it. Don't add `__del__`. |+| A long-lived service holder (news scheduler, etc.) | Wrap construction in `try/finally` at the cycle boundary. Don't store the LLM if you can recreate it cheaply. |+| A DB session | Use `with get_user_db_session(username) as session:`. Don't bypass via `get_settings_manager(username=...)` without `owns_session=False` (see #3023). |+| An asyncio event loop | Use the existing one. If you genuinely need a new one (background thread fallback), call `loop.close()` in a `finally` — see `news_strategy.py` for the reference pattern (post-#3018). |++---++## Anti-patterns++These look reasonable but break specific things in this codebase:++- **Adding `__del__` to a class with `close()`.** At interpreter exit+ the `logger`, `httpx`, and event-loop modules may already be torn+ down. `__del__` can run after them and raise. Use explicit close in+ a `finally` instead.+- **Closing a shared httpx client.** ChatOpenAI / ChatAnthropic share+ one httpx pool across instances via `@lru_cache`. Closing it kills+ every other live LLM in the same process. The Ollama check in+ `_close_base_llm` exists exactly to gate this.+- **Truthy idempotency sentinels on Mock objects.** `Mock()` without a+ `spec` auto-generates child Mocks for any attribute access, so+ `getattr(client, "_ldr_closed", False)` returns a truthy Mock and+ short-circuits the close. Always use `is True` / `is None` checks+ for sentinels — see the pattern in `_close_base_llm`.+- **Skipping `super().close()` in a search-engine subclass.**+ `BaseSearchEngine.close()` is what cascades into preview/content+ filters. Override it without calling super and you leak every+ filter's resources (this was a Copilot finding on #3818).+- **Treating `asyncio.run()` as safe inside an event loop.** It raises+ `RuntimeError` if called from a thread that already has a running+ loop. The pattern in `_close_base_llm` is: detect a running loop+ with `get_running_loop()`, skip the async close in that branch (the+ loop owner will close), only call `asyncio.run` in the no-loop case.++---++## History++The FD-leak campaign spans roughly four months of iterative work. Each+fix narrowed the remaining surface; each subsequent issue was found in+a corner the previous wave hadn't touched.++### Wave 1 — initial leak inventory (Jan 2026)++- **#1832, #1849, #1856, #1860** — first comprehensive sweep. Identified+ seven distinct leak sources: `auth_db` engine, `download_management`+ DB, search cache, subprocess zombies, HTTP sessions in+ `SemanticScholarSearchEngine` and `BaseDownloader`, Socket.IO threads.+ Established context-manager + `try/finally` patterns. Added a+ pre-commit hook to catch missing cleanup at commit time.++### Wave 2 — thread-local engine accumulation (Mar 2026)++- **#2495** — diagnosed that Flask's teardown only cleaned the+ request-scoped `g.db_session` while a separate `_thread_engines` dict+ accumulated NullPool engines per thread, leaking ~3 FDs per request.+ Added `cleanup_current_thread()` across six tier-1 paths.+- **#2591** — dead-thread engines (when threads crashed they left+ engines behind) plus `stream=True` socket holds in the generic+ downloader. Added a throttled dead-thread sweep, removed `stream=True`,+ raised the Docker ulimit from 1024 to 65536.++### Wave 3 — LLM wrapper lifecycle (Mar 2026)++- **#2708** — diagnosed `ChatOllama` → `httpx.Client` chains with no+ `__del__`. With the news scheduler triggering 50–300+ `quick_summary()` calls per hour, a 1024-FD container exhausted in+ 3–4 hours. Wrapped four programmatic API entry points in+ `try/finally` with explicit close.+- **#2712** — extracted `close_llm()` to a shared utility. Added+ `close()` and `_owns_llm` to `NewsAnalyzer`, `HeadlineGenerator`,+ `TopicGenerator`, `JournalReputationFilter`, `DomainClassifier`,+ `GitHubSearchEngine`, `IntegratedReportGenerator`,+ `ElasticsearchSearchEngine`, and the benchmark graders.+- **#2756** — wrapped bare `.close()` calls in `finally` blocks with+ `safe_close()` to prevent masking the original exception.+- **#2732** — moved `close()` into `ProcessingLLMWrapper` and+ `RateLimitedLLMWrapper` directly; eliminated the standalone+ `close_llm()` free function.++### Wave 4 — DB session leaks + per-call patterns (late Mar / early Apr 2026)++- **#3018** — `get_settings_manager(username=...)` was bypassing+ `g.db_session` and creating QueuePool sessions per-thread; live+ diagnostics showed 321 sockets allocated, only 66 in use.+ `DownloadService.close()` leaked the inner `SettingsManager` session.+ Also fixed `TopicBasedRecommender._create_recommendation_card()`+ (per-call LLM with no cleanup) and an `asyncio.new_event_loop()` in+ `news_strategy.py` that never closed.+- **#3204** — test fixtures using `return` instead of `yield` left+ engines un-disposed. Migrated 8 test files to `yield` ++ `engine.dispose()`.++### Wave 5 — DB pool architecture (Apr 2026)++- **#3340** — kept QueuePool but minimized FDs (`pool_size=1`,+ `max_overflow=2`, periodic `dispose()` every 30 min).+- **#3337** (closed) — proposed switching SQLCipher engines to+ NullPool for zero persistent FDs. Superseded by #3441.+- **#3441** — removed per-thread NullPool engines entirely+ (~2,100 lines of sweep logic deleted) and routed metrics through a+ single shared per-user QueuePool with bounded sizing+ (`pool_size=20`, `max_overflow=40`).+- **#3477** — created [ADR-0004](../decisions/0004-nullpool-for-sqlcipher.md)+ capturing the final pool model and updated stale FD calculations+ across docs.++### Wave 6 — async client close (May 2026)++- **#3818** (open, declined for merge) — proposed session-pooling+ around `safe_get`/`safe_post` to address #3816. The session refactor+ is reasonable in isolation, but the lsof in #3816 showed ~72% of+ leaked FDs as `a_inode [eventpoll]` selectors, not HTTP request+ sockets — pointing at async-client transports rather than `safe_get`+ callers (whose response bodies were already consumed). See+ [the PR comment](https://github.com/LearningCircuit/local-deep-research/pull/3818#issuecomment-4402290677)+ for the full reasoning.+- **#3855** — extended `_close_base_llm()` to also close+ `ChatOllama._async_client` (the actual gap the lsof pointed to).+ Added the `IntegratedReportGenerator` close that was missing from the+ per-research `finally` block. Idempotency via `_ldr_closed` sentinels+ on the inner httpx clients.++---++## Intentionally not done (deferred)++These showed up during planning and were deliberately *not* done. If+they get rediscovered as "missing work" by future contributors, please+reference this section first.++- **`weakref.finalize` defense-in-depth on the LLM wrappers.** Designed+ and verified safe (no `__del__` conflicts, `__getattr__` doesn't+ intercept `_finalizer`, no reference cycles). Deferred until a+ fourth wave of "missed close" leaks justifies adding a new pattern+ that future contributors must understand. Current explicit-close+ discipline has held since #2712 / #2732 / #3018.+- **LLM caching in `get_llm()`.** Bounding total `ChatOllama` instances+ to N=distinct configs would make leak shapes architecturally+ impossible. Orthogonal optimization, deferred — adds complexity+ around settings invalidation and multi-tenant isolation.+- **Pre-commit hook flagging `get_llm()` callers without `close()`.**+ Useful in principle, deferred — high false-positive risk+ (caller-passed LLMs, lazy-init holders, factory-returned LLMs all+ legitimately don't close). Needs a careful design.+- **Dedicated `/api/v1/health/fd` diagnostic with eventpoll-inode+ dedupe.** PR #3033 stalled at a basic version (Windows + RLIM_INFINITY+ bugs); PR #3036 added `utilities/fd_monitor.py` for cross-platform FD+ reading. A type/inode-breakdown extension is feasible but deferred+ until an active leak hunt actually needs it.++---++## Glossary++- **`_owns_llm`** — instance flag set in `__init__` to `True` when the+ class fetched its own LLM via `get_llm()`, `False` when an LLM was+ injected by the caller. Gates whether `close()` actually closes the+ LLM.+- **`safe_close(resource, name)`** — helper in `utilities/resource_utils.py`+ that calls `resource.close()` inside a try/except, logging on failure.+ Never raises. Used in every `finally` block.+- **`_ldr_closed`** — sentinel attribute set on inner httpx clients by+ `_close_base_llm` to make the function idempotent. Checked with+ `is True` (not truthy) so Mock objects without a `spec` don't trip+ the guard.+- **eventpoll FD** — Linux `a_inode` file descriptor type for+ `epoll_create`'d kernel objects. Each asyncio event loop registers+ one. Leaked AsyncClients hold them via the loop's selector.