~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(library): project ResearchResource whole-table scans to bound memory (#4560) (#4724)
LearningCircuit8c4029204455
diff --git a/changelog.d/+bound-research-resource-loads.bugfix.md b/changelog.d/+bound-research-resource-loads.bugfix.mdnew file mode 100644index 0000000000..9b25a35591--- /dev/null+++ b/changelog.d/+bound-research-resource-loads.bugfix.md@@ -0,0 +1 @@+The "download all" and "queue all undownloaded" library actions now project only the columns they use instead of loading full `ResearchResource` rows, avoiding materializing every resource's `content_preview` text on a whole-table scan for large libraries (#4560).diff --git a/src/local_deep_research/research_library/routes/library_routes.py b/src/local_deep_research/research_library/routes/library_routes.pyindex f01b870150..168bea8ff2 100644--- a/src/local_deep_research/research_library/routes/library_routes.py+++ b/src/local_deep_research/research_library/routes/library_routes.py@@ -615,8 +615,15 @@ def download_all_text():try:# Get all undownloaded resourceswith get_user_db_session(username) as session:- # Get resources that don't have text files yet- all_resources = session.query(ResearchResource).all()+ # Project only the columns the loop uses (id/url/title) rather+ # than loading full ``ResearchResource`` entities — the+ # ``content_preview`` Text column would otherwise materialize+ # for every resource on this whole-table scan (#4560).+ all_resources = session.query(+ ResearchResource.id,+ ResearchResource.url,+ ResearchResource.title,+ ).all()# Filter to only downloadable resources (academic/PDF)resources = [r for r in all_resources if is_downloadable_url(r.url)@@ -1027,9 +1034,16 @@ def queue_all_undownloaded():logger.info(f"queue_all_undownloaded called for user {username}")with get_user_db_session(username) as db_session:- # Find all resources that don't have a completed download+ # Find all resources that don't have a completed download. Project only+ # the columns the filter/loop use (id/url/research_id) instead of full+ # ``ResearchResource`` entities so the ``content_preview`` Text column+ # isn't materialized for every row on this whole-table scan (#4560).undownloaded = (- db_session.query(ResearchResource)+ db_session.query(+ ResearchResource.id,+ ResearchResource.url,+ ResearchResource.research_id,+ ).outerjoin(Document,(