Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(library): decouple downloader from database session in download_resource (#6560) (#6564)

S1cbe094c61bb
diff --git a/changelog.d/6564.bugfix.md b/changelog.d/6564.bugfix.md
new file mode 100644
index 0000000000..897831898c
--- /dev/null
+++ b/changelog.d/6564.bugfix.md
@@ -0,0 +1,49 @@
+Move `downloader.download_with_result` outside the database session in `download_resource` (#6560, #6564).
+Previously, wrapping `_download_pdf` inside `get_user_db_session(...)` pinned a checked-out connection
+from QueuePool across multi-second HTTP transfers, exhausting the pool and starving concurrent FastAPI
+request handlers under bulk downloads. Pre-download checks now execute in a brief transaction, the
+HTTP transfer runs outside any session, and short transactions stage the `DownloadAttempt` and document
+records only after download completion. The success path now commits document staging and download
+queue status separately, widening a crash window that can strand a PROCESSING queue row (recoverable
+via re-run).
+
+Follow-up review fixes:
+- Flush the new `DownloadTracker` row before expunging it on the pre-download path, so the
+ `DownloadAttempt.url_hash` foreign key is satisfied on first-time downloads (with FK enforcement
+ the prior code raised `FOREIGN KEY constraint failed`).
+- Snapshot `target_resource.title` into a plain local before staging commit so post-commit text
+ production does not lazy-refresh and re-checkout a QueuePool connection immediately before the
+ arXiv network call / CPU extraction.
+- Wrap the new-tracker INSERT in `session.begin_nested()` so the unconditional pre-block
+ `session.commit()` releases the QueuePool connection without conflating the pre-check write
+ with any caller-staged row on the shared thread-local session. The savepoint scopes the
+ pre-check's only write so a failure inside it cannot drag the caller's outer transaction
+ with it, and the structural change is pinned by `test_pre_check_savepoint_scopes_new_tracker_write`.
+- Stop passing the raw exception text through `sanitize_error_for_client()` to the
+ caller-visible `skip_reason`. That function scrubs credential shapes only and explicitly
+ does not strip SQL text, target URLs, or dependency internals — the SQLAlchemy `IntegrityError`
+ on the staging path surfaced `INSERT INTO document_collections ...` verbatim into the
+ retry-manager and bulk-stream SSE messages. The exception is now mapped to a fixed safe
+ token via `_client_safe_download_message()` (e.g. `database_constraint`, `network_reset`,
+ `pdf_parse_failed`); the full sanitized text is preserved server-side in
+ `DownloadAttempt.error_message` and `logger.exception` for debugging. The same mapping
+ now covers the sibling text-extraction failure paths (`_try_arxiv_text_extraction`,
+ `_try_api_text_extraction`, `_fallback_pdf_extraction`, whose `Failed to save text`
+ strings flow into `download_as_text` and the bulk-stream SSE `error` field) and the
+ HTML downloader's exception path (via the shared `security.client_safe_errors`
+ table, which both call sites import so the copies cannot drift again);
+ the CWE-209 guard comments in `web/routers/library.py` now state the fixed-token
+ guarantee instead of claiming `sanitize_error_for_client()`-scrubbed is safe.
+ `FailureClassifier` now recognises the token vocabulary (`network_timeout` →
+ `timeout` 30-min instead of `unknown_error` 1h, plus the database/pdf/
+ `filesystem_error:`/`download_error:` shapes), and `RetryManager.record_attempt`
+ passes single-token skip_reasons through as `error_type` instead of collapsing
+ them to the literal `"str"`. The HTML downloader's tokenized failure logs the
+ exception class (`exc_class=`) for operators while still never logging `str(e)`
+ or the traceback.
+- Added `test_first_time_download_persists_tracker_before_attempt_with_fk_on` (FK-ON engine)
+ and `test_download_sqlalchemy_exception_does_not_leak_sql_in_skip_reason` to pin the two
+ follow-up fixes. The FK-ON test is mutation-verified for the combined savepoint+flush
+ invariant (removing both fails with `FOREIGN KEY constraint failed`); the explicit
+ `session.flush()` alone is defense-in-depth over the savepoint-exit auto-flush, and the
+ leak test is mutation-verified against restoring `sanitize_error_for_client(str(e))`.
diff --git a/src/local_deep_research/library/download_management/failure_classifier.py b/src/local_deep_research/library/download_management/failure_classifier.py
index 1ba6c73b36..890ec57d9b 100644
--- a/src/local_deep_research/library/download_management/failure_classifier.py
+++ b/src/local_deep_research/library/download_management/failure_classifier.py
@@ -87,6 +87,72 @@ class RateLimitFailure(TemporaryFailure):
self.domain = domain
+# Caller-safe skip-reason tokens emitted by
+# ``security.client_safe_errors.client_safe_download_message`` (PR #6564).
+# ``RetryManager.record_attempt`` receives the token as the string
+# ``details`` (with ``error_type`` collapsed to the literal ``"str"``), so
+# the classifier must recognise the token vocabulary in *both* fields —
+# otherwise ``network_timeout`` falls through to ``unknown_error`` (1h +
+# "Unclassified error" warning) where the pre-token text classified as
+# ``timeout`` (30-min). All entries stay Temporary (no retryable -> terminal
+# flip); the token itself is preserved as ``error_type`` so the UI
+# distinction survives.
+_CLIENT_SAFE_TOKEN_FAILURES: dict[str, tuple[str, str, timedelta]] = {
+ "network_timeout": ("timeout", "Request timed out", timedelta(minutes=30)),
+ "network_unavailable": (
+ "network_error",
+ "Network connectivity issue",
+ timedelta(minutes=5),
+ ),
+ "network_reset": (
+ "network_error",
+ "Network connectivity issue",
+ timedelta(minutes=5),
+ ),
+ "http_error": ("http_error", "HTTP error", timedelta(hours=1)),
+ "network_redirect_limit": (
+ "network_redirect_limit",
+ "Too many redirects",
+ timedelta(hours=1),
+ ),
+ "database_constraint": (
+ "database_constraint",
+ "Database constraint violation",
+ timedelta(hours=1),
+ ),
+ "database_unavailable": (
+ "database_unavailable",
+ "Database temporarily unavailable",
+ timedelta(hours=1),
+ ),
+ "database_error": (
+ "database_error",
+ "Database error",
+ timedelta(hours=1),
+ ),
+ "database_data_error": (
+ "database_data_error",
+ "Database data error",
+ timedelta(hours=1),
+ ),
+ "database_interface_error": (
+ "database_interface_error",
+ "Database interface error",
+ timedelta(hours=1),
+ ),
+ "database_invalid_request": (
+ "database_invalid_request",
+ "Database invalid request",
+ timedelta(hours=1),
+ ),
+ "pdf_parse_failed": (
+ "pdf_parse_failed",
+ "PDF parsing failed",
+ timedelta(hours=1),
+ ),
+}
+
+
class FailureClassifier:
"""Classifies download failures into appropriate types based on error patterns"""
@@ -135,6 +201,47 @@ class FailureClassifier:
timedelta(hours=1),
)
+ # Caller-safe token vocabulary (PR #6564). Tokens arrive as the
+ # ``details`` string with ``error_type == "str"`` via
+ # ``RetryManager.record_attempt`` (or as ``error_type`` directly
+ # once callers pass it through), so check both fields. Exact match
+ # only — free-text details must keep falling through to the pattern
+ # branches below.
+ for candidate in (error_type, details):
+ token = (candidate or "").strip().lower()
+ if not token:
+ continue
+ if token in _CLIENT_SAFE_TOKEN_FAILURES:
+ err, msg, cooldown = _CLIENT_SAFE_TOKEN_FAILURES[token]
+ return TemporaryFailure(err, msg, cooldown)
+ if token.startswith("filesystem_error:"):
+ return TemporaryFailure(
+ "filesystem_error",
+ "Filesystem error",
+ timedelta(hours=1),
+ )
+ if token.startswith("download_error:"):
+ cls_name = token.split(":", 1)[1].strip()
+ if "timeout" in cls_name or "timed out" in cls_name:
+ return TemporaryFailure(
+ "timeout", "Request timed out", timedelta(minutes=30)
+ )
+ if (
+ "connection" in cls_name
+ or "network" in cls_name
+ or "reset" in cls_name
+ ):
+ return TemporaryFailure(
+ "network_error",
+ "Network connectivity issue",
+ timedelta(minutes=5),
+ )
+ return TemporaryFailure(
+ f"download_error:{cls_name}",
+ f"Download failed ({cls_name})",
+ timedelta(hours=1),
+ )
+
# Error message pattern classifications
error_lower = error_type.lower()
details_lower = details.lower()
diff --git a/src/local_deep_research/library/download_management/retry_manager.py b/src/local_deep_research/library/download_management/retry_manager.py
index 0c2a721d89..a4bfc78059 100644
--- a/src/local_deep_research/library/download_management/retry_manager.py
+++ b/src/local_deep_research/library/download_management/retry_manager.py
@@ -134,11 +134,32 @@ class RetryManager:
self.status_tracker.mark_success(resource_id, session=session)
logger.info(f"Resource {resource_id} marked as successful")
else:
- # Failed download - classify the failure
+ # Failed download - classify the failure. ``error_message`` here
+ # is the caller-visible ``skip_reason`` string (PR #6564 fixed
+ # tokens like ``network_timeout``), so ``type(...).__name__`` is
+ # always the literal ``"str"``. Pass single-token values through
+ # as ``error_type`` so ``FailureClassifier`` can restore the
+ # pre-token granularity (``network_timeout`` -> ``timeout``
+ # 30-min instead of ``unknown_error`` 1h). Multi-word free text
+ # keeps the historic ``"str"`` shape; the classifier reads it via
+ # ``details``.
+ if error_message and isinstance(error_message, str):
+ stripped = error_message.strip()
+ if (
+ stripped
+ and " " not in stripped
+ and "\n" not in stripped
+ and "\t" not in stripped
+ ):
+ resolved_error_type = stripped
+ else:
+ resolved_error_type = type(error_message).__name__
+ else:
+ resolved_error_type = (
+ type(error_message).__name__ if error_message else "unknown"
+ )
failure = self.failure_classifier.classify_failure(
- error_type=type(error_message).__name__
- if error_message
- else "unknown",
+ error_type=resolved_error_type,
status_code=status_code,
url=url,
details=details or (error_message or "Unknown error"),
diff --git a/src/local_deep_research/research_library/downloaders/html.py b/src/local_deep_research/research_library/downloaders/html.py
index 41c0a73373..1d4d1e7a0b 100644
--- a/src/local_deep_research/research_library/downloaders/html.py
+++ b/src/local_deep_research/research_library/downloaders/html.py
@@ -19,11 +19,30 @@ from .base import (
)
from .extraction.pipeline import extract_content_with_metadata
from ...constants import BROWSER_USER_AGENT
-from ...security import sanitize_error_for_client
+from ...security.client_safe_errors import (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES,
+ client_safe_download_message,
+)
from ...security.ssrf_validator import redact_url_for_log
from ...utilities.resource_utils import safe_close
+# Single source of truth for caller-safe tokens lives in
+# ``security.client_safe_errors`` (PR #6564 review: this local copy drifted
+# from the services table — missing DataError/InterfaceError/
+# InvalidRequestError and the OSError errno branch). Aliases below preserve
+# the historic names; the mapping itself is shared so it cannot drift again.
+# Importing from ``security`` (not ``research_library.services``) avoids a
+# services <-> downloaders cycle. sanitize_error_for_client() is NOT a
+# substitute — it scrubs credential shapes only, not SQL/paths/URLs.
+_CLIENT_SAFE_HTML_MESSAGES = CLIENT_SAFE_DOWNLOAD_MESSAGES
+
+
+def _client_safe_html_message(exc: BaseException) -> str:
+ """Back-compat alias for :func:`client_safe_download_message`."""
+ return client_safe_download_message(exc)
+
+
def _as_markdown_text(value: str) -> str:
"""Collapse page-supplied metadata to a single line.
@@ -132,10 +151,14 @@ class HTMLDownloader(BaseDownloader):
return None
- except Exception:
+ except Exception as e:
+ # Log the exception class only (zero-risk: the token already
+ # exposes it to the client). Never log str(e) or the traceback:
+ # either can re-embed the credentialed URL / SQL text / paths.
logger.opt(exception=False).error(
- "html.download_failed url={url}",
+ "html.download_failed url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
return None
@@ -206,16 +229,16 @@ class HTMLDownloader(BaseDownloader):
except Exception as e:
logger.opt(exception=False).error(
- "html.download_result_failed url={url}",
+ "html.download_result_failed url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
# skip_reason propagates to the browser via the download SSE
- # stream; the fetch URL can carry credentials — scrub before
- # returning (the exception detail is deliberately NOT logged
- # above; a traceback could re-embed the credentialed URL).
- return DownloadResult(
- skip_reason=sanitize_error_for_client(f"Error: {str(e)}")
- )
+ # stream; the exception text can carry SQL text, target URLs, or
+ # file paths — return a fixed token only (never str(e); only the
+ # class name is logged above, as a traceback could re-embed the
+ # credentialed URL).
+ return DownloadResult(skip_reason=_client_safe_html_message(e))
def _fetch_html(self, url: str) -> Optional[str]:
"""Fetch raw HTML content from URL."""
@@ -411,13 +434,15 @@ class HTMLDownloader(BaseDownloader):
# The caller handles the failure, so it is not an error
# here (ERROR records reach the user's browser).
logger.warning(
- "html.fetch_error url={url}",
+ "html.fetch_error url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
else:
logger.opt(exception=False).error(
- "html.fetch_error url={url}",
+ "html.fetch_error url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
self.rate_tracker.record_outcome(
engine_type=engine_type,
@@ -546,10 +571,11 @@ class HTMLDownloader(BaseDownloader):
"url": url,
}
- except Exception:
+ except Exception as e:
logger.opt(exception=False).error(
- "html.extract_failed url={url}",
+ "html.extract_failed url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
return None
@@ -606,9 +632,10 @@ class HTMLDownloader(BaseDownloader):
return metadata
- except Exception:
+ except Exception as e:
logger.opt(exception=False).error(
- "html.metadata_extract_failed url={url}",
+ "html.metadata_extract_failed url={url} exc_class={exc_class}",
url=redact_url_for_log(url),
+ exc_class=type(e).__name__,
)
return {"url": url}
diff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.py
index fde7c5559a..c3ecb9ff7c 100644
--- a/src/local_deep_research/research_library/services/download_service.py
+++ b/src/local_deep_research/research_library/services/download_service.py
@@ -8,6 +8,7 @@ Handles downloading PDFs from various academic sources with:
- File organization and storage
"""
+from contextlib import nullcontext
import hashlib
import os
import re
@@ -40,6 +41,10 @@ from ...database.models.download_tracker import (
# that bypass, and `is_downloadable_domain` upstream is a relevance
# filter, not a security control.
from ...security import safe_get, sanitize_error_for_client
+from ...security.client_safe_errors import (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES,
+ client_safe_download_message,
+)
from ...database.models.library import (
Collection,
Document as Document,
@@ -108,6 +113,18 @@ def _arxiv_text_provenance(source: ArxivTextSource) -> tuple[str, str]:
assert_never(unreachable)
+# Single source of truth for caller-safe skip-reason tokens lives in
+# ``security.client_safe_errors`` (PR #6564 review: the services and HTML
+# downloader copies drifted). Aliases below preserve the historic import
+# locations for back-compat; new code should import from ``security``.
+_CLIENT_SAFE_DOWNLOAD_MESSAGES = CLIENT_SAFE_DOWNLOAD_MESSAGES
+
+
+def _client_safe_download_message(exc: BaseException) -> str:
+ """Back-compat alias for :func:`client_safe_download_message`."""
+ return client_safe_download_message(exc)
+
+
class DownloadService:
"""Service for downloading and managing research PDFs."""
@@ -700,26 +717,81 @@ class DownloadService:
)
if not tracker:
- tracker = DownloadTracker(
- url=resource.url,
- url_hash=url_hash,
- first_resource_id=resource.id,
- is_downloaded=False,
- )
- session.add(tracker)
- session.commit()
+ # Wrap the new-tracker INSERT in a SAVEPOINT so it is
+ # structurally separated from the shared thread-local
+ # session's outer transaction. Three reasons:
+ # 1. The savepoint commit makes the new tracker row
+ # durable even if a later step on this thread rolls
+ # back the outer transaction.
+ # 2. If the INSERT itself raises (e.g. UNIQUE conflict
+ # on a concurrent same-URL insert), only the savepoint
+ # rolls back — the caller's outer transaction state
+ # is untouched.
+ # 3. The savepoint scopes ``session.expunge(tracker)``
+ # below: the pending INSERT is part of the savepoint's
+ # write-set, so the expunge does not cancel a write
+ # that was about to promote.
+ # Note: the savepoint does NOT prevent the unconditional
+ # ``session.commit()`` at the bottom of this block from
+ # promoting caller-staged rows already in the outer
+ # transaction — SQLAlchemy's savepoint commit only absorbs
+ # the inner writes; the outer commit still flushes
+ # everything. None of the three current call sites stage
+ # rows across the call (see callers in web/routers/library.py);
+ # the discipline is documented at the call sites.
+ # Honesty note on durability: with nothing else staged, the
+ # SAVEPOINT is the first statement in autocommit so its
+ # RELEASE commits immediately (a second connection already
+ # sees the row; a later rollback() is a no-op) — durability
+ # here rests on that driver behavior, not on savepoint
+ # nesting alone.
+ with session.begin_nested():
+ tracker = DownloadTracker(
+ url=resource.url,
+ url_hash=url_hash,
+ first_resource_id=resource.id,
+ is_downloaded=False,
+ )
+ session.add(tracker)
+ # Explicit flush inside the savepoint — the savepoint's
+ # __exit__ does auto-flush its writes into the outer
+ # transaction, but a future SQLAlchemy change or an
+ # autoflush=False caller would silently drop the INSERT
+ # before ``session.expunge(tracker)`` cancels it. The
+ # flush makes the FK-on-first-download invariant explicit
+ # at this site rather than relying on savepoint-exit
+ # side effects. See PR #6564 P1 review.
+ session.flush()
+
+ # Snapshot plain values for the post-download block (#6560, #6564)
+ resource_id_val = resource.id
+ resource_url = resource.url
+
+ # Expunge instances so their loaded attributes do not expire on commit
+ # and accessing them outside the session will not trigger lazy reload.
+ if hasattr(session, "expunge"):
+ session.expunge(resource)
+ session.expunge(tracker)
+
+ # Unconditionally commit to release the connection back to QueuePool (#6564).
+ # The new-tracker INSERT is enclosed in the savepoint above and is
+ # therefore already durable; this commit only closes the outer
+ # transaction and does not promote any caller-staged row.
+ session.commit()
- # Attempt download
- success, skip_reason, status_code = self._download_pdf(
- resource, tracker, session, collection_id
- )
+ # Attempt download outside database session (#6560)
+ success, skip_reason, status_code = self._download_pdf(
+ resource, tracker, None, collection_id
+ )
+ # Open short transaction for post-download queue and retry updates
+ with get_user_db_session(self.username, self.password) as session:
# Record attempt with retry manager for smart failure tracking
self.retry_manager.record_attempt(
- resource_id=resource.id,
+ resource_id=resource_id_val,
result=(success, skip_reason),
status_code=status_code,
- url=resource.url,
+ url=resource_url,
details=skip_reason
or (
"Successfully downloaded" if success else "Download failed"
@@ -730,7 +802,7 @@ class DownloadService:
# Update queue status if exists
queue_entry = (
session.query(LibraryDownloadQueue)
- .filter_by(resource_id=resource_id)
+ .filter_by(resource_id=resource_id_val)
.first()
)
@@ -753,7 +825,7 @@ class DownloadService:
# Get the document that was just created
doc = (
session.query(Document)
- .filter_by(resource_id=resource_id)
+ .filter_by(resource_id=resource_id_val)
.order_by(Document.created_at.desc())
.first()
)
@@ -786,14 +858,17 @@ class DownloadService:
# DB work off-thread, so neither dirties this session.)
safe_rollback(session, "download_resource auto-index")
logger.exception("Failed to trigger auto-indexing")
+ finally:
+ # Release any checked-out connection from the auto-index query
+ session.commit()
- return success, skip_reason
+ return success, skip_reason
def _download_pdf(
self,
resource: ResearchResource,
tracker: DownloadTracker,
- session: Session,
+ session: Optional[Session] = None,
collection_id: Optional[str] = None,
) -> Tuple[bool, Optional[str], Optional[int]]:
"""
@@ -802,13 +877,19 @@ class DownloadService:
Args:
resource: The research resource to download
tracker: Download tracker for this URL
- session: Database session
+ session: Database session (optional; if None, short transaction opened after download)
collection_id: Optional target collection ID (defaults to Library if not provided)
Returns:
Tuple of (success: bool, skip_reason: Optional[str], status_code: Optional[int])
"""
url = resource.url
+ # Snapshot the title before any session work. When the caller passes
+ # session=None this instance is detached, and its columns are expired
+ # by the caller's own commit, so the value has to be read while it is
+ # still loaded. Canonical text production below uses it for its log
+ # lines and runs with no session open.
+ resource_title = resource.title
# Egress policy gate at the network-fire point. The entry caller
# (download_resource) does not gate, so this is the true PEP — a
@@ -821,18 +902,10 @@ class DownloadService:
)
return False, f"egress_policy_denied:{reason}", None
- # Log attempt
- attempt = DownloadAttempt(
- url_hash=tracker.url_hash,
- attempt_number=tracker.download_attempts.count() + 1
- if hasattr(tracker, "download_attempts")
- else 1,
- attempted_at=datetime.now(UTC),
- )
- session.add(attempt)
-
+ attempted_at = datetime.now(UTC)
try:
- # Use modular downloaders with detailed skip reasons
+ # Use modular downloaders with detailed skip reasons.
+ # Download occurs outside the database session (#6560).
pdf_content = None
selected_downloader = None
skip_reason = None
@@ -865,300 +938,404 @@ class DownloadService:
logger.error(f"No downloader found for {url}")
skip_reason = "No compatible downloader available"
- if not pdf_content:
- error_msg = skip_reason or "Failed to download PDF content"
- # Store skip reason in attempt for retrieval
- attempt.error_message = error_msg
- attempt.succeeded = False
- session.commit()
- logger.info(f"Download failed with reason: {error_msg}")
- return False, error_msg, status_code
-
# Produce the canonical text BEFORE the persistence block.
# download_text_with_source may issue a network (HTML) request,
# and holding an uncommitted write transaction (Document row +
# save_pdf blob) open across that round-trip extends the write
- # lock for the full network latency. Producing text first keeps
- # the PDF persistence a short transaction that commits before
- # the best-effort text save opens. Text production and
- # persistence both stay best-effort: a failure in either must
- # not fail the PDF download itself (see the commit and the
- # guard around _save_text_with_db below).
+ # lock for the full network latency. No session is open here at
+ # all, so that request also cannot sit on a checked-out
+ # connection (#6560). Production and persistence both stay
+ # best-effort: a failure in either must not fail the PDF
+ # download itself.
text_details: tuple[str, str, str] | None = None
- try:
- if isinstance(selected_downloader, ArxivDownloader):
- logger.info(
- f"Producing canonical arXiv text for: {resource.title[:50]}"
- )
- arxiv_text = selected_downloader.download_text_with_source(
- url,
- pdf_content=pdf_content,
- )
- if arxiv_text is not None:
- extraction_method, extraction_source = (
- _arxiv_text_provenance(arxiv_text.source)
+ if pdf_content:
+ try:
+ if isinstance(selected_downloader, ArxivDownloader):
+ logger.info(
+ f"Producing canonical arXiv text for: {resource_title[:50]}"
)
- text_details = (
- arxiv_text.text,
- extraction_method,
- extraction_source,
+ arxiv_text = (
+ selected_downloader.download_text_with_source(
+ url,
+ pdf_content=pdf_content,
+ )
)
- else:
- logger.info(
- f"Extracting text from downloaded PDF for: {resource.title[:50]}"
+ if arxiv_text is not None:
+ extraction_method, extraction_source = (
+ _arxiv_text_provenance(arxiv_text.source)
+ )
+ text_details = (
+ arxiv_text.text,
+ extraction_method,
+ extraction_source,
+ )
+ else:
+ logger.info(
+ f"Extracting text from downloaded PDF for: {resource_title[:50]}"
+ )
+ text = self._extract_text_from_pdf(pdf_content)
+ if text:
+ text_details = (text, "pdf_extraction", "local_pdf")
+
+ if text_details is None:
+ logger.warning(
+ f"Text production returned empty text for: {resource_title[:50]}"
+ )
+ except Exception:
+ logger.exception(
+ "Failed to produce text after PDF download, "
+ "but PDF download succeeded"
)
- text = self._extract_text_from_pdf(pdf_content)
- if text:
- text_details = (text, "pdf_extraction", "local_pdf")
+ text_details = None
+
+ # Open short transaction only after download completion to stage
+ # the DownloadAttempt and update document records (#6560).
+ session_cm = (
+ nullcontext(session)
+ if session is not None
+ else get_user_db_session(self.username, self.password)
+ )
- if text_details is None:
- logger.warning(
- f"Text production returned empty text for: {resource.title[:50]}"
+ with session_cm as sess:
+ if session is None:
+ db_resource = sess.get(ResearchResource, resource.id)
+ db_tracker = (
+ sess.query(DownloadTracker)
+ .filter_by(url_hash=tracker.url_hash)
+ .first()
+ )
+ target_resource = (
+ db_resource if db_resource is not None else resource
+ )
+ target_tracker = (
+ db_tracker if db_tracker is not None else tracker
+ )
+ else:
+ target_resource = resource
+ target_tracker = tracker
+
+ # Snapshot scalar resource values used AFTER the staging commit
+ # so attribute access does not lazy-refresh a checked-out
+ # connection through the staging session immediately before
+ # text production's network/CPU work. PR #6564 P2 review.
+ # Only the title is read in log lines after the staging
+ # commit; the freshly fetched `db_resource` below already has
+ # its scalar columns eagerly loaded, so passing it to
+ # `_save_text_with_db` / `get_document_for_resource` is
+ # safe (those read `id` and `document_id`, never lazy-loaded).
+ staging_resource_title = target_resource.title
+
+ # Helper for attempt number calculation
+ if hasattr(target_tracker, "download_attempts") and hasattr(
+ target_tracker.download_attempts, "count"
+ ):
+ attempt_num = target_tracker.download_attempts.count() + 1
+ elif hasattr(tracker, "download_attempts") and hasattr(
+ tracker.download_attempts, "count"
+ ):
+ attempt_num = tracker.download_attempts.count() + 1
+ else:
+ attempt_num = 1
+ if not pdf_content:
+ error_msg = skip_reason or "Failed to download PDF content"
+ attempt = DownloadAttempt(
+ url_hash=tracker.url_hash,
+ attempt_number=attempt_num,
+ attempted_at=attempted_at,
+ succeeded=False,
+ error_message=error_msg,
+ )
+ sess.add(attempt)
+ # Record the failure in its own transaction so the row
+ # survives whatever the caller does with the session
+ # after this early return.
+ sess.commit()
+ logger.info(f"Download failed with reason: {error_msg}")
+ return False, error_msg, status_code
+
+ # Get PDF storage mode setting. Resolve through the operator gate
+ # BEFORE any PDF is written: a stored value or an
+ # LDR_RESEARCH_LIBRARY_PDF_STORAGE_MODE env override could still
+ # read "filesystem" even though the settings UI withholds that
+ # option, so the unencrypted mode is coerced to the encrypted
+ # "database" default here unless the operator opted in. This is
+ # the true enforcement point — hiding the dropdown alone is not
+ # enough.
+ pdf_storage_mode = resolve_pdf_storage_mode(
+ self.settings.get_setting(
+ "research_library.pdf_storage_mode", "none"
)
- except Exception:
- logger.exception(
- "Failed to produce text after PDF download, "
- "but PDF download succeeded"
- )
- text_details = None
-
- # Get PDF storage mode setting. Resolve through the operator gate
- # BEFORE any PDF is written: a stored value or an
- # LDR_RESEARCH_LIBRARY_PDF_STORAGE_MODE env override could still
- # read "filesystem" even though the settings UI withholds that
- # option, so the unencrypted mode is coerced to the encrypted
- # "database" default here unless the operator opted in. This is
- # the true enforcement point — hiding the dropdown alone is not
- # enough.
- pdf_storage_mode = resolve_pdf_storage_mode(
- self.settings.get_setting(
- "research_library.pdf_storage_mode", "none"
)
- )
- max_pdf_size_mb = int(
- self.settings.get_setting(
- "research_library.max_pdf_size_mb",
- DEFAULT_MAX_PDF_SIZE_MB,
+ max_pdf_size_mb = int(
+ self.settings.get_setting(
+ "research_library.max_pdf_size_mb",
+ DEFAULT_MAX_PDF_SIZE_MB,
+ )
)
- )
- logger.info(
- f"[DOWNLOAD_SERVICE] PDF storage mode: {pdf_storage_mode}"
- )
-
- # Update tracker
- import hashlib
-
- tracker.file_hash = hashlib.sha256(pdf_content).hexdigest()
- tracker.file_size = len(pdf_content)
- tracker.is_downloaded = True
- tracker.downloaded_at = datetime.now(UTC)
-
- # Initialize PDF storage manager. Writes go to the per-user root;
- # legacy_root gives read fallback for pre-isolation files (#5521).
- pdf_storage_manager = PDFStorageManager(
- library_root=self.library_root,
- storage_mode=pdf_storage_mode,
- max_pdf_size_mb=max_pdf_size_mb,
- legacy_root=self.legacy_library_root,
- )
-
- # Update attempt with success info
- attempt.succeeded = True
-
- # Check if library document already exists
- existing_doc = get_document_for_resource(session, resource)
-
- if existing_doc:
- # Update existing document. Only replace document_hash when
- # transitioning from FAILED — that's the placeholder hash from
- # _record_failed_text_extraction. For any other prior state
- # the hash is already a real content hash and clobbering it
- # risks UNIQUE-constraint collisions (issue #3827).
- was_failed = existing_doc.status == DocumentStatus.FAILED
- if was_failed:
- existing_doc.document_hash = tracker.file_hash
- existing_doc.file_size = len(pdf_content)
- existing_doc.status = DocumentStatus.COMPLETED
- existing_doc.processed_at = datetime.now(UTC)
-
- # Save PDF using storage manager (updates storage_mode and file_path)
- file_path_result, _ = pdf_storage_manager.save_pdf(
- pdf_content=pdf_content,
- document=existing_doc,
- session=session,
- filename=f"{resource.id}.pdf",
- url=url,
- resource_id=resource.id,
+ logger.info(
+ f"[DOWNLOAD_SERVICE] PDF storage mode: {pdf_storage_mode}"
)
# Update tracker
- tracker.file_path = (
- file_path_result if file_path_result else None
+ import hashlib
+
+ file_hash_val = hashlib.sha256(pdf_content).hexdigest()
+ file_size_val = len(pdf_content)
+ downloaded_at_val = datetime.now(UTC)
+
+ target_tracker.file_hash = file_hash_val
+ target_tracker.file_size = file_size_val
+ target_tracker.is_downloaded = True
+ target_tracker.downloaded_at = downloaded_at_val
+ if hasattr(tracker, "file_hash"):
+ tracker.file_hash = file_hash_val
+ tracker.file_size = file_size_val
+ tracker.is_downloaded = True
+ tracker.downloaded_at = downloaded_at_val
+
+ # Initialize PDF storage manager. Writes go to the per-user root;
+ # legacy_root gives read fallback for pre-isolation files (#5521).
+ pdf_storage_manager = PDFStorageManager(
+ library_root=self.library_root,
+ storage_mode=pdf_storage_mode,
+ max_pdf_size_mb=max_pdf_size_mb,
+ legacy_root=self.legacy_library_root,
)
- tracker.file_name = (
- Path(file_path_result).name
- if file_path_result and file_path_result != "database"
- else None
+
+ # Update attempt with success info
+ attempt = DownloadAttempt(
+ url_hash=tracker.url_hash,
+ attempt_number=attempt_num,
+ attempted_at=attempted_at,
+ succeeded=True,
+ bytes_downloaded=len(pdf_content),
)
- else:
- # Get source type ID for research downloads
- try:
- source_type_id = get_source_type_id(
- self.username, "research_download", self.password
+ sess.add(attempt)
+
+ # Check if library document already exists
+ existing_doc = get_document_for_resource(sess, target_resource)
+
+ if existing_doc:
+ # Update existing document. Only replace document_hash when
+ # transitioning from FAILED — that's the placeholder hash from
+ # _record_failed_text_extraction. For any other prior state
+ # the hash is already a real content hash and clobbering it
+ # risks UNIQUE-constraint collisions (issue #3827).
+ was_failed = existing_doc.status == DocumentStatus.FAILED
+ if was_failed:
+ existing_doc.document_hash = target_tracker.file_hash
+ existing_doc.file_size = len(pdf_content)
+ existing_doc.status = DocumentStatus.COMPLETED
+ existing_doc.processed_at = datetime.now(UTC)
+
+ # Save PDF using storage manager (updates storage_mode and file_path)
+ file_path_result, _ = pdf_storage_manager.save_pdf(
+ pdf_content=pdf_content,
+ document=existing_doc,
+ session=sess,
+ filename=f"{target_resource.id}.pdf",
+ url=url,
+ resource_id=target_resource.id,
)
- # Use provided collection_id or default to Library
- library_collection_id = (
- collection_id
- or get_default_library_id(self.username, self.password)
+
+ # Update tracker
+ target_tracker.file_path = (
+ file_path_result if file_path_result else None
)
- except Exception:
- logger.exception(
- "Failed to get source type or library collection"
+ target_tracker.file_name = (
+ Path(file_path_result).name
+ if file_path_result and file_path_result != "database"
+ else None
)
- raise
-
- # Create new unified document entry
- doc_id = str(uuid.uuid4())
- doc = Document(
- id=doc_id,
- source_type_id=source_type_id,
- resource_id=resource.id,
- research_id=resource.research_id,
- document_hash=tracker.file_hash,
- original_url=url,
- file_size=len(pdf_content),
- file_type="pdf",
- mime_type="application/pdf",
- title=resource.title,
- status=DocumentStatus.COMPLETED,
- processed_at=datetime.now(UTC),
- storage_mode=pdf_storage_mode,
- )
- session.add(doc)
- session.flush() # Ensure doc.id is available for blob storage
+ if hasattr(tracker, "file_path"):
+ tracker.file_path = target_tracker.file_path
+ tracker.file_name = target_tracker.file_name
+ else:
+ # Get source type ID for research downloads
+ try:
+ source_type_id = get_source_type_id(
+ self.username, "research_download", self.password
+ )
+ # Use provided collection_id or default to Library
+ library_collection_id = (
+ collection_id
+ or get_default_library_id(
+ self.username, self.password
+ )
+ )
+ except Exception:
+ logger.exception(
+ "Failed to get source type or library collection"
+ )
+ raise
- # Save PDF using storage manager (updates storage_mode and file_path)
- file_path_result, _ = pdf_storage_manager.save_pdf(
- pdf_content=pdf_content,
- document=doc,
- session=session,
- filename=f"{resource.id}.pdf",
- url=url,
- resource_id=resource.id,
- )
+ # Create new unified document entry
+ doc_id = str(uuid.uuid4())
+ doc = Document(
+ id=doc_id,
+ source_type_id=source_type_id,
+ resource_id=target_resource.id,
+ research_id=target_resource.research_id,
+ document_hash=target_tracker.file_hash,
+ original_url=url,
+ file_size=len(pdf_content),
+ file_type="pdf",
+ mime_type="application/pdf",
+ title=target_resource.title,
+ status=DocumentStatus.COMPLETED,
+ processed_at=datetime.now(UTC),
+ storage_mode=pdf_storage_mode,
+ )
+ sess.add(doc)
+ sess.flush() # Ensure doc.id is available for blob storage
+ # Save PDF using storage manager (updates storage_mode and file_path)
+ file_path_result, _ = pdf_storage_manager.save_pdf(
+ pdf_content=pdf_content,
+ document=doc,
+ session=sess,
+ filename=f"{target_resource.id}.pdf",
+ url=url,
+ resource_id=target_resource.id,
+ )
- # Update tracker
- tracker.file_path = (
- file_path_result if file_path_result else None
- )
- tracker.file_name = (
- Path(file_path_result).name
- if file_path_result and file_path_result != "database"
- else None
- )
+ # Update tracker
+ target_tracker.file_path = (
+ file_path_result if file_path_result else None
+ )
+ target_tracker.file_name = (
+ Path(file_path_result).name
+ if file_path_result and file_path_result != "database"
+ else None
+ )
+ if hasattr(tracker, "file_path"):
+ tracker.file_path = target_tracker.file_path
+ tracker.file_name = target_tracker.file_name
- # Link document to default Library collection
- ensure_in_collection(session, doc_id, library_collection_id)
-
- # Commit the PDF persistence as its own transaction BEFORE the
- # best-effort text save below. Until this point the attempt
- # row, the Document row, the PDF blob, and the collection
- # link are all uncommitted — the only commit used to be the
- # caller's, at the end of download_resource. _save_text_with_db
- # rolls the whole session back before re-raising (issue
- # #3827), so a DB error in the text save would have discarded
- # every one of those writes while the caller went on to mark
- # the queue COMPLETED and record a retry success — the user is
- # told success and the database keeps nothing. Committing here
- # bounds that rollback to the text changes. download_resource's
- # later commit stays correct: it simply opens a fresh
- # transaction and persists the retry/queue bookkeeping that
- # follows this call.
- session.commit()
+ # Link document to default Library collection
+ ensure_in_collection(sess, doc_id, library_collection_id)
- if text_details is not None:
- # Text persistence stays best-effort, mirroring the
- # production step above. _save_text_with_db re-raises
- # after its own full-session rollback (issue #3827); with
- # the PDF transaction already committed above, that
- # rollback can no longer discard the PDF persistence. The
- # re-raised DB error (IntegrityError on autoflush,
- # OperationalError, ...) is still a local text-save
- # problem that must not fail an already-successful PDF
- # download, which is what the outer except would do: flip
- # tracker.is_accessible to a false 404 signal and consume
- # the retry budget. Catch, log, and keep the successful
- # PDF outcome; the text is re-extracted on a later pass.
- try:
- text, extraction_method, extraction_source = text_details
- # Get the document ID we just created/updated
- pdf_doc = get_document_for_resource(session, resource)
- pdf_document_id = pdf_doc.id if pdf_doc else None
-
- # Save text to encrypted database
- self._save_text_with_db(
- resource=resource,
- text=text,
- session=session,
- extraction_method=extraction_method,
- extraction_source=extraction_source,
- pdf_document_id=pdf_document_id,
+ if pdf_storage_mode == "database":
+ logger.info(
+ f"Successfully stored PDF in database: {resource.url}"
)
+ elif pdf_storage_mode == "filesystem":
+ logger.info(f"Successfully downloaded: {tracker.file_path}")
+ else:
logger.info(
- f"Successfully produced and saved text for: {resource.title[:50]}"
+ f"Successfully extracted text from: {resource.url}"
)
+
+ # Commit the PDF persistence as its own transaction BEFORE
+ # the best-effort text save below. Until this point the
+ # attempt row, the Document row, the PDF blob and the
+ # collection link are all uncommitted. _save_text_with_db
+ # rolls its session back before re-raising (issue #3827),
+ # so without this commit a DB error in the text save would
+ # discard every one of those writes while the caller went on
+ # to mark the queue COMPLETED and record a retry success —
+ # the user is told success and the database keeps nothing.
+ # The caller's later commit stays correct: it opens a fresh
+ # transaction for the retry/queue bookkeeping that follows.
+ sess.commit()
+
+ # Save the produced text only after the PDF has been stored and
+ # committed, in its own transaction.
+ if text_details is not None:
+ text, extraction_method, extraction_source = text_details
+ text_session_cm = (
+ nullcontext(session)
+ if session is not None
+ else get_user_db_session(self.username, self.password)
+ )
+ try:
+ with text_session_cm as text_sess:
+ pdf_doc = get_document_for_resource(
+ text_sess, target_resource
+ )
+ pdf_document_id = pdf_doc.id if pdf_doc else None
+
+ # Save text to encrypted database
+ self._save_text_with_db(
+ resource=target_resource,
+ text=text,
+ session=text_sess,
+ extraction_method=extraction_method,
+ extraction_source=extraction_source,
+ pdf_document_id=pdf_document_id,
+ )
+ if session is None:
+ text_sess.commit()
+ logger.info(
+ f"Successfully produced and saved text for: {staging_resource_title[:50]}"
+ )
except Exception:
logger.exception(
- "Failed to save text after PDF download, "
- "but PDF download succeeded"
+ "Failed to save text after PDF download, but PDF download succeeded"
)
- # Update attempt
- attempt.succeeded = True
- attempt.bytes_downloaded = len(pdf_content)
-
- if pdf_storage_mode == "database":
- logger.info(
- f"Successfully stored PDF in database: {resource.url}"
- )
- elif pdf_storage_mode == "filesystem":
- logger.info(f"Successfully downloaded: {tracker.file_path}")
- else:
- logger.info(f"Successfully extracted text from: {resource.url}")
-
return True, None, status_code
except Exception as e:
logger.exception(f"Download failed for {url}")
- # The error may have come from a failed flush/commit in the
- # success path above (e.g. save_pdf / ensure_in_collection),
- # leaving the *shared* session in PendingRollbackError. The caller
- # (download_resource) keeps using this same session and commits
- # again, so roll back here or that commit cascades. This swallow-
- # and-return path is invisible to get_user_db_session's own
- # rollback (we don't propagate), hence the explicit recovery.
- safe_rollback(session, "_download_pdf")
+ # Two distinct outputs:
+ # * server-side diagnostic (persisted in DownloadAttempt.error_message
+ # and emitted via logger.exception above) — full str(e), with
+ # credential scrubbing applied so secrets don't reach the DB.
+ # * caller-safe skip_reason (the second tuple element, and what
+ # flows into retry-manager / bulk-stream SSE messages) — must
+ # never carry SQL text, target URLs, or file paths. The Zotero
+ # sync_service uses the same shape: look up str(exc) in a fixed
+ # map of author-written constants and otherwise return just the
+ # exception class name. sanitize_error_for_client() is not a
+ # substitute — its docstring says it scrubs credentials only.
+ server_error = sanitize_error_for_client(str(e))
+ skip_reason = _client_safe_download_message(e)
tracker.is_accessible = False
- # request/network errors can echo a resource URL carrying an
- # api_key/token or user:pass@host — scrub before returning. The
- # message is surfaced to the browser via the download SSE stream.
- safe_error = sanitize_error_for_client(str(e))
- # The rollback discarded the pending ``attempt`` row added above,
- # so re-record the failed attempt on the now-clean session for
- # download_resource to commit.
- session.add(
- DownloadAttempt(
- url_hash=tracker.url_hash,
- attempt_number=tracker.download_attempts.count() + 1
- if hasattr(tracker, "download_attempts")
- else 1,
- attempted_at=datetime.now(UTC),
- succeeded=False,
- error_type=type(e).__name__,
- error_message=safe_error,
+ if session is not None:
+ safe_rollback(session, "_download_pdf")
+ session.add(
+ DownloadAttempt(
+ url_hash=tracker.url_hash,
+ attempt_number=tracker.download_attempts.count() + 1
+ if hasattr(tracker, "download_attempts")
+ else 1,
+ attempted_at=datetime.now(UTC),
+ succeeded=False,
+ error_type=type(e).__name__,
+ error_message=server_error,
+ )
)
- )
- return False, safe_error, None
+ else:
+ try:
+ with get_user_db_session(
+ self.username, self.password
+ ) as err_sess:
+ db_tracker = (
+ err_sess.query(DownloadTracker)
+ .filter_by(url_hash=tracker.url_hash)
+ .first()
+ )
+ if db_tracker:
+ db_tracker.is_accessible = False
+ err_sess.add(
+ DownloadAttempt(
+ url_hash=tracker.url_hash,
+ attempt_number=tracker.download_attempts.count()
+ + 1
+ if hasattr(tracker, "download_attempts")
+ else 1,
+ attempted_at=datetime.now(UTC),
+ succeeded=False,
+ error_type=type(e).__name__,
+ error_message=server_error,
+ )
+ )
+ err_sess.commit()
+ except Exception:
+ logger.exception(
+ "Failed to record failed attempt in database"
+ )
+ return False, skip_reason, None
def _extract_text_from_pdf(self, pdf_content: bytes) -> Optional[str]:
"""
@@ -1684,9 +1861,12 @@ class DownloadService:
except Exception as exc:
safe_rollback(session, "_try_arxiv_text_extraction")
logger.exception("Canonical arXiv text extraction failed")
- safe_error = sanitize_error_for_client(
- f"Failed to extract arXiv text: {exc}"
- )
+ # Caller-visible: fixed token only. sanitize_error_for_client()
+ # scrubs credentials but leaves SQL text / URLs / paths intact,
+ # and this string flows into download_as_text's return and the
+ # bulk-stream SSE `error` field — route through the same mapping
+ # as _download_pdf (PR #6564 follow-up).
+ safe_error = _client_safe_download_message(exc)
return False, safe_error
if arxiv_text is None:
@@ -1708,9 +1888,9 @@ class DownloadService:
except Exception as exc:
safe_rollback(session, "_try_arxiv_text_extraction")
logger.exception("Failed to save canonical arXiv text")
- safe_error = sanitize_error_for_client(
- f"Failed to save text: {exc}"
- )
+ # Same caller-safe mapping as above — never str(exc) (see comment
+ # on the fetch-failure path).
+ safe_error = _client_safe_download_message(exc)
return False, safe_error
return True, None
@@ -1924,10 +2104,10 @@ class DownloadService:
# after we return, so it must be clean.
safe_rollback(session, "_try_api_text_extraction")
logger.exception(f"Failed to save text for resource {resource.id}")
- # Sanitize error message before returning to API
- safe_error = sanitize_error_for_client(
- f"Failed to save text: {str(e)}"
- )
+ # Caller-visible: fixed token only (CWE-209). sanitize_error_for_client
+ # leaves SQL text intact and this flows into download_as_text's
+ # return and the bulk-stream SSE `error` field.
+ safe_error = _client_safe_download_message(e)
return False, safe_error
def _fallback_pdf_extraction(
@@ -2010,10 +2190,9 @@ class DownloadService:
# we return.
safe_rollback(session, "_fallback_pdf_extraction")
logger.exception(f"Failed to save text for resource {resource.id}")
- # Sanitize error message before returning to API
- safe_error = sanitize_error_for_client(
- f"Failed to save text: {str(e)}"
- )
+ # Caller-visible: fixed token only (CWE-209) — same mapping as
+ # _download_pdf; never str(e).
+ safe_error = _client_safe_download_message(e)
return False, safe_error
def _get_downloader(self, url: str):
@@ -2403,7 +2582,11 @@ class DownloadService:
else:
extraction_quality = "low"
- # Find the document by pdf_document_id or resource_id
+ # Find the document by pdf_document_id or resource_id. Both reads
+ # touch scalar columns (`document_id`, `id`) that are eagerly
+ # loaded when `resource` was queried, so passing a detached
+ # `resource` from a snapshot block (post-staging-commit) is safe
+ # and does not lazy-refresh through this session. PR #6564 P2.
doc = None
if pdf_document_id:
doc = (
diff --git a/src/local_deep_research/security/__init__.py b/src/local_deep_research/security/__init__.py
index ec6c77252b..3b86cce517 100644
--- a/src/local_deep_research/security/__init__.py
+++ b/src/local_deep_research/security/__init__.py
@@ -46,6 +46,10 @@ from .log_sanitizer import (
strip_control_chars,
)
from .filename_sanitizer import sanitize_filename, UnsafeFilenameError
+from .client_safe_errors import (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES,
+ client_safe_download_message,
+)
from .module_whitelist import (
get_safe_module_class,
ModuleNotAllowedError,
@@ -112,6 +116,8 @@ __all__ = [
"sanitize_for_log",
"scrub_error",
"strip_control_chars",
+ "CLIENT_SAFE_DOWNLOAD_MESSAGES",
+ "client_safe_download_message",
"sanitize_filename",
"UnsafeFilenameError",
]
diff --git a/src/local_deep_research/security/client_safe_errors.py b/src/local_deep_research/security/client_safe_errors.py
new file mode 100644
index 0000000000..d60fa13950
--- /dev/null
+++ b/src/local_deep_research/security/client_safe_errors.py
@@ -0,0 +1,81 @@
+"""Caller-safe download skip-reason tokens (single source of truth).
+
+PR #6564 follow-up: ``research_library.services.download_service`` and
+``research_library.downloaders.html`` each carried their own copy of the
+exception-class -> fixed-token table. The copies drifted (``DataError``,
+``InterfaceError``, ``InvalidRequestError`` and the ``OSError`` errno branch
+existed only on the services side), and the "mirrors" comment overstated the
+parity. This module owns the table and the mapping function; both call sites
+import from here so they cannot drift again.
+
+Contract (CWE-209): never pass ``str(exc)`` to the caller. The exception text
+can carry SQL text (``INSERT INTO document_collections ...``), target URLs,
+or absolute file paths. Caller-visible ``skip_reason`` values are fixed
+tokens or ``download_error:<ClassName>`` only. Full diagnostics stay
+server-side via ``logger.exception`` and ``DownloadAttempt.error_message``.
+``sanitize_error_for_client()`` is NOT a substitute — it scrubs credential
+shapes only, not SQL/paths/URLs.
+"""
+
+import os
+
+# Client-safe skip-reason mapping for download exception paths.
+#
+# sanitize_error_for_client() scrubs credential *shapes* but explicitly does
+# NOT remove SQL text, provider endpoints, or dependency internals. The
+# exception's text here can be a SQLAlchemy IntegrityError
+# ("...INSERT INTO document_collections..."), a requests ConnectionError
+# carrying the target host, or a PDF library message with absolute paths.
+CLIENT_SAFE_DOWNLOAD_MESSAGES: dict[str, str] = {
+ # Network surface — never echo the URL / host the caller already knows.
+ "ConnectionError": "network_unavailable",
+ "ConnectionResetError": "network_reset",
+ "Timeout": "network_timeout",
+ "HTTPError": "http_error",
+ "ChunkedEncodingError": "network_unavailable",
+ "ContentDecodingError": "network_unavailable",
+ "TooManyRedirects": "network_redirect_limit",
+ # SQL surface — never echo SQL text, table names, or driver internals.
+ # Maps to the SQLAlchemy class names; the IntegrityError case is the
+ # one the FK-ON regression surfaced ("FOREIGN KEY constraint failed
+ # [SQL: INSERT INTO document_collections ...]").
+ "IntegrityError": "database_constraint",
+ "OperationalError": "database_unavailable",
+ "DatabaseError": "database_error",
+ "DataError": "database_data_error",
+ "InterfaceError": "database_interface_error",
+ "InvalidRequestError": "database_invalid_request",
+ # PDF / extraction surface — never echo absolute paths or parser internals.
+ "PdfReadError": "pdf_parse_failed",
+ "PdfStreamError": "pdf_parse_failed",
+}
+
+
+def client_safe_download_message(exc: BaseException) -> str:
+ """Return a caller-safe skip_reason for *exc*.
+
+ Author-written constant for known classes; the exception's class name
+ for everything else. Never passes ``str(exc)`` through — that can carry
+ SQL text, target URLs, or file paths the caller should not see. The full
+ exception text is preserved server-side via ``logger.exception`` and
+ stored in ``DownloadAttempt.error_message`` only after the same safe
+ mapping (see ``download_service._download_pdf``'s outer except).
+
+ ``OSError`` is the one exception family where ``str(exc)`` is *not*
+ free of app-controlled text — Python formats it as
+ ``"[Errno N] <strerror>: <filename>"`` and the filename field is
+ arbitrary. We can't safely pass ``str(exc)`` through. What we *can*
+ safely surface is ``exc.strerror`` when it matches the OS-level
+ canonical message (i.e. ``os.strerror(exc.errno)``). That string is
+ provided by the OS — not by the application — so it carries no SQL,
+ URL, or path content. Custom constructions like
+ ``OSError(28, "leak: secret file contents")`` deliberately don't match the
+ canonical message and fall through to the class-name default.
+ """
+ if isinstance(exc, OSError) and exc.errno is not None:
+ canonical = os.strerror(exc.errno)
+ if exc.strerror and exc.strerror == canonical:
+ return f"filesystem_error:{exc.strerror}"
+ return CLIENT_SAFE_DOWNLOAD_MESSAGES.get(
+ type(exc).__name__, f"download_error:{type(exc).__name__}"
+ )
diff --git a/src/local_deep_research/web/routers/library.py b/src/local_deep_research/web/routers/library.py
index 652b6700cf..3d3bb67aa4 100644
--- a/src/local_deep_research/web/routers/library.py
+++ b/src/local_deep_research/web/routers/library.py
@@ -915,8 +915,10 @@ def download_all_text(
# Its every error-return path (see download_service.py
# download_as_text/_try_api_text_extraction/
# _fallback_pdf_extraction) is either a static string or
- # already scrubbed via sanitize_error_for_client() — no
- # raw exception text reaches `error` here.
+ # a fixed token from _client_safe_download_message()
+ # (never str(e) — sanitize_error_for_client() scrubs
+ # credentials only and leaves SQL text intact) — no raw
+ # exception text reaches `error` here.
success, error = download_service.download_as_text(rid)
if success:
@@ -958,8 +960,9 @@ def download_all_text(
# CWE-209 (CodeQL "Information exposure through an exception"): the SSE
# body streamed here never carries a raw exception message — see the
# `except Exception as e:` block in generate() above, which only ever
- # surfaces `type(e).__name__` or an already-scrubbed string from
- # download_service.
+ # surfaces `type(e).__name__` or a fixed token from
+ # download_service._client_safe_download_message (sanitize_error_for_client
+ # alone would not suffice — it scrubs credentials only, not SQL/paths).
return WorkerCleanupStreamingResponse(
generate(),
media_type="text/event-stream",
@@ -1397,8 +1400,10 @@ async def download_bulk(
# CWE-209 (CodeQL "Information exposure through an exception"): the SSE
# body streamed here never carries a raw exception message. `skip_reason`
# either comes straight from download_service (already a static string or
- # sanitize_error_for_client()-scrubbed — see download_resource /
- # download_as_text and their helpers in download_service.py) or, on the
+ # a fixed token from _client_safe_download_message() — see
+ # download_resource / download_as_text and their helpers in
+ # download_service.py; sanitize_error_for_client() alone would not
+ # suffice since it scrubs credentials only, not SQL/paths) or, on the
# `except Exception as e:` path above, is built from `type(e).__name__`
# only — never the raw `error_msg = str(e)`.
return WorkerCleanupStreamingResponse(
diff --git a/tests/library/download_management/test_failure_classifier_tokens_6564.py b/tests/library/download_management/test_failure_classifier_tokens_6564.py
new file mode 100644
index 0000000000..97ea83b5d4
--- /dev/null
+++ b/tests/library/download_management/test_failure_classifier_tokens_6564.py
@@ -0,0 +1,106 @@
+"""PR #6564 optional #2: classifier knows the fixed-token vocabulary."""
+
+from datetime import timedelta
+from unittest.mock import patch
+
+from local_deep_research.library.download_management.failure_classifier import (
+ FailureClassifier,
+ TemporaryFailure,
+)
+import local_deep_research.library.download_management.retry_manager as rm_module
+
+
+class TestTokenVocabulary:
+ def setup_method(self):
+ self.c = FailureClassifier()
+
+ def test_network_timeout_restores_30min(self):
+ f = self.c.classify_failure("str", details="network_timeout")
+ assert isinstance(f, TemporaryFailure)
+ assert f.error_type == "timeout"
+ assert f.retry_after == timedelta(minutes=30)
+
+ def test_network_tokens_map_to_5min(self):
+ for token in ("network_unavailable", "network_reset"):
+ f = self.c.classify_failure("str", details=token)
+ assert f.error_type == "network_error", token
+ assert f.retry_after == timedelta(minutes=5), token
+
+ def test_database_tokens_classified_without_warning(self):
+ for token in (
+ "database_constraint",
+ "database_unavailable",
+ "database_error",
+ "database_data_error",
+ "database_interface_error",
+ "database_invalid_request",
+ ):
+ f = self.c.classify_failure("str", details=token)
+ assert f.error_type == token, token
+ assert f.retry_after == timedelta(hours=1), token
+
+ def test_download_error_unwraps_class_name(self):
+ f = self.c.classify_failure("str", details="download_error:Timeout")
+ assert f.error_type == "timeout"
+ assert f.retry_after == timedelta(minutes=30)
+
+ f = self.c.classify_failure(
+ "str", details="download_error:ConnectionError"
+ )
+ assert f.error_type == "network_error"
+ assert f.retry_after == timedelta(minutes=5)
+
+ def test_filesystem_error_prefix(self):
+ f = self.c.classify_failure(
+ "str", details="filesystem_error:No such file or directory"
+ )
+ assert f.error_type == "filesystem_error"
+ assert f.retry_after == timedelta(hours=1)
+
+ def test_error_type_field_also_recognised(self):
+ f = self.c.classify_failure("network_timeout")
+ assert f.error_type == "timeout"
+ assert f.retry_after == timedelta(minutes=30)
+
+
+class TestRetryManagerPassthrough:
+ def test_token_passed_as_error_type(self):
+ seen = {}
+
+ class FakeClassifier:
+ def classify_failure(self, error_type=None, **kwargs):
+ seen["error_type"] = error_type
+ seen["details"] = kwargs.get("details")
+ return TemporaryFailure("x", "y", timedelta(minutes=5))
+
+ with (
+ patch.object(rm_module, "ResourceStatusTracker"),
+ patch.object(
+ rm_module, "FailureClassifier", return_value=FakeClassifier()
+ ),
+ ):
+ m = rm_module.RetryManager("u")
+ m.record_attempt(
+ 1, (False, "network_timeout"), details="network_timeout"
+ )
+ assert seen["error_type"] == "network_timeout"
+
+ def test_free_text_keeps_str_shape(self):
+ seen = {}
+
+ class FakeClassifier:
+ def classify_failure(self, error_type=None, **kwargs):
+ seen["error_type"] = error_type
+ return TemporaryFailure("x", "y", timedelta(minutes=5))
+
+ with (
+ patch.object(rm_module, "ResourceStatusTracker"),
+ patch.object(
+ rm_module, "FailureClassifier", return_value=FakeClassifier()
+ ),
+ ):
+ m = rm_module.RetryManager("u")
+ m.record_attempt(
+ 1, (False, "Request timed out"), details="Connection timeout"
+ )
+ assert seen["error_type"] == "str"
diff --git a/tests/research_library/downloaders/test_html_downloader_deep_coverage.py b/tests/research_library/downloaders/test_html_downloader_deep_coverage.py
index c93d04bdbe..c298ad4ffc 100644
--- a/tests/research_library/downloaders/test_html_downloader_deep_coverage.py
+++ b/tests/research_library/downloaders/test_html_downloader_deep_coverage.py
@@ -183,19 +183,20 @@ class TestDownloadWithResult:
assert b"Real content" in result.content
def test_exception(self, downloader):
- """Exception returns error skip reason."""
+ """Exception returns fixed-token skip reason (never str(e))."""
with patch.object(
downloader, "_fetch_html", side_effect=ValueError("bad url")
):
result = downloader.download_with_result("https://example.com")
assert result.is_success is False
- assert "Error" in result.skip_reason
+ assert result.skip_reason == "download_error:ValueError"
def test_exception_skip_reason_scrubs_credentials(self, downloader):
"""The exception path's skip_reason propagates to the browser via the
- download SSE stream, so a credential echoed in the error (e.g. a fetch
- URL with an api_key) must be redacted, not leaked. Regression guard
- for the download-service credential-leak fix."""
+ download SSE stream, so exception text (e.g. a fetch URL with an
+ api_key, SQL text, or file paths) must never reach the caller.
+ Regression guard for the download-service CWE-209 fix (PR #6564
+ follow-up): the path returns a fixed token, not a scrubbed message."""
secret = "supersecret1234567890"
boom = ValueError(
f"fetch failed for https://api.example.com/doc?api_key={secret}"
@@ -206,8 +207,8 @@ class TestDownloadWithResult:
)
assert result.is_success is False
assert secret not in result.skip_reason
- # Still an error message, just with the credential masked.
- assert result.skip_reason.startswith("Error:")
+ # Fixed token only — no raw exception text survives.
+ assert result.skip_reason == "download_error:ValueError"
# ---------------------------------------------------------------------------
diff --git a/tests/research_library/services/test_download_service_arxiv_provenance.py b/tests/research_library/services/test_download_service_arxiv_provenance.py
index d439ec6d11..427b74c15e 100644
--- a/tests/research_library/services/test_download_service_arxiv_provenance.py
+++ b/tests/research_library/services/test_download_service_arxiv_provenance.py
@@ -828,10 +828,6 @@ def test_arxiv_typed_exception_is_terminal_and_records_once(service):
"_check_url_against_policy",
return_value=(True, "allowed"),
),
- patch(
- f"{MODULE}.sanitize_error_for_client",
- return_value="arXiv text extraction failed",
- ),
patch(f"{MODULE}.safe_rollback") as rollback,
patch.object(service, "_try_existing_pdf_extraction") as existing_pdf,
patch.object(service, "_try_api_text_extraction") as api_fallback,
@@ -839,12 +835,14 @@ def test_arxiv_typed_exception_is_terminal_and_records_once(service):
):
result = service.download_as_text(resource.id)
- assert result == (False, "arXiv text extraction failed")
+ # Caller-visible reason is a fixed token (PR #6564 follow-up), never
+ # str(exc) — even though the exception text carries a secret shape here.
+ assert result == (False, "download_error:RuntimeError")
service.retry_manager.record_attempt.assert_called_once_with(
resource_id=resource.id,
result=result,
url=resource.url,
- details="arXiv text extraction failed",
+ details="download_error:RuntimeError",
session=session,
)
rollback.assert_called_once_with(session, "_try_arxiv_text_extraction")
diff --git a/tests/research_library/services/test_download_service_coverage.py b/tests/research_library/services/test_download_service_coverage.py
index 210e43332a..e674c7e605 100644
--- a/tests/research_library/services/test_download_service_coverage.py
+++ b/tests/research_library/services/test_download_service_coverage.py
@@ -1233,13 +1233,12 @@ class TestTryApiTextExtraction:
patch.object(
svc, "_save_text_with_db", side_effect=RuntimeError("db error")
),
- patch(
- f"{MODULE}.sanitize_error_for_client", return_value="safe error"
- ),
):
result = svc._try_api_text_extraction(session, resource)
assert result[0] is False
- assert "safe error" in result[1]
+ # Caller-visible reason is a fixed token (PR #6564 follow-up),
+ # never the raw exception text.
+ assert result[1] == "download_error:RuntimeError"
# ============================================================
@@ -1344,10 +1343,11 @@ class TestFallbackPdfExtraction:
patch.object(
svc, "_save_text_with_db", side_effect=RuntimeError("db")
),
- patch(f"{MODULE}.sanitize_error_for_client", return_value="safe"),
):
success, msg = svc._fallback_pdf_extraction(session, resource)
assert success is False
+ # Fixed token, never str(e) (PR #6564 follow-up).
+ assert msg == "download_error:RuntimeError"
def test_download_fails_no_skip_reason(self, svc):
session = MagicMock()
@@ -2152,8 +2152,16 @@ class TestDownloadPdf:
resource, tracker, session
)
assert success is False
- assert reason == "safe error"
+ # Caller-visible reason is the safe mapped token, NOT the
+ # sanitized text (PR #6564 follow-up: caller surfaces must not
+ # carry SQL/dependency internals).
+ assert reason == "download_error:RuntimeError"
assert tracker.is_accessible is False
+ # The persisted DownloadAttempt.error_message keeps the
+ # sanitized diagnostic for server-side debugging.
+ added = session.add.call_args[0][0]
+ assert added.error_message == "safe error"
+ assert added.error_type == "RuntimeError"
def test_text_extraction_failure_doesnt_fail_download(self, svc):
"""Text extraction failure after successful PDF download should not fail the download."""
diff --git a/tests/research_library/services/test_download_service_deep_coverage.py b/tests/research_library/services/test_download_service_deep_coverage.py
index 7ed151a1f8..779bf62403 100644
--- a/tests/research_library/services/test_download_service_deep_coverage.py
+++ b/tests/research_library/services/test_download_service_deep_coverage.py
@@ -777,7 +777,15 @@ class TestDownloadPdfDeep:
engine.dispose()
def test_outer_exception_caught_and_sanitized(self, svc):
- """Exceptions inside the try block are caught and sanitized via sanitize_error_for_client."""
+ """Exceptions inside the try block are caught; the caller-visible reason
+ is a safe mapped token, not the sanitized exception text.
+
+ The persisted DownloadAttempt.error_message keeps the sanitized
+ text so the operator can debug from the row, but the
+ caller-visible reason flows into retry-manager / bulk-stream SSE
+ and must not carry SQL text, dependency internals, or the raw
+ exception class message (PR #6564 follow-up).
+ """
session = MagicMock()
resource = MagicMock()
resource.id = 16
@@ -810,7 +818,14 @@ class TestDownloadPdfDeep:
resource, tracker, session
)
assert success is False
- assert reason == "redacted error"
+ # Caller-visible: safe mapped token, never the sanitized text
+ # (which would carry SQL/dependency internals for non-RuntimeError
+ # exceptions).
+ assert reason == "download_error:RuntimeError"
+ # Server-side: persisted DownloadAttempt keeps the sanitized text.
+ added = session.add.call_args[0][0]
+ assert added.error_message == "redacted error"
+ assert added.error_type == "RuntimeError"
def test_none_storage_mode_logs_text_extraction(self, svc):
"""Storage mode 'none' logs text extraction success message."""
diff --git a/tests/research_library/services/test_download_service_pool_exhaustion_6560.py b/tests/research_library/services/test_download_service_pool_exhaustion_6560.py
new file mode 100644
index 0000000000..762eff9cdf
--- /dev/null
+++ b/tests/research_library/services/test_download_service_pool_exhaustion_6560.py
@@ -0,0 +1,1178 @@
+"""Regression tests for issue #6560: Connection Pool Exhaustion in download_resource.
+
+Validates that:
+1. download_resource does NOT hold a database session during downloader.download_with_result.
+2. Under bulk downloads, QueuePool connections are released across HTTP transfers so concurrent
+ request handlers are not starved.
+3. DownloadAttempt and document records are staged in a short transaction strictly AFTER download completion.
+4. Failed downloads and network exceptions stage attempts without holding or leaking connections.
+"""
+
+import concurrent.futures
+import threading
+import time
+import uuid
+from contextlib import contextmanager
+from unittest.mock import MagicMock, patch
+
+import pytest
+from sqlalchemy import create_engine, event
+from sqlalchemy.orm import sessionmaker
+from sqlalchemy.pool import QueuePool
+
+from local_deep_research.database.models import Base
+from local_deep_research.database.models.download_tracker import (
+ DownloadAttempt,
+ DownloadTracker,
+)
+from local_deep_research.database.models.library import (
+ Collection,
+ Document,
+ DocumentStatus,
+ DownloadQueue, # noqa: F401 - registers table on Base.metadata
+ SourceType,
+)
+from local_deep_research.database.models.research import (
+ ResearchHistory,
+ ResearchResource,
+)
+from local_deep_research.research_library.services.download_service import (
+ DownloadService,
+)
+
+MODULE = "local_deep_research.research_library.services.download_service"
+
+
+def service_stub():
+ """DownloadService stub wired for testing download_resource."""
+ with patch.object(DownloadService, "__init__", lambda self, *a, **kw: None):
+ s = DownloadService.__new__(DownloadService)
+ s.username = "test_user"
+ s.password = "test_pass"
+ s.library_root = "/tmp/ldr-test-library"
+ s.legacy_library_root = None
+ s._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ s.settings = settings
+ s._check_url_against_policy = lambda url: (True, "ok")
+ s.retry_manager = MagicMock()
+ return s
+
+
+def test_download_resource_does_not_hold_db_session_during_download(
+ service_stub,
+):
+ """downloader.download_with_result must execute OUTSIDE any active database session.
+
+ If a database session is held across download_with_result, this test asserts False.
+ """
+ events = []
+ session_active = False
+
+ @contextmanager
+ def tracking_get_user_db_session(*args, **kwargs):
+ nonlocal session_active
+ session_active = True
+ events.append("session_enter")
+ mock_session = MagicMock()
+
+ # Mock query return values
+ mock_session.get.return_value = MagicMock(
+ id=1,
+ url="https://arxiv.org/abs/2401.0001",
+ title="Test Paper",
+ research_id="res-1",
+ document_id=None,
+ )
+ mock_session.query.return_value.filter_by.return_value.first.return_value = None
+ mock_session.query.return_value.filter_by.return_value.order_by.return_value.first.return_value = None
+
+ try:
+ yield mock_session
+ finally:
+ session_active = False
+ events.append("session_exit")
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+
+ def mock_download(url, content_type):
+ events.append("download_start")
+ # CRITICAL ASSERTION: No database session may be active during network transfer
+ assert not session_active, (
+ "Database session was held open during downloader.download_with_result! "
+ "This violates #6560 resolution plan."
+ )
+ events.append("download_end")
+ return MagicMock(
+ is_success=True,
+ content=b"%PDF-1.4 mock content",
+ status_code=200,
+ skip_reason=None,
+ )
+
+ downloader.download_with_result.side_effect = mock_download
+ service_stub.downloaders = [downloader]
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=tracking_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value="src-1"),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+ success, reason = service_stub.download_resource(1)
+ assert success is True
+ assert reason is None
+ # Verify the lifecycle order:
+ # 1. pre-checks session enters then exits
+ # 2. download happens outside session
+ # 3. post-download staging session enters and exits
+ assert events.index("session_exit") < events.index("download_start")
+ assert events.index("download_end") < len(events) - 1
+
+
+def test_queue_pool_exhaustion_prevented_under_bounded_pool():
+ """A bounded QueuePool (size=1) must not be exhausted while a slow download is in-flight.
+
+ Simulates a multi-second HTTP transfer on worker 1, while worker 2 (FastAPI handler)
+ checks out a session from the same pool. In the old code, worker 2 would time out (pool exhausted).
+ With the fix, worker 2 succeeds immediately because worker 1 released its connection.
+ """
+ engine = create_engine(
+ "sqlite:///:memory:",
+ poolclass=QueuePool,
+ pool_size=1,
+ max_overflow=0,
+ pool_timeout=1.0, # 1 second timeout
+ connect_args={"check_same_thread": False},
+ )
+ Base.metadata.create_all(engine)
+ Session = sessionmaker(bind=engine)
+
+ # Seed required rows
+ with Session() as seed_session:
+ rh = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="q",
+ mode="quick",
+ status="completed",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(rh)
+ resource = ResearchResource(
+ research_id=rh.id,
+ title="Test Paper",
+ url="https://example.com/slow_paper.pdf",
+ source_type="academic",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(resource)
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ seed_session.add(source_type)
+ seed_session.commit()
+ resource_id = resource.id
+ source_type_id = source_type.id
+
+ thread_local = threading.local()
+
+ def get_thread_session():
+ if not hasattr(thread_local, "session") or thread_local.session is None:
+ thread_local.session = Session()
+ return thread_local.session
+
+ @contextmanager
+ def production_get_user_db_session(*args, **kwargs):
+ sess = get_thread_session()
+ try:
+ yield sess
+ except Exception:
+ sess.rollback()
+ raise
+
+ service = DownloadService.__new__(DownloadService)
+ service.username = "test_user"
+ service.password = "test_pass"
+ service.library_root = "/tmp/ldr-test-library"
+ service.legacy_library_root = None
+ service._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ service.settings = settings
+ service._check_url_against_policy = lambda url: (True, "ok")
+ service.retry_manager = MagicMock()
+
+ worker2_acquired_connection = False
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+
+ def slow_download(url, content_type):
+ nonlocal worker2_acquired_connection
+ # While download is in-flight on worker 1, simulate worker 2 (FastAPI request handler)
+ # attempting to check out a database session from the pool.
+ # Pool size is 1, so if worker 1 pinned the connection, worker 2 would fail with TimeoutError!
+ try:
+ with production_get_user_db_session() as w2_sess:
+ count = w2_sess.query(ResearchResource).count()
+ assert count == 1
+ w2_sess.commit()
+ worker2_acquired_connection = True
+ except Exception as exc:
+ worker2_acquired_connection = False
+ raise AssertionError(
+ f"Worker 2 starved by worker 1 download: {exc}"
+ ) from exc
+
+ return MagicMock(
+ is_success=True,
+ content=b"%PDF-1.4 downloaded data",
+ status_code=200,
+ skip_reason=None,
+ )
+
+ downloader.download_with_result.side_effect = slow_download
+ service.downloaders = [downloader]
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=production_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value=source_type_id),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+ success, reason = service.download_resource(resource_id)
+
+ assert success is True
+ assert worker2_acquired_connection is True, (
+ "Worker 2 was unable to acquire connection while worker 1 was downloading. "
+ "Connection pool was exhausted."
+ )
+ # Assert item 2: expire_on_commit was not permanently mutated on the shared session
+ assert get_thread_session().expire_on_commit is True
+
+ # Verify attempt and document were properly persisted
+ with Session() as verify_session:
+ attempts = verify_session.query(DownloadAttempt).all()
+ assert len(attempts) == 1
+ assert attempts[0].succeeded is True
+ assert attempts[0].bytes_downloaded == len(b"%PDF-1.4 downloaded data")
+ docs = (
+ verify_session.query(Document)
+ .filter_by(resource_id=resource_id)
+ .all()
+ )
+ assert len(docs) == 1
+ assert docs[0].status == DocumentStatus.COMPLETED
+
+ engine.dispose()
+
+
+def test_queue_pool_exhaustion_prevented_on_retry_with_preexisting_tracker():
+ """A pre-existing DownloadTracker (the retry/re-download path) must not pin the connection (#6564 item 1).
+
+ Drives production session semantics (thread-local session reused without auto-commit or auto-close
+ on context manager exit). Under a size-1 QueuePool, verifies that when tracker already exists,
+ the pre-check transaction still releases its connection so concurrent requests are not starved.
+ Also verifies session.expire_on_commit is not permanently mutated (#6564 item 2).
+ """
+ engine = create_engine(
+ "sqlite:///:memory:",
+ poolclass=QueuePool,
+ pool_size=1,
+ max_overflow=0,
+ pool_timeout=1.0,
+ connect_args={"check_same_thread": False},
+ )
+ Base.metadata.create_all(engine)
+ Session = sessionmaker(bind=engine)
+
+ # Seed required rows including a pre-existing tracker
+ with Session() as seed_session:
+ rh = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="q",
+ mode="quick",
+ status="completed",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(rh)
+ resource = ResearchResource(
+ research_id=rh.id,
+ title="Retry Paper",
+ url="https://example.com/retry_paper.pdf",
+ source_type="academic",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(resource)
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ seed_session.add(source_type)
+ seed_session.commit()
+ resource_id = resource.id
+ source_type_id = source_type.id
+
+ service = DownloadService.__new__(DownloadService)
+ service.username = "test_user"
+ service.password = "test_pass"
+ service.library_root = "/tmp/ldr-test-library"
+ service.legacy_library_root = None
+ service._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ service.settings = settings
+ service._check_url_against_policy = lambda url: (True, "ok")
+ service.retry_manager = MagicMock()
+
+ url_hash = service._get_url_hash("https://example.com/retry_paper.pdf")
+ with Session() as seed_session:
+ tracker = DownloadTracker(
+ url="https://example.com/retry_paper.pdf",
+ url_hash=url_hash,
+ first_resource_id=resource_id,
+ is_downloaded=False,
+ )
+ seed_session.add(tracker)
+ seed_session.commit()
+
+ thread_local = threading.local()
+
+ def get_thread_session():
+ if not hasattr(thread_local, "session") or thread_local.session is None:
+ thread_local.session = Session()
+ return thread_local.session
+
+ @contextmanager
+ def production_get_user_db_session(*args, **kwargs):
+ sess = get_thread_session()
+ try:
+ yield sess
+ except Exception:
+ sess.rollback()
+ raise
+
+ worker2_acquired_connection = False
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+
+ def slow_download(url, content_type):
+ nonlocal worker2_acquired_connection
+
+ def worker2_fn():
+ nonlocal worker2_acquired_connection
+ try:
+ with production_get_user_db_session() as w2_sess:
+ count = w2_sess.query(ResearchResource).count()
+ assert count == 1
+ w2_sess.commit()
+ worker2_acquired_connection = True
+ except Exception as exc:
+ worker2_acquired_connection = False
+ raise AssertionError(
+ f"Worker 2 starved on retry path: {exc}"
+ ) from exc
+
+ t = threading.Thread(target=worker2_fn)
+ t.start()
+ t.join()
+
+ return MagicMock(
+ is_success=True,
+ content=b"%PDF-1.4 downloaded data",
+ status_code=200,
+ skip_reason=None,
+ )
+
+ downloader.download_with_result.side_effect = slow_download
+ service.downloaders = [downloader]
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=production_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value=source_type_id),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+ success, reason = service.download_resource(resource_id)
+
+ assert success is True
+ assert worker2_acquired_connection is True, (
+ "Worker 2 was starved while worker 1 was downloading on retry path. "
+ "The pre-existing tracker branch did not commit its pre-check transaction."
+ )
+ assert get_thread_session().expire_on_commit is True
+
+ engine.dispose()
+
+
+def test_download_failure_stages_attempt_after_download(service_stub):
+ """When download fails (e.g. 404 / skip reason), the attempt is staged only after download completes."""
+ events = []
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+
+ def fail_download(url, content_type):
+ events.append("network_download_attempt")
+ return MagicMock(
+ is_success=False,
+ content=None,
+ status_code=404,
+ skip_reason="HTTP 404 Not Found",
+ )
+
+ downloader.download_with_result.side_effect = fail_download
+ service_stub.downloaders = [downloader]
+
+ mock_session = MagicMock()
+ mock_session.get.return_value = MagicMock(
+ id=2,
+ url="https://example.com/not_found.pdf",
+ title="404 Paper",
+ document_id=None,
+ )
+ mock_session.query.return_value.filter_by.return_value.first.return_value = None
+
+ @contextmanager
+ def mock_db_sess(*args, **kwargs):
+ events.append("session_active")
+ yield mock_session
+
+ with patch(f"{MODULE}.get_user_db_session", side_effect=mock_db_sess):
+ success, reason = service_stub.download_resource(2)
+
+ assert success is False
+ assert "404" in reason
+
+ # Verify that DownloadAttempt was added to session
+ added_attempts = [
+ call.args[0]
+ for call in mock_session.add.call_args_list
+ if call.args and isinstance(call.args[0], DownloadAttempt)
+ ]
+ assert len(added_attempts) >= 1
+ assert added_attempts[0].succeeded is False
+ assert "404" in added_attempts[0].error_message
+
+
+def test_download_network_exception_records_failure_without_session_pinning(
+ service_stub,
+):
+ """When download_with_result raises an exception (e.g. connection reset),
+ the exception is caught and failed attempt is staged without leaking sessions.
+ """
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+ downloader.download_with_result.side_effect = ConnectionResetError(
+ "Connection reset by peer"
+ )
+ service_stub.downloaders = [downloader]
+
+ mock_session = MagicMock()
+ mock_session.get.return_value = MagicMock(
+ id=3,
+ url="https://example.com/network_error.pdf",
+ title="Error Paper",
+ document_id=None,
+ )
+ mock_session.query.return_value.filter_by.return_value.first.return_value = None
+
+ @contextmanager
+ def mock_db_sess(*args, **kwargs):
+ yield mock_session
+
+ with patch(f"{MODULE}.get_user_db_session", side_effect=mock_db_sess):
+ success, reason = service_stub.download_resource(3)
+
+ assert success is False
+ # Skip reason must be a caller-safe token, NOT the raw exception text.
+ # sanitize_error_for_client scrubs credentials but explicitly does not
+ # strip SQL text or dependency internals; the failure path now maps the
+ # exception class to a fixed token (see #6564 follow-up).
+ assert reason == "network_reset"
+ assert "Connection reset by peer" not in reason
+
+ added_attempts = [
+ call.args[0]
+ for call in mock_session.add.call_args_list
+ if call.args and isinstance(call.args[0], DownloadAttempt)
+ ]
+ assert len(added_attempts) >= 1
+ assert added_attempts[0].succeeded is False
+ assert added_attempts[0].error_type == "ConnectionResetError"
+ # Server-side persisted error_message keeps the diagnostic text (with
+ # credential scrubbing); the caller-visible skip_reason does not.
+ assert "Connection reset by peer" in added_attempts[0].error_message
+
+
+def test_download_sqlalchemy_exception_does_not_leak_sql_in_skip_reason(
+ service_stub,
+):
+ """The caller-visible skip_reason for a SQLAlchemy failure must not carry SQL text.
+
+ Reviewer finding (post-#6564): ``sanitize_error_for_client(str(e))`` scrubbed
+ credentials but left SQL text — e.g. ``"INSERT INTO document_collections ..."`` —
+ verbatim, and the result flowed into retry-manager and bulk-stream SSE messages.
+ The fix maps the exception class to a fixed token; this test asserts that
+ mapping is actually applied, not bypassed for SQLAlchemy errors.
+
+ Mutation: removing ``_client_safe_download_message`` and restoring the old
+ ``sanitize_error_for_client(str(e))`` would put the SQL text back into the
+ returned skip_reason; this test fails.
+ """
+ from sqlalchemy.exc import IntegrityError
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+ downloader.download_with_result.side_effect = IntegrityError(
+ "INSERT INTO document_collections (document_id, collection_id) "
+ "VALUES (?, ?, ?)",
+ params=("doc-1", "coll-1"),
+ orig=Exception("FOREIGN KEY constraint failed"),
+ )
+ service_stub.downloaders = [downloader]
+
+ mock_session = MagicMock()
+ mock_session.get.return_value = MagicMock(
+ id=4,
+ url="https://example.com/fk_failure.pdf",
+ title="FK Paper",
+ document_id=None,
+ )
+ mock_session.query.return_value.filter_by.return_value.first.return_value = None
+
+ @contextmanager
+ def mock_db_sess(*args, **kwargs):
+ yield mock_session
+
+ with patch(f"{MODULE}.get_user_db_session", side_effect=mock_db_sess):
+ success, reason = service_stub.download_resource(4)
+
+ assert success is False
+ # Skip reason is the safe mapped token, NOT the SQL text.
+ assert reason == "database_constraint"
+ forbidden_substrings = [
+ "INSERT INTO",
+ "document_collections",
+ "FOREIGN KEY",
+ "VALUES",
+ "doc-1",
+ "coll-1",
+ ]
+ for needle in forbidden_substrings:
+ assert needle not in reason, (
+ f"Caller-visible skip_reason leaked SQL fragment {needle!r}: "
+ f"{reason!r}"
+ )
+
+ # The persisted DownloadAttempt.error_message keeps the server-side
+ # diagnostic (with credential scrubbing) so the operator can debug the
+ # staging failure from the row — but the retry-manager and bulk-stream
+ # surfaces see only the safe mapped token above.
+ added_attempts = [
+ call.args[0]
+ for call in mock_session.add.call_args_list
+ if call.args and isinstance(call.args[0], DownloadAttempt)
+ ]
+ assert len(added_attempts) >= 1
+ assert added_attempts[0].succeeded is False
+ assert added_attempts[0].error_type == "IntegrityError"
+
+
+def test_text_extraction_save_failures_do_not_leak_sql_in_error(
+ service_stub,
+):
+ """Sibling text-extraction paths must use the same fixed-token mapping.
+
+ Reviewer finding (post-#6564 at 4abcb707): only ``_download_pdf``'s
+ except path was mapped — ``_try_arxiv_text_extraction``,
+ ``_try_api_text_extraction`` and ``_fallback_pdf_extraction`` shaped
+ ``sanitize_error_for_client(f"Failed to save text: {str(e)}")``, which
+ keeps SQL text verbatim, and ``download_as_text`` returns it into the
+ bulk-stream SSE ``error`` field (text_only mode). All three now route
+ through ``_client_safe_download_message``.
+
+ Mutation: restoring ``sanitize_error_for_client(f"Failed to save text:
+ {e}")`` on any of these paths puts ``INSERT INTO`` back into the
+ returned error; this test fails.
+ """
+ from sqlalchemy.exc import IntegrityError
+
+ def _integrity_error():
+ return IntegrityError(
+ "INSERT INTO document_collections (document_id, collection_id) "
+ "VALUES (?, ?, ?)",
+ params=("doc-1", "coll-1"),
+ orig=Exception("FOREIGN KEY constraint failed"),
+ )
+
+ forbidden = [
+ "INSERT INTO",
+ "document_collections",
+ "FOREIGN KEY",
+ "VALUES",
+ "doc-1",
+ "coll-1",
+ ]
+
+ # _try_api_text_extraction save failure.
+ session = MagicMock()
+ resource = MagicMock()
+ resource.url = "https://example.com/api_text"
+ resource.id = 11
+ resource.title = "API Text Paper With Long Title For Slicing"
+ downloader = MagicMock()
+ downloader.download_with_result.return_value = MagicMock(
+ is_success=True, content="text from api"
+ )
+ with (
+ patch.object(service_stub, "_get_downloader", return_value=downloader),
+ patch.object(
+ service_stub,
+ "_save_text_with_db",
+ side_effect=_integrity_error(),
+ ),
+ ):
+ ok, err = service_stub._try_api_text_extraction(session, resource)
+ assert ok is False
+ assert err == "database_constraint"
+ for needle in forbidden:
+ assert needle not in err
+
+ # _fallback_pdf_extraction save failure.
+ session = MagicMock()
+ resource = MagicMock()
+ resource.url = "https://example.com/fallback_pdf"
+ resource.id = 12
+ resource.title = "Fallback PDF Paper With Long Title For Slicing"
+ downloader = MagicMock()
+ downloader.download_with_result.return_value = MagicMock(
+ is_success=True, content=b"%PDF-1.4 bytes"
+ )
+ with (
+ patch.object(service_stub, "_get_downloader", return_value=downloader),
+ patch.object(
+ service_stub, "_extract_text_from_pdf", return_value="text"
+ ),
+ patch.object(
+ service_stub,
+ "_save_text_with_db",
+ side_effect=_integrity_error(),
+ ),
+ ):
+ ok, err = service_stub._fallback_pdf_extraction(session, resource)
+ assert ok is False
+ assert err == "database_constraint"
+ for needle in forbidden:
+ assert needle not in err
+
+
+def test_first_time_download_persists_tracker_before_attempt_with_fk_on(
+ tmp_path,
+):
+ """FK-ON regression pin for #6564 P1.
+
+ On a first-time download, ``download_resource`` adds the new
+ ``DownloadTracker`` and then ``session.expunge``s it so its attributes
+ remain accessible without a lazy refresh. If the INSERT is still
+ pending at that point, ``expunge`` cancels it and the later
+ ``DownloadAttempt.url_hash`` insert fails with ``FOREIGN KEY constraint
+ failed`` (production runs with ``PRAGMA foreign_keys = ON``; this
+ engine mirrors that). The fix makes the tracker row durable before the
+ expunge via two cooperating mechanisms: an explicit ``session.flush()``
+ inside a ``session.begin_nested()`` savepoint. The savepoint's exit
+ auto-flushes its writes, so the invariant is doubly defended — removing
+ the explicit flush alone does NOT fail this test (verified by run), and
+ removing both the savepoint wrapper and the explicit flush DOES fail it:
+ the attempt INSERT hits the missing FK parent, the staging commit
+ aborts, and the assertions below see 0 trackers / 0 attempts.
+
+ The explicit flush is retained as defense-in-depth so the FK-on-first-
+ download invariant is stated at this site rather than relying on
+ savepoint-exit side effects (see the comment at the pre-check block).
+
+ The earlier pool-exhaustion tests do not catch this because they run
+ on engines that do not enable ``PRAGMA foreign_keys`` (SQLite default
+ is OFF), so the broken INSERT silently no-ops.
+ """
+ db_file = tmp_path / "fk_on_test.db"
+ engine = create_engine(
+ f"sqlite:///{db_file}",
+ connect_args={"check_same_thread": False},
+ )
+
+ # Mirror production: every connection in sqlcipher_utils turns
+ # PRAGMA foreign_keys on at connect time. Without this listener the
+ # SQLite default (OFF) silently turns the broken sequence into a
+ # vacuous pass.
+ @event.listens_for(engine, "connect")
+ def _enable_fk(dbapi_connection, _connection_record):
+ dbapi_connection.execute("PRAGMA foreign_keys = ON")
+
+ Base.metadata.create_all(engine)
+
+ # Sanity-check the pragma actually took effect on a fresh connection.
+ with engine.connect() as conn:
+ fk_state = conn.exec_driver_sql("PRAGMA foreign_keys").scalar_one()
+ assert fk_state == 1, (
+ "PRAGMA foreign_keys did not turn on — FK-ON regression test "
+ "would be vacuous."
+ )
+
+ Session = sessionmaker(bind=engine)
+
+ # Seed required rows.
+ with Session() as seed_session:
+ rh = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="q",
+ mode="quick",
+ status="completed",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(rh)
+ resource = ResearchResource(
+ research_id=rh.id,
+ title="First Download Paper",
+ url="https://example.com/first_paper.pdf",
+ source_type="academic",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(resource)
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ source_type_id = source_type.id
+ seed_session.add(source_type)
+ # FK target for ensure_in_collection (document_collections.collection_id).
+ collection = Collection(
+ id="lib-1",
+ name="Library",
+ collection_type="default_library",
+ is_default=True,
+ )
+ seed_session.add(collection)
+ seed_session.commit()
+ resource_id = resource.id
+
+ service = DownloadService.__new__(DownloadService)
+ service.username = "test_user"
+ service.password = "test_pass"
+ service.library_root = "/tmp/ldr-test-library"
+ service.legacy_library_root = None
+ service._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ service.settings = settings
+ service._check_url_against_policy = lambda url: (True, "ok")
+ service.retry_manager = MagicMock()
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+ downloader.download_with_result.return_value = MagicMock(
+ is_success=True,
+ content=b"%PDF-1.4 first download",
+ status_code=200,
+ skip_reason=None,
+ )
+ service.downloaders = [downloader]
+
+ thread_local = threading.local()
+
+ def get_thread_session():
+ if not hasattr(thread_local, "session") or thread_local.session is None:
+ thread_local.session = Session()
+ return thread_local.session
+
+ @contextmanager
+ def production_get_user_db_session(*args, **kwargs):
+ sess = get_thread_session()
+ try:
+ yield sess
+ except Exception:
+ sess.rollback()
+ raise
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=production_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value=source_type_id),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+ success, reason = service.download_resource(resource_id)
+
+ assert success is True, (
+ f"First-time download failed under FK enforcement: {reason!r}. "
+ "Likely cause: the pending DownloadTracker INSERT was cancelled "
+ "by session.expunge() before commit, leaving no parent row for "
+ "DownloadAttempt.url_hash (the #6564 P1 fix is the explicit "
+ "session.flush() inside the begin_nested() savepoint before "
+ "session.expunge). This is mutation-verified — removing both the "
+ "savepoint wrapper and the explicit flush causes this test "
+ "to fail with FOREIGN KEY constraint failed on download_attempts."
+ )
+
+ # On-disk assertions: tracker + attempt + document must all exist. The
+ # broken sequence produces 0 trackers (expunge cancels the pending
+ # INSERT) and 0 attempts (FK failure aborts the staging commit).
+ with Session() as verify_session:
+ trackers = verify_session.query(DownloadTracker).all()
+ attempts = verify_session.query(DownloadAttempt).all()
+ docs = (
+ verify_session.query(Document)
+ .filter_by(resource_id=resource_id)
+ .all()
+ )
+
+ assert len(trackers) == 1, (
+ f"Expected exactly 1 DownloadTracker row, found {len(trackers)}. "
+ "The pending INSERT was likely cancelled by expunge."
+ )
+ assert len(attempts) == 1, (
+ f"Expected exactly 1 DownloadAttempt row, found {len(attempts)}. "
+ "FOREIGN KEY constraint failed on the attempt INSERT — the "
+ "tracker row was not durable at the time of the attempt."
+ )
+ assert attempts[0].succeeded is True
+ assert attempts[0].bytes_downloaded == len(b"%PDF-1.4 first download")
+ assert len(docs) == 1
+ assert docs[0].status == DocumentStatus.COMPLETED
+
+ engine.dispose()
+
+
+def test_pre_check_savepoint_scopes_new_tracker_write(tmp_path):
+ """The new-tracker INSERT must be scoped to a SAVEPOINT.
+
+ Reviewer finding (post-#6564): ``download_resource`` issues an
+ unconditional ``session.commit()`` to release the connection back to
+ QueuePool. The new-tracker INSERT is the only write the pre-check
+ performs on a first-time download, and it must be enclosed in a
+ savepoint so a failure INSIDE the savepoint (e.g. a future UNIQUE
+ constraint conflict, an autoflush interaction, or a manual rollback
+ by another block) cannot drag the caller's outer transaction with
+ it.
+
+ Without the savepoint, ``session.add(tracker)`` puts the pending
+ INSERT directly on the shared thread-local session's outer
+ transaction. The unconditional ``session.commit()`` at the end
+ promotes it together with any other row the caller has staged — a
+ structural surprise rather than a deliberate commit. None of the
+ three current call sites stage rows across the call, but the
+ reviewer asked for a structural guarantee.
+
+ This test asserts the savepoint is in place by checking that the
+ session's nested-transaction counter advances during the pre-check
+ block. Removing the ``with session.begin_nested():`` wrapper drops
+ the nested-transaction count to zero on a first-time download.
+ """
+ db_file = tmp_path / "savepoint_scope_test.db"
+ engine = create_engine(
+ f"sqlite:///{db_file}",
+ connect_args={"check_same_thread": False},
+ )
+ Base.metadata.create_all(engine)
+ Session = sessionmaker(bind=engine)
+
+ with Session() as seed_session:
+ rh = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="q",
+ mode="quick",
+ status="completed",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(rh)
+ resource = ResearchResource(
+ research_id=rh.id,
+ title="Savepoint Scope Paper",
+ url="https://example.com/savepoint_scope_paper.pdf",
+ source_type="academic",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_session.add(resource)
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ source_type_id = source_type.id
+ seed_session.add(source_type)
+ collection = Collection(
+ id="lib-1",
+ name="Library",
+ collection_type="default_library",
+ is_default=True,
+ )
+ seed_session.add(collection)
+ seed_session.commit()
+ resource_id = resource.id
+
+ service = DownloadService.__new__(DownloadService)
+ service.username = "test_user"
+ service.password = "test_pass"
+ service.library_root = "/tmp/ldr-test-library"
+ service.legacy_library_root = None
+ service._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ service.settings = settings
+ service._check_url_against_policy = lambda url: (True, "ok")
+ service.retry_manager = MagicMock()
+
+ # Probe: count savepoint begin_nested calls during the pre-check.
+ nested_count = {"value": 0}
+
+ downloader = MagicMock()
+ downloader.can_handle.return_value = True
+ downloader.download_with_result.return_value = MagicMock(
+ is_success=True,
+ content=b"%PDF-1.4 savepoint scope",
+ status_code=200,
+ skip_reason=None,
+ )
+ service.downloaders = [downloader]
+
+ thread_local = threading.local()
+
+ def get_thread_session():
+ if not hasattr(thread_local, "session") or thread_local.session is None:
+ thread_local.session = Session()
+ return thread_local.session
+
+ @contextmanager
+ def production_get_user_db_session(*args, **kwargs):
+ sess = get_thread_session()
+ # Wrap begin_nested so we can observe the pre-check using it.
+ original = sess.begin_nested
+
+ @contextmanager
+ def counting_begin_nested(*a, **kw):
+ nested_count["value"] += 1
+ with original(*a, **kw) as sp:
+ yield sp
+
+ sess.begin_nested = counting_begin_nested
+ try:
+ yield sess
+ except Exception:
+ sess.rollback()
+ raise
+ finally:
+ sess.begin_nested = original
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=production_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value=source_type_id),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+ success, reason = service.download_resource(resource_id)
+
+ assert success is True, (
+ f"download_resource failed: {reason!r}. The savepoint structure "
+ "should isolate the pre-check writes."
+ )
+
+ # First-time download must have opened exactly one savepoint around
+ # the new-tracker INSERT. A retry path (existing tracker) opens zero,
+ # so this asserts the FIRST-DOWNLOAD case specifically — which is the
+ # one with the write to scope.
+ assert nested_count["value"] == 1, (
+ f"Expected exactly 1 begin_nested() call during the pre-check, "
+ f"got {nested_count['value']}. The new-tracker INSERT is no "
+ f"longer scoped to a savepoint — caller-staged rows are no "
+ f"longer isolated from the unconditional commit."
+ )
+
+ # End-to-end correctness: the new-tracker INSERT must still be durable.
+ with Session() as verify_session:
+ trackers = verify_session.query(DownloadTracker).all()
+ attempts = verify_session.query(DownloadAttempt).all()
+ assert len(trackers) == 1
+ assert len(attempts) == 1
+ assert attempts[0].succeeded is True
+
+ engine.dispose()
+
+
+def test_bulk_concurrent_downloads_do_not_exhaust_connection_pool(tmp_path):
+ """Bulk downloads (10 concurrent workers) on a pool of max 4 connections must not starve
+ concurrent database requests.
+ """
+ db_file = tmp_path / "bulk_test.db"
+ engine = create_engine(
+ f"sqlite:///{db_file}",
+ poolclass=QueuePool,
+ pool_size=2,
+ max_overflow=2, # Max 4 total connections
+ pool_timeout=5.0,
+ connect_args={"check_same_thread": False},
+ )
+ Base.metadata.create_all(engine)
+ Session = sessionmaker(bind=engine)
+
+ # Seed 10 resources
+ with Session() as seed_sess:
+ rh = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="bulk",
+ mode="quick",
+ status="completed",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_sess.add(rh)
+ res_ids = []
+ for i in range(10):
+ r = ResearchResource(
+ research_id=rh.id,
+ title=f"Bulk Paper {i}",
+ url=f"https://example.com/paper_{i}.pdf",
+ source_type="academic",
+ created_at="2026-05-09T00:00:00",
+ )
+ seed_sess.add(r)
+ seed_sess.flush()
+ res_ids.append(r.id)
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ source_type_id = source_type.id
+ seed_sess.add(source_type)
+ seed_sess.commit()
+
+ thread_local = threading.local()
+
+ def get_thread_session():
+ if not hasattr(thread_local, "session") or thread_local.session is None:
+ thread_local.session = Session()
+ return thread_local.session
+
+ @contextmanager
+ def production_get_user_db_session(*args, **kwargs):
+ sess = get_thread_session()
+ try:
+ yield sess
+ except Exception:
+ sess.rollback()
+ raise
+
+ def make_service():
+ s = DownloadService.__new__(DownloadService)
+ s.username = "test_user"
+ s.password = "test_pass"
+ s.library_root = "/tmp/ldr-test-library"
+ s.legacy_library_root = None
+ s._closed = False
+ settings = MagicMock()
+ settings.get_setting = lambda key, default=None: {
+ "research_library.pdf_storage_mode": "database",
+ "research_library.max_pdf_size_mb": 50,
+ }.get(key, default)
+ s.settings = settings
+ s._check_url_against_policy = lambda url: (True, "ok")
+ s.retry_manager = MagicMock()
+
+ dl = MagicMock()
+ dl.can_handle.return_value = True
+
+ def download_fn(url, content_type):
+ time.sleep(0.05) # 50ms simulated HTTP transfer
+ return MagicMock(
+ is_success=True,
+ content=f"%PDF-bulk data {url}".encode(),
+ status_code=200,
+ skip_reason=None,
+ )
+
+ dl.download_with_result.side_effect = download_fn
+ s.downloaders = [dl]
+ return s
+
+ with (
+ patch(
+ f"{MODULE}.get_user_db_session",
+ side_effect=production_get_user_db_session,
+ ),
+ patch(f"{MODULE}.PDFStorageManager") as mock_psm,
+ patch(f"{MODULE}.get_source_type_id", return_value=source_type_id),
+ patch(f"{MODULE}.get_default_library_id", return_value="lib-1"),
+ ):
+ mock_psm.return_value.save_pdf.return_value = ("database", None)
+
+ def run_download(rid):
+ svc = make_service()
+ return svc.download_resource(rid)
+
+ def run_fastapi_request():
+ with production_get_user_db_session() as sess:
+ count = sess.query(ResearchResource).count()
+ sess.commit()
+ return count
+
+ # Run 10 downloads concurrently with 15 concurrent simulated FastAPI requests
+ with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
+ download_futures = [
+ executor.submit(run_download, rid) for rid in res_ids
+ ]
+ fastapi_futures = [
+ executor.submit(run_fastapi_request) for _ in range(15)
+ ]
+
+ # All FastAPI requests must complete without timing out or pool exhaustion
+ for f in concurrent.futures.as_completed(fastapi_futures):
+ count = f.result()
+ assert count == 10
+
+ # All downloads must succeed
+ for f in concurrent.futures.as_completed(download_futures):
+ success, reason = f.result()
+ assert success is True, f"Download failed: {reason}"
+ assert reason is None
+
+ engine.dispose()
diff --git a/tests/security/test_client_safe_errors_shared_6564.py b/tests/security/test_client_safe_errors_shared_6564.py
new file mode 100644
index 0000000000..9e303fd254
--- /dev/null
+++ b/tests/security/test_client_safe_errors_shared_6564.py
@@ -0,0 +1,95 @@
+"""PR #6564 optional #1: single source of truth for safe download tokens."""
+
+import os
+
+from local_deep_research.security.client_safe_errors import (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES,
+ client_safe_download_message,
+)
+
+
+class TestSharedTable:
+ def test_services_and_html_aliases_share_identity(self):
+ from local_deep_research.research_library.services.download_service import (
+ _CLIENT_SAFE_DOWNLOAD_MESSAGES as svc,
+ _client_safe_download_message as svc_fn,
+ )
+ from local_deep_research.research_library.downloaders.html import (
+ _CLIENT_SAFE_HTML_MESSAGES as html,
+ _client_safe_html_message as html_fn,
+ )
+
+ assert svc is CLIENT_SAFE_DOWNLOAD_MESSAGES
+ assert html is CLIENT_SAFE_DOWNLOAD_MESSAGES
+
+ exc = ValueError("INSERT INTO document_collections ...")
+ assert svc_fn(exc) == "download_error:ValueError"
+ assert html_fn(exc) == "download_error:ValueError"
+ assert client_safe_download_message(exc) == "download_error:ValueError"
+
+ def test_previously_drifted_keys_present(self):
+ # These existed only on the services side before the shared module.
+ assert (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES["DataError"] == "database_data_error"
+ )
+ assert (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES["InterfaceError"]
+ == "database_interface_error"
+ )
+ assert (
+ CLIENT_SAFE_DOWNLOAD_MESSAGES["InvalidRequestError"]
+ == "database_invalid_request"
+ )
+
+ class DataError(Exception):
+ pass
+
+ DataError.__name__ = "DataError"
+ assert (
+ client_safe_download_message(DataError()) == "database_data_error"
+ )
+
+ def test_oserror_canonical_branch(self):
+ err_no = 2 # ENOENT
+ canonical = os.strerror(err_no)
+ exc = OSError(err_no, canonical)
+ assert (
+ client_safe_download_message(exc) == f"filesystem_error:{canonical}"
+ )
+
+ # Custom message must NOT leak through.
+ leaked = OSError(28, "leak /etc/passwd")
+ assert client_safe_download_message(leaked) == "download_error:OSError"
+
+ def test_never_echoes_exception_text(self):
+ exc = RuntimeError("INSERT INTO document_collections VALUES (1)")
+ msg = client_safe_download_message(exc)
+ assert msg == "download_error:RuntimeError"
+ assert "INSERT INTO" not in msg
+
+
+class TestHtmlOperatorLogCarriesClass:
+ """PR #6564 optional #3: tokenized HTML failure logs its class."""
+
+ def test_download_with_result_logs_exc_class_not_text(self):
+ from unittest.mock import patch
+
+ from local_deep_research.research_library.downloaders.html import (
+ HTMLDownloader,
+ )
+ import local_deep_research.research_library.downloaders.html as html_mod
+
+ downloader = HTMLDownloader()
+ secret = "supersecret1234567890"
+ boom = ValueError(f"fetch failed api_key={secret}")
+ with (
+ patch.object(downloader, "_fetch_html", side_effect=boom),
+ patch.object(html_mod, "logger") as mock_logger,
+ ):
+ result = downloader.download_with_result("https://example.com")
+ assert result.skip_reason == "download_error:ValueError"
+ error_call = mock_logger.opt.return_value.error.call_args
+ kwargs = error_call.kwargs
+ assert kwargs.get("exc_class") == "ValueError"
+ # Neither the message template nor the kwargs carry the text/secret.
+ assert secret not in str(error_call.args) + str(kwargs)