~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(library): keep refused PDF storage retryable (#6913)
LearningCircuit86acdc7697d5
diff --git a/changelog.d/+pdf-storage-refusal-status.bugfix.md b/changelog.d/+pdf-storage-refusal-status.bugfix.mdnew file mode 100644index 0000000000..0ecf4a6f6c--- /dev/null+++ b/changelog.d/+pdf-storage-refusal-status.bugfix.md@@ -0,0 +1 @@+A research download whose PDF the PDF storage size limit (`research_library.max_pdf_size_mb`) refuses still keeps the PDF's extracted text, searchable and indexed as before. The document is no longer labelled as storing a PDF it does not hold: it is a text-only document that awaits its PDF, so the Download Manager offers the PDF for download again. While the limit still refuses a PDF of the recorded size, downloading it fetches nothing and reports that the PDF is larger than the storage limit, with a message saying to raise "Maximum PDF Size (MB)" (in the bulk download progress and the single-download endpoints) instead of a raw `pdf_storage_size_limit` token or a generic "Download failed". Once the limit is raised above the PDF's size, the next download stores the PDF into the same document. The refusal sets no retry cooldown, so the text downloads for the resource are never held back by it; should a caller record the `pdf_storage_size_limit` reason with the retry manager, it is classified as an operator-fixable failure that is held back one day at a time but never made permanent or counted toward the retry limit. The same holds when the refused PDF matches a text-only research download by content (the duplicate is linked to it, as before; a matching upload or Zotero import is linked and left as it is) and for a refill of a document that "Sync library" left awaiting its PDF, which now keeps awaiting it instead of settling as text-only. With research-source auto-persistence on, another resource that cites the same refused PDF (by URL or by content) is linked to the text-only document without downloading the PDF again, as before; once the limit is raised, downloading that resource stores the PDF into the document. A PDF that the limit refuses during the research's own fetch, before the source is saved as a research resource, stays a text-only document and is not re-stored when the limit is raised. A collection upload whose PDF the storage manager refuses no longer reports `pdf_stored: true` or labels the document as stored in the database; it is a text-only upload. Not healed: documents that an earlier version completed in database or filesystem mode with no PDF stored (a refused PDF before this fix) still count as downloaded; delete and download them again to store their PDF once the limit allows it.diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.mdindex 6b7e22f616..89befa4c19 100644--- a/docs/CONFIGURATION.md+++ b/docs/CONFIGURATION.md@@ -222,7 +222,7 @@ They are not available in the Web UI because they are needed to start the applic| `research_library.auto_index_enabled` | `LDR_RESEARCH_LIBRARY_AUTO_INDEX_ENABLED` | `false` | Automatically index documents for RAG search when they are added to collections (via upload or download). Indexing runs in the background without blocking uploads. Disable to defer indexing and improve upload speed for large batches. | APP || `research_library.auto_persist_research_sources` | `LDR_RESEARCH_LIBRARY_AUTO_PERSIST_RESEARCH_SOURCES` | `false` | Automatically persist research sources (PDFs and extracted text) into the Library when a research run completes. Default off. When enabled, PDF downloads still honor the scheduler preference document_scheduler.download_pdfs and the PDF storage mode. | APP || `research_library.confirm_deletions` | `LDR_RESEARCH_LIBRARY_CONFIRM_DELETIONS` | `true` | Show confirmation dialogs before delete operations. Recommended to keep enabled to prevent accidental data loss. | APP |-| `research_library.max_pdf_size_mb` | `LDR_RESEARCH_LIBRARY_MAX_PDF_SIZE_MB` | `3072` | Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage. | APP |+| `research_library.max_pdf_size_mb` | `LDR_RESEARCH_LIBRARY_MAX_PDF_SIZE_MB` | `3072` | Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored; their extracted text is still kept, and a research download whose PDF was refused this way is stored when it is next downloaded after this limit is raised above its size. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage. | APP || `research_library.pdf_storage_mode` | `RESEARCH_LIBRARY_PDF_STORAGE_MODE` | `database` | Choose how PDFs are stored. Each user has their own encrypted database. Database mode stores PDFs encrypted within it (secure, portable, isolated per-user). Filesystem mode stores unencrypted files on disk (faster for large files). None stores only extracted text (smallest footprint). | APP || `research_library.shared_library` | `RESEARCH_LIBRARY_SHARED_LIBRARY` | `false` | ⚠️ WARNING: When enabled, all users share the same library directory. Documents from all users will be visible to each other. Only enable in trusted environments. | APP || `research_library.storage_path` | `RESEARCH_LIBRARY_STORAGE_PATH` | `~/Documents/LocalDeepResearch/Library` | Operator-controlled base path for research library documents. Existing saved locations remain in use; set LDR_RESEARCH_LIBRARY_STORAGE_PATH to change the path. Each user has a subdirectory unless shared library is enabled. | APP |diff --git a/src/local_deep_research/defaults/research_library/library_settings.json b/src/local_deep_research/defaults/research_library/library_settings.jsonindex eec07618c3..8e66393d47 100644--- a/src/local_deep_research/defaults/research_library/library_settings.json+++ b/src/local_deep_research/defaults/research_library/library_settings.json@@ -87,7 +87,7 @@},"research_library.max_pdf_size_mb": {"category": "research_library",- "description": "Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage.",+ "description": "Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored; their extracted text is still kept, and a research download whose PDF was refused this way is stored when it is next downloaded after this limit is raised above its size. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage.","editable": true,"max_value": 10240,"min_value": 1,diff --git a/src/local_deep_research/library/download_management/__init__.py b/src/local_deep_research/library/download_management/__init__.pyindex cff5a4506e..6b2c214d8e 100644--- a/src/local_deep_research/library/download_management/__init__.py+++ b/src/local_deep_research/library/download_management/__init__.py@@ -16,6 +16,7 @@ from .failure_classifier import (PermanentFailure,TemporaryFailure,RateLimitFailure,+ OperatorFixableFailure,FailureClassifier,)from .status_tracker import ResourceStatusTracker, ResourceDownloadStatus@@ -30,6 +31,7 @@ __all__ = ["PermanentFailure","TemporaryFailure","RateLimitFailure",+ "OperatorFixableFailure","FailureClassifier","ResourceStatusTracker","ResourceDownloadStatus",diff --git a/src/local_deep_research/library/download_management/failure_classifier.py b/src/local_deep_research/library/download_management/failure_classifier.pyindex ca798407b6..ded674bea5 100644--- a/src/local_deep_research/library/download_management/failure_classifier.py+++ b/src/local_deep_research/library/download_management/failure_classifier.py@@ -91,6 +91,23 @@ class RateLimitFailure(TemporaryFailure):self.domain = domain+class OperatorFixableFailure(TemporaryFailure):+ """A refusal that only a change to the operator's own settings cures.++ The source served the file; a local setting (the PDF storage size+ limit) refused it. Repeating the download changes nothing until the+ operator changes that setting, and it says nothing about the source, so+ the status tracker does not treat it as a failing source (see+ ``ResourceStatusTracker._apply_failure``): every refusal holds the+ resource back for the same ``cooldown``, so a bulk run does not fetch+ the file again right away, but it never escalates along+ ``compute_retry_cooldown``'s schedule, is never promoted to+ ``permanently_failed`` and does not count toward ``MAX_TOTAL_RETRIES``.+ Once the operator changes the setting, the resource is eligible again+ after one ``cooldown``, however often it was refused before.+ """++# Caller-safe skip-reason tokens emitted by# ``security.client_safe_errors.client_safe_download_message`` (PR #6564).# ``RetryManager.record_attempt`` receives the token as the string@@ -100,7 +117,10 @@ class RateLimitFailure(TemporaryFailure):# "Unclassified error" warning) where the pre-token text classified as# ``timeout`` (30-min). All entries stay Temporary (no retryable -> terminal# flip); the token itself is preserved as ``error_type`` so the UI-# distinction survives.+# distinction survives. Every token that ``CLIENT_SAFE_DOWNLOAD_MESSAGES``+# can emit needs an entry here (a test sweeps the whole vocabulary). The+# ones listed in ``_OPERATOR_FIXABLE_TOKENS`` are refused by the operator's+# own settings rather than failed by the source._CLIENT_SAFE_TOKEN_FAILURES: dict[str, tuple[str, str, timedelta]] = {"network_timeout": ("timeout", "Request timed out", timedelta(minutes=30)),"network_unavailable": (@@ -154,8 +174,27 @@ _CLIENT_SAFE_TOKEN_FAILURES: dict[str, tuple[str, str, timedelta]] = {"PDF parsing failed",timedelta(hours=1),),+ # The configured PDF storage size limit refuses the PDF (see+ # ``download_service.PDFStorageLimitError``). ``download_resource``+ # returns it, without downloading anything, for a document that keeps+ # its text because the limit refused its PDF, and does not record it+ # with the retry manager: the limit itself decides when the PDF is+ # downloaded again, and a cooldown would also hold back the text routes.+ # The entry keeps the token classified, as operator-fixable, for any+ # caller that does record it: held back a day at a time, never+ # escalated or made permanent.+ "pdf_storage_size_limit": (+ "pdf_storage_size_limit",+ "PDF exceeds the configured storage size limit",+ timedelta(days=1),+ ),}+# Tokens in ``_CLIENT_SAFE_TOKEN_FAILURES`` that name a refusal by the+# operator's own settings. They classify as ``OperatorFixableFailure``: no+# escalation, no permanent failure, no retry-budget use.+_OPERATOR_FIXABLE_TOKENS = frozenset({"pdf_storage_size_limit"})+class FailureClassifier:"""Classifies download failures into appropriate types based on error patterns"""@@ -221,6 +260,8 @@ class FailureClassifier:continueif token in _CLIENT_SAFE_TOKEN_FAILURES:err, msg, cooldown = _CLIENT_SAFE_TOKEN_FAILURES[token]+ if token in _OPERATOR_FIXABLE_TOKENS:+ return OperatorFixableFailure(err, msg, cooldown)return TemporaryFailure(err, msg, cooldown)if token.startswith("filesystem_error:"):return TemporaryFailure(diff --git a/src/local_deep_research/library/download_management/status_tracker.py b/src/local_deep_research/library/download_management/status_tracker.pyindex 6d17b710c4..4bf956c700 100644--- a/src/local_deep_research/library/download_management/status_tracker.py+++ b/src/local_deep_research/library/download_management/status_tracker.py@@ -12,7 +12,7 @@ from loguru import loggerfrom sqlalchemy.orm import sessionmaker, Sessionfrom .models import ResourceDownloadStatus, Base-from .failure_classifier import BaseFailure+from .failure_classifier import BaseFailure, OperatorFixableFailureMAX_TOTAL_RETRIES = 5@@ -27,6 +27,9 @@ def compute_retry_cooldown(attempt 2: 1 dayattempt 3-4: 30 daysattempt >= 5: None (permanent failure)++ It applies to failures of the source. An ``OperatorFixableFailure``+ never goes through it (see ``ResourceStatusTracker._apply_failure``)."""if attempt >= MAX_TOTAL_RETRIES:return None@@ -134,6 +137,10 @@ class ResourceStatusTracker:status = ResourceDownloadStatus(resource_id=resource_id)session.add(status)+ # Whether this failure uses up part of the retry budget+ # (MAX_TOTAL_RETRIES). Only failures of the source do.+ uses_retry_budget = True+# Update status informationif failure.is_permanent():status.status = "permanently_failed"@@ -144,6 +151,26 @@ class ResourceStatusTracker:logger.info(f"[STATUS_TRACKER] Marked resource {resource_id} as permanently failed: {failure.error_type}")+ elif isinstance(failure, OperatorFixableFailure):+ # The operator's own settings refused a file the source served,+ # so repeating it says nothing about the source: hold the+ # resource back for the same cooldown every time, and never+ # escalate it, promote it to permanently failed or spend the+ # retry budget on it. Otherwise the fifth refusal would end+ # the resource's retries however the operator changes the+ # setting.+ uses_retry_budget = False+ status.failure_type = failure.error_type+ status.failure_message = failure.message+ status.status = "temporarily_failed"+ status.retry_after_timestamp = (+ datetime.now(UTC) + failure.retry_after+ )+ logger.info(+ f"[STATUS_TRACKER] Marked resource {resource_id} as temporarily failed: "+ f"{failure.error_type}, retry after: {failure.retry_after} "+ "(operator-fixable, not counted toward the retry limit)"+ )else:status.failure_type = failure.error_typestatus.failure_message = failure.message@@ -171,7 +198,8 @@ class ResourceStatusTracker:# Ensure total_retry_count is initialized (handle None from legacy data)if status.total_retry_count is None:status.total_retry_count = 0- status.total_retry_count += 1+ if uses_retry_budget:+ status.total_retry_count += 1# Check if last attempt was today (before overwriting last_attempt_at)today = datetime.now(UTC).date()diff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.pyindex fb9568f9fd..f9f2179dd1 100644--- a/src/local_deep_research/research_library/services/download_service.py+++ b/src/local_deep_research/research_library/services/download_service.py@@ -256,6 +256,71 @@ def _client_safe_download_message(exc: BaseException) -> str:return client_safe_download_message(exc)+class PDFStorageLimitError(ValueError):+ """The configured PDF storage size limit refuses a PDF.++ It names the fixed reason ``pdf_storage_size_limit`` (see+ ``CLIENT_SAFE_DOWNLOAD_MESSAGES``) that ``DownloadService.download_resource``+ returns, without downloading anything, for a document whose PDF the+ current limit would refuse again (see+ ``DownloadService._pdf_refused_at_current_limit``).+ """+++#: The reason ``download_resource`` returns for a PDF the limit refuses.+PDF_STORAGE_SIZE_LIMIT_REASON = client_safe_download_message(+ PDFStorageLimitError("PDF exceeds the configured storage size limit")+)+++def _pdf_storage_refused(file_path: str | None, storage_mode: str) -> bool:+ """Whether ``save_pdf`` refused a PDF that its effective mode stores.++ ``storage_mode`` is the manager's effective mode+ (``PDFStorageManager.storage_mode``, which coerces any spelling it does+ not know to "none"). In "database" and "filesystem" mode ``save_pdf``+ returns no path only for a PDF over the size limit; in "none" mode it+ never stores one, and that is no refusal.+ """+ return file_path is None and storage_mode in ("filesystem", "database")+++def _keep_refused_pdf_text_only(+ document: Document,+ session: Session,+ pdf_storage_manager: Any,+ pdf_size: int,+) -> bool:+ """Keep a research download whose PDF the size limit refused, text-only.++ The document stays COMPLETED with the text its download extracts, and+ is labelled the way a text-only document is (``file_type`` "pdf",+ ``storage_mode`` "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), which+ is also the shape of a document that awaits its PDF (see+ ``document_awaits_pdf``). So the Download Manager offers its PDF for+ download, and ``file_size`` records how large the refused PDF was:+ ``DownloadService._pdf_refused_at_current_limit`` answers a download of+ it without fetching it again until the operator raises the limit above+ that size.++ Only a research download (with a ``resource_id``) that stores no PDF is+ relabelled. A document that already stores a PDF keeps it, a text+ extraction keeps its label, and an upload or a Zotero import (no+ ``resource_id``) has nothing that would download its PDF. Returns+ whether the document was relabelled.+ """+ if document.resource_id is None or document.file_type == "text":+ return False+ if pdf_storage_manager.has_pdf(document, session):+ return False+ document.file_type = "pdf"+ document.mime_type = "application/pdf"+ document.storage_mode = "none"+ document.file_path = FILE_PATH_TEXT_ONLY+ document.file_size = pdf_size+ return True++_PDF_PERSIST_EXECUTOR: Optional[ThreadPoolExecutor] = None_EXECUTOR_LOCK = threading.Lock()# Bound the background persistence queue: full PDF payloads park in RAM@@ -1585,6 +1650,62 @@ class DownloadService:return queued+ def _links_without_fetch(self, document) -> bool:+ """Whether a resource that finds ``document`` by URL or content links to it.++ The dedup link sites of ``persist_research_sources`` and the+ auto-persist tracker dedup in ``download_resource`` link a resource+ to a completed document instead of downloading it again. A document+ that awaits its PDF (see ``document_awaits_pdf``) is not linked+ there, so that downloading the resource refills it, unless the+ current PDF storage size limit refuses its PDF+ (``_pdf_refused_at_current_limit``): downloading it would then+ fetch the whole PDF only for the limit to refuse it again, so the+ resource is linked as it was before the refusal kept the document+ awaiting its PDF. Once the limit is raised, the linked resource's+ download refills the document through its ``document_id``.+ """+ return not document_awaits_pdf(+ document+ ) or self._pdf_refused_at_current_limit(document)++ def _pdf_refused_at_current_limit(self, document) -> bool:+ """Whether the current PDF storage size limit refuses ``document``'s PDF.++ True for a document that awaits its PDF (see ``document_awaits_pdf``)+ whose recorded ``file_size``, the size of the PDF it last downloaded+ (the one the limit refused, see ``_keep_refused_pdf_text_only``),+ exceeds ``research_library.max_pdf_size_mb`` while the storage mode+ stores PDFs. ``PDFStorageManager.save_pdf`` would refuse those bytes+ again, so downloading them is pointless until the operator raises+ the limit; then this is False and the PDF is downloaded again. A+ storage mode that stores no PDF ("none", or a spelling+ ``PDFStorageManager`` does not know) refuses nothing here.+ """+ if document is None or not document_awaits_pdf(document):+ return False+ size = document.file_size+ if not isinstance(size, int) or size <= 0:+ return False+ storage_mode = resolve_pdf_storage_mode(+ self.settings.get_setting(+ "research_library.pdf_storage_mode", "none"+ )+ )+ if storage_mode not in ("database", "filesystem"):+ return False+ try:+ max_pdf_size_mb = int(+ self.settings.get_setting(+ "research_library.max_pdf_size_mb", DEFAULT_MAX_PDF_SIZE_MB+ )+ )+ except (TypeError, ValueError):+ # Not a limit this check can compare against: leave the+ # decision to the download, which reads the same setting.+ return False+ return size > max_pdf_size_mb * 1024 * 1024+def _is_downloadable(self, resource: ResearchResource) -> bool:"""Check if a resource is likely downloadable as PDF.@@ -1834,7 +1955,7 @@ class DownloadService:).first())- if existing_by_hash and not document_awaits_pdf(+ if existing_by_hash and self._links_without_fetch(existing_by_hash):r.document_id = existing_by_hash.id@@ -1852,7 +1973,7 @@ class DownloadService:.order_by(Document.created_at.desc()).first())- if existing_by_url and not document_awaits_pdf(+ if existing_by_url and self._links_without_fetch(existing_by_url):r.document_id = existing_by_url.id@@ -1868,6 +1989,7 @@ class DownloadService:enable_pdf_storageand is_dland document_awaits_pdf(doc)+ and not self._pdf_refused_at_current_limit(doc)):items_to_process.append((r.id, r.url, is_dl))else:@@ -1887,9 +2009,16 @@ class DownloadService:f"Auto-persisted PDF for resource {resource_id} ({redact_url_for_log(url)})")else:- logger.info(- f"PDF download failed for resource {resource_id} ({failure_reason_for_log(error)}), attempting text fallback"- )+ if error == PDF_STORAGE_SIZE_LIMIT_REASON:+ logger.info(+ f"PDF of resource {resource_id} not stored: it "+ "exceeds the configured PDF storage size "+ "limit; attempting text fallback"+ )+ else:+ logger.info(+ f"PDF download failed for resource {resource_id} ({failure_reason_for_log(error)}), attempting text fallback"+ )text_success, text_error = self.download_as_text(resource_id)@@ -1971,6 +2100,25 @@ class DownloadService:)return True, None+ # A document that keeps its text because the PDF storage size+ # limit refused its PDF is not downloaded again while the limit+ # still refuses a PDF of that size. Nothing is fetched or+ # recorded: the retry manager keeps no cooldown for it, so the+ # text routes that ask it are not held back, and the PDF is+ # downloaded as soon as the operator raises the limit.+ # A resource linked to another resource's document by content+ # (``document_id``) refills that document, which the query+ # above leaves out.+ refill_doc = existing_doc+ if refill_doc is None and resource.document_id:+ refill_doc = session.get(Document, resource.document_id)+ if self._pdf_refused_at_current_limit(refill_doc):+ logger.info(+ f"Not downloading the PDF of resource {resource_id}: it "+ "exceeds the configured PDF storage size limit"+ )+ return False, PDF_STORAGE_SIZE_LIMIT_REASON+# Get collection_id from queue entry if it existsqueue_entry = (session.query(LibraryDownloadQueue)@@ -2027,7 +2175,7 @@ class DownloadService:if (first_docand first_doc.status == DocumentStatus.COMPLETED- and not document_awaits_pdf(first_doc)+ and self._links_without_fetch(first_doc)):# Link without skipping bookkeeping: the caller may# have queued this resource into a specific collection,@@ -2558,6 +2706,10 @@ class DownloadService:sess.add(attempt)canonical_retry = False+ # Set when the size limit refuses this PDF: the document then+ # keeps its text and awaits its PDF until the limit allows it+ # (see _keep_refused_pdf_text_only).+ pdf_refused = False# Older placeholder recovery could leave a nonempty PDF# labelled "unknown"; that content still needs preservation.if (@@ -2668,12 +2820,28 @@ class DownloadService:# document_awaits_pdf from asking for the PDF again.existing_doc.file_type = "pdf"existing_doc.mime_type = "application/pdf"+ elif _pdf_storage_refused(+ file_path_result, pdf_storage_manager.storage_mode+ ):+ # The size limit refused the PDF. Keep the document+ # and the text saved below, labelled as storing no+ # PDF, and record the refused size, so the PDF is+ # downloaded again only once the limit allows it. A+ # refill of a document that awaits its PDF stays+ # awaiting it.+ pdf_refused = True+ _keep_refused_pdf_text_only(+ existing_doc,+ sess,+ pdf_storage_manager,+ len(pdf_content),+ )elif refills_missing_pdf:# The PDF of a document that still needed one was- # downloaded but not stored, as in storage mode- # "none" or for a PDF over the size limit. Leave the- # document as a "none"-mode download leaves one, so- # that the download paths stop fetching it again.+ # downloaded but not stored, because storage mode+ # "none" stores no PDF. Leave the document as a+ # "none"-mode download leaves one, so that the+ # download paths stop fetching it again.existing_doc.file_path = None# Update tracker@@ -2780,6 +2948,22 @@ class DownloadService:resource_id=target_resource.id,))+ if _pdf_storage_refused(+ file_path_result,+ pdf_storage_manager.storage_mode,+ ):+ # The size limit refused the upgrade.+ # Link to the canonical document as+ # before, and keep it text-only until+ # the limit allows its PDF (see+ # _keep_refused_pdf_text_only).+ pdf_refused = True+ _keep_refused_pdf_text_only(+ existing_by_hash,+ sess,+ pdf_storage_manager,+ len(pdf_content),+ )target_tracker.file_path = (file_path_resultif file_path_result@@ -2871,11 +3055,17 @@ class DownloadService:# re-download, not data loss.target_tracker.file_path = Nonetarget_tracker.file_name = None- if document_awaits_pdf(existing_by_hash):- # Storage was disabled or refused this PDF's- # size. Settle a linked refill as a text-only- # download, just like the own-document path,- # so every bulk run does not fetch it again.+ if (+ document_awaits_pdf(existing_by_hash)+ and not pdf_refused+ ):+ # Storage was disabled. Settle a linked+ # refill as a text-only download, just like+ # the own-document path, so every bulk run+ # does not fetch it again. One the size+ # limit refused keeps awaiting its PDF; the+ # limit check in download_resource stops the+ # fetches until the limit allows it.existing_by_hash.file_path = Noneexcept Exception as exc:log_exception_type(@@ -2908,6 +3098,19 @@ class DownloadService:)pdf_document_id = doc.idtarget_resource.document_id = doc.id+ if _pdf_storage_refused(+ file_path_result, pdf_storage_manager.storage_mode+ ):+ # The size limit refused the PDF: keep the new+ # document and its text, awaiting its PDF until+ # the limit allows it.+ pdf_refused = True+ _keep_refused_pdf_text_only(+ doc,+ sess,+ pdf_storage_manager,+ len(pdf_content),+ )target_tracker.file_path = (file_path_result if file_path_result else None)@@ -2921,7 +3124,13 @@ class DownloadService:tracker.file_path = target_tracker.file_pathtracker.file_name = target_tracker.file_name- if pdf_storage_mode == "database":+ if pdf_refused:+ logger.warning(+ "PDF not stored: it exceeds the configured PDF "+ f"storage size limit ({max_pdf_size_mb} MB); keeping "+ f"its text: {redact_url_for_log(resource.url)}"+ )+ elif pdf_storage_mode == "database":logger.info(f"Successfully stored PDF in database: {redact_url_for_log(resource.url)}")diff --git a/src/local_deep_research/research_library/utils/__init__.py b/src/local_deep_research/research_library/utils/__init__.pyindex 97a4132b27..603eb650fd 100644--- a/src/local_deep_research/research_library/utils/__init__.py+++ b/src/local_deep_research/research_library/utils/__init__.py@@ -346,12 +346,17 @@ def document_awaits_pdf(document) -> bool:a PDF download completed before ``_download_pdf`` labelled suchdocuments "pdf", is picked up too.- No other writer leaves a research download in that shape: a- "none"-mode download keeps ``file_path=None``, "Remove PDF to save- space" writes ``blob_deleted``, and a text extraction is ``file_type``- "text". A download into the document that stores no PDF either, as in- storage mode "none" or for a PDF over the size limit, clears the mark,- see ``DownloadService._download_pdf``.+ The one other writer of that shape is a download whose PDF the PDF+ storage size limit refuses (``_keep_refused_pdf_text_only`` in the+ download service): its document keeps the extracted text and awaits+ its PDF too, with the refused size in ``file_size``, and+ ``download_resource`` does not fetch it again while the limit still+ refuses that size (``DownloadService._pdf_refused_at_current_limit``).+ Otherwise a "none"-mode download keeps ``file_path=None``, "Remove PDF+ to save space" writes ``blob_deleted``, and a text extraction is+ ``file_type`` "text". A download into the document in storage mode+ "none", which stores no PDF, clears the mark, see+ ``DownloadService._download_pdf``."""return (document.status == DocumentStatus.COMPLETEDdiff --git a/src/local_deep_research/security/client_safe_errors.py b/src/local_deep_research/security/client_safe_errors.pyindex 262edf598f..a2a220e945 100644--- a/src/local_deep_research/security/client_safe_errors.py+++ b/src/local_deep_research/security/client_safe_errors.py@@ -49,6 +49,7 @@ CLIENT_SAFE_DOWNLOAD_MESSAGES: dict[str, str] = {"InterfaceError": "database_interface_error","InvalidRequestError": "database_invalid_request",# PDF / extraction surface — never echo absolute paths or parser internals.+ "PDFStorageLimitError": "pdf_storage_size_limit","PdfReadError": "pdf_parse_failed","PdfStreamError": "pdf_parse_failed",}@@ -94,3 +95,28 @@ def client_safe_download_message(exc: BaseException) -> str:return CLIENT_SAFE_DOWNLOAD_MESSAGES.get(type(exc).__name__, f"download_error:{type(exc).__name__}")+++# What a person reads for a skip-reason token they can act on themselves.+# The tokens stay what the services return and the retry manager classifies;+# the routes that show a download result to a person (the bulk download+# stream and the single-download endpoints) show this text instead.+CLIENT_SAFE_DOWNLOAD_USER_MESSAGES: dict[str, str] = {+ "pdf_storage_size_limit": (+ "The PDF is larger than the PDF storage size limit, so it was not "+ "stored (its text is kept when it could be extracted). Raise "+ '"Maximum PDF Size (MB)" in the research library settings to '+ "store it."+ ),+}+++def download_reason_for_user(reason):+ """Return the text to show a person for a download skip *reason*.++ A token listed in ``CLIENT_SAFE_DOWNLOAD_USER_MESSAGES`` becomes its+ message; any other value is returned unchanged.+ """+ if isinstance(reason, str):+ return CLIENT_SAFE_DOWNLOAD_USER_MESSAGES.get(reason, reason)+ return reasondiff --git a/src/local_deep_research/web/routers/library.py b/src/local_deep_research/web/routers/library.pyindex 0b6adf1951..d6e4e60c4d 100644--- a/src/local_deep_research/web/routers/library.py+++ b/src/local_deep_research/web/routers/library.py@@ -47,6 +47,10 @@ from ...research_library.services.pdf_storage_manager import (resolve_pdf_storage_mode,)from ...security import failure_reason_for_log, redact_url_for_log+from ...security.client_safe_errors import (+ CLIENT_SAFE_DOWNLOAD_USER_MESSAGES,+ download_reason_for_user,+)from ...research_library.utils import (apply_user_subdir,document_awaits_pdf_clause,@@ -777,6 +781,13 @@ def download_single_resource({"success": False, "error": "Resource not found"},status_code=404,)+ if error in CLIENT_SAFE_DOWNLOAD_USER_MESSAGES:+ # A refusal the operator can lift (the PDF storage size limit):+ # say how, rather than suggest trying again.+ return JSONResponse(+ {"success": False, "error": download_reason_for_user(error)},+ status_code=409,+ )# The reason may be exception text that keeps the URL's path and# query; download_service logs the cause itself where it is caught.logger.warning(@@ -1442,7 +1453,11 @@ async def download_bulk(}# Add skip reason if availableif skip_reason:- update_data["error"] = skip_reason+ # A token the operator can act on (the PDF storage+ # size limit) is shown as the message that says how.+ update_data["error"] = download_reason_for_user(+ skip_reason+ )logger.info("Sending skip reason to UI: "f"{failure_reason_for_log(skip_reason)}"@@ -2105,6 +2120,12 @@ async def download_source(if success:return {"success": True, "message": "Download completed"}+ if message in CLIENT_SAFE_DOWNLOAD_USER_MESSAGES:+ # A refusal the operator can lift: say how.+ return {+ "success": False,+ "message": download_reason_for_user(message),+ }# Log internal message, but show only generic message to userreturn {"success": False, "message": "Download failed"}diff --git a/src/local_deep_research/web/routers/rag.py b/src/local_deep_research/web/routers/rag.pyindex b7315676b4..1da56c3590 100644--- a/src/local_deep_research/web/routers/rag.py+++ b/src/local_deep_research/web/routers/rag.py@@ -2931,21 +2931,39 @@ def _upload_to_collection_sync(pdf_stored = Falseif store_pdf_in_db:try:- pdf_storage_manager.save_pdf(+ stored_path, _ = pdf_storage_manager.save_pdf(pdf_content=file_content,document=new_doc,session=db_session,filename=filename,)- pdf_stored = True- logger.info(- f"PDF stored in encrypted database for {filename}"- )+ # A PDF the manager refuses (over its size+ # limit) is reported by returning no path, not+ # by raising, so only a returned path means+ # the PDF was stored.+ pdf_stored = bool(stored_path)+ if pdf_stored:+ logger.info(+ f"PDF stored in encrypted database for {filename}"+ )+ else:+ logger.warning(+ f"PDF for {filename} was not stored "+ "(over the storage size limit); "+ "keeping the extracted text only"+ )except Exception:logger.exception(f"Failed to store PDF in database for {filename}")# Continue without PDF storage - text is still saved+ if not pdf_stored:+ # The document was created labelled as holding+ # its PDF in the database. Nothing was stored,+ # so label it as the text-only upload it is,+ # as for an upload with PDF storage off.+ new_doc.storage_mode = "none"+ new_doc.file_path = FILE_PATH_TEXT_ONLY# Add to collectionensure_in_collection(diff --git a/tests/library/download_management/test_pdf_storage_limit_failure_6913.py b/tests/library/download_management/test_pdf_storage_limit_failure_6913.pynew file mode 100644index 0000000000..046f443ead--- /dev/null+++ b/tests/library/download_management/test_pdf_storage_limit_failure_6913.py@@ -0,0 +1,212 @@+"""PR #6913: a PDF storage-size refusal is classified deliberately and stays retryable.++``DownloadService.download_resource`` answers a download whose PDF the+configured storage size limit refuses with the fixed reason+``pdf_storage_size_limit``. That is a setting of the operator, not a fault of+the source. ``download_resource`` itself does not record that answer with the+retry manager (the limit decides when the PDF is fetched again, see+``tests/research_library/test_download_service_contracts.py``), but the token+stays classified for any caller that does, and the retry layer has to keep+the resource retryable on its own terms:++* the reason must be a token the classifier knows. Left out, it fell through+ to ``unknown_error`` with an "Unclassified error" warning;+* classifying it as an ordinary temporary failure is not enough either.+ Every temporary failure follows the same schedule (its own cooldown once,+ then 1 day, 30 days, 30 days, and a permanent failure at the fifth), so+ the fifth refusal would end the resource's retries whatever the operator+ does with the limit.++Every assertion on the refusal is paired with the same drive of an ordinary+temporary failure (``database_error``), which does go permanent, so a+harness that could not see a permanent failure cannot pass here.+"""++from datetime import UTC, datetime, timedelta+from unittest.mock import patch++from sqlalchemy import create_engine+from sqlalchemy.orm import sessionmaker++from local_deep_research.library.download_management import (+ failure_classifier,+ retry_manager as retry_manager_module,+)+from local_deep_research.library.download_management.failure_classifier import (+ FailureClassifier,+)+from local_deep_research.library.download_management.models import (+ Base,+ ResourceDownloadStatus,+)+from local_deep_research.library.download_management.status_tracker import (+ MAX_TOTAL_RETRIES,+ ResourceStatusTracker,+)+from local_deep_research.security.client_safe_errors import (+ CLIENT_SAFE_DOWNLOAD_MESSAGES,+)++# The reason DownloadService returns for a refusal, read from the table that+# produces it so the two cannot drift apart unnoticed.+REFUSAL = CLIENT_SAFE_DOWNLOAD_MESSAGES["PDFStorageLimitError"]+# A token for an ordinary temporary failure of the source side.+ORDINARY = "database_error"+++def _retry_manager():+ """A real ``RetryManager``: real classifier, real tracker, in-memory SQLite."""+ engine = create_engine("sqlite:///:memory:")+ Base.metadata.create_all(engine)+ tracker = ResourceStatusTracker.__new__(ResourceStatusTracker)+ tracker.Session = sessionmaker(bind=engine)+ with patch.object(+ retry_manager_module, "ResourceStatusTracker", return_value=tracker+ ):+ return retry_manager_module.RetryManager("user")+++def _fail(manager, reason, resource_id=1):+ """Record a failed attempt with the arguments ``download_resource`` passes.++ ``DownloadService.download_resource`` records every failed download+ this way, but it answers a size-limit refusal without recording it, so+ this stands in for any caller that does record the refusal reason.+ """+ with manager.status_tracker.Session() as session:+ manager.record_attempt(+ resource_id=resource_id,+ result=(False, reason),+ status_code=None,+ url="https://example.com/huge.pdf",+ details=reason,+ session=session,+ )+++def _status(manager, resource_id=1):+ return manager.status_tracker.get_resource_status(resource_id)+++def _let_two_days_pass(manager, resource_id=1):+ """Move the stored attempt and retry times two days into the past."""+ with manager.status_tracker.Session() as session:+ row = (+ session.query(ResourceDownloadStatus)+ .filter_by(resource_id=resource_id)+ .one()+ )+ row.last_attempt_at -= timedelta(days=2)+ if row.retry_after_timestamp is not None:+ row.retry_after_timestamp -= timedelta(days=2)+ session.commit()+++class TestClassification:+ def test_the_refusal_token_is_classified_not_unclassified(self):+ classifier = FailureClassifier()+ # RetryManager.record_attempt hands the token over in ``details``+ # with ``error_type`` "str", or in ``error_type`` as well.+ for error_type, details in (+ ("str", REFUSAL),+ (REFUSAL, REFUSAL),+ (REFUSAL, ""),+ ):+ with patch.object(failure_classifier, "logger") as log:+ failure = classifier.classify_failure(+ error_type, details=details+ )+ assert failure.error_type == REFUSAL, (error_type, details)+ assert failure.retry_after == timedelta(days=1)+ assert not failure.is_permanent()+ log.warning.assert_not_called()++ # Control: the same spy does see the fall-through for an unknown token.+ with patch.object(failure_classifier, "logger") as log:+ unknown = classifier.classify_failure(+ "str", details="not_a_known_token"+ )+ assert unknown.error_type == "unknown_error"+ log.warning.assert_called_once()+ assert "Unclassified error" in log.warning.call_args.args[0]++ def test_every_token_the_download_service_can_emit_is_classified(self):+ classifier = FailureClassifier()+ tokens = set(CLIENT_SAFE_DOWNLOAD_MESSAGES.values())+ assert REFUSAL in tokens+ unclassified = {+ token+ for token in tokens+ if classifier.classify_failure("str", details=token).error_type+ == "unknown_error"+ }+ assert unclassified == set()+++class TestRefusalStaysRetryable:+ def test_repeated_refusals_never_become_permanent(self):+ manager = _retry_manager()++ # More refusals than the schedule allows an ordinary failure.+ for attempt in range(1, MAX_TOTAL_RETRIES + 4):+ _fail(manager, REFUSAL)+ status = _status(manager)+ assert status["status"] == "temporarily_failed", (attempt, status)++ assert status["failure_type"] == REFUSAL+ # Held back for one cooldown after the latest refusal, not for the+ # 30 days an ordinary failure's third attempt would earn.+ wait = datetime.fromisoformat(+ status["retry_after_timestamp"]+ ) - datetime.now(UTC)+ assert timedelta(hours=23) < wait <= timedelta(days=1)+ decision = manager.should_retry_resource(1)+ assert decision.can_retry is False+ assert "Cooldown active" in decision.reason++ def test_an_ordinary_temporary_failure_does_go_permanent(self):+ """Control for the test above: this harness can see a permanent failure."""+ manager = _retry_manager()++ for _ in range(MAX_TOTAL_RETRIES):+ _fail(manager, ORDINARY)++ assert _status(manager)["status"] == "permanently_failed"+ decision = manager.should_retry_resource(1)+ assert decision.can_retry is False+ assert "Permanently failed" in decision.reason++ def test_a_refused_resource_is_retryable_again_after_one_cooldown(self):+ refused = _retry_manager()+ for _ in range(MAX_TOTAL_RETRIES + 3):+ _fail(refused, REFUSAL)+ assert refused.should_retry_resource(1).can_retry is False++ _let_two_days_pass(refused)++ assert refused.should_retry_resource(1).can_retry is True++ # Control: the same passage of time does not reopen an ordinary+ # failure that used up its schedule.+ ordinary = _retry_manager()+ for _ in range(MAX_TOTAL_RETRIES):+ _fail(ordinary, ORDINARY)+ _let_two_days_pass(ordinary)+ assert ordinary.should_retry_resource(1).can_retry is False++ def test_refusals_neither_use_nor_reset_the_budget_of_source_failures(self):+ manager = _retry_manager()+ for _ in range(MAX_TOTAL_RETRIES - 1):+ _fail(manager, ORDINARY)+ assert _status(manager)["total_retry_count"] == MAX_TOTAL_RETRIES - 1++ for _ in range(3):+ _fail(manager, REFUSAL)++ status = _status(manager)+ assert status["total_retry_count"] == MAX_TOTAL_RETRIES - 1+ assert status["status"] == "temporarily_failed"++ # The fifth failure of the source still ends the retries.+ _fail(manager, ORDINARY)+ assert _status(manager)["status"] == "permanently_failed"diff --git a/tests/research_library/routes/test_rag_routes_deep_main_port.py b/tests/research_library/routes/test_rag_routes_deep_main_port.pyindex 50165ca188..02e85a740b 100644--- a/tests/research_library/routes/test_rag_routes_deep_main_port.py+++ b/tests/research_library/routes/test_rag_routes_deep_main_port.py@@ -468,7 +468,8 @@ class TestUploadPdfStorageDatabase:))pdf_manager = Mock()- pdf_manager.save_pdf = Mock()+ # ``save_pdf`` returns ``(path or None, size)``; a path means stored.+ pdf_manager.save_pdf = Mock(return_value=("database", 17))result = self._call(files_data=[self._file("test.pdf", b"%PDF-test-content")],diff --git a/tests/research_library/routes/test_rag_routes_upload_main_port.py b/tests/research_library/routes/test_rag_routes_upload_main_port.pyindex 90a3247935..4df9784c36 100644--- a/tests/research_library/routes/test_rag_routes_upload_main_port.py+++ b/tests/research_library/routes/test_rag_routes_upload_main_port.py@@ -244,6 +244,15 @@ def _fail_document_flush(db_session, filename, message):return failure+def _added(db_session, model):+ """Every ``model`` instance the route added to the session, in order."""+ return [+ call.args[0]+ for call in db_session.add.call_args_list+ if isinstance(call.args[0], model)+ ]++def _make_settings_mock(overrides=None):"""Settings manager mock with the RAG defaults the upload path reads."""from local_deep_research.constants import (@@ -1769,8 +1778,15 @@ class TestUploadToCollection:db_session.query = Mock(side_effect=query_side_effect)+ from local_deep_research.database.models.library import Document++ pdf_bytes = b"%PDF-1.4 real pdf content"mock_pdf_manager = Mock()- mock_pdf_manager.save_pdf = Mock()+ # ``save_pdf`` returns ``(path or None, size)``; the path is what+ # tells a stored PDF from a refused one.+ mock_pdf_manager.save_pdf = Mock(+ return_value=("database", len(pdf_bytes))+ )with _upload_env(db_session=db_session,@@ -1783,10 +1799,7 @@ class TestUploadToCollection:],):status, rdata = _body(- _run_upload(- [("report.pdf", b"%PDF-1.4 real pdf content")],- pdf_storage="database",- )+ _run_upload([("report.pdf", pdf_bytes)], pdf_storage="database"))assert status == 200@@ -1795,6 +1808,99 @@ class TestUploadToCollection:assert rdata["uploaded"][0]["status"] == "uploaded"assert rdata["uploaded"][0]["pdf_stored"] is Truemock_pdf_manager.save_pdf.assert_called_once()+ # A stored PDF keeps the label the document was created with.+ (doc,) = _added(db_session, Document)+ assert (doc.storage_mode, doc.file_path) == ("database", None)++ @pytest.mark.parametrize(+ "max_mb, stored",+ [(1, False), (3072, True)],+ ids=["over-the-limit", "under-the-limit"],+ )+ def test_upload_pdf_over_the_storage_limit_is_text_only(+ self, tmp_path, max_mb, stored+ ):+ """A PDF the storage manager refuses must not be reported as stored.++ ``save_pdf`` signals a refusal (the PDF is over its size limit) by+ returning no path, not by raising. The upload read that as success:+ ``pdf_stored=True`` on a Document labelled ``storage_mode="database"``+ with no blob behind it, the false "stored" state that PR #6913 removes+ from downloads. The refusal here comes from a real+ ``PDFStorageManager`` with a 1 MB limit; the control (a limit the PDF+ is under) goes through the identical path and does store it.+ """+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.database.models.library import (+ Document,+ DocumentBlob,+ )+ from local_deep_research.research_library.services.pdf_storage_manager import (+ PDFStorageManager,+ )++ mock_coll = Mock()+ mock_coll.id = "coll-1"++ mock_source = Mock()+ mock_source.id = "src-005"++ db_session = _make_db_session()+ call_count = {"n": 0}++ def query_side_effect(*args):+ call_count["n"] += 1+ q = _build_mock_query()+ if call_count["n"] == 1:+ q.first.return_value = mock_coll+ elif call_count["n"] == 2:+ q.first.return_value = None+ elif call_count["n"] == 3:+ q.first.return_value = mock_source+ return q++ db_session.query = Mock(side_effect=query_side_effect)++ pdf_bytes = b"%PDF-1.4 " + b"\x00" * (1024 * 1024)+ manager = PDFStorageManager(+ library_root=tmp_path,+ storage_mode="database",+ max_pdf_size_mb=max_mb,+ )++ with _upload_env(+ db_session=db_session,+ settings_overrides={+ "research_library.upload_pdf_storage": "database"+ },+ extra_patches=[+ *_extraction_patches(text="PDF extracted text"),+ patch(_PDF_MGR, return_value=manager),+ ],+ ):+ status, rdata = _body(+ _run_upload([("huge.pdf", pdf_bytes)], pdf_storage="database")+ )++ assert status == 200+ assert rdata["success"] is True+ assert rdata["errors"] == []+ assert len(rdata["uploaded"]) == 1+ assert rdata["uploaded"][0]["status"] == "uploaded"+ assert rdata["uploaded"][0]["pdf_stored"] is stored+ (doc,) = _added(db_session, Document)+ blobs = _added(db_session, DocumentBlob)+ # The extracted text is kept either way.+ assert doc.text_content == "PDF extracted text"+ if stored:+ assert (doc.storage_mode, doc.file_path) == ("database", None)+ assert [blob.pdf_binary for blob in blobs] == [pdf_bytes]+ else:+ assert (doc.storage_mode, doc.file_path) == (+ "none",+ FILE_PATH_TEXT_ONLY,+ )+ assert blobs == []def test_upload_pdf_storage_failure_continues(self):"""Ported from ``...::test_upload_pdf_storage_failure_continues``.@@ -1850,6 +1956,15 @@ class TestUploadToCollection:assert len(rdata["uploaded"]) == 1assert rdata["uploaded"][0]["status"] == "uploaded"assert rdata["uploaded"][0]["pdf_stored"] is False+ # Nothing was stored, so the document must not claim a database blob.+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.database.models.library import Document++ (doc,) = _added(db_session, Document)+ assert (doc.storage_mode, doc.file_path) == (+ "none",+ FILE_PATH_TEXT_ONLY,+ )def test_upload_auto_index_triggered(self):"""Ported from ``...::test_upload_auto_index_triggered``.diff --git a/tests/research_library/test_download_service_contracts.py b/tests/research_library/test_download_service_contracts.pyindex 25723ecd3f..b60553c151 100644--- a/tests/research_library/test_download_service_contracts.py+++ b/tests/research_library/test_download_service_contracts.py@@ -43,9 +43,13 @@ Covered:a large body: a too-large ``Content-Length`` is rejected before thebody is read at all, and a chunked-forever body with no``Content-Length`` is cut off by the installed body guard;- * a download whose bytes are then REJECTED by the storage size limit- is still recorded as a COMPLETED document with nothing stored- (DEFECT -- xfail);+ * a PDF the storage size limit refuses keeps its extracted text in a+ COMPLETED document labelled as storing no PDF and awaiting one (see+ ``document_awaits_pdf``), with the refused size recorded: it is not+ downloaded again while the limit refuses that size, sets no retry+ cooldown for the text routes, and is stored once the limit is raised.+ A refill of a document that awaits its PDF stays awaiting it, while a+ "none"-mode refill still settles as a text-only download;* a ``.pdf`` URL alone does not make a response a PDF, but a``Content-Type: application/pdf`` header alone DOES: the body isnever magic-byte checked when the header claims PDF, and the@@ -68,7 +72,7 @@ Covered:import hashlibimport uuid-from contextlib import contextmanager+from contextlib import contextmanager, nullcontextfrom datetime import timedeltafrom pathlib import Pathfrom types import SimpleNamespace@@ -271,7 +275,12 @@ def _tree(root: Path):@pytest.fixturedef db(tmp_path):- """Real on-disk SQLite, FK enforcement on, plus a fresh-connection reader."""+ """Real on-disk SQLite, FK enforcement on, plus a fresh-connection reader.++ ``download_session`` is a separate session that ``_one_download_session``+ hands the download service for every scope, the way production reuses+ one thread-local session; the fixture closes it at teardown.+ """db_dir = tmp_path / "db"db_dir.mkdir()engine = create_engine(f"sqlite:///{db_dir / 'library.db'}")@@ -293,9 +302,14 @@ def db(tmp_path):# `with Session() as session` rather than try/finally + session.close():# same teardown, and it satisfies the check-session-context-manager hook.- with Session() as session:+ with Session() as session, Session() as download_session:try:- yield SimpleNamespace(engine=engine, session=session, fresh=fresh)+ yield SimpleNamespace(+ engine=engine,+ session=session,+ fresh=fresh,+ download_session=download_session,+ )finally:engine.dispose()@@ -721,72 +735,689 @@ def test_chunked_forever_body_is_cut_off_by_the_body_guard(monkeypatch):assert served.total <= 4096 + 512-def test_storage_size_rejection_is_reported_as_a_completed_download(- db, store, seeded+SEARCHABLE_TEXT = "Searchable text extracted from the oversized PDF."+# 1 MiB of padding against a 1 MB limit: the same+# ``file_size > self.max_pdf_size_bytes`` branch a multi-gigabyte PDF takes,+# without moving multiple gigabytes.+OVERSIZED_PDF = PDF_BYTES + b"\x00" * (1024 * 1024)+REFUSAL = "pdf_storage_size_limit"+++def _one_download_session(db, monkeypatch):+ """Give the download service one reused session, as production does.++ ``get_user_db_session`` hands every scope on a thread the same+ thread-local session and does not close it at scope exit.+ ``_download_pdf`` saves the extracted text in a later scope than the one+ that stored the PDF, through the resource instance that scope loaded.+ With a fresh session per scope (``db.fresh()``) that instance is+ detached and expired by then, so ``_save_text_with_db`` fails on it, the+ failure is logged and swallowed as best-effort, and the text is never+ saved: the tests that patched ``get_user_db_session`` that way read+ ``text_content`` as None. ``test_library_sync_contracts`` documents the+ same trap on its ``one_download_session`` fixture.+ """+ from local_deep_research.research_library.services import download_service++ monkeypatch.setattr(+ download_service,+ "get_user_db_session",+ lambda *a, **k: nullcontext(db.download_session),+ )+++def _refusing_service(db, store, monkeypatch, fixture, mode, *, existing):+ """A service whose 1 MB limit refuses ``OVERSIZED_PDF`` in ``mode``.++ ``existing`` keeps the seeded FAILED document (the retry branch);+ otherwise it is deleted so the download creates a new document. Text+ extraction returns ``SEARCHABLE_TEXT``, so a test can tell whether the+ text survived the refusal. Every ``record_attempt`` call is collected.+ The service gets one reused session (see ``_one_download_session``).+ """+ from local_deep_research.database.models.library import Collection+ from local_deep_research.research_library.services import download_service++ collection_id = "retry-library"+ db.session.add(Collection(id=collection_id, name="Retry library"))+ if not existing:+ db.session.delete(fixture.document)+ db.session.commit()+ monkeypatch.setattr(+ download_service,+ "get_source_type_id",+ lambda *a: fixture.source_type.id,+ )+ monkeypatch.setattr(+ download_service, "get_default_library_id", lambda *a: collection_id+ )+ _one_download_session(db, monkeypatch)+ downloader = StubDownloader(+ DownloadResult(content=OVERSIZED_PDF, is_success=True, status_code=200)+ )+ service = make_service(store, downloader, storage_mode=mode, max_mb=1)+ service._extract_text_from_pdf = lambda content: SEARCHABLE_TEXT+ recorded = []+ service.retry_manager = SimpleNamespace(+ record_attempt=lambda **kwargs: recorded.append(kwargs)+ )+ return service, downloader, recorded++[email protected]("mode", ["database", "filesystem"])+ "existing", [False, True], ids=["new-document", "failed-document"]+)+def test_a_refused_pdf_keeps_its_text_in_a_document_awaiting_the_pdf(+ db, store, seeded, filesystem_mode, monkeypatch, mode, existing):- """CHARACTERIZATION of a defect (see the xfail below for the contract).-- ``save_pdf`` returns ``(None, size)`` when the bytes exceed- ``research_library.max_pdf_size_mb``, but ``_download_pdf`` has- already flipped the Document to COMPLETED and does not look at that- return value: the call reports success, the row says COMPLETED, and- there is no blob and no file. ``download_resource`` then short-circuits- on that COMPLETED row forever, so the PDF is never fetched again.-- Driven with a 1 MiB payload against a 1 MB limit -- the same- ``file_size > self.max_pdf_size_bytes`` branch a multi-gigabyte PDF- takes, without moving multiple gigabytes.+ """The size limit refuses the PDF, not the download.++ The document is COMPLETED with the extracted text saved, so it stays+ searchable, and it is labelled as storing no PDF and awaiting one+ (``document_awaits_pdf``), with the refused PDF's size recorded. No+ bytes are stored anywhere, and the retry manager records a success."""- fixture = seeded("https://evil.example.com/huge.pdf")- oversized = PDF_BYTES + b"\x00" * (1024 * 1024)+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.research_library.utils import document_awaits_pdf++ fixture = seeded("https://example.com/huge.pdf")+ resource_id = fixture.resource.id+ tracker_id = fixture.tracker.id+ service, downloader, recorded = _refusing_service(+ db, store, monkeypatch, fixture, mode, existing=existing+ )++ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 1+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=resource_id).one()+ assert doc.status is DocumentStatus.COMPLETED+ assert doc.text_content == SEARCHABLE_TEXT+ assert (doc.file_type, doc.storage_mode, doc.file_path) == (+ "pdf",+ "none",+ FILE_PATH_TEXT_ONLY,+ )+ assert doc.file_size == len(OVERSIZED_PDF)+ assert document_awaits_pdf(doc)+ assert other.query(DocumentBlob).count() == 0+ tracker = other.get(DownloadTracker, tracker_id)+ assert (tracker.is_downloaded, tracker.file_path) == (True, None)+ assert other.query(DownloadAttempt).one().succeeded is True+ assert _tree(store.sandbox) == set()+ assert [call["result"] for call in recorded] == [(True, None)]++[email protected]("mode", ["database", "filesystem"])+def test_a_refused_pdf_is_not_downloaded_again_while_the_limit_refuses_it(+ db, store, seeded, filesystem_mode, monkeypatch, mode+):+ """No daily (or per-run) re-download of a PDF the limit still refuses.++ ``download_resource`` answers with the refusal before fetching anything,+ and records nothing with the retry manager, as long as the limit is+ below the recorded size. The document and its text are left as they are.+ """+ from local_deep_research.research_library.utils import document_awaits_pdf++ fixture = seeded("https://example.com/huge.pdf")+ resource_id = fixture.resource.id+ service, downloader, recorded = _refusing_service(+ db, store, monkeypatch, fixture, mode, existing=False+ )+ assert service.download_resource(resource_id) == (True, None)++ for _ in range(3):+ assert service.download_resource(resource_id) == (False, REFUSAL)+ assert len(downloader.urls) == 1+ assert len(recorded) == 1+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=resource_id).one()+ assert document_awaits_pdf(doc)+ assert doc.text_content == SEARCHABLE_TEXT+ assert other.query(DownloadAttempt).count() == 1+ assert _tree(store.sandbox) == set()++[email protected]("mode", ["database", "filesystem"])+ "existing", [False, True], ids=["new-document", "failed-document"]+)+def test_a_refused_pdf_is_stored_once_the_limit_is_raised(+ db, store, seeded, filesystem_mode, monkeypatch, mode, existing+):+ """Raising the limit above the PDF's size makes the next download store it.++ The PDF goes into the same document, which keeps its text and no+ longer awaits its PDF.+ """+ from local_deep_research.research_library.utils import document_awaits_pdf++ fixture = seeded("https://example.com/huge.pdf")+ resource_id = fixture.resource.id+ service, downloader, _recorded = _refusing_service(+ db, store, monkeypatch, fixture, mode, existing=existing+ )+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ doc_id = (+ other.query(Document).filter_by(resource_id=resource_id).one().id+ )++ service.settings._values["research_library.max_pdf_size_mb"] = 2+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=resource_id).one()+ assert doc.id == doc_id+ assert doc.status is DocumentStatus.COMPLETED+ assert not document_awaits_pdf(doc)+ assert doc.storage_mode == mode+ assert doc.text_content == SEARCHABLE_TEXT+ manager = PDFStorageManager(library_root=store.root, storage_mode=mode)+ assert manager.load_pdf(doc, other) == OVERSIZED_PDF+++def test_a_refusal_sets_no_cooldown_for_the_text_routes(+ db, store, seeded, monkeypatch+):+ """The refusal must not hold back the text routes through the retry layer.++ ``download_as_text`` and ``_try_arxiv_text_extraction`` both ask+ ``RetryManager.should_retry_resource`` before any network access. With+ a real ``RetryManager`` recording every ``download_resource`` call, the+ first (refused) download records a success, and the answers while the+ limit refuses the PDF record nothing, however often they are asked, so+ the resource stays retryable for those routes.+ """+ from local_deep_research.library.download_management import (+ retry_manager as retry_manager_module,+ )+ from local_deep_research.library.download_management.models import (+ Base as RetryBase,+ )+ from local_deep_research.library.download_management.status_tracker import (+ MAX_TOTAL_RETRIES,+ ResourceStatusTracker,+ )++ RetryBase.metadata.create_all(db.engine)+ tracker = ResourceStatusTracker.__new__(ResourceStatusTracker)+ tracker.Session = sessionmaker(bind=db.engine)+ monkeypatch.setattr(+ retry_manager_module,+ "ResourceStatusTracker",+ lambda *args, **kwargs: tracker,+ )+ retry_manager = retry_manager_module.RetryManager(USERNAME)++ fixture = seeded("https://example.com/huge.pdf")+ resource_id = fixture.resource.id+ service, downloader, _recorded = _refusing_service(+ db, store, monkeypatch, fixture, "database", existing=False+ )+ service.retry_manager = retry_manager++ assert service.download_resource(resource_id) == (True, None)+ for _ in range(MAX_TOTAL_RETRIES + 2):+ assert service.download_resource(resource_id) == (False, REFUSAL)+ assert len(downloader.urls) == 1+ assert retry_manager.should_retry_resource(resource_id).can_retry is True+ status = retry_manager.status_tracker.get_resource_status(resource_id)+ assert status is None or status["status"] == "completed"+++ "size,refused",+ [(1024 * 1024, False), (1024 * 1024 + 1, True)],+ ids=["at-the-limit", "one-byte-over"],+)+def test_the_limit_check_refuses_only_a_pdf_over_the_limit(+ store, size, refused+):+ """The check refuses exactly what ``save_pdf`` refuses: more than the+ limit (``file_size > max_pdf_size_bytes``), not a PDF of exactly the+ limit's size."""+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY++ document = Document(+ status=DocumentStatus.COMPLETED,+ file_type="pdf",+ storage_mode="none",+ file_path=FILE_PATH_TEXT_ONLY,+ resource_id=1,+ file_size=size,+ )+ service = make_service(+ store, StubDownloader(), storage_mode="database", max_mb=1+ )+ assert service._pdf_refused_at_current_limit(document) is refused+ manager = PDFStorageManager(+ library_root=store.root, storage_mode="database", max_pdf_size_mb=1+ )+ assert (size > manager.max_pdf_size_bytes) is refused+++def _second_resource_on(db, fixture, url):+ """Another research resource citing ``url``, with no tracker of its own."""+ resource = ResearchResource(+ research_id=fixture.resource.research_id,+ title="Same Paper, Cited Again",+ url=url,+ source_type="academic",+ created_at="2026-08-01T00:00:00",+ )+ db.session.add(resource)+ db.session.commit()+ return resource.id+++def test_auto_persist_links_a_second_resource_to_a_refused_pdf_without_fetching(+ db, store, seeded, monkeypatch+):+ """Auto-persist tracker dedup in ``download_resource``.++ A second resource on the same URL finds the document whose PDF the+ limit refused. It is linked to it without downloading the PDF again,+ and once the limit is raised, its download stores the PDF there.+ """+ from local_deep_research.research_library.utils import document_awaits_pdf++ url = "https://example.com/huge.pdf"+ fixture = seeded(url)+ first_id = fixture.resource.id+ service, downloader, _recorded = _refusing_service(+ db, store, monkeypatch, fixture, "database", existing=False+ )+ service._settings_snapshot = {+ "research_library.auto_persist_research_sources": True+ }+ assert service.download_resource(first_id) == (True, None)+ second_id = _second_resource_on(db, fixture, url)++ assert service.download_resource(+ second_id, collection_id="retry-library"+ ) == (True, None)+ assert len(downloader.urls) == 1+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=first_id).one()+ assert other.get(ResearchResource, second_id).document_id == doc.id+ assert document_awaits_pdf(doc)+ assert other.query(Document).count() == 1++ service.settings._values["research_library.max_pdf_size_mb"] = 2+ assert service.download_resource(+ second_id, collection_id="retry-library"+ ) == (True, None)+ assert len(downloader.urls) == 2+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=first_id).one()+ assert not document_awaits_pdf(doc)+ manager = PDFStorageManager(+ library_root=store.root, storage_mode="database"+ )+ assert manager.load_pdf(doc, other) == OVERSIZED_PDF++[email protected]("link", ["content-hash", "url"])+def test_persist_research_sources_links_to_a_refused_pdf_without_fetching(+ db, store, seeded, monkeypatch, link+):+ """The two dedup links of ``persist_research_sources``.++ ``content-hash``: the URL's tracker names the refused PDF's hash, and+ the document's own URL is another one, so only the hash link can find+ it. ``url``: the tracker has no hash, so only the URL link can. Either+ way the second resource is linked to the document without downloading+ the PDF again, and the first resource's own document, which the limit+ still refuses, is not downloaded again either.+ """+ url = "https://example.com/huge.pdf"+ fixture = seeded(url)+ first_id = fixture.resource.id+ tracker_id = fixture.tracker.id+ service, downloader, _recorded = _refusing_service(+ db, store, monkeypatch, fixture, "database", existing=False+ )+ service.settings._values["document_scheduler.download_pdfs"] = True+ assert service.download_resource(first_id) == (True, None)+ second_id = _second_resource_on(db, fixture, url)+ with db.fresh() as other:+ doc = other.query(Document).filter_by(resource_id=first_id).one()+ doc_id = doc.id+ if link == "content-hash":+ doc.original_url = "https://mirror.example.org/huge.pdf"+ else:+ other.get(DownloadTracker, tracker_id).file_hash = None+ other.commit()++ summary = service.persist_research_sources(fixture.resource.research_id)++ assert len(downloader.urls) == 1+ assert summary["failed"] == 0+ with db.fresh() as other:+ assert other.get(ResearchResource, second_id).document_id == doc_id+ assert other.query(Document).count() == 1++[email protected]("mode", ["database", "filesystem"])+def test_dedup_storage_refusal_links_keeps_the_text_and_stores_after_raise(+ db, store, seeded, filesystem_mode, monkeypatch, mode+):+ """A duplicate whose bytes match a text-only research download.++ The limit refuses the upgrade of the canonical document to a stored+ PDF. The duplicate is linked to it as before (collection, queue,+ tracker), the canonical keeps its text and now awaits its PDF, neither+ resource downloads it again while the limit refuses it, and once the+ limit is raised the duplicate's download stores the PDF in the+ canonical document.+ """+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.database.models.library import (+ Collection,+ DocumentCollection,+ DownloadQueue,+ )+ from local_deep_research.research_library.utils import document_awaits_pdf++ canonical = seeded("https://example.com/canonical.pdf")+ duplicate = seeded("https://example.com/duplicate.pdf")+ canonical_id = canonical.document.id+ canonical_resource_id = canonical.resource.id+ duplicate_resource_id = duplicate.resource.id+ canonical_tracker_id = canonical.tracker.id+ duplicate_tracker_id = duplicate.tracker.id+ duplicate_url_hash = duplicate.tracker.url_hash+ db.session.delete(duplicate.document)+ db.session.add_all(+ [+ Collection(id="original-collection", name="Original"),+ Collection(id="target-collection", name="Target"),+ ]+ )+ db.session.flush()+ db.session.add_all(+ [+ DocumentCollection(+ document_id=canonical_id, collection_id="original-collection"+ ),+ DownloadQueue(+ resource_id=duplicate_resource_id,+ research_id=duplicate.resource.research_id,+ collection_id="target-collection",+ status=DocumentStatus.PENDING,+ ),+ ]+ )+ db.session.commit()downloader = StubDownloader(- DownloadResult(content=oversized, is_success=True, status_code=200)+ DownloadResult(content=OVERSIZED_PDF, is_success=True, status_code=200))- service = make_service(store, downloader, max_mb=1)+ service = make_service(store, downloader, storage_mode="none", max_mb=1)+ service._extract_text_from_pdf = lambda content: SEARCHABLE_TEXT+ service.retry_manager = SimpleNamespace(+ record_attempt=lambda **kwargs: None+ )+ _one_download_session(db, monkeypatch)+ assert service.download_resource(canonical_resource_id) == (True, None)- success, reason, _status = service._download_pdf(- fixture.resource, fixture.tracker, db.session, None+ service.settings._values["research_library.pdf_storage_mode"] = mode+ assert service.download_resource(duplicate_resource_id) == (True, None)+ with db.fresh() as other:+ doc = other.get(Document, canonical_id)+ assert other.query(Document).count() == 1+ assert doc.status is DocumentStatus.COMPLETED+ assert doc.text_content == SEARCHABLE_TEXT+ assert (doc.storage_mode, doc.file_path) == (+ "none",+ FILE_PATH_TEXT_ONLY,+ )+ assert document_awaits_pdf(doc)+ assert other.query(DocumentBlob).count() == 0+ assert (+ other.get(ResearchResource, duplicate_resource_id).document_id+ == canonical_id+ )+ tracker = other.get(DownloadTracker, duplicate_tracker_id)+ assert (tracker.is_downloaded, tracker.file_path) == (True, None)+ assert {+ row.collection_id for row in other.query(DocumentCollection).all()+ } == {"original-collection", "target-collection"}+ assert (+ other.query(DownloadQueue).one().status is DocumentStatus.COMPLETED+ )+ attempt = (+ other.query(DownloadAttempt)+ .filter_by(url_hash=duplicate_url_hash)+ .one()+ )+ assert attempt.succeeded is True+ assert _tree(store.sandbox) == set()++ # Neither resource fetches the PDF again while the limit refuses it.+ assert service.download_resource(duplicate_resource_id) == (False, REFUSAL)+ assert service.download_resource(canonical_resource_id) == (False, REFUSAL)+ assert len(downloader.urls) == 2++ service.settings._values["research_library.max_pdf_size_mb"] = 2+ assert service.download_resource(duplicate_resource_id) == (True, None)+ assert len(downloader.urls) == 3+ with db.fresh() as other:+ doc = other.get(Document, canonical_id)+ assert other.query(Document).count() == 1+ assert doc.storage_mode == mode+ assert not document_awaits_pdf(doc)+ assert doc.text_content == SEARCHABLE_TEXT+ trackers = [+ other.get(DownloadTracker, tid)+ for tid in (canonical_tracker_id, duplicate_tracker_id)+ ]+ assert all(tracker.is_downloaded for tracker in trackers)+ assert trackers[0].file_path == trackers[1].file_path+ manager = PDFStorageManager(library_root=store.root, storage_mode=mode)+ assert manager.load_pdf(doc, other) == OVERSIZED_PDF+++def _document_awaiting_its_pdf(db, seeded, site):+ """A document "Sync library" left awaiting its PDF, and the refill.++ ``own-document`` refills it through its own resource with changed+ bytes; ``linked-canonical`` through a resource linked to it whose+ bytes the content-hash dedup matches to it.+ """+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.database.models.library import Collection+ from local_deep_research.research_library.utils import document_awaits_pdf++ owner = seeded("https://example.com/awaiting.pdf")+ awaiting_id = owner.document.id+ awaiting = owner.document+ awaiting.status = DocumentStatus.COMPLETED+ awaiting.storage_mode = "none"+ awaiting.file_path = FILE_PATH_TEXT_ONLY+ awaiting.text_content = "text kept by the sync"+ refill = owner+ if site == "own-document":+ awaiting.document_hash = hashlib.sha256(b"the lost PDF").hexdigest()+ else:+ awaiting.document_hash = hashlib.sha256(OVERSIZED_PDF).hexdigest()+ refill = seeded("https://example.com/linked-awaiting.pdf")+ db.session.delete(refill.document)+ db.session.flush()+ refill.resource.document_id = awaiting_id+ db.session.add(Collection(id="refill-collection", name="Refill"))+ db.session.commit()+ assert document_awaits_pdf(db.session.get(Document, awaiting_id))+ return awaiting_id, refill++[email protected]("mode", ["database", "filesystem"])[email protected]("site", ["own-document", "linked-canonical"])+def test_a_refused_refill_keeps_the_document_awaiting_its_pdf(+ db, store, seeded, filesystem_mode, mode, site+):+ """A refill the size limit refuses leaves the document awaiting its PDF.++ It keeps its text and records the refused size, so ``download_resource``+ stops fetching it while the limit refuses that size and fetches it+ again once the limit is raised above it.+ """+ from local_deep_research.constants import FILE_PATH_TEXT_ONLY+ from local_deep_research.database.models.library import DocumentCollection+ from local_deep_research.research_library.utils import document_awaits_pdf++ awaiting_id, refill = _document_awaiting_its_pdf(db, seeded, site)+ tracker_id = refill.tracker.id+ url_hash = refill.tracker.url_hash++ downloader = StubDownloader(+ DownloadResult(content=OVERSIZED_PDF, is_success=True, status_code=200)+ )+ service = make_service(store, downloader, storage_mode=mode, max_mb=1)+ success, reason, _ = service._download_pdf(+ refill.resource, refill.tracker, db.session, "refill-collection")db.session.commit()+ assert (success, reason) == (True, None)+ with db.fresh() as other:+ doc = other.get(Document, awaiting_id)+ assert other.query(Document).count() == 1+ assert doc.status is DocumentStatus.COMPLETED+ assert doc.text_content == "text kept by the sync"+ assert (doc.storage_mode, doc.file_path) == (+ "none",+ FILE_PATH_TEXT_ONLY,+ )+ assert doc.file_size == len(OVERSIZED_PDF)+ assert document_awaits_pdf(doc)+ assert other.query(DocumentBlob).count() == 0+ assert {+ row.collection_id+ for row in other.query(DocumentCollection).filter_by(+ document_id=awaiting_id+ )+ } == {"refill-collection"}+ tracker = other.get(DownloadTracker, tracker_id)+ assert (tracker.is_downloaded, tracker.file_path) == (True, None)+ attempt = (+ other.query(DownloadAttempt).filter_by(url_hash=url_hash).one()+ )+ assert attempt.succeeded is True++ assert service._pdf_refused_at_current_limit(doc) is True+ service.settings._values["research_library.max_pdf_size_mb"] = 2+ assert service._pdf_refused_at_current_limit(doc) is False+ assert _tree(store.sandbox) == set()+++def test_a_none_mode_refill_still_settles_text_only(db, store, seeded):+ """Storage mode "none" stores no PDF, and that is no refusal.++ A refill there still settles the document as a text-only download+ (``file_path`` None), so the download paths stop fetching it, and the+ limit check does not hold it back either.+ """+ from local_deep_research.research_library.utils import document_awaits_pdf++ awaiting_id, refill = _document_awaiting_its_pdf(db, seeded, "own-document")+ downloader = StubDownloader(+ DownloadResult(content=OVERSIZED_PDF, is_success=True, status_code=200)+ )+ service = make_service(store, downloader, storage_mode="none", max_mb=1)+ success, reason, _ = service._download_pdf(+ refill.resource, refill.tracker, db.session, "refill-collection"+ )+ db.session.commit()++ assert (success, reason) == (True, None)+ with db.fresh() as other:+ doc = other.get(Document, awaiting_id)+ assert (doc.storage_mode, doc.file_path) == ("none", None)+ assert not document_awaits_pdf(doc)+ assert doc.text_content == "text kept by the sync"+ assert service._pdf_refused_at_current_limit(doc) is False+ assert _tree(store.sandbox) == set()+++def test_text_only_mode_does_not_require_pdf_storage(db, store, seeded):+ """Storage mode "none" completes the download without treating the+ unstored PDF as refused: the document does not await its PDF."""+ from local_deep_research.research_library.utils import document_awaits_pdf++ fixture = seeded("https://example.com/text-only.pdf")+ downloader = StubDownloader(+ DownloadResult(content=PDF_BYTES, is_success=True, status_code=200)+ )+ service = make_service(store, downloader, storage_mode="none")+ success, reason, _ = service._download_pdf(+ fixture.resource, fixture.tracker, db.session+ )+ db.session.commit()assert success is True and reason is Nonewith db.fresh() as other:+ assert other.query(DocumentBlob).count() == 0doc = other.get(Document, fixture.document.id)- blob = (- other.query(DocumentBlob)- .filter_by(document_id=fixture.document.id)- .first()- )- tracker = other.get(DownloadTracker, fixture.tracker.id)- assert doc.status == DocumentStatus.COMPLETED- assert doc.file_size == len(oversized)- assert blob is None, "no bytes were stored"- assert doc.file_path is None- assert tracker.is_downloaded is True and tracker.file_path is None- assert _tree(store.sandbox) == set(), "and nothing on disk either"+ assert doc.file_path is None+ assert not document_awaits_pdf(doc)+ assert _tree(store.sandbox) == set()++[email protected]("raw_mode", ["NONE", "None", "none ", "Database"])+def test_non_canonical_storage_mode_spellings_stay_text_only(+ db, store, seeded, raw_mode+):+ """A storage mode spelling the manager does not know refuses nothing.++ ``resolve_pdf_storage_mode`` returns these values unchanged and+ ``PDFStorageManager`` coerces each to an effective "none" (with a+ warning), so nothing is stored and nothing counts as refused: the+ download succeeds and the document is not made to await its PDF.++ What this guards: a refusal check that reads the raw setting and+ treats anything other than the exact string "none" as a mode that+ stores PDFs (every spelling here would then count as refused). It does+ not tell the raw setting from the manager's effective mode in general:+ for the canonical "database" and "filesystem" the two are the same,+ and the raw spellings here are not "database" or "filesystem" either,+ so a check against those two raw strings passes it too. The env+ override (LDR_RESEARCH_LIBRARY_PDF_STORAGE_MODE) can carry these+ values, so the premise is reachable.+ """+ from local_deep_research.research_library.utils import document_awaits_pdf++ fixture = seeded("https://example.com/noncanon.pdf")+ downloader = StubDownloader(+ DownloadResult(content=PDF_BYTES, is_success=True, status_code=200)+ )+ service = make_service(store, downloader, storage_mode=raw_mode)+ success, reason, _ = service._download_pdf(+ fixture.resource, fixture.tracker, db.session+ )+ db.session.commit()+ assert success is True and reason is None+ with db.fresh() as other:+ doc = other.get(Document, fixture.document.id)+ assert not document_awaits_pdf(doc)+ assert service._pdf_refused_at_current_limit(doc) is False- strict=True,- reason=(- "DEFECT: _download_pdf ignores save_pdf's None return, so a PDF "- "rejected by the storage size limit is committed as a COMPLETED "- "Document with no blob and no file. download_resource's "- "already-downloaded short-circuit then never retries it."- ),-)def test_a_completed_document_always_has_retrievable_bytes(db, store, seeded):- """CONTRACT: COMPLETED means the bytes are somewhere.+ """CONTRACT: COMPLETED means the bytes are somewhere, or it says so.Either the call reports failure, or the row is not COMPLETED, or the- content is retrievable. Right now none of the three holds.+ content is retrievable, or the document is labelled as storing no PDF+ and awaiting one (``document_awaits_pdf``), as a document whose PDF+ the storage size limit refused is."""- fixture = seeded("https://evil.example.com/huge.pdf")- oversized = PDF_BYTES + b"\x00" * (1024 * 1024)+ from local_deep_research.research_library.utils import document_awaits_pdf+ fixture = seeded("https://evil.example.com/huge.pdf")downloader = StubDownloader(- DownloadResult(content=oversized, is_success=True, status_code=200)+ DownloadResult(content=OVERSIZED_PDF, is_success=True, status_code=200))service = make_service(store, downloader, max_mb=1)@@ -802,8 +1433,9 @@ def test_a_completed_document_always_has_retrievable_bytes(db, store, seeded):)retrievable = manager.has_pdf(doc, other)completed = doc.status == DocumentStatus.COMPLETED+ says_no_pdf = doc.storage_mode == "none" and document_awaits_pdf(doc)- assert not (success and completed) or retrievable+ assert not (success and completed) or retrievable or says_no_pdf# ===========================================================================diff --git a/tests/research_library/test_library_sync_integration.py b/tests/research_library/test_library_sync_integration.pyindex 56f2f07681..44163bddff 100644--- a/tests/research_library/test_library_sync_integration.py+++ b/tests/research_library/test_library_sync_integration.py@@ -63,6 +63,18 @@ def test_linked_refill_settles_storage_and_requested_collection(storage,limit,):+ """A refill through a linked resource settles storage and the requested+ collection.++ ``("database", 0)``: the size limit refuses every PDF. Before #6913 a+ refused refill settled the document as a text-only download and+ reported success. #6913 changed that premise on purpose: a document+ awaiting its PDF whose recorded ``file_size`` the current limit refuses+ keeps awaiting it, and ``download_resource`` answers+ ``pdf_storage_size_limit`` without fetching anything (see+ ``DownloadService._pdf_refused_at_current_limit``), so that case now+ asserts the refusal and the untouched document, queue and collection.+ """owner = world.add_resource("https://example.com/refill-owner.pdf")downloader = StubDownloader(TEXT_PDF_BYTES)service = make_service(store, downloader, storage_mode="filesystem")@@ -96,6 +108,36 @@ def test_linked_refill_settles_storage_and_requested_collection(})+ if limit == 0:+ assert service.download_resource(linked) == (+ False,+ "pdf_storage_size_limit",+ )+ with db.fresh() as session:+ assert session.query(Document).count() == 1+ document = session.get(Document, document_id)+ assert (+ document.document_hash+ == hashlib.sha256(TEXT_PDF_BYTES).hexdigest()+ )+ assert document.text_content == EXTRACTED_TEXT+ assert document_awaits_pdf(document)+ assert (+ session.query(DocumentCollection)+ .filter_by(document_id=document_id, collection_id="requested")+ .count()+ == 0+ )+ assert (+ session.query(DownloadQueue)+ .filter_by(resource_id=linked)+ .one()+ .status+ is DocumentStatus.PENDING+ )+ assert downloader.urls == ["https://example.com/refill-owner.pdf"]+ return+assert service.download_resource(linked) == (True, None)with db.fresh() as session:assert session.query(Document).count() == 1diff --git a/tests/security/test_client_safe_errors_shared_6564.py b/tests/security/test_client_safe_errors_shared_6564.pyindex b2de8000ef..d36aa38189 100644--- a/tests/security/test_client_safe_errors_shared_6564.py+++ b/tests/security/test_client_safe_errors_shared_6564.py@@ -121,6 +121,32 @@ class TestSharedTable:assert msg == "download_error:RuntimeError"assert "INSERT INTO" not in msg+ def test_pdf_storage_refusal_uses_a_fixed_reason(self):+ from local_deep_research.research_library.services.download_service import (+ PDFStorageLimitError,+ )++ exc = PDFStorageLimitError("private URL https://example.com/token")+ assert client_safe_download_message(exc) == "pdf_storage_size_limit"++ def test_pdf_storage_refusal_is_shown_as_how_to_lift_it(self):+ """A person sees which setting to raise, never the raw token, and+ every token the user-message table names is a real token."""+ from local_deep_research.security.client_safe_errors import (+ CLIENT_SAFE_DOWNLOAD_USER_MESSAGES,+ download_reason_for_user,+ )++ message = download_reason_for_user("pdf_storage_size_limit")+ assert "Maximum PDF Size (MB)" in message+ assert "pdf_storage_size_limit" not in message+ assert set(CLIENT_SAFE_DOWNLOAD_USER_MESSAGES) <= set(+ CLIENT_SAFE_DOWNLOAD_MESSAGES.values()+ )+ # Any other reason is passed through unchanged.+ assert download_reason_for_user("network_timeout") == "network_timeout"+ assert download_reason_for_user(None) is None+class TestHtmlOperatorLogCarriesClass:"""PR #6564 optional #3: tokenized HTML failure logs its class."""diff --git a/tests/settings/golden_master_settings.json b/tests/settings/golden_master_settings.jsonindex 6eb8680cd2..acbb89ab32 100644--- a/tests/settings/golden_master_settings.json+++ b/tests/settings/golden_master_settings.json@@ -2806,7 +2806,7 @@},"research_library.max_pdf_size_mb": {"category": "research_library",- "description": "Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage.",+ "description": "Maximum allowed PDF file size in megabytes for the research library. PDFs larger than this will not be stored; their extracted text is still kept, and a research download whose PDF was refused this way is stored when it is next downloaded after this limit is raised above its size. Uploads are also gated by the per-file upload cap (LDR_SECURITY_UPLOAD_MAX_FILE_SIZE_MB, default 3072 MB / 3 GB) — raising this value above that cap has no effect because uploads are rejected before they reach storage.","editable": true,"max_value": 10240,"min_value": 1,diff --git a/tests/web/routers/test_library_api_response_shapes.py b/tests/web/routers/test_library_api_response_shapes.pyindex 0ae13dd3ee..bbf730b20b 100644--- a/tests/web/routers/test_library_api_response_shapes.py+++ b/tests/web/routers/test_library_api_response_shapes.py@@ -562,6 +562,26 @@ def test_download_single_resource_missing_resource_is_a_404_not_a_500():}+def test_download_single_resource_storage_limit_refusal_says_to_raise_it():+ """A PDF the storage size limit refuses is not a server fault, and the+ operator can lift it: the reply names the setting to raise, not the+ raw ``pdf_storage_size_limit`` token or "please try again"."""+ context, _service = _download_service_context(+ (False, "pdf_storage_size_limit")+ )++ with _patched(DownloadService=Mock(return_value=context)):+ response = download_single_resource(+ _get_request(), 42, username="alice"+ )++ assert response.status_code == 409+ body = json.loads(response.body)+ assert body["success"] is False+ assert "Maximum PDF Size (MB)" in body["error"]+ assert "pdf_storage_size_limit" not in body["error"]++def test_download_single_resource_success_is_a_plain_success():"""Positive control: the two failure branches above are real branches,not a route that refuses everything."""diff --git a/tests/web/routers/test_library_download_outcomes.py b/tests/web/routers/test_library_download_outcomes.pyindex 3de113b63b..df9337b344 100644--- a/tests/web/routers/test_library_download_outcomes.py+++ b/tests/web/routers/test_library_download_outcomes.py@@ -891,6 +891,19 @@ class TestDownloadSourceOutcomeMessages:assert "No space left on device" not in renderedassert "/srv/pdfs" not in rendered+ def test_a_storage_limit_refusal_says_which_setting_to_raise(self):+ """The fixed ``pdf_storage_size_limit`` reason is not a generic+ failure: the operator can lift it, so the message names the+ setting instead of "Download failed" or the raw token."""+ response, _session, _service = _call_download_source(+ existing_document=None,+ download_result=(False, "pdf_storage_size_limit"),+ )++ assert response["success"] is False+ assert "Maximum PDF Size (MB)" in response["message"]+ assert "pdf_storage_size_limit" not in response["message"]+def test_a_failure_is_not_reported_as_completed(self):response, _session, _service = _call_download_source(existing_document=None, download_result=(False, "parser error")diff --git a/tests/web/routers/test_library_sse_stream_outcomes.py b/tests/web/routers/test_library_sse_stream_outcomes.pyindex 9994f81eff..a8fd45b399 100644--- a/tests/web/routers/test_library_sse_stream_outcomes.py+++ b/tests/web/routers/test_library_sse_stream_outcomes.py@@ -398,6 +398,30 @@ def test_a_successful_item_reports_success():assert "error" not in item+def test_a_storage_limit_refusal_is_shown_as_how_to_lift_it():+ """``download_resource`` returns the fixed ``pdf_storage_size_limit``+ reason for a PDF the storage size limit refuses. The row must tell the+ person which setting to raise, not show them the raw token."""+ session = _BulkSession(+ _QueueQuery(lambda: 1, lambda: [_queue_item(1, 101)])+ )+ download_service = MagicMock()+ download_service.queue_research_downloads.return_value = 0+ download_service.download_resource.return_value = (+ False,+ "pdf_storage_size_limit",+ )++ events = _run_download_bulk(+ session, download_service, {"research_ids": ["r1"], "mode": "pdf"}+ )++ item = [e for e in events if "status" in e][0]+ assert item["status"] == "skipped"+ assert "Maximum PDF Size (MB)" in item["error"]+ assert "pdf_storage_size_limit" not in item["error"]++def test_text_only_mode_extracts_text_instead_of_downloading():"""``mode`` selects the service method. A mode check that collapsed wouldsilently download PDFs for a user who asked for text only."""diff --git a/tests/web/test_route_table_parity.py b/tests/web/test_route_table_parity.pyindex f6f5b169cf..92006a05c9 100644--- a/tests/web/test_route_table_parity.py+++ b/tests/web/test_route_table_parity.py@@ -1753,7 +1753,9 @@ EXPECTED_STATUS_CODES_GAINED = {("GET", "/library/api/zotero/groups"): [400],# Flask had no "resource not found" branch here and fell through to a# generic 500; the port answers 404. A bug fix, undocumented.- ("POST", "/library/api/download/{}"): [404],+ # A PDF the storage size limit refuses answers 409 with a message that+ # names the "Maximum PDF Size (MB)" setting (#6913); the text is kept.+ ("POST", "/library/api/download/{}"): [404, 409],# Werkzeug applied MAX_CONTENT_LENGTH implicitly and Flask rendered it# through `@app.errorhandler(413)`; FastAPI has no implicit gate, so# the same cap is spelled out in the handler. Parity restored, not a