~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(library): keep documents that still hold content during library sync (#7246)
LearningCircuit4d30f5205df4
diff --git a/changelog.d/+library-sync-keeps-content.bugfix.md b/changelog.d/+library-sync-keeps-content.bugfix.mdnew file mode 100644index 0000000000..396ee93b2a--- /dev/null+++ b/changelog.d/+library-sync-keeps-content.bugfix.md@@ -0,0 +1,89 @@+"Sync library" no longer deletes downloaded documents that still hold+content. It deleted a document whose PDF it could not find even when the+document still had its extracted text or a PDF stored in the database, and+the text, stored PDF, collection memberships and search index entries went+with it. This hit documents whose PDF had been removed to save space,+documents re-downloaded with PDF storage set to "none", failed downloads+later completed with text, documents labelled as stored in the database+without a stored PDF, and Zotero imports with a source URL or DOI whose PDF+is stored in the database.++A document whose PDF file was removed from disk but whose extracted text+was not now keeps its text, collection memberships and search index+entries. If its research still has the resource it was downloaded for, the+sync counts it under "Files Missing" and makes it a text-only document, so+the library no longer offers a PDF that is not there, and the Download+Manager counts it as available to download rather than as already+downloaded. The Download Manager's PDF downloads then put its PDF back into+the same document, also when the PDF downloaded again differs from the one+removed. Each download run downloads it once, also when several of its+researches cite the paper; two runs over different researches at the same+time can each download it. The run downloads it through the document's+own resource, or through another resource for the same paper when the own+one cannot succeed: its URL is not downloadable or is refused by the+egress policy, the retry manager holds it back, or its latest download+attempt failed. A download that a restart of the server cut off does not+keep the document's other resources from downloading its PDF. "Extract+Text to DB" skips such a document, since it already has the text. If the+resource is gone, the sync still makes the document text-only but does+not count it under "Files Missing", because the Download Manager has no+resource to download its PDF from. A later download of the identical PDF+for another research then stores the PDF in it. A document whose PDF file+was removed and that has no extracted text and no PDF stored elsewhere is+still deleted, so it can be downloaded again. A document whose PDF was+removed with "Remove PDF to save space" is kept as it is, and its PDF is+not offered for download again.++When a resource links to a newer document, an older retained document+cannot hide the newer document's missing PDF. Download queues, direct+refills and download counts follow the explicit link, while preserving+the older document for resources that still use it. Cached PDF text+extraction follows the same link, so it cannot report success while the+linked document still lacks text.++Refills preserve canonical document reuse and retry bookkeeping. A regenerated+PDF updates the document whose file was lost and joins the requested collection;+if another document owns those bytes, the refill reports a conflict and keeps+both documents intact. Refills that cannot store a PDF because storage is disabled+or its size limit is exceeded settle as text-only downloads. When accepted text+changes, its old search entries stay invalid until reindexing finishes.++The sync now checks each document against its own PDF: its file, or its+blob when it stores the PDF in the database. Several documents can share+a URL, such as the documents of two researches that cite the same paper.+When another document with the same URL had stored its PDF later, a+document whose file was removed counted as found as long as that other+file was still there, so its missing PDF was never reported or downloaded+again.++A failed download that a later PDF download completes is now recorded as a+PDF document, and so is a document that "Extract Text to DB" created once+a later PDF download stores its PDF in it, so library search results link+to their PDFs.++In the Download Manager, the "downloaded" counts under a research's+"Sources by Domain" and the "Downloaded" badges of its PDFs now show the+PDFs that are downloaded. They compared the stored status with a string+it never equals, so they showed none as downloaded. Five more places made+the same comparison and now read the status correctly:++- `GET /library/api/get-research-sources/{research_id}` and+ `POST /library/api/check-downloads` reported every source as not+ downloaded. They now report a source as downloaded when the Download+ Manager counts its document as downloaded, which a document that awaits+ its PDF is not.+- `POST /library/api/download-source` never answered "Already+ downloaded". For a downloaded source it queued the source again and+ answered "Download completed". It now answers "Already downloaded" when+ the Download Manager counts the source's document as downloaded, and+ downloads the PDF of a document that awaits it.+- The document details page showed the "Download Status" of every+ document as a warning badge reading "Documentstatus.completed" or the+ like. It now shows "Completed", or another status such as "Failed" in a+ warning badge.+- Text extraction, such as "Text Only (from PDFs)" or the document+ scheduler's, never read the PDF file that a downloaded document without+ text keeps on disk. It asked the source for the text or the PDF again,+ or gave up when the retry manager held the source back. It now extracts+ the text from that file before it asks the source. For an arXiv paper+ the text is still requested from arXiv first, as before.diff --git a/changelog.d/+library-sync-missing-file-paths.security.md b/changelog.d/+library-sync-missing-file-paths.security.mdnew file mode 100644index 0000000000..3270d407b8--- /dev/null+++ b/changelog.d/+library-sync-missing-file-paths.security.md@@ -0,0 +1,10 @@+"Sync library" (`POST /library/api/sync-library`) no longer returns the+absolute server-side path of a missing file. Each `missing_files` entry+carried the resolved location of the file on the server, which describes+the host's directory layout rather than the document. It now gives the+path stored for the file, relative to the library root, as+`GET /library/api/documents` does with `file_path`, or `invalid` when that+path does not resolve inside the library root. For a document whose PDF was+stored in the database it gives `database`. This matches what+`/library/api/check-downloads` and, since #6462, the library document API+do.diff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.pyindex e4e24240e7..c86ddb37c5 100644--- a/src/local_deep_research/research_library/services/download_service.py+++ b/src/local_deep_research/research_library/services/download_service.py@@ -16,14 +16,15 @@ import timeimport uuidfrom datetime import datetime, UTCfrom pathlib import Path-from typing import Any, Dict, Optional, Tuple, assert_never+from collections.abc import Set as AbstractSet+from typing import Any, Dict, NamedTuple, Optional, Tuple, assert_neverfrom urllib.parse import urlparseimport requestsfrom loguru import loggerfrom ...security import redact_url_for_log-from sqlalchemy import func-from sqlalchemy.orm import Session, load_only+from sqlalchemy import and_, func, or_+from sqlalchemy.orm import Session, defer, load_onlyimport pdfplumberfrom pypdf import PdfReader@@ -73,9 +74,14 @@ from ...database.library_init import get_source_type_id, get_default_library_idfrom ...database.session_context import get_user_db_session, safe_rollbackfrom ...utilities.db_utils import get_settings_managerfrom ...library.download_management import RetryManager+from ...library.download_management.models import (+ DownloadStatus as RetryStatus,+)from ...config.paths import get_library_directoryfrom ..utils import (apply_user_subdir,+ document_awaits_pdf,+ download_claim_is_live,ensure_in_collection,find_tracker_by_url,get_document_for_resource,@@ -111,6 +117,41 @@ from ...utilities.pdf_extraction_limits import ()+class RefillRank(NamedTuple):+ """Where a resource stands among the resources that refill a document.++ A document that awaits its PDF (see ``document_awaits_pdf``) is+ refilled by a download of its own resource (``Document.resource_id``)+ or of a resource linked to it by content+ (``ResearchResource.document_id``), and one download refills it. Unless+ one of them is already being downloaded or queued, the queueing paths+ give that download to the resource whose rank sorts first (see+ ``DownloadService._refilled_by_another_resource``). The fields sort in+ order, False before True.+ """++ #: Its URL cannot be queued: it is not downloadable, or the egress+ #: policy refuses it.+ cannot_queue: bool+ #: The retry manager holds it back: a cooldown, a permanent failure or+ #: a retry limit.+ held_back: bool+ #: Its latest download attempt failed.+ failed_last: bool+ #: It is not the document's own resource.+ linked: bool+ resource_id: int++ @property+ def viable(self) -> bool:+ """Whether nothing known stands in the way of its download.++ Its URL can be queued, the retry manager does not hold it back,+ and its latest download attempt did not fail.+ """+ return not (self.cannot_queue or self.held_back or self.failed_last)++def _text_extraction_quality(method: str | None, source: str | None) -> str:if method in {"native_api", "html_extraction"}:return "high"@@ -484,12 +525,21 @@ class DownloadService:self._policy_locked = Truereturn None- def _check_url_against_policy(self, url: str) -> Tuple[bool, str]:+ def _check_url_against_policy(+ self, url: str, *, record_denial: bool = True+ ) -> Tuple[bool, str]:"""Run evaluate_url against self._egress_context. Returns(allowed, reason). When no context is configured (legacycallers OR tests that mock __init__), returns (True, "no_context")so back-compat callers aren't broken — but new callers passinga snapshot DO get gated. Logged to policy_audit on denial.++ ``record_denial=False`` returns the same decision without recording+ a denial: it is not logged, here or by ``evaluate_url``, and does+ not count toward the run's denied-fetch quota. Only ``refill_rank``+ passes it, since it fetches nothing on the decision. Every other+ caller checks a URL before it is queued or fetched, and records its+ denials."""# ``_policy_locked`` is set by _build_egress_context when a# snapshot was supplied but the policy itself raised. Without@@ -512,8 +562,10 @@ class DownloadService:from ...security.egress.policy import evaluate_urlfrom ...security.ssrf_validator import redact_url_for_log- decision = evaluate_url(url, egress_context)- if not decision.allowed:+ decision = evaluate_url(+ url, egress_context, record_denial=record_denial+ )+ if not decision.allowed and record_denial:logger.bind(policy_audit=True).warning("DownloadService URL denied by egress policy",# Redact userinfo creds / query-string tokens from the log.@@ -635,7 +687,12 @@ class DownloadService:return Nonedef queue_research_downloads(- self, research_id: str, collection_id: Optional[str] = None+ self,+ research_id: str,+ collection_id: Optional[str] = None,+ *,+ text_only: bool = False,+ run_research_ids: Optional[AbstractSet[str]] = None,) -> int:"""Queue all downloadable PDFs from a research session.@@ -643,6 +700,19 @@ class DownloadService:Args:research_id: The research session IDcollection_id: Optional target collection ID (defaults to Library if not provided)+ text_only: True when the queued resources only get their text+ extracted (``download_bulk``'s "text_only" mode). A completed+ document that awaits its PDF (see ``document_awaits_pdf``)+ already holds its text, so it then counts as done, like any+ completed document. A text extraction would not fetch its PDF+ either, so it would be queued and reported again on every run.+ run_research_ids: The research sessions whose queued downloads+ the same run downloads, as a set (``download_bulk`` builds+ one from its ``research_ids`` for the whole run and passes it+ for each research), or None when nothing downloads the queue+ right away. It is only looked up, never copied, unless it+ lacks ``research_id``. Decides which resource refills a+ document that awaits its PDF, see the notes.Returns:Number of items queued@@ -650,14 +720,44 @@ class DownloadService:Notes:Resets existing FAILED/COMPLETED queue entries for the research backto PENDING, effectively retrying them. Resources that already have a- PENDING queue entry or a COMPLETED Document are skipped. As of #4685,+ PENDING queue entry or a COMPLETED Document are skipped, except a+ COMPLETED Document that awaits its PDF when PDFs are queued. As of #4685,``download_bulk`` calls this unconditionally on every bulk run, sopreviously-failed downloads are automatically retried each time.A PROCESSING entry is in flight in another ``download_bulk`` stream,so it is left alone rather than reset: resetting it would let thisrun re-claim and re-download it (#4691).++ A document that awaits its PDF can be refilled through its own+ resource or any resource linked to it by content+ (``resource.document_id``), and one download refills it. So a+ run leaves the document to one of them (see+ ``_refilled_by_another_resource``): to one that a download holds,+ else to one already queued for the run, unless that one is not+ viable and another ranks before it, else to the one that ranks+ first in ``refill_rank``. That is the document's own resource+ unless it cannot succeed (its URL cannot be queued, the retry+ manager holds it back, or its latest attempt failed) while a+ linked one can. Once the document holds its PDF again,+ ``download_resource`` reports the document's own resource as+ already downloaded, and for a linked one whose URL the egress+ policy allows, ``_download_pdf`` reuses that stored PDF instead+ of fetching it. Of this research's resources the document's own+ one is considered first."""queued = 0+ # The research sessions whose queue the same run downloads (see+ # _refilled_by_another_resource). download_bulk passes one set for+ # the whole run, so it is only looked up here: copying it for each+ # research made the bulk pre-pass quadratic in the number of+ # research sessions.+ run = run_research_ids+ if run is not None and research_id not in run:+ run = run | {research_id}+ # The refill_rank of each resource compared below, by resource and+ # document owner, so that the egress policy and the retry manager+ # are asked once per resource in this call.+ ranks: Dict[Tuple[int, Optional[int]], RefillRank] = {}# Get default library collection if no collection_id providedif not collection_id:@@ -673,11 +773,53 @@ class DownloadService:.all())- for resource in resources:+ # A document's own resource before the resources linked to it+ # by content, so that the own resource of a document awaiting+ # its PDF is considered first when this research has both. A+ # resource can also point to its own document through+ # document_id. The sort is stable and leaves the other+ # resources in order. It reads the owner of each document these+ # resources point to, in one query of bare columns: loading the+ # documents would load their text (#4560).+ owner_of_linked_document = (+ dict(+ session.query(ResearchResource.id, Document.resource_id)+ .join(Document, ResearchResource.document_id == Document.id)+ .filter(ResearchResource.research_id == research_id)+ .all()+ )+ if any(res.document_id for res in resources)+ else {}+ )++ def linked_to_another_resources_document(res) -> bool:+ return bool(res.document_id) and (+ owner_of_linked_document.get(res.id) != res.id+ )++ for resource in sorted(+ resources, key=linked_to_another_resources_document+ ):if self._is_downloadable(resource):- # Library resources linked via document_id are already done+ # The document awaiting its PDF that this resource would+ # refill, if any (see document_awaits_pdf).+ awaiting = None+ # Library resources linked via document_id are already+ # done, unless PDFs are queued and the linked document+ # still needs its PDF (see document_awaits_pdf).if resource.document_id:- continue+ # Not its text, which nothing here reads (#4560).+ linked = (+ None+ if text_only+ else session.query(Document)+ .options(defer(Document.text_content))+ .filter(Document.id == resource.document_id)+ .first()+ )+ if linked is None or not document_awaits_pdf(linked):+ continue+ awaiting = linked# Egress policy gate before queueing.allowed, reason = self._check_url_against_policy(@@ -700,18 +842,45 @@ class DownloadService:.first())- # Check if already downloaded (trust the database status)+ # Check if already downloaded (trust the database status).+ # When PDFs are queued, a completed document that still+ # needs its PDF does not count (see document_awaits_pdf).+ # An explicit link is authoritative; an older retained+ # document can still name this resource as its owner.existing_doc = (- session.query(Document)+ awaiting+ if resource.document_id+ else session.query(Document).filter_by(resource_id=resource.id,status=DocumentStatus.COMPLETED,).first())+ if (+ not text_only+ and existing_doc is not None+ and document_awaits_pdf(existing_doc)+ ):+ awaiting = existing_doc+ existing_doc = None# Queue if not already queued and not marked as completedif not existing_queue and not existing_doc:+ # One download refills a document awaiting its PDF:+ # leave it to the resource that does.+ if (+ awaiting is not None+ and self._refilled_by_another_resource(+ session,+ awaiting,+ resource,+ run,+ research_id,+ ranks,+ )+ ):+ continue# Check one more time if ANY queue entry exists (regardless of status)any_queue = (session.query(LibraryDownloadQueue)@@ -772,6 +941,166 @@ class DownloadService:"""return is_downloadable_url(resource.url)+ def refill_rank(+ self, resource_id: int, url: Optional[str], owner_id: Optional[int]+ ) -> RefillRank:+ """The rank of a resource that would refill a document awaiting its PDF.++ ``owner_id`` is the document's own resource (``Document.resource_id``).+ See ``RefillRank``: a viable resource ranks before one that is not,+ and among viable resources the document's own one ranks first. Of+ the others, one that can be queued ranks before one that cannot, and+ one the retry manager would retry now before one it holds back. The+ rank asks the egress policy and the retry manager, whose record of+ attempts the "Download All New" retry-policy filter reads too.+ ``queue_research_downloads`` and the "Download All New" route both+ choose the resource that refills a document by this rank.++ Ranking fetches nothing, so it records no denial of the egress+ policy: a refused URL is neither written to the audit log nor+ counted toward the run's denied-fetch quota (see+ ``_check_url_against_policy``). Fetching the URL still checks it and+ records the denial, and so does queueing it in+ ``queue_research_downloads``.+ """+ cannot_queue = not (+ url+ and is_downloadable_url(url)+ and self._check_url_against_policy(url, record_denial=False)[0]+ )+ held_back = failed_last = False+ if self.retry_manager:+ held_back = not self.retry_manager.should_retry_resource(+ resource_id+ ).can_retry+ status = self.retry_manager.status_tracker.get_resource_status(+ resource_id+ )+ failed_last = status is not None and status["status"] in (+ RetryStatus.TEMPORARILY_FAILED.value,+ RetryStatus.PERMANENTLY_FAILED.value,+ )+ return RefillRank(+ cannot_queue=cannot_queue,+ held_back=held_back,+ failed_last=failed_last,+ linked=resource_id != owner_id,+ resource_id=resource_id,+ )++ def _refilled_by_another_resource(+ self,+ session: Session,+ document: Document,+ resource: ResearchResource,+ run: Optional[AbstractSet[str]],+ research_id: str,+ ranks: Dict[Tuple[int, Optional[int]], RefillRank],+ ) -> bool:+ """Whether a resource other than ``resource`` refills ``document``.++ A document that awaits its PDF (see ``document_awaits_pdf``) is+ refilled by a download of its own resource (``Document.resource_id``,+ unless that resource points to another document through+ ``document_id``) or of any resource linked to it by content+ (``ResearchResource.document_id``): each downloads into it, see+ ``get_document_for_resource``. One such download refills it, so+ ``queue_research_downloads`` queues ``resource`` only when this is+ False. ``research_id`` is the research being queued, ``run`` the+ research sessions of the run (see ``queue_research_downloads``),+ and ``ranks`` holds the ``refill_rank`` of the resources compared+ so far in that call. True when another of these resources++ * has a PROCESSING queue entry that a download still holds (see+ ``download_claim_is_live``): another stream is downloading it.+ An entry that a restart left PROCESSING does not count, and that+ resource is not queued either, since no reset touches a+ PROCESSING entry;+ * has a PENDING queue entry that the same run downloads, i.e. of a+ research session in ``run``, and is viable or ranks before+ ``resource`` (see ``RefillRank``). Without a run (``run`` None,+ when nothing downloads the queue right away) any PENDING entry+ counts: the run that downloads the queue later checks again with+ its own research sessions. Within a run, a PENDING entry of a+ research session outside it does not count, since the run would+ not download it;+ * ranks before ``resource`` and would be queued by the same call or+ run: its research session is ``research_id`` or one in ``run``,+ and it has no queue entry PENDING or PROCESSING. Ranking before+ ``resource``, which can be queued, it can be queued too.++ So, unless one of them is being downloaded, a resource that cannot+ succeed does not keep a viable one from refilling the document, and+ of the viable ones in the run the document's own resource refills+ it, whichever research the run queues first, unless another viable+ one is already queued for the run.+ """+ # Bare columns: the full rows would load each resource's+ # content_preview (#4560).+ others = (+ session.query(+ ResearchResource.id,+ ResearchResource.url,+ ResearchResource.research_id,+ )+ .filter(+ ResearchResource.id != resource.id,+ or_(+ ResearchResource.document_id == document.id,+ and_(+ ResearchResource.id == document.resource_id,+ ResearchResource.document_id.is_(None),+ ),+ ),+ )+ .all()+ )+ if not others:+ return False+ entries = {+ entry.resource_id: entry+ for entry in session.query(+ LibraryDownloadQueue.resource_id,+ LibraryDownloadQueue.status,+ LibraryDownloadQueue.research_id,+ LibraryDownloadQueue.last_attempt_at,+ ).filter(+ LibraryDownloadQueue.resource_id.in_(+ [other.id for other in others]+ ),+ LibraryDownloadQueue.status.in_(+ (DocumentStatus.PENDING, DocumentStatus.PROCESSING)+ ),+ )+ }++ def rank_of(res) -> RefillRank:+ key = (res.id, document.resource_id)+ if key not in ranks:+ ranks[key] = self.refill_rank(+ res.id, res.url, document.resource_id+ )+ return ranks[key]++ own_rank = rank_of(resource)+ queued_now = run if run is not None else {research_id}+ for other in others:+ entry = entries.get(other.id)+ if entry is not None and entry.status == DocumentStatus.PROCESSING:+ if download_claim_is_live(entry.last_attempt_at):+ return True+ continue+ rank = rank_of(other)+ if entry is not None:+ if (run is None or entry.research_id in run) and (+ rank.viable or rank < own_rank+ ):+ return True+ continue+ if rank < own_rank and other.research_id in queued_now:+ return True+ return False+def download_resource(self, resource_id: int) -> Tuple[bool, Optional[str]]:"""Download a specific resource.@@ -785,16 +1114,22 @@ class DownloadService:logger.error(f"Resource {resource_id} not found")return False, "Resource not found"- # Check if already downloaded (trust the database after sync)- existing_doc = (- session.query(Document)- .filter_by(- resource_id=resource_id, status=DocumentStatus.COMPLETED- )- .first()+ # Check if already downloaded (trust the database after sync).+ # A completed document that the sync made text-only because its+ # PDF file was missing still needs its PDF, so it is downloaded+ # again into that document (see document_awaits_pdf).+ existing_query = session.query(Document).filter_by(+ resource_id=resource_id, status=DocumentStatus.COMPLETED)-- if existing_doc:+ if resource.document_id:+ # Never short-circuit on a retained older document. Keep+ # linked-document retries on the normal path so they repair+ # queue, tracker, and collection bookkeeping.+ existing_query = existing_query.filter(+ Document.id == resource.document_id+ )+ existing_doc = existing_query.first()+ if existing_doc and not document_awaits_pdf(existing_doc):logger.info("Resource already downloaded (according to database)")@@ -1227,6 +1562,37 @@ class DownloadService:file_size_val = len(pdf_content)downloaded_at_val = datetime.now(UTC)+ existing_doc = get_document_for_resource(sess, target_resource)+ if (+ existing_doc+ and document_awaits_pdf(existing_doc)+ and existing_doc.document_hash != file_hash_val+ and sess.query(Document.id)+ .filter(Document.document_hash == file_hash_val)+ .first()+ is not None+ ):+ # Keeping the old hash while storing different bytes+ # poisons later content deduplication: a download of+ # the original PDF would then serve the replacement.+ # Preserve both documents until the caller can relink+ # to the canonical owner, before changing any storage,+ # text, or tracker state.+ error_msg = (+ "PDF refill conflicts with another library document"+ )+ sess.add(+ DownloadAttempt(+ url_hash=target_tracker.url_hash,+ attempt_number=attempt_num,+ attempted_at=attempted_at,+ succeeded=False,+ error_message=error_msg,+ )+ )+ sess.commit()+ return False, error_msg, status_code+target_tracker.file_hash = file_hash_valtarget_tracker.file_size = file_size_valtarget_tracker.is_downloaded = True@@ -1254,14 +1620,13 @@ class DownloadService:)sess.add(attempt)- # Check if library document already exists- existing_doc = get_document_for_resource(sess, target_resource)canonical_retry = False# Older placeholder recovery could leave a nonempty PDF# labelled "unknown"; that content still needs preservation.if (existing_docand existing_doc.document_hash != file_hash_val+ and not document_awaits_pdf(existing_doc)and (existing_doc.status == DocumentStatus.COMPLETEDor (@@ -1275,6 +1640,8 @@ class DownloadService:# failed, or the link appeared while the download was in# flight. Resolve changed bytes by hash without rewriting# the document used by other resources.+ # A sync-marked missing PDF may refill the same record;+ # the collision guard above already protects other owners.existing_doc = Nonecanonical_retry = Trueif (@@ -1309,6 +1676,7 @@ class DownloadService:if existing_doc:pdf_document_id = existing_doc.id+ refills_missing_pdf = document_awaits_pdf(existing_doc)# Update existing document. Only replace document_hash when# transitioning from FAILED — that's the placeholder hash from# _record_failed_text_extraction. For any other prior state@@ -1322,6 +1690,28 @@ class DownloadService:existing_doc.status = DocumentStatus.COMPLETEDexisting_doc.processed_at = datetime.now(UTC)+ # A document that the library sync made text-only after+ # its PDF file went missing gets its PDF back here, in+ # the same document, whatever the bytes hash to (see+ # document_awaits_pdf). A publisher or arXiv can serve+ # a regenerated PDF, so the bytes can differ from the+ # lost ones. The document then takes their hash, as a+ # new download would. A hash already owned by another+ # document was refused before any writes above.+ if (+ refills_missing_pdf+ and existing_doc.document_hash != file_hash_val+ ):+ existing_doc.document_hash = file_hash_val++ if refills_missing_pdf and collection_id:+ # A linked resource may request a different collection.+ # Keep its membership when changed bytes refill this+ # record instead of taking the canonical dedup path.+ ensure_in_collection(+ sess, existing_doc.id, collection_id+ )+# Save PDF using storage manager (updates storage_mode and file_path)file_path_result, _ = pdf_storage_manager.save_pdf(pdf_content=pdf_content,@@ -1331,6 +1721,23 @@ class DownloadService:url=url,resource_id=target_resource.id,)+ if file_path_result:+ # The PDF is now stored in this document, so label+ # it a PDF, as a new download is. A failed-download+ # placeholder is labelled "unknown" and a text+ # extraction "text". Either label hides the stored+ # PDF from PDFStorageManager.has_pdf. Once the sync+ # has made the document text-only, it also keeps+ # document_awaits_pdf from asking for the PDF again.+ existing_doc.file_type = "pdf"+ existing_doc.mime_type = "application/pdf"+ elif refills_missing_pdf:+ # The PDF of a document that still needed one was+ # downloaded but not stored, as in storage mode+ # "none" or for a PDF over the size limit. Leave the+ # document as a "none"-mode download leaves one, so+ # that the download paths stop fetching it again.+ existing_doc.file_path = None# Update trackertarget_tracker.file_path = (@@ -1447,6 +1854,14 @@ class DownloadService:and file_path_result != "database"else None)+ if file_path_result:+ # The PDF is now stored in this document:+ # label it a PDF, as the existing-document+ # branch above does.+ existing_by_hash.file_type = "pdf"+ existing_by_hash.mime_type = (+ "application/pdf"+ )# Keep sibling trackers for the same bytes# consistent: the canonical URL's tracker still# has file_path=None from its none-mode download,@@ -1519,6 +1934,12 @@ class DownloadService:# re-download, not data loss.target_tracker.file_path = Nonetarget_tracker.file_name = None+ if document_awaits_pdf(existing_by_hash):+ # Storage was disabled or refused this PDF's+ # size. Settle a linked refill as a text-only+ # download, just like the own-document path,+ # so every bulk run does not fetch it again.+ existing_by_hash.file_path = Noneexcept Exception as exc:log_exception_type("Failed to reuse/upgrade deduped PDF storage",@@ -2410,12 +2831,24 @@ class DownloadService:def _try_existing_pdf_extraction(self, session, resource, resource_id: int) -> Optional[Tuple[bool, Optional[str]]]:- """Try extracting text from existing PDF in database."""- pdf_document = (- session.query(Document).filter_by(resource_id=resource_id).first()- )+ """Extract text from the PDF file of the resource's document.++ Reads the file that the completed document's own ``file_path``+ names (filesystem storage), without a request, and saves its text+ into that document. A PDF stored in the database is not read here+ (``_text_from_cached_pdf`` reads one for an arXiv paper). Returns+ None, so that the caller tries its other sources, when the resource+ has no document, the document is not completed or names no file,+ its file is missing or yields no text, or the read or the save+ fails.+ """+ # An older retained snapshot may still name this resource as its+ # owner after the resource links to a newer document.+ pdf_document = get_document_for_resource(session, resource)- if not pdf_document or pdf_document.status != "completed":+ # A loaded status is a DocumentStatus member, which never equals+ # the string "completed".+ if not pdf_document or pdf_document.status != DocumentStatus.COMPLETED:return None# Validate path to prevent path traversal attacks. Resolve against thediff --git a/src/local_deep_research/research_library/services/library_service.py b/src/local_deep_research/research_library/services/library_service.pyindex 09a1dff648..09be4153cd 100644--- a/src/local_deep_research/research_library/services/library_service.py+++ b/src/local_deep_research/research_library/services/library_service.py@@ -13,10 +13,10 @@ from typing import Dict, List, Optionalfrom urllib.parse import urlparsefrom loguru import logger-from sqlalchemy import or_, func, case+from sqlalchemy import and_, or_, func, casefrom sqlalchemy.orm import aliased, defer-from ...constants import FILE_PATH_SENTINELS+from ...constants import FILE_PATH_SENTINELS, FILE_PATH_TEXT_ONLYfrom ...utilities.arxiv import is_arxiv_paper_urlfrom ...database.models.download_tracker import DownloadTrackerfrom ...database.models.library import (@@ -35,9 +35,12 @@ from ...utilities.resource_utils import safe_closefrom ...security import url_authority_without_userinfofrom ...utilities.sql_utils import escape_like as _escape_likefrom ..utils import (+ document_awaits_pdf_clause,+ document_counts_as_downloaded,find_tracker_by_url,get_absolute_path_from_settings,open_file_location,+ resource_document_clause,)# Filter dropdowns (library page + ``/api/research-list``) load research@@ -55,6 +58,97 @@ _DROPDOWN_RESEARCH_LIMIT = 1000_DOMAIN_SCAN_BATCH_SIZE = 1000+def _names_a_file(file_path: Optional[str]) -> bool:+ """Whether a ``file_path`` value names a file on disk.++ ``None``, ``"database"`` and the sentinels (``blob_deleted``,+ ``text_only_not_stored``, ``metadata_only``) name no file.+ """+ return (+ bool(file_path)+ and file_path != "database"+ and file_path not in FILE_PATH_SENTINELS+ )+++def _tracker_describes_storage(+ tracker_path: Optional[str], doc: Document+) -> bool:+ """Whether a URL's tracker path names where ``doc`` stores its PDF.++ Every document with the URL shares its tracker, and a download that+ stores a PDF writes to the tracker where it stored it:++ * filesystem storage: the file's path, which+ ``PDFStorageManager.save_pdf`` also writes to the document, so the+ two are equal (rows written before database storage was added,+ 8d7bb38d1a, copied the tracker's path into the document);+ * database storage: ``"database"``, while ``save_pdf`` labels the+ document ``storage_mode`` "database" and leaves its ``file_path``+ None, as it has since database storage was added (8d7bb38d1a).+ ``upgrade_to_pdf`` labels a document it stores a PDF in the same+ way, and the content-hash dedup of ``_download_pdf`` writes+ ``"database"`` for a matched document with that label too.++ So the tracker describes a document when it names the document's own+ path, or when it is ``"database"`` and the document is labelled as+ stored in the database, without a path. The label alone does not say+ that a blob was stored (a failed-download placeholder keeps the+ column's ``"database"`` default), so the sync still checks the+ document's own blob. ``"database"`` can describe several documents of+ the URL at once.+ """+ if tracker_path == doc.file_path:+ return True+ return (+ tracker_path == "database"+ and doc.storage_mode == "database"+ and doc.file_path is None+ )+++def _missing_file_entry(+ doc: Document, stored_path: Optional[str], resolved_path+) -> Dict:+ """The sync's ``missing_files`` entry for a document whose file is gone.++ ``POST /library/api/sync-library`` returns these entries to the client.+ ``path`` is the stored path, which is relative to the user's library+ root (the library listing returns the same value as ``file_path``), or+ ``"invalid"`` when it does not resolve inside that root+ (``resolved_path`` is ``None``). For a document whose PDF the URL's+ tracker records as stored in the database, the stored value is+ ``"database"``, which names no file. It is never ``resolved_path``+ itself: the absolute path is the server's directory layout, which+ ``/library/api/check-downloads`` and the library listing withhold from+ the client (#6462).+ """+ return {+ "id": doc.id,+ "title": doc.title,+ "path": stored_path if resolved_path and stored_path else "invalid",+ "url": doc.original_url,+ }+++def _counts_as_downloaded():+ """SQL condition: a ``Document`` the Download Manager counts as downloaded.++ A completed document, unless it awaits its PDF (see+ ``document_awaits_pdf``): "Sync library" kept its text after its PDF+ file went missing and reported the file missing, and "Download All New"+ queues it, so the Download Manager counts it as available to download.+ Without a matching document (the outer join's NULL row) the condition is+ not true, so nothing is counted. ``document_counts_as_downloaded``+ (``research_library.utils``) is the same rule for a loaded+ ``Document``.+ """+ return and_(+ Document.status == DocumentStatus.COMPLETED,+ ~document_awaits_pdf_clause(),+ )++class LibraryService:"""Service for managing and querying the document library."""@@ -71,6 +165,125 @@ class LibraryService:is not None)+ def _remaining_content(+ self, session, doc: Document, settings_manager+ ) -> Optional[str]:+ """What a download stores itself, whatever its URL's tracker names.++ Returns ``"pdf"`` when its PDF is still stored (a blob in the+ database, or a file at the document's own ``file_path``),+ ``"text"`` when only its extracted text is left, and ``None`` when+ nothing is.++ The document's ``storage_mode`` and its tracker's ``file_path`` can+ both be stale: a re-download in "none" mode clears the tracker path+ but keeps the stored PDF, and a failed-download placeholder keeps+ the ``"database"`` column default after text is saved into it. The+ tracker, which every document with the URL shares, can also name+ another document's file. So this checks what is actually stored.+ Deleting a document that holds any of it would also delete that+ content, its chunks and its vectors.+ """+ if self._has_blob_in_db(session, doc.id):+ return "pdf"+ if _names_a_file(doc.file_path):+ resolved = get_absolute_path_from_settings(+ doc.file_path, self.username, settings_manager=settings_manager+ )+ if resolved and resolved.is_file():+ return "pdf"+ has_text = (+ session.query(Document.id)+ .filter(+ Document.id == doc.id,+ Document.text_content.isnot(None),+ Document.text_content != "",+ )+ .first()+ is not None+ )+ return "text" if has_text else None++ def _keep_text_report_missing_pdf(+ self,+ doc: Document,+ tracker,+ stored_path: Optional[str],+ resolved_path,+ stats: Dict,+ ) -> None:+ """Keep a document's text, drop its dangling PDF and report it.++ For a research download whose PDF file is gone but whose extracted+ text is not, and that still has its research resource+ (``resource_id``), from which the download manager can download+ the PDF again. The document keeps its row, text, collection links,+ chunks and vectors. It is labelled a PDF and stored the way+ text-only documents are (``file_type`` "pdf", ``storage_mode``+ "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), so the library no+ longer offers a PDF that is not there, and ``tracker`` is reset+ to not downloaded unless it is None (see ``_clear_missing_pdf``).+ It is counted in ``files_missing`` and listed in ``missing_files``+ under ``stored_path``, the document's own path that named the+ missing file (see ``_missing_file_entry``), and+ ``document_awaits_pdf`` holds for it, so the download manager+ downloads its PDF into it again.++ The label is set here because what went missing was a PDF: the+ sync calls this only for a missing file that the document's own+ ``file_path`` names, and only PDF storage writes a file path into+ a document. A+ failed-download placeholder that a PDF download completed before+ ``_download_pdf`` labelled such documents "pdf" is still labelled+ "unknown", and would otherwise never be downloaded again.+ """+ doc.file_type = "pdf"+ doc.mime_type = "application/pdf"+ self._clear_missing_pdf(doc, tracker, stats)+ stats["files_missing"] += 1+ stats["missing_files"].append(+ _missing_file_entry(doc, stored_path, resolved_path)+ )++ def _clear_missing_pdf(self, doc: Document, tracker, stats: Dict) -> None:+ """Make a document whose PDF file is gone a text-only document.++ It is stored the way text-only documents are (``storage_mode``+ "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), and ``tracker``,+ unless it is None, is reset to not downloaded, so neither names the+ missing file any more. The sync passes the URL's tracker when it+ names this document's path or no path, and None when it describes+ where other documents store their PDF (another document's path, or+ ``"database"`` for a document not stored there, see+ ``_tracker_describes_storage``), which is left as it is. The reset+ counts in ``trackers_updated`` only when it changes the tracker: a+ tracker without a path can be reset already, for example by the+ deletion of another document of the URL whose file it named, when+ the sync read that document first. Counted again, that tracker+ counted twice in that read order and once in the other.++ The sync calls this directly, without reporting the document, for+ a download that keeps its extracted text but no longer has its+ research resource (``resource_id`` is NULL, e.g. after+ ``delete_resource``). No download path asks for such a document's+ PDF (``document_awaits_pdf`` needs the resource), and it counts in+ ``files_found``, as every text-only document does. A later download+ of the same PDF bytes for another resource still reaches it through+ the content-hash match in ``DownloadService._download_pdf``. In a+ storage mode that keeps PDFs, that match stores the PDF into a+ matched document stored without one (``storage_mode`` "none").+ Left naming the missing file, the document made that match copy the+ dangling path into the new resource's tracker and store nothing.+ """+ doc.storage_mode = "none"+ doc.file_path = FILE_PATH_TEXT_ONLY+ if tracker is not None and (+ tracker.is_downloaded or tracker.file_path is not None+ ):+ tracker.is_downloaded = False+ tracker.file_path = None+ stats["trackers_updated"] += 1+def _get_safe_absolute_path(self, file_path: str) -> Optional[str]:"""Get the absolute path for a file, safely handling invalid paths.@@ -637,6 +850,9 @@ class LibraryService:) -> List[Dict]:"""Get research sessions with download statistics.+ ``downloaded_count`` leaves out a document that awaits its PDF (see+ ``_counts_as_downloaded``).+Args:limit: Maximum number of results (0 = no limit, for backwards compat).offset: Number of rows to skip. Only applied when limit > 0.@@ -652,7 +868,7 @@ class LibraryService:func.count(func.distinct(case(- (Document.status == "completed", Document.id),+ (_counts_as_downloaded(), Document.id),else_=None,))@@ -679,8 +895,7 @@ class LibraryService:).outerjoin(Document,- (ResearchResource.id == Document.resource_id)- | (ResearchResource.document_id == Document.id),+ resource_document_clause(),).group_by(ResearchHistory.id).order_by(ResearchHistory.created_at.desc())@@ -769,7 +984,10 @@ class LibraryService:This is a lightweight query that only returns totals — used forthe download-manager header so stats remain accurate regardless- of which page the user is viewing.+ of which page the user is viewing. A document that awaits its PDF+ does not count as downloaded (see ``_counts_as_downloaded``): it is+ left out of ``already_downloaded``, so it is not subtracted from+ ``available_to_download``."""with get_user_db_session(self.username) as session:row = (@@ -783,7 +1001,7 @@ class LibraryService:func.count(func.distinct(case(- (Document.status == "completed", Document.id),+ (_counts_as_downloaded(), Document.id),else_=None,))@@ -811,8 +1029,7 @@ class LibraryService:).outerjoin(Document,- (ResearchResource.id == Document.resource_id)- | (ResearchResource.document_id == Document.id),+ resource_document_clause(),).one())@@ -835,8 +1052,15 @@ class LibraryService:"""Batch-fetch PDF documents and domain breakdowns for multiple research sessions.Returns a dict keyed by research_id with:- - "pdf_sources": list of document dicts (capped at limit_per_research)+ - "pdf_sources": list of document dicts (capped at limit_per_research),+ each with "document_title", "domain", "file_type" and+ "downloaded" (whether the Download Manager shows it as+ downloaded, with its "Downloaded" badge)- "domains": dict of domain -> {total, pdfs, downloaded}++ "downloaded" follows the rule of the page's other counts (see+ ``document_counts_as_downloaded``): a completed document counts,+ unless it awaits its PDF."""if not research_ids:return {}@@ -895,7 +1119,8 @@ class LibraryService:entry["domains"][domain]["total"] += 1if doc.file_type == "pdf":entry["domains"][domain]["pdfs"] += 1- if doc.status == "completed":+ downloaded = document_counts_as_downloaded(doc)+ if downloaded:entry["domains"][domain]["downloaded"] += 1# PDF sources preview (capped)@@ -911,7 +1136,7 @@ class LibraryService:"document_title": title,"domain": domain,"file_type": doc.file_type,- "download_status": doc.status or "unknown",+ "downloaded": downloaded,})@@ -1002,7 +1227,10 @@ class LibraryService:"domain": self._extract_domain(resource.url)if resourceelse "User Upload",- "download_status": doc.status,+ # The status's string value, which the details page+ # compares with 'completed' and prints: a loaded+ # DocumentStatus member never equals that string.+ "download_status": doc.status.value,"downloaded_at": doc.processed_at.isoformat()if doc.processed_atand hasattr(doc.processed_at, "isoformat")@@ -1216,6 +1444,59 @@ class LibraryService:Sync library database with filesystem.Check which PDF files exist and update database accordingly.+ Every document with the same URL shares that URL's tracker. The+ tracker is used for a document only when it names where the+ document stores its PDF (see ``_tracker_describes_storage``): the+ document's own path, or ``"database"`` for a document labelled as+ stored in the database. Each document is judged by what it stores+ itself, and a document the tracker does not describe leaves the+ tracker unchanged.++ A download whose PDF file is missing is handled by what it still+ stores (see ``_remaining_content``):++ * its PDF, as a database blob or a file at its own ``file_path``:+ kept unchanged;+ * only its extracted text, while its own ``file_path`` names a+ file, and it still has its research resource: kept with its text+ but made text-only, the URL's tracker reset if it names that file+ or no file, and counted in ``files_missing`` and+ ``missing_files``, so that the download manager downloads its PDF+ again (see ``_keep_text_report_missing_pdf``);+ * only its extracted text, while its own ``file_path`` names a+ file, with no research resource (e.g. a download whose resource+ was deleted): kept with its text and made text-only, the URL's+ tracker reset as above, and counted in ``files_found``. No+ download path asks for its PDF, so it is not reported, but a+ later download of the same PDF bytes stores the PDF in it (see+ ``_clear_missing_pdf``);+ * only its extracted text, with no file named (a "database" label+ without a blob): kept unchanged;+ * nothing: deleted, so that its URL can be downloaded again, and+ counted in ``files_missing``. The URL's tracker is reset if it+ names the document's file. If it is ``"database"`` and the+ document is stored there, it is reset only when, once every+ document has been checked, no other completed document under+ that tracker still holds a PDF blob: until then the URL's PDF is+ still stored in the database, as the tracker records.++ Left alone before any of these checks: a document whose own+ ``file_path`` is a sentinel, such as ``blob_deleted`` after "Remove+ PDF to save space", and a document whose ``storage_mode`` is "none"+ or unset (a text-only document), unless the URL's tracker names its+ path.++ ``missing_files`` lists every document counted in ``files_missing``+ whose ``file_path`` names a file, by the path stored for it,+ relative to the library root, never by the absolute path on the+ server (see ``_missing_file_entry``). It also lists every document+ deleted while the URL's tracker records it as stored in the+ database, as ``"database"``.++ ``trackers_updated`` counts the trackers the sync changed, each+ once, whichever document of their URL it reads first (see+ ``_clear_missing_pdf``).+Returns:Statistics about the sync operation"""@@ -1246,6 +1527,12 @@ class LibraryService:# Documents deleted below — their FAISS vectors + chunk rows are# purged AFTER this transaction commits (see the post-commit block).removed_doc_ids: List[str] = []+ # The documents checked under each "database" tracker, by+ # tracker id, and the "database" trackers whose document was+ # deleted below. Whether such a tracker is reset is decided after+ # the loop, once every document under it has been checked.+ database_tracker_documents: Dict[int, List[str]] = {}+ emptied_database_trackers: Dict[int, DownloadTracker] = {}# Sync documents with filesystem.#@@ -1269,16 +1556,46 @@ class LibraryService:# legacy-hash rows so pre-upgrade documents are not# mistaken for tracker-less orphans and deleted below.tracker = find_tracker_by_url(session, doc.original_url)+ if tracker is not None and tracker.file_path == "database":+ database_tracker_documents.setdefault(+ tracker.id, []+ ).append(doc.id)+ # A URL has one tracker, shared by every document with+ # that URL: the documents of two researches that cite+ # the same paper, or a failed-download placeholder that+ # "Extract Text to DB" completed with text next to the+ # document that holds the PDF. The tracker's path is the+ # one written last. It describes this document only when+ # it names where the document stores its PDF: the+ # document's own path, which every download that stores+ # a PDF file writes to both, or "database" for a+ # document labelled as stored in the database (see+ # _tracker_describes_storage). Any other document is+ # judged by what it stores itself, in the ``else``+ # branch, and leaves the tracker alone. Judged by+ # another document's file, it was counted found while+ # its own file was missing, or labelled a PDF, made+ # text-only and reported missing for a file it never+ # had, which the download manager then downloaded into+ # it as a second copy.+ if (+ tracker is not None+ and tracker.file_path+ and not _tracker_describes_storage(+ tracker.file_path, doc+ )+ ):+ tracker = Noneif tracker and tracker.file_path:- # Sentinel tracker paths (``blob_deleted``,+ # The tracker names where this document stores its+ # PDF: its own path, or the database.+ # Sentinel paths (``blob_deleted``,# ``text_only_not_stored``, ``metadata_only``) name# no file by design — e.g. a blob purge keeps the- # document complete with ``file_path="blob_deleted"``- # while a dedup-mirrored duplicate tracker could- # still carry that value. Resolving one as a- # relative path always misses and would run- # ``delete_document_completely`` on the+ # document complete with ``file_path="blob_deleted"``.+ # Resolving one as a relative path always misses and+ # would run ``delete_document_completely`` on the# text/chunks/vectors the sentinel was protecting# (issue #6760 follow-up). Leave them alone like the# text-only branch below.@@ -1286,17 +1603,16 @@ class LibraryService:stats["files_found"] += 1continue# Database-stored PDFs have no filesystem file to- # sync — the blob is the source of truth. Treat a- # present blob as found instead of falling into the- # missing-file path that deletes the document.+ # sync — the blob is the source of truth. A present+ # blob counts as found. ``DocumentBlob`` is the+ # module's import: importing it here would make it a+ # local name of the whole function, which the+ # post-loop check below could read only after this+ # branch had run.if (tracker.file_path == "database"or doc.storage_mode == "database"):- from ...database.models.library import (- DocumentBlob,- )-has_blob = (session.query(DocumentBlob.document_id).filter_by(document_id=doc.id)@@ -1319,25 +1635,69 @@ class LibraryService:self.username,settings_manager=settings_manager,)- if file_path and file_path.is_file():+ # "database" names no file: a file of that name in+ # the library root is not this document's PDF.+ if (+ _names_a_file(tracker.file_path)+ and file_path+ and file_path.is_file()+ ):+ stats["files_found"] += 1+ continue+ # The document's file is gone, or its blob is. What+ # else it stores decides.+ remaining = self._remaining_content(+ session, doc, settings_manager+ )+ if remaining == "text" and _names_a_file(+ tracker.file_path+ ):+ # Only the text is left of a PDF that was a+ # file. Keep the text and clear the dangling+ # PDF reference.+ if doc.resource_id is not None:+ # Its resource can download the PDF again:+ # report the PDF missing, under the stored+ # path that named the missing file.+ self._keep_text_report_missing_pdf(+ doc,+ tracker,+ tracker.file_path,+ file_path,+ stats,+ )+ else:+ # No download path asks for its PDF, but a+ # later download of the same bytes stores+ # the PDF in it (see _clear_missing_pdf).+ self._clear_missing_pdf(doc, tracker, stats)+ stats["files_found"] += 1+ elif remaining is not None:+ # Its PDF is still stored as a blob, or its text+ # is left and its path is the "database" label,+ # which names no file. Keep it and its tracker+ # unchanged.stats["files_found"] += 1else:# File missing or path invalid - mark for re-downloadstats["files_missing"] += 1stats["missing_files"].append(- {- "id": doc.id,- "title": doc.title,- "path": str(file_path)- if file_path- else "invalid",- "url": doc.original_url,- }+ _missing_file_entry(+ doc, tracker.file_path, file_path+ ))- # Reset tracker- tracker.is_downloaded = False- tracker.file_path = None+ if tracker.file_path == "database":+ # "database" describes every document of+ # the URL stored there, and another one can+ # still hold its PDF blob. Decided after the+ # loop.+ emptied_database_trackers[tracker.id] = tracker+ else:+ # Reset tracker+ tracker.is_downloaded = False+ tracker.file_path = None+ stats["trackers_updated"] += 1# Delete the document entry so it can be re-queuedfrom ..deletion.utils.cascade_helper import (@@ -1348,19 +1708,86 @@ class LibraryService:session, doc.id)removed_doc_ids.append(doc.id)- stats["trackers_updated"] += 1else:- # No tracker or path. Text-only documents- # (storage_mode "none") never have a file by design —- # deleting them here would destroy text, chunks, and- # vectors that are recoverable only by re-download- # (dead links = permanent loss). Leave them alone.+ # No tracker, a tracker without a path, or one that+ # names where other documents store their PDF:+ # another document's path, or "database" for a+ # document not stored there. That tracker is not+ # passed on (``tracker`` is None). Text-only+ # documents (storage_mode "none") never have a file+ # by design — deleting them here would destroy+ # text, chunks, and vectors that are recoverable+ # only by re-download (dead links = permanent loss).+ # Leave them alone. A sentinel path on the document+ # means the same: "Remove PDF to save space"+ # (delete_blob_only) writes ``blob_deleted`` to the+ # document but leaves the tracker naming the copy it+ # removed. That PDF was removed on purpose, so it is+ # neither missing nor to be downloaded again.if (doc.storage_mode in (None, "none")) or (doc.file_path in FILE_PATH_SENTINELS):stats["files_found"] += 1continue+ # Any other label can be stale, e.g. a lost tracker+ # path after a "none"-mode re-download, or the+ # "database" default on a placeholder that later got+ # text. What the document still stores decides.+ if settings_manager is None and doc.file_path:+ settings_manager = get_settings_manager(+ username=self.username+ )+ remaining = self._remaining_content(+ session, doc, settings_manager+ )+ if remaining == "text" and _names_a_file(doc.file_path):+ # Its own PDF file is gone and only the text is+ # left. Keep the text and clear the dangling+ # PDF reference.+ if doc.resource_id is not None:+ # Its resource can download the PDF again:+ # report the PDF missing.+ self._keep_text_report_missing_pdf(+ doc,+ tracker,+ doc.file_path,+ get_absolute_path_from_settings(+ doc.file_path,+ self.username,+ settings_manager=settings_manager,+ ),+ stats,+ )+ else:+ # No download path asks for its PDF, but a+ # later download of the same bytes stores+ # the PDF in it (see _clear_missing_pdf).+ self._clear_missing_pdf(doc, tracker, stats)+ stats["files_found"] += 1+ continue+ if remaining is not None:+ # Its PDF is still stored, or its text is left+ # and it names no file (e.g. the "database"+ # default on a placeholder that later got text).+ stats["files_found"] += 1+ continue+ # Nothing of it is left: delete it so that its URL+ # can be downloaded again. A file its own path names+ # is reported missing, as the branch above reports+ # one that the tracker names.stats["files_missing"] += 1+ if _names_a_file(doc.file_path):+ stats["missing_files"].append(+ _missing_file_entry(+ doc,+ doc.file_path,+ get_absolute_path_from_settings(+ doc.file_path,+ self.username,+ settings_manager=settings_manager,+ ),+ )+ )from ..deletion.utils.cascade_helper import (CascadeHelper,)@@ -1374,6 +1801,32 @@ class LibraryService:settings_manager, "settings manager", allow_none=True)+ # A "database" tracker records that its URL's PDF is stored in+ # the database, which can hold it for several documents of the+ # URL. One of them was deleted above with nothing left. While+ # another still holds a PDF blob, the record is true and the+ # tracker is left as it is; otherwise it is reset. Every+ # document under the tracker has been checked by now, so the+ # order in which the sync read them does not change whether it+ # is reset.+ removed = set(removed_doc_ids)+ for tracker_id, tracker in emptied_database_trackers.items():+ others = [+ document_id+ for document_id in database_tracker_documents[tracker_id]+ if document_id not in removed+ ]+ if others and (+ session.query(DocumentBlob.document_id)+ .filter(DocumentBlob.document_id.in_(others))+ .first()+ is not None+ ):+ continue+ tracker.is_downloaded = False+ tracker.file_path = None+ stats["trackers_updated"] += 1+session.commit()logger.info(f"Library sync completed: {stats['files_found']} found, {stats['files_missing']} missing"diff --git a/src/local_deep_research/research_library/utils/__init__.py b/src/local_deep_research/research_library/utils/__init__.pyindex 97960aacdc..661b74fb9d 100644--- a/src/local_deep_research/research_library/utils/__init__.py+++ b/src/local_deep_research/research_library/utils/__init__.py@@ -5,23 +5,31 @@ import osimport socketimport subprocessimport sys+from datetime import UTC, datetimefrom pathlib import Pathfrom typing import Optionalfrom urllib.parse import unquote, urlparsefrom fastapi.responses import JSONResponsefrom loguru import logger+from sqlalchemy import and_, func, or_from ...security import redact_url_for_logfrom ...config.paths import get_library_directory+from ...constants import FILE_PATH_TEXT_ONLY+from ...database.models.research import ResearchResource+from ...database.models.library import (+ Document,+ DocumentCollection,+ DocumentStatus,+)from ...utilities.download_url import (normalize_url_for_hash as normalize_url_for_hash,get_url_hash as get_url_hash,get_legacy_url_hashes as get_legacy_url_hashes,get_url_hash_candidates as get_url_hash_candidates,)-from ...database.models.library import Document, DocumentCollectionfrom ...security.ip_ranges import PRIVATE_IP_RANGESfrom ...security.network_utils import is_private_ipfrom ...security.path_validator import PathValidator@@ -262,6 +270,171 @@ def get_document_for_resource(session, resource):return session.query(Document).filter_by(resource_id=resource.id).first()+def resource_document_clause():+ """Join resources to their canonical document, honoring explicit links.++ Mirrors ``get_document_for_resource``: retained documents can still+ name a resource as their owner after that resource links elsewhere.+ """+ return or_(+ ResearchResource.document_id == Document.id,+ and_(+ ResearchResource.document_id.is_(None),+ ResearchResource.id == Document.resource_id,+ ),+ )+++def document_awaits_pdf(document) -> bool:+ """Whether a completed research download still needs its PDF.++ "Sync library" keeps a downloaded document whose PDF file was removed+ from disk if the document still has its extracted text and its own+ research resource (``resource_id``). It labels the document a PDF and+ stores it the way text-only documents are stored (``file_type`` "pdf",+ ``storage_mode`` "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), resets+ the URL's tracker unless it names where other documents store their+ PDF, and reports the file missing, so every document it reports that+ way matches here+ (see ``LibraryService._keep_text_report_missing_pdf``). Everywhere else+ a completed document counts as downloaded, so ``download_resource`` and+ the PDF queueing paths (``queue_research_downloads`` and the "Download+ All New" route, through ``document_awaits_pdf_clause``) ask this, for+ the document's own resource and for the resources linked to it by+ content (``ResearchResource.document_id``), and download the PDF again+ into the same document. One download refills it, so the queueing paths+ queue one of those resources (see ``DownloadService.refill_rank``).+ Once the document holds its PDF again, ``download_resource`` reports+ the document's own resource as already downloaded, and for a linked+ one whose URL the egress policy allows, ``DownloadService._download_pdf``+ reuses that stored PDF instead of fetching it. The+ Download Manager asks too (``LibraryService``'s+ ``get_research_list_with_stats``,+ ``get_download_manager_summary_stats`` and ``get_pdf_previews_batch``):+ such a document counts as available to download, not as downloaded,+ and shows no "Downloaded" badge. So do the library routes+ ``get_research_sources`` and ``check_downloads``, which report it as+ not downloaded, and ``download_source``, which downloads it again+ rather than answer "Already downloaded" (see+ ``document_counts_as_downloaded``). Text extraction does not ask: such+ a document already holds its text.++ A document whose resource is gone (``resource_id`` NULL) never matches,+ so nothing asks for its PDF. The sync stores such a document text-only+ too, without reporting it, so that a later download of the same PDF+ bytes for another resource stores the PDF in it (see+ ``LibraryService._clear_missing_pdf``).++ This checks ``file_type`` "pdf", not only that storage marker. Zotero+ imports and uploads write the same marker for PDFs and differ only in+ having no ``resource_id``. A text extraction (``file_type`` "text") has+ the same ``file_path`` and differs only in the ``"database"`` default of+ ``storage_mode``, so a writer that set it to "none", the value for a+ document without a stored PDF, would make every extracted text ask for+ a PDF. The sync can set the label itself because what it found missing+ was a PDF: only ``PDFStorageManager.save_pdf`` writes a file path into a+ document, and the sync converts a document only for a missing file+ that its own ``file_path`` names. The URL's tracker, which every+ document with that URL shares, counts for a document only when it+ names that path, or when it is ``"database"`` and the document is+ labelled as stored in the database, which names no file and so leads+ to no conversion. So a document it converts with another label, such as a+ failed-download placeholder that a PDF download completed before+ ``_download_pdf`` labelled such documents "pdf", is picked up too.++ No other writer leaves a research download in that shape: a+ "none"-mode download keeps ``file_path=None``, "Remove PDF to save+ space" writes ``blob_deleted``, and a text extraction is ``file_type``+ "text". A download into the document that stores no PDF either, as in+ storage mode "none" or for a PDF over the size limit, clears the mark,+ see ``DownloadService._download_pdf``.+ """+ return (+ document.status == DocumentStatus.COMPLETED+ and document.file_type == "pdf"+ and document.storage_mode == "none"+ and document.file_path == FILE_PATH_TEXT_ONLY+ and document.resource_id is not None+ )+++def document_awaits_pdf_clause():+ """:func:`document_awaits_pdf` as a SQL condition on ``Document``.++ The nullable columns are compared through ``coalesce`` and ``status``+ is NOT NULL, so on a ``Document`` row the condition is never NULL and+ its negation is true for every document the function rejects.+ """+ return and_(+ Document.status == DocumentStatus.COMPLETED,+ func.coalesce(Document.file_type, "") == "pdf",+ func.coalesce(Document.storage_mode, "") == "none",+ func.coalesce(Document.file_path, "") == FILE_PATH_TEXT_ONLY,+ Document.resource_id.isnot(None),+ )+++def document_counts_as_downloaded(document) -> bool:+ """Whether a loaded ``Document`` counts as downloaded.++ A completed document counts, unless it awaits its PDF (see+ ``document_awaits_pdf``), which the Download Manager offers for+ download again. Its counts apply the same rule in SQL+ (``LibraryService``'s ``_counts_as_downloaded``), and the source+ previews under each research (``get_pdf_previews_batch``) apply this+ function. So do the library routes that tell whether a research's+ source is downloaded (``get_research_sources`` and ``check_downloads``)+ and ``download_source``, which answers "Already downloaded" only for a+ document that counts.++ ``status`` loads as a ``DocumentStatus`` member. ``DocumentStatus`` is+ a plain ``enum.Enum``, so a member never equals its string value+ (``DocumentStatus.COMPLETED == "completed"`` is False), and the status+ is compared with the member.+ """+ return document.status == DocumentStatus.COMPLETED and not (+ document_awaits_pdf(document)+ )+++# When this server process started. This module is imported while the app+# starts, before any download can claim a queue row. See+# download_claim_is_live.+_PROCESS_STARTED_AT = datetime.now(UTC)+++def download_claim_is_live(claimed_at: Optional[datetime]) -> bool:+ """Whether a PROCESSING download-queue row is still being downloaded.++ ``claimed_at`` is the row's ``last_attempt_at``, which the download+ that claims it sets (``_claim_download_queue_item`` in the library+ routes). A download finishes or releases the row it claimed before it+ ends, and it cannot outlive this server process, which serves with one+ worker (see ``web/app.py``). So a row claimed before this process+ started was cut off by a restart, as is one claimed before claims+ recorded their time (``claimed_at`` None): nothing finishes or+ releases it, so it does not stand for a download in progress. A row+ whose release failed in this process (which is logged) still counts+ as in progress until the next restart.++ Both times are wall-clock readings, so a clock that goes back defeats+ the rule. If this process started with its clock behind the time an+ earlier process recorded for a claim, as on a device without a+ real-time clock that starts with a stale time and has it corrected+ later, that claim counts as in progress until the next restart, and no+ other resource of its document refills the document until then. A+ clock set back while this process runs can make a claim it made count+ as cut off, so that another resource of the document can be queued+ while that download runs. The queue row records nothing else that+ tells the process that claimed it.+ """+ if claimed_at is None:+ return False+ if claimed_at.tzinfo is None:+ claimed_at = claimed_at.replace(tzinfo=UTC)+ return claimed_at >= _PROCESS_STARTED_AT++def find_tracker_by_url(session, url: str):"""Find a ``DownloadTracker`` for ``url`` across hash schemes.diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.pyindex 2723810b65..cb473d8584 100644--- a/src/local_deep_research/security/egress/policy.py+++ b/src/local_deep_research/security/egress/policy.py@@ -1587,13 +1587,22 @@ def _quota_ctx(ctx: EgressContext) -> EgressContext:return active if active is not None else ctx-def evaluate_url(url: str, ctx: EgressContext) -> Decision:+def evaluate_url(+ url: str, ctx: EgressContext, *, record_denial: bool = True+) -> Decision:"""Decide whether an arbitrary URL may be fetched (e.g., by ``fetch_content``).Enforces the run's denied-fetch quota to prevent exhaustion attacksvia malicious indexed documents that loop the agent through hundredsof denied fetches.++ ``record_denial=False`` returns the same decision without recording a+ denial: it is neither counted toward that quota nor written to the+ audit log (an internal error is still logged). Only for a caller that+ fetches nothing on the decision, such as one that ranks URLs; a fetch+ must be decided with the default, so that its denials are counted."""+ deny = _record_denial if record_denial else _unrecorded_denialtry:# Quota check: hard fail after MAX_DENIED_FETCHES_PER_RUN. Anchor the# counter to the run's active context (``_quota_ctx``) so the budget is@@ -1605,19 +1614,19 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:return Decision(False, "denial_quota_exceeded")if not isinstance(url, str) or not url:- return _record_denial(ctx, url, "url_malformed")+ return deny(ctx, url, "url_malformed")try:parsed = urlsplit(url)except Exception:- return _record_denial(ctx, url, "url_malformed")+ return deny(ctx, url, "url_malformed")if parsed.scheme.lower() in _DANGEROUS_SCHEMES:- return _record_denial(ctx, url, "dangerous_scheme")+ return deny(ctx, url, "dangerous_scheme")if parsed.scheme.lower() not in ("http", "https"):- return _record_denial(ctx, url, "unsupported_scheme")+ return deny(ctx, url, "unsupported_scheme")if not parsed.hostname:- return _record_denial(ctx, url, "no_hostname")+ return deny(ctx, url, "no_hostname")# HTTP client libraries (requests/urllib3) percent-DECODE the host# in the netloc before the socket connect, but urlsplit().hostname@@ -1650,7 +1659,7 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:if is_ip_blocked(host, allow_localhost=True, allow_private_ips=True):- return _record_denial(ctx, url, "blocked_metadata_ip")+ return deny(ctx, url, "blocked_metadata_ip")# is_ip_blocked only parses canonical IP notation, so an# alternate-encoded metadata literal (octal/hex/integer) slips# through above. Normalize it to the dotted-quad the resolver@@ -1666,14 +1675,14 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:alt, allow_localhost=True, allow_private_ips=True)):- return _record_denial(ctx, url, "blocked_metadata_ip")+ return deny(ctx, url, "blocked_metadata_ip")# Metadata endpoints reachable by hostname (GCP) — is_ip_blocked# can't see these because they aren't IP literals. Block by name so# the invariant holds under PUBLIC_ONLY/BOTH too (the SSRF validator# backstops the actual fetch, but evaluate_url's own guarantee must# not depend on it).if host.lower() in _METADATA_HOSTNAMES:- return _record_denial(ctx, url, "blocked_metadata_ip")+ return deny(ctx, url, "blocked_metadata_ip")except Exception: # noqa: silent-exception - defensive, see below# Non-IP host or helper error: fall through to normal# classification (DNS path has its own handling).@@ -1695,27 +1704,37 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:# bigger risk than false-positives on DOI redirects.if classification is True:return Decision(True, "allowed_private_host_under_strict")- return _record_denial(ctx, url, "strict_public_host")+ return deny(ctx, url, "strict_public_host")if ctx.scope == EgressScope.PUBLIC_ONLY:if classification is False:return Decision(True, "allowed_public_host")- return _record_denial(ctx, url, "scope_mismatch_public_only")+ return deny(ctx, url, "scope_mismatch_public_only")if ctx.scope == EgressScope.PRIVATE_ONLY:if classification is True:return Decision(True, "allowed_private_host")- return _record_denial(ctx, url, "scope_mismatch_private_only")+ return deny(ctx, url, "scope_mismatch_private_only")# BOTH: any classified host is fine.if classification is None:- return _record_denial(ctx, url, "host_unclassified")+ return deny(ctx, url, "host_unclassified")return Decision(True, "allowed_both_scope")except Exception: # pragma: no cover - defensivelogger.bind(policy_audit=True).exception("evaluate_url internal error")return Decision(False, "internal_error")+def _unrecorded_denial(+ ctx: EgressContext, url: object, reason: str+) -> Decision:+ """A denial for ``evaluate_url(record_denial=False)``.++ Neither counted toward the denied-fetch quota nor logged.+ """+ return Decision(False, reason)++def host_not_classified_public(url: str, ctx: EgressContext) -> bool:"""True when ``url`` is an http(s) URL whose host the policy does NOTclassify public (private, or unclassifiable), decided WITHOUT recordingdiff --git a/src/local_deep_research/web/routers/library.py b/src/local_deep_research/web/routers/library.pyindex 7721deb0e8..d9ff7f8472 100644--- a/src/local_deep_research/web/routers/library.py+++ b/src/local_deep_research/web/routers/library.py@@ -46,7 +46,11 @@ from ...research_library.services.pdf_storage_manager import PDFStorageManagerfrom ...security import failure_reason_for_log, redact_url_for_logfrom ...research_library.utils import (apply_user_subdir,+ document_awaits_pdf_clause,+ document_counts_as_downloaded,+ download_claim_is_live,get_document_for_resource,+ resource_document_clause,handle_api_error,is_downloadable_domain,is_downloadable_url,@@ -1019,12 +1023,22 @@ def _claim_download_queue_item(db_session, queue_item_id):for every row (issue #4691). The row-count guard makes thePENDING -> PROCESSING transition the single point where exactly one streamwins each row; the loser sees 0 updated rows and skips it.++ The claim also records its time in ``last_attempt_at``. Nothing reclaims+ a row left PROCESSING, e.g. by a restart during the download, and the+ queueing of a document that awaits its PDF (see ``document_awaits_pdf``)+ lets another resource refill it only while no download holds one of its+ resources. ``download_claim_is_live`` tells a held row from one a restart+ left behind by this time."""claimed = (db_session.query(LibraryDownloadQueue).filter_by(id=queue_item_id, status=DocumentStatus.PENDING).update(- {LibraryDownloadQueue.status: DocumentStatus.PROCESSING},+ {+ LibraryDownloadQueue.status: DocumentStatus.PROCESSING,+ LibraryDownloadQueue.last_attempt_at: datetime.now(UTC),+ },synchronize_session=False,))@@ -1155,11 +1169,20 @@ async def download_bulk(# a COMPLETED Document, and re-queues any remaining row (e.g. a# prior FAILED attempt) back to PENDING. Calling it BEFORE counting# ensures the total reflects what will actually be processed- # (issue #4660).+ # (issue #4660). In PDF mode it also queues a COMPLETED Document+ # that still needs its PDF (see document_awaits_pdf), through+ # one resource per such document across all the research+ # sessions of this run (run_research_ids); a text extraction+ # already has that document's text, so text_only mode skips it.+ # The set is built once for the run: each call only looks it up.+ run_research_ids = frozenset(research_ids)for research_id in research_ids:try:download_service.queue_research_downloads(- research_id, collection_id+ research_id,+ collection_id,+ text_only=mode == "text_only",+ run_research_ids=run_research_ids,)except Exception:queue_failures += 1@@ -1486,6 +1509,110 @@ async def mark_for_redownload(return {"success": True, "marked": count}+def _one_resource_per_awaiting_document(+ db_session, resources, username, password+):+ """Keep one of the resources that would refill the same document.++ A document that awaits its PDF (see ``document_awaits_pdf``) does not+ count as a download, so "Download All New" finds its own resource+ (``Document.resource_id``) and every resource linked to it by content+ (``ResearchResource.document_id``). Each of them downloads into that+ document (``get_document_for_resource``), and one download refills it.+ So of the ``resources`` that would refill the same document only one is+ kept, the one ``queue_research_downloads`` would leave the document to+ (see ``DownloadService._refilled_by_another_resource``): one that a+ download holds (a PROCESSING queue entry, see+ ``download_claim_is_live``), else a viable one that is already PENDING+ in the download queue, else the one that ranks first in+ ``DownloadService.refill_rank``. So a resource that cannot succeed (its+ URL cannot be queued, the retry manager holds it back, or its latest+ attempt failed) is kept only when it is being downloaded or no viable+ one is there. A resource whose PROCESSING entry a restart left behind+ comes last, since no reset touches that entry. The others count as+ skipped. (Once the document holds its PDF again,+ ``DownloadService.download_resource`` reports the document's own+ resource as already downloaded, and for a linked one whose URL the+ egress policy allows, ``DownloadService._download_pdf`` reuses that+ stored PDF instead of fetching it.)++ ``resources`` have passed the route's retry-policy and downloadability+ checks. The ranking also asks the egress policy and the resources'+ retry records, through a ``DownloadService`` for ``username``, which is+ built only when some document has more than one of them.++ Returns the kept resources in their order, and how many were left out.+ """+ awaiting = {+ resource_id: (document_id, owner_id)+ for resource_id, document_id, owner_id in (+ db_session.query(+ ResearchResource.id, Document.id, Document.resource_id+ )+ .join(+ Document,+ resource_document_clause(),+ )+ .filter(document_awaits_pdf_clause())+ .all()+ )+ }+ candidates = {}+ for resource in resources:+ if resource.id in awaiting:+ document_id = awaiting[resource.id][0]+ candidates.setdefault(document_id, []).append(resource)+ shared = [rows for rows in candidates.values() if len(rows) > 1]+ if not shared:+ return resources, 0++ entries = {+ entry.resource_id: entry+ for entry in db_session.query(+ LibraryDownloadQueue.resource_id,+ LibraryDownloadQueue.status,+ LibraryDownloadQueue.last_attempt_at,+ )+ .filter(+ LibraryDownloadQueue.resource_id.in_(+ [row.id for rows in shared for row in rows]+ ),+ LibraryDownloadQueue.status.in_(+ (DocumentStatus.PENDING, DocumentStatus.PROCESSING)+ ),+ )+ .all()+ }+ left_out = set()+ with DownloadService(username, password) as download_service:+ for rows in shared:+ owner_id = awaiting[rows[0].id][1]+ choices = []+ for row in rows:+ rank = download_service.refill_rank(row.id, row.url, owner_id)+ entry = entries.get(row.id)+ if entry is None:+ tier = 2+ elif entry.status == DocumentStatus.PROCESSING:+ tier = (+ 0+ if download_claim_is_live(entry.last_attempt_at)+ else 3+ )+ else:+ tier = 1 if rank.viable else 2+ choices.append(((tier, rank), row.id))+ kept_id = min(choices)[1]+ left_out.update(row.id for row in rows if row.id != kept_id)+ for resource_id in sorted(left_out):+ logger.debug(+ f"Skipping resource {resource_id}: another resource of its "+ "document is queued to download its PDF"+ )+ kept = [resource for resource in resources if resource.id not in left_out]+ return kept, len(left_out)++@router.post("/api/queue-all-undownloaded")def queue_all_undownloaded(request: Request, username: Annotated[str, Depends(require_auth)]@@ -1499,6 +1626,11 @@ def queue_all_undownloaded(# the columns the filter/loop use (id/url/research_id) instead of full# ``ResearchResource`` entities so the ``content_preview`` Text column# isn't materialized for every row on this whole-table scan (#4560).+ # A completed document that still needs its PDF (the library sync+ # kept its text after its PDF file went missing) is not counted as+ # a download, see document_awaits_pdf. Only one of the resources+ # that would refill such a document is queued, see+ # _one_resource_per_awaiting_document.undownloaded = (db_session.query(ResearchResource.id,@@ -1507,11 +1639,9 @@ def queue_all_undownloaded().outerjoin(Document,- (- (ResearchResource.id == Document.resource_id)- | (ResearchResource.document_id == Document.id)- )- & (Document.status == "completed"),+ resource_document_clause()+ & (Document.status == "completed")+ & ~document_awaits_pdf_clause(),).filter(Document.id.is_(None)).all()@@ -1540,6 +1670,8 @@ def queue_all_undownloaded(# Convert filter_results to dict for O(1) lookup instead of O(n²)filter_results_by_id = {r.resource_id: r for r in filter_results}+ # The resources that pass every check below, in scan order.+ eligible = []for resource in undownloaded:# Check if resource passed the smart filterfilter_result = filter_results_by_id.get(resource.id)@@ -1587,6 +1719,16 @@ def queue_all_undownloaded()continue+ eligible.append(resource)++ # One download refills a document that awaits its PDF, so only one+ # of the resources that would refill the same document is queued.+ eligible, covered = _one_resource_per_awaiting_document(+ db_session, eligible, username, user_password+ )+ skipped_count += covered++ for resource in eligible:# Check if already in queue (any status)existing_queue = (db_session.query(LibraryDownloadQueue)@@ -1714,7 +1856,10 @@ def get_research_sources("file_type": None,}- if document and document.status == "completed":+ # The Download Manager's rule: a completed document, unless it+ # awaits its PDF (see document_counts_as_downloaded). A loaded+ # status never equals the string "completed".+ if document and document_counts_as_downloaded(document):source_data.update({"downloaded": True,@@ -1760,7 +1905,10 @@ async def check_downloads()for resource in resources:document = get_document_for_resource(db_session, resource)- if document and document.status == "completed":+ # A document that awaits its PDF is reported not downloaded,+ # with the resource_id to download it by (see+ # document_counts_as_downloaded).+ if document and document_counts_as_downloaded(document):# Do NOT return document.file_path — it's the absolute# server path, which leaks directory layout to any# authenticated user. The client fetches via@@ -1821,10 +1969,12 @@ async def download_source({"error": "Resource not found"}, status_code=404)- # Check if already downloaded+ # Check if already downloaded, by the Download Manager's rule+ # (see document_counts_as_downloaded). A document that awaits+ # its PDF is downloaded again, into the same document.existing = get_document_for_resource(db_session, resource)- if existing and existing.status == "completed":+ if existing and document_counts_as_downloaded(existing):return {"success": True,"message": "Already downloaded",diff --git a/src/local_deep_research/web/templates/pages/download_manager.html b/src/local_deep_research/web/templates/pages/download_manager.htmlindex d7d5225841..4fd6ae2f33 100644--- a/src/local_deep_research/web/templates/pages/download_manager.html+++ b/src/local_deep_research/web/templates/pages/download_manager.html@@ -284,7 +284,7 @@<i aria-hidden="true" class="fas fa-file-pdf"></i><span class="ldr-source-title">{{ source.document_title|truncate(60) }}</span><span class="ldr-source-domain">{{ source.domain }}</span>- {% if source.download_status == 'completed' %}+ {% if source.downloaded %}<span class="ldr-badge ldr-badge-success">Downloaded</span>{% endif %}</div>diff --git a/tests/research_library/services/test_download_service_arxiv_provenance.py b/tests/research_library/services/test_download_service_arxiv_provenance.pyindex 0df0c9fc11..76e557be44 100644--- a/tests/research_library/services/test_download_service_arxiv_provenance.py+++ b/tests/research_library/services/test_download_service_arxiv_provenance.py@@ -461,9 +461,9 @@ def test_arxiv_retry_denial_reads_a_database_stored_pdf(service):session = MagicMock()document = MagicMock()document.id = "database-pdf-document"- # The string form satisfies both stored-PDF readers' status checks, so+ # The loaded member satisfies both stored-PDF readers' status checks, so# the filesystem strategy is stopped by the missing file_path alone.- document.status = "completed"+ document.status = DocumentStatus.COMPLETEDdocument.file_type = "pdf"document.file_path = None# First lookup is the resource; any later one is its stored document.diff --git a/tests/research_library/services/test_download_service_coverage.py b/tests/research_library/services/test_download_service_coverage.pyindex 68ecaa0b9c..a3a251352f 100644--- a/tests/research_library/services/test_download_service_coverage.py+++ b/tests/research_library/services/test_download_service_coverage.py@@ -962,7 +962,7 @@ class TestTryExistingPdfExtraction:def test_not_completed(self, svc):doc = MagicMock()- doc.status = "pending"+ doc.status = DocumentStatus.PENDINGsession = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (doc@@ -972,7 +972,7 @@ class TestTryExistingPdfExtraction:def test_no_file_path(self, svc):doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = Nonesession = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -983,7 +983,7 @@ class TestTryExistingPdfExtraction:def test_sentinel_file_path(self, svc):doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = "text_only_not_stored"session = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -997,7 +997,7 @@ class TestTryExistingPdfExtraction:def test_path_traversal_blocked(self, svc):doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = "../../etc/passwd"session = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -1019,7 +1019,7 @@ class TestTryExistingPdfExtraction:def test_file_not_found(self, svc, tmp_path):doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = str(tmp_path / "missing.pdf")session = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -1043,7 +1043,7 @@ class TestTryExistingPdfExtraction:pdf_file.write_bytes(b"fake pdf")doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = str(pdf_file)doc.id = "doc-1"session = MagicMock()@@ -1071,7 +1071,7 @@ class TestTryExistingPdfExtraction:pdf_file.write_bytes(b"bad pdf")doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = str(pdf_file)session = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -1096,7 +1096,7 @@ class TestTryExistingPdfExtraction:pdf_file.write_bytes(b"pdf")doc = MagicMock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = str(pdf_file)session = MagicMock()session.query.return_value.filter_by.return_value.first.return_value = (@@ -1919,6 +1919,10 @@ class TestDownloadResource:session = MagicMock()resource = MagicMock()resource.id = 1+ # No explicit document link (ResearchResource.document_id). A bare+ # MagicMock attribute is truthy and would send download_resource+ # down its explicit-link lookup, which these mocks do not stub.+ resource.document_id = Nonesession.get.return_value = resourceexisting_doc = MagicMock()@@ -1940,6 +1944,7 @@ class TestDownloadResource:resource = MagicMock()resource.id = 1resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resource# Return None for existing_doc check, then various other queries@@ -1980,6 +1985,7 @@ class TestDownloadResource:resource = MagicMock()resource.id = 1resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcecall_count = [0]diff --git a/tests/research_library/services/test_download_service_deep_coverage.py b/tests/research_library/services/test_download_service_deep_coverage.pyindex 3d96269fec..5f73350148 100644--- a/tests/research_library/services/test_download_service_deep_coverage.py+++ b/tests/research_library/services/test_download_service_deep_coverage.py@@ -62,6 +62,10 @@ class TestDownloadResourceDeep:resource = MagicMock()resource.id = 5resource.url = "https://example.com/paper.pdf"+ # No explicit document link (ResearchResource.document_id). A bare+ # MagicMock attribute is truthy and would send download_resource+ # down its explicit-link lookup, which these mocks do not stub.+ resource.document_id = Nonesession.get.return_value = resourceexisting_tracker = MagicMock()@@ -100,6 +104,7 @@ class TestDownloadResourceDeep:resource = MagicMock()resource.id = 2resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcetracker = MagicMock()@@ -137,6 +142,7 @@ class TestDownloadResourceDeep:resource = MagicMock()resource.id = 3resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcetracker = MagicMock()@@ -180,6 +186,7 @@ class TestDownloadResourceDeep:resource = MagicMock()resource.id = 4resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcetracker = MagicMock()@@ -214,6 +221,7 @@ class TestDownloadResourceDeep:resource = MagicMock()resource.id = 6resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcetracker = MagicMock()diff --git a/tests/research_library/services/test_download_service_logic.py b/tests/research_library/services/test_download_service_logic.pyindex e956d2417f..3dc5a5f8e9 100644--- a/tests/research_library/services/test_download_service_logic.py+++ b/tests/research_library/services/test_download_service_logic.py@@ -12,6 +12,8 @@ from unittest.mock import Mock, patchimport pytest+from local_deep_research.database.models.library import DocumentStatus+# ---------------------------------------------------------------------------# Helpers@@ -235,11 +237,11 @@ class TestTryExistingPdfExtraction:service.close()def test_not_completed_returns_none(self, tmp_path):- """Document with status != 'completed' -> returns None."""+ """A Document that is not completed -> returns None."""service = _create_service(tmp_path)doc = Mock()- doc.status = "pending"+ doc.status = DocumentStatus.PENDINGmock_session = Mock()mock_query = Mock()@@ -256,7 +258,7 @@ class TestTryExistingPdfExtraction:service = _create_service(tmp_path)doc = Mock()- doc.status = "completed"+ doc.status = DocumentStatus.COMPLETEDdoc.file_path = "pdfs/nonexistent.pdf"mock_session = Mock()diff --git a/tests/research_library/services/test_download_service_strategy_coverage.py b/tests/research_library/services/test_download_service_strategy_coverage.pyindex 84f0db4b6b..f6e8f8bf86 100644--- a/tests/research_library/services/test_download_service_strategy_coverage.py+++ b/tests/research_library/services/test_download_service_strategy_coverage.py@@ -17,6 +17,7 @@ from unittest.mock import MagicMock, patchimport pytest+from local_deep_research.database.models.library import DocumentStatusfrom local_deep_research.research_library.services.download_service import (DownloadService,)@@ -81,6 +82,10 @@ class TestDownloadResourceArxivStrategy:resource = MagicMock()resource.id = 1resource.url = "https://arxiv.org/abs/2301.00001"+ # No explicit document link (ResearchResource.document_id). A bare+ # MagicMock attribute is truthy and would send download_resource+ # down its explicit-link lookup, which these mocks do not stub.+ resource.document_id = Nonesession.get.return_value = resource# Simulate: existing_doc=None, queue=None, tracker=None, post-download queue=None@@ -279,6 +284,7 @@ class TestDownloadResourcePubmedStrategy:resource = MagicMock()resource.id = 2resource.url = "https://pubmed.ncbi.nlm.nih.gov/12345678"+ resource.document_id = Nonesession.get.return_value = resourcesession.query.return_value.filter_by.return_value.first.return_value = (None@@ -507,6 +513,7 @@ class TestDownloadResourceDedupCheck:resource = MagicMock()resource.id = 10resource.url = "https://arxiv.org/abs/2301.99999"+ resource.document_id = Nonesession.get.return_value = resourceexisting_doc = MagicMock()@@ -531,6 +538,7 @@ class TestDownloadResourceDedupCheck:resource = MagicMock()resource.id = 11resource.url = "https://arxiv.org/abs/2301.12345"+ resource.document_id = Nonesession.get.return_value = resourceexisting_tracker = MagicMock()@@ -574,6 +582,7 @@ class TestDownloadResourceDedupCheck:resource = MagicMock()resource.id = 12resource.url = "https://example.com/new-paper.pdf"+ resource.document_id = Nonesession.get.return_value = resource# All filter_by queries return None (no existing doc, queue, or tracker)@@ -671,7 +680,9 @@ class TestDownloadAsTextSuccess:pdf_path.write_bytes(b"%PDF-stored")pdf_document = MagicMock()pdf_document.id = "stored-pdf-document"- pdf_document.status = "completed"+ # The member a loaded row holds, which the filesystem strategy+ # compares with (the string "completed" never equals it).+ pdf_document.status = DocumentStatus.COMPLETEDpdf_document.file_path = "stored.pdf"session.query.return_value.filter_by.return_value.first.side_effect = [resource,@@ -903,6 +914,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 20resource.url = "https://arxiv.org/abs/2301.11111"+ resource.document_id = Nonesession.get.return_value = resourcesession.query.return_value.filter_by.return_value.first.return_value = (None@@ -929,6 +941,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 21resource.url = "https://example.com/paywalled.pdf"+ resource.document_id = Nonesession.get.return_value = resourcesession.query.return_value.filter_by.return_value.first.return_value = (None@@ -959,6 +972,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 22resource.url = "https://pubmed.ncbi.nlm.nih.gov/77777777"+ resource.document_id = Nonesession.get.return_value = resourcesession.query.return_value.filter_by.return_value.first.return_value = (None@@ -983,6 +997,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 23resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcesession.query.return_value.filter_by.return_value.first.return_value = (None@@ -1007,6 +1022,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 24resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcequeue_entry = MagicMock()@@ -1043,6 +1059,7 @@ class TestRetryFailedDownload:resource = MagicMock()resource.id = 25resource.url = "https://example.com/paper.pdf"+ resource.document_id = Nonesession.get.return_value = resourcequeue_entry = MagicMock()diff --git a/tests/research_library/services/test_library_service_collections_stats.py b/tests/research_library/services/test_library_service_collections_stats.pyindex 072ded666d..813d97a01a 100644--- a/tests/research_library/services/test_library_service_collections_stats.py+++ b/tests/research_library/services/test_library_service_collections_stats.py@@ -746,7 +746,16 @@ class TestGetPdfPreviewsBatch:assert total_domain_docs == 5def test_domain_breakdown_counts(self, mocker):- """Domain breakdown correctly counts total/pdfs/downloaded."""+ """Domain breakdown correctly counts total/pdfs/downloaded.++ ``status`` is a ``DocumentStatus`` member, as the ORM loads it. Set+ to the string "completed", it passed a comparison with that string+ that a loaded document never passes.+ """+ from local_deep_research.database.models.library import (+ DocumentStatus,+ )+service = _make_service()mock_session = MagicMock()@@ -754,7 +763,7 @@ class TestGetPdfPreviewsBatch:doc1.id = "d1"doc1.research_id = "r1"doc1.file_type = "pdf"- doc1.status = "completed"+ doc1.status = DocumentStatus.COMPLETEDdoc1.original_url = Nonedoc1.filename = "a.pdf"@@ -762,7 +771,7 @@ class TestGetPdfPreviewsBatch:doc2.id = "d2"doc2.research_id = "r1"doc2.file_type = "pdf"- doc2.status = "pending"+ doc2.status = DocumentStatus.PENDINGdoc2.original_url = Nonedoc2.filename = "b.pdf"@@ -788,6 +797,10 @@ class TestGetPdfPreviewsBatch:assert arxiv["total"] == 2assert arxiv["pdfs"] == 2assert arxiv["downloaded"] == 1 # only doc1 is completed+ # The "Downloaded" badge of each source.+ assert [+ source["downloaded"] for source in result["r1"]["pdf_sources"]+ ] == [True, False]def test_resource_none_falls_back_to_doc_filename(self, mocker):"""When resource is None, title falls back to doc.filename."""diff --git a/tests/research_library/services/test_library_service_coverage.py b/tests/research_library/services/test_library_service_coverage.pyindex 87f0c7b37c..1b9f363084 100644--- a/tests/research_library/services/test_library_service_coverage.py+++ b/tests/research_library/services/test_library_service_coverage.py@@ -14,6 +14,8 @@ Focuses narrowly on uncovered branches:from contextlib import contextmanagerfrom unittest.mock import Mock, MagicMock, patch+from local_deep_research.database.models.library import DocumentStatus+MODULE = "local_deep_research.research_library.services.library_service"@@ -69,7 +71,7 @@ def _setup_get_document_by_id(mock_doc.file_type = "pdf"mock_doc.mime_type = "application/pdf"mock_doc.text_content = None- mock_doc.status = "completed"+ mock_doc.status = DocumentStatus.COMPLETEDmock_doc.processed_at = Nonemock_doc.favorite = Falsemock_doc.tags = []@@ -285,7 +287,7 @@ class TestGetDocumentByIdResearchCreatedAt:mock_doc.file_type = "pdf"mock_doc.mime_type = "application/pdf"mock_doc.text_content = None- mock_doc.status = "completed"+ mock_doc.status = DocumentStatus.COMPLETEDmock_doc.processed_at = Nonemock_doc.favorite = Falsemock_doc.tags = []diff --git a/tests/research_library/services/test_library_service_crud_deep.py b/tests/research_library/services/test_library_service_crud_deep.pyindex 27bb0079e3..34eb8dbe54 100644--- a/tests/research_library/services/test_library_service_crud_deep.py+++ b/tests/research_library/services/test_library_service_crud_deep.py@@ -16,6 +16,7 @@ from local_deep_research.constants import (FILE_PATH_METADATA_ONLY,FILE_PATH_TEXT_ONLY,)+from local_deep_research.database.models.library import DocumentStatus# ============== Helper ==============@@ -302,7 +303,7 @@ class TestGetDocumentByIdHappyPaths:mock_doc.text_content = doc_overrides.get("text_content", "one two three")- mock_doc.status = "completed"+ mock_doc.status = DocumentStatus.COMPLETEDmock_doc.processed_at = Nonemock_doc.favorite = Falsemock_doc.tags = []@@ -365,6 +366,8 @@ class TestGetDocumentByIdHappyPaths:assert result is not Noneassert result["id"] == "doc-123"assert result["document_title"] == "Test Paper"+ # The details page compares the status's string value.+ assert result["download_status"] == "completed"assert "has_pdf" in resultassert "word_count" in resultassert "collections" in result@@ -517,6 +520,22 @@ class TestGetLibraryStatsValues:# ============== sync_library_with_filesystem ==============+# The sync deletes a missing-file download only when the document holds+# nothing else (no text, no blob, no file of its own). A Mock document has+# no real columns or rows for that check to read, so the deletion tests+# below pin it to "holds nothing else" (``None``). The rule itself is+# covered against a real database in+# tests/research_library/test_library_sync_contracts.py.+#+# The sync uses a tracker's file path for a document only when the+# document stores the same path, as every download that stores a PDF file+# writes it. The Mock documents below whose tracker names a file store+# that path.+_REMAINING_CONTENT = (+ "local_deep_research.research_library.services.library_service."+ "LibraryService._remaining_content"+)+class TestSyncLibraryWithFilesystem:"""Tests for sync_library_with_filesystem logic."""@@ -543,6 +562,7 @@ class TestSyncLibraryWithFilesystem:mock_doc.id = "doc-1"mock_doc.title = "Test"mock_doc.original_url = "https://example.com/doc.pdf"+ mock_doc.file_path = "pdfs/doc.pdf"mock_tracker = Mock()mock_tracker.file_path = "pdfs/doc.pdf"@@ -583,7 +603,7 @@ class TestSyncLibraryWithFilesystem:assert result["files_found"] == 1def test_doc_with_tracker_file_missing(self, mocker):- """Doc with tracker but missing file → files_missing + cascade delete."""+ """Doc with tracker, missing file, nothing else → cascade delete."""service = _make_service()mock_session = MagicMock()@@ -591,6 +611,7 @@ class TestSyncLibraryWithFilesystem:mock_doc.id = "doc-1"mock_doc.title = "Test"mock_doc.original_url = "https://example.com/doc.pdf"+ mock_doc.file_path = "pdfs/gone.pdf"mock_tracker = Mock()mock_tracker.file_path = "pdfs/gone.pdf"@@ -621,14 +642,19 @@ class TestSyncLibraryWithFilesystem:mocker.patch("local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely")+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)result = service.sync_library_with_filesystem()assert result["files_missing"] == 1assert result["trackers_updated"] == 1+ remaining.assert_called_once()def test_doc_without_tracker(self, mocker):- """Doc with no tracker → files_missing + cascade delete."""+ """Doc with no tracker that holds nothing else → cascade delete.++ The missing file its own path names is reported under that path.+ """service = _make_service()mock_session = MagicMock()@@ -636,6 +662,7 @@ class TestSyncLibraryWithFilesystem:mock_doc.id = "doc-1"mock_doc.title = "Test"mock_doc.original_url = "https://example.com/doc.pdf"+ mock_doc.file_path = "pdfs/gone.pdf"call_count = {"n": 0}@@ -654,16 +681,27 @@ class TestSyncLibraryWithFilesystem:mock_session.query.side_effect = query_router_mock_session_cm(mocker, mock_session)+ mock_path = MagicMock()+ mock_path.is_file.return_value = False+ mocker.patch(+ "local_deep_research.research_library.services.library_service.get_absolute_path_from_settings",+ return_value=mock_path,+ )mocker.patch("local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely")+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)result = service.sync_library_with_filesystem()assert result["files_missing"] == 1+ assert [+ (entry["id"], entry["path"]) for entry in result["missing_files"]+ ] == [("doc-1", "pdfs/gone.pdf")]+ remaining.assert_called_once()def test_multiple_mixed_docs(self, mocker):- """2 found + 1 missing → correct totals."""+ """2 found + 1 missing that holds nothing else → correct totals."""service = _make_service()mock_session = MagicMock()@@ -673,6 +711,7 @@ class TestSyncLibraryWithFilesystem:d.id = f"doc-{i}"d.title = f"Doc {i}"d.original_url = f"https://example.com/doc{i}.pdf"+ d.file_path = f"pdfs/doc{i}.pdf"docs.append(d)trackers = [Mock(file_path=f"pdfs/doc{i}.pdf") for i in range(3)]@@ -718,12 +757,15 @@ class TestSyncLibraryWithFilesystem:mocker.patch("local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely")+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)result = service.sync_library_with_filesystem()assert result["total_documents"] == 3assert result["files_found"] == 2assert result["files_missing"] == 1+ # Asked only for the document whose file is missing.+ remaining.assert_called_once()def test_settings_manager_resolved_once_not_per_document(self, mocker):"""Regression test for the #6095 follow-up (issue 1).@@ -749,6 +791,7 @@ class TestSyncLibraryWithFilesystem:d.id = f"doc-{i}"d.title = f"Doc {i}"d.original_url = f"https://example.com/doc{i}.pdf"+ d.file_path = f"pdfs/doc{i}.pdf"docs.append(d)trackers = [Mock(file_path=f"pdfs/doc{i}.pdf") for i in range(3)]diff --git a/tests/research_library/test_library_sync_contracts.py b/tests/research_library/test_library_sync_contracts.pynew file mode 100644index 0000000000..bfeae4c341--- /dev/null+++ b/tests/research_library/test_library_sync_contracts.py@@ -0,0 +1,4238 @@+"""Contracts for what "Sync library" does with downloads whose PDF is missing.++``LibraryService.sync_library_with_filesystem`` is there to deal with a+downloaded document whose PDF file was removed from disk, so that its URL+can be downloaded again. Before PDF and text documents were merged into+one ``Document`` row, the sync checked PDF records only and left text+documents alone. Deleting the merged row also deletes its text, its PDF+blob, its collection links and, after the commit, its chunks and vectors.++The contract pinned here: the sync deletes a download whose file is+missing only when nothing else of it is stored (no extracted text, no PDF+blob in the database, no file at the document's own path). The+``storage_mode`` label and the tracker's ``file_path`` cannot decide this+on their own, because production writers leave both stale:++ * R1: re-downloading an existing document in "none" mode, or in an+ unrecognised mode that ``PDFStorageManager`` runs as "none", clears+ the tracker path. The document keeps its stored PDF and its label.+ * R2: ``_record_failed_text_extraction`` creates its placeholder with+ the ``"database"`` column default. ``_save_text_with_db`` later+ completes it with text and never stores a PDF.+ * R3: rows already written with ``storage_mode="database"`` and no blob+ keep that shape; nothing repairs them. A content-hash duplicate+ download copies ``"database"`` into the shared tracker.+ * A Zotero import stores its PDF blob and the item's source URL but+ creates no tracker at all.+ * A URL has one tracker, which every document with that URL shares: the+ documents of two researches that cite the same paper, or a+ failed-download placeholder that "Extract Text to DB" completed next+ to the document holding the PDF. Its path names the file written+ last, so it describes a document only when it is the document's own+ path, or when it is ``"database"`` and the document is labelled as+ stored in the database, which can be several documents. The sync+ judges every other document by what it stores itself and leaves the+ tracker alone. A ``"database"`` tracker is reset only when no+ document under it still holds a PDF blob.++A download whose file was removed while its extracted text was not keeps+the text. If it still has its research resource, the sync makes it+text-only, resets its tracker and reports the file missing, and the+download manager's PDF downloads put the PDF back into the same document,+also when the PDF downloaded again has other bytes. A run downloads it+once, also when a second research's resource is linked to the document:+through the document's own resource, unless that one cannot succeed or a+restart left its queue entry PROCESSING, and the linked one can.+Choosing that resource records no denial of the egress policy and loads+no document's text. The Download Manager counts such a document as+available to download, in its totals and in its research's sources, and+so do the library's source routes, through which ``download_source``+puts its PDF back. Every document the sync reports this way is one+``document_awaits_pdf`` asks for, including a failed-download placeholder+that a PDF download completed. Text extraction leaves such a document+alone: it already has the text. Without its resource no download path+asks for the PDF, so the sync makes the document text-only without+reporting it, and a later download of the same PDF bytes for another+resource stores the PDF in it. A PDF removed with "Remove PDF to save+space" is not a missing file: the document is kept as it is, and its PDF+is not downloaded again. The sync's response names each missing file by+its stored path, relative to the library root, never by its absolute+path on the server, and counts each tracker it changes once in+``trackers_updated``, whichever document of the tracker's URL it reads+first.++A loaded ``Document.status`` is a ``DocumentStatus`` member, which never+equals its string value, so it is compared with the member: text+extraction reads the PDF file that a completed download stores before it+asks the source, and the document details page shows the document's+status.++The deletion the sync is for still happens. A filesystem-mode download+with no text whose file was removed is deleted and can be downloaded+again, and so can a database-mode download with no blob and no text,+which the sync reports as ``"database"``.++The harness follows ``test_download_service_contracts.py``: a real+on-disk SQLite file with foreign keys on, a real ``tmp_path`` library+store, the production ``DownloadService`` writers (built with ``__new__``+around a stub downloader, so no request is ever made), the production+sync, ``delete_blob_only``, download queueing, the bulk download route,+the Download Manager page, the library's source routes and the document+details page. Only the session, settings and password sources are+replaced, plus the retry-policy filter of the "Download All New" route+and the routes' ``DownloadService`` factory. The service's retry manager+has no attempts on record, except where a case uses the production one+over the test database, and its egress policy allows every URL, except+where a case refuses one, through a stand-in or, in one case, the+production policy. Rows are read back through a fresh session.+"""++import asyncio+import hashlib+import json+import re+import uuid+from collections.abc import Set as AbstractSet+from contextlib import contextmanager, nullcontext+from datetime import UTC, datetime, timedelta+from types import SimpleNamespace++import pytest+from loguru import logger+from sqlalchemy import create_engine, event+from sqlalchemy.orm import sessionmaker+from starlette.requests import Request++from local_deep_research.constants import (+ FILE_PATH_BLOB_DELETED,+ FILE_PATH_TEXT_ONLY,+)+from local_deep_research.database.models import Base+from local_deep_research.database.models.download_tracker import (+ DownloadAttempt,+ DownloadTracker,+)+from local_deep_research.database.models.library import (+ Collection,+ Document,+ DocumentBlob,+ DocumentChunk,+ DocumentCollection,+ DocumentStatus,+ DownloadQueue,+ EmbeddingProvider,+ SourceType,+)+from local_deep_research.database.models.research import (+ ResearchHistory,+ ResearchResource,+)+from local_deep_research.library.download_management import (+ FailureClassifier,+ ResourceDownloadStatus,+ ResourceFilter,+ ResourceStatusTracker,+ RetryDecision,+ RetryManager,+)+from local_deep_research.library.download_management import (+ models as retry_models,+)+from local_deep_research.research_library import utils as library_utils+from local_deep_research.research_library.deletion.services import (+ document_deletion,+)+from local_deep_research.research_library.downloaders.base import (+ ContentType,+ DownloadResult,+)+from local_deep_research.research_library.services import (+ download_service,+ library_service,+)+from local_deep_research.research_library.services.download_service import (+ DownloadService,+)+from local_deep_research.research_library.services.pdf_storage_manager import (+ PDFStorageManager,+ resolve_pdf_storage_mode,+)+from local_deep_research.research_library.utils import (+ document_awaits_pdf,+ find_tracker_by_url,+ get_url_hash,+)+from local_deep_research.security.egress.audit_hook import (+ clear_active_context,+)+from local_deep_research.security.egress.policy import (+ EgressContext,+ EgressScope,+)+from local_deep_research.utilities import db_utils+from local_deep_research.web.routers import library as library_router++USERNAME = "library_sync_user"+LIBRARY_COLLECTION_ID = "library-sync-contracts"+PDF_BYTES = (+ b"%PDF-1.4\n1 0 obj<</Type/Catalog>>endobj\ntrailer<</Root 1 0 R>>\n%%EOF\n"+)+PDF_HASH = hashlib.sha256(PDF_BYTES).hexdigest()+SAVED_TEXT = "Text that only this document holds."+EXTRACTED_TEXT = "Text extracted from the downloaded PDF."+++def _text_pdf(text):+ """A one-page PDF with a real text layer.++ ``PDF_BYTES`` has no pages, so the download's text extraction finds no+ text in it. This one goes through the real extraction and yields+ ``text``, the usual case for a downloaded paper.+ """+ objects = [+ b"<< /Type /Catalog /Pages 2 0 R >>",+ b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",+ b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 400 200] "+ b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",+ b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",+ ]+ stream = f"BT /F1 12 Tf 10 100 Td ({text}) Tj ET".encode("latin-1")+ objects.append(+ b"<< /Length "+ + str(len(stream)).encode()+ + b" >>\nstream\n"+ + stream+ + b"\nendstream"+ )+ out = bytearray(b"%PDF-1.4\n")+ offsets = []+ for number, body in enumerate(objects, start=1):+ offsets.append(len(out))+ out += f"{number} 0 obj\n".encode() + body + b"\nendobj\n"+ xref_offset = len(out)+ out += f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode()+ for offset in offsets:+ out += f"{offset:010d} 00000 n \n".encode()+ out += (+ f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"+ f"startxref\n{xref_offset}\n%%EOF\n"+ ).encode()+ return bytes(out)+++TEXT_PDF_BYTES = _text_pdf(EXTRACTED_TEXT)+# The same paper served again as a regenerated PDF: other bytes, other text.+REGENERATED_TEXT = "Text extracted from the regenerated PDF."+REGENERATED_PDF_BYTES = _text_pdf(REGENERATED_TEXT)+# Hard-coded for the same reason as in test_download_service_contracts.py:+# a renamed gate must fail the filesystem cases, not turn them into+# database-mode cases.+FILESYSTEM_GATE_ENV = "LDR_RESEARCH_LIBRARY_ALLOW_FILESYSTEM_PDF_STORAGE"+++class StubSettings:+ """Minimal stand-in for SettingsManager.get_setting."""++ def __init__(self, values):+ self._values = values++ def get_setting(self, key, default=None):+ return self._values.get(key, default)+++class StubDownloader:+ """Returns ``content`` for every URL and records the calls.++ A URL in ``unreachable`` fails the way a server that is down does+ (HTTP 503).+ """++ def __init__(self, content=PDF_BYTES, unreachable=()):+ self.content = content+ self.unreachable = set(unreachable)+ self.urls = []++ def can_handle(self, url):+ return True++ def download_with_result(self, url, content_type=ContentType.PDF):+ self.urls.append(url)+ if url in self.unreachable:+ return DownloadResult(+ skip_reason="Service unavailable", status_code=503+ )+ return DownloadResult(+ content=self.content, is_success=True, status_code=200+ )+++class NoRetryHistory:+ """A retry manager with no attempts on record: it holds nothing back."""++ status_tracker = SimpleNamespace(get_resource_status=lambda _id: None)++ def record_attempt(self, **kwargs):+ pass++ def should_retry_resource(self, resource_id):+ return RetryDecision(can_retry=True)+++def store(tmp_path):+ """The per-user library store the download and the sync both use."""+ base = tmp_path / "library"+ root = base / USERNAME+ (root / "pdfs").mkdir(parents=True)+ return SimpleNamespace(base=base, root=root)+++def _tree(root):+ """Every file under ``root``, as posix-relative strings."""+ return {+ p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file()+ }+++def db(tmp_path):+ """Real on-disk SQLite, FK enforcement on, plus a fresh-session reader."""+ db_dir = tmp_path / "db"+ db_dir.mkdir()+ engine = create_engine(f"sqlite:///{db_dir / 'library.db'}")++ @event.listens_for(engine, "connect")+ def _fk_on(dbapi_connection, _record):+ cur = dbapi_connection.cursor()+ cur.execute("PRAGMA foreign_keys = ON")+ cur.close()++ Base.metadata.create_all(engine)+ Session = sessionmaker(bind=engine)++ @contextmanager+ def fresh():+ """A brand-new session on the same file, never the mutated one."""+ with Session() as other:+ yield other++ with Session() as session:+ try:+ yield SimpleNamespace(engine=engine, session=session, fresh=fresh)+ finally:+ engine.dispose()+++def world(db, monkeypatch):+ """A research session, the Library collection and resource factory.++ The download service's own session, source-type and default-collection+ lookups are pointed at this database.+ """+ session = db.session+ source_type = SourceType(+ id=str(uuid.uuid4()),+ name="research_download",+ display_name="Research Download",+ )+ research = ResearchHistory(+ id=str(uuid.uuid4()),+ query="library sync contract",+ mode="quick",+ status="completed",+ created_at="2026-08-01T00:00:00",+ )+ session.add_all(+ [+ source_type,+ research,+ Collection(id=LIBRARY_COLLECTION_ID, name="Library"),+ ]+ )+ session.commit()+ source_type_id = source_type.id+ research_id = research.id++ monkeypatch.setattr(+ download_service, "get_source_type_id", lambda *a, **k: source_type_id+ )+ monkeypatch.setattr(+ download_service,+ "get_default_library_id",+ lambda *a, **k: LIBRARY_COLLECTION_ID,+ )+ monkeypatch.setattr(+ download_service, "get_user_db_session", lambda *a, **k: db.fresh()+ )++ def add_resource(url, *, tracker=True, downloaded=False):+ """A cited resource, with the URL's tracker unless one exists."""+ resource = ResearchResource(+ research_id=research_id,+ title="Sync Contract Paper",+ url=url,+ source_type="academic",+ created_at="2026-08-01T00:00:00",+ )+ session.add(resource)+ session.commit()+ if tracker:+ session.add(+ DownloadTracker(+ url=url,+ url_hash=get_url_hash(url),+ first_resource_id=resource.id,+ is_downloaded=downloaded,+ file_hash=PDF_HASH if downloaded else None,+ )+ )+ session.commit()+ return resource.id++ return SimpleNamespace(+ source_type_id=source_type_id,+ research_id=research_id,+ add_resource=add_resource,+ )+++def one_download_session(db, world, monkeypatch):+ """Give the download service one reused session, as production does.++ ``get_user_db_session`` hands every scope on a thread the same+ thread-local session and does not close it at scope exit. The+ download saves the extracted text in a later scope than the one that+ stored the PDF, through the resource instance that scope loaded. With+ the fresh session per scope that ``world`` sets up, that instance is+ detached by then and the text is never saved. The cases whose+ documents must hold the text of a real extraction use this.+ """+ session = sessionmaker(bind=db.engine)()+ monkeypatch.setattr(+ download_service,+ "get_user_db_session",+ lambda *a, **k: nullcontext(session),+ )+ try:+ yield session+ finally:+ session.close()+++def filesystem_mode(monkeypatch):+ """Open the operator gate for unencrypted filesystem PDF storage."""+ monkeypatch.setenv(FILESYSTEM_GATE_ENV, "true")+ assert resolve_pdf_storage_mode("filesystem") == "filesystem", (+ "the filesystem-storage operator gate did not open; the filesystem "+ "cases would silently run in database mode"+ )+++def retry_history(db):+ """The production retry manager, keeping its record in this database.++ ``download_resource`` records each attempt with it, and the refill+ ranking and the "Download All New" retry-policy filter read it back.+ Only its database source is replaced.+ """+ retry_models.Base.metadata.create_all(db.engine)+ tracker = ResourceStatusTracker.__new__(ResourceStatusTracker)+ tracker.username = USERNAME+ tracker.password = None+ tracker.engine = db.engine+ tracker.Session = sessionmaker(bind=db.engine)+ manager = RetryManager.__new__(RetryManager)+ manager.username = USERNAME+ manager.failure_classifier = FailureClassifier()+ manager.status_tracker = tracker+ return manager+++def make_service(+ store, downloader, *, storage_mode, retry_manager=None, refused_urls=()+):+ """A ``DownloadService`` reduced to what ``download_resource`` uses.++ ``retry_manager`` defaults to one without attempts on record. The+ egress policy refuses ``refused_urls`` and allows every other URL, at+ queueing and at the download alike.+ """+ service = DownloadService.__new__(DownloadService)+ service.username = USERNAME+ service.password = None+ service._closed = False+ service.settings = StubSettings(+ {+ "research_library.pdf_storage_mode": storage_mode,+ "research_library.max_pdf_size_mb": 3072,+ }+ )+ service.library_root = str(store.root)+ service.downloaders = [downloader]+ service.retry_manager = retry_manager or NoRetryHistory()+ service._egress_context = None+ if refused_urls:+ refused = set(refused_urls)+ service._check_url_against_policy = lambda url, record_denial=True: (+ (False, "refused_by_test_policy")+ if url in refused+ else (True, "no_context")+ )+ return service+++def make_library_service(db, store, monkeypatch):+ """A ``LibraryService`` whose real sync runs against this database.++ Only the session and settings sources are replaced. The keep/delete+ rule, tracker lookup, blob check, path resolution against ``store``+ and the post-commit purge are the production code.+ """+ monkeypatch.setattr(+ library_service, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ monkeypatch.setattr(+ document_deletion, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ settings = StubSettings({"research_library.storage_path": str(store.base)})+ monkeypatch.setattr(+ library_service,+ "get_settings_manager",+ lambda **_kwargs: SimpleNamespace(+ get_setting=settings.get_setting, close=lambda: None+ ),+ )+ service = library_service.LibraryService.__new__(+ library_service.LibraryService+ )+ service.username = USERNAME+ return service+++def _library_links(session, document_id):+ return (+ session.query(DocumentCollection)+ .filter_by(document_id=document_id)+ .count()+ )+++def _add_chunk(db, document_id):+ """A search-index chunk row of the document, as indexing writes one."""+ text = f"chunk of {document_id}"+ chunk = DocumentChunk(+ chunk_hash=hashlib.sha256(text.encode()).hexdigest(),+ source_type="document",+ source_id=document_id,+ collection_name=f"collection_{LIBRARY_COLLECTION_ID}",+ chunk_text=text,+ chunk_index=0,+ start_char=0,+ end_char=len(text),+ word_count=len(text.split()),+ embedding_id=str(uuid.uuid4()),+ embedding_model="fake-model",+ embedding_model_type=EmbeddingProvider.SENTENCE_TRANSFORMERS,+ embedding_dimension=8,+ )+ db.session.add(chunk)+ db.session.commit()+ return chunk.id+++def _cite_in_another_research(+ db, url, query="second research citing the paper"+):+ """A further research session citing ``url``, and its resource."""+ research = ResearchHistory(+ id=str(uuid.uuid4()),+ query=query,+ mode="quick",+ status="completed",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(research)+ db.session.commit()+ resource = ResearchResource(+ research_id=research.id,+ title="Sync Contract Paper",+ url=url,+ source_type="academic",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(resource)+ db.session.commit()+ return research.id, resource.id+++def _add_database_row_without_pdf(+ db, world, resource_id, research_id, url, content+):+ """An existing database-mode download that holds no PDF and no text.++ The R3 shape without text: a database-mode download whose PDF the size+ limit refused, and that had no text to extract, left a completed row+ labelled ``"database"``, with no ``file_path``, no blob and no text.+ Nothing repairs such rows. ``content`` is the refused PDF, whose hash+ the row keeps, so a later download of the same bytes is linked to it.+ """+ document_id = str(uuid.uuid4())+ db.session.add(+ Document(+ id=document_id,+ source_type_id=world.source_type_id,+ resource_id=resource_id,+ research_id=research_id,+ document_hash=hashlib.sha256(content).hexdigest(),+ original_url=url,+ file_size=len(content),+ file_type="pdf",+ mime_type="application/pdf",+ title="Sync Contract Paper",+ status=DocumentStatus.COMPLETED,+ storage_mode="database",+ file_path=None,+ )+ )+ db.session.add(+ DocumentCollection(+ document_id=document_id, collection_id=LIBRARY_COLLECTION_ID+ )+ )+ db.session.commit()+ return document_id+++class PassingResourceFilter:+ """The retry-policy filter, letting every resource through."""++ def __init__(self, *args, **kwargs):+ pass++ def filter_downloadable_resources(self, resources):+ return [+ SimpleNamespace(resource_id=row.id, can_retry=True, reason="ok")+ for row in resources+ ]++ def get_filter_summary(self, resources):+ return SimpleNamespace(+ to_dict=lambda: {},+ permanently_failed_count=0,+ temporarily_failed_count=0,+ )++ def get_skipped_resources_info(self, resources):+ return []+++def claim_queue_entry(db, resource_id, research_id):+ """Queue ``resource_id`` and claim it as a bulk download stream does.++ The claim is the production one (``_claim_download_queue_item``), so+ the entry is PROCESSING and carries the time the claim records.+ """+ entry = DownloadQueue(+ resource_id=resource_id,+ research_id=research_id,+ collection_id=LIBRARY_COLLECTION_ID,+ status=DocumentStatus.PENDING,+ )+ db.session.add(entry)+ db.session.commit()+ assert library_router._claim_download_queue_item(db.session, entry.id)+ with db.fresh() as other:+ claimed = other.get(DownloadQueue, entry.id)+ assert claimed.status is DocumentStatus.PROCESSING+ assert claimed.last_attempt_at is not None+ return entry.id+++class RetryPolicyFilter(ResourceFilter):+ """The production retry-policy filter, over the given retry manager."""++ def __init__(self, retry_manager):+ self.username = USERNAME+ self.retry_manager = retry_manager+++# The ranking-only DownloadService of "Download All New" stores nothing.+_NO_STORE = SimpleNamespace(root="unused-library-root", base="unused-base")+++def download_all_new(db, monkeypatch, *, retry_policy=None, **service_options):+ """Queue what the download manager's "Download All New" would.++ Runs the ``queue_all_undownloaded`` route against this database. Its+ session, the user's password and the retry-policy filter are replaced:+ the filter lets every resource through, or is the production filter+ over ``retry_policy``. So is the ``DownloadService`` the route ranks a+ document's resources with (``service_options`` as for+ ``make_service``). Which resources count as not downloaded, and which+ of a document's resources is queued, are the production code.+ """+ monkeypatch.setattr(+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ monkeypatch.setattr(+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"+ )+ monkeypatch.setattr(+ library_router,+ "ResourceFilter",+ PassingResourceFilter+ if retry_policy is None+ else (lambda *a, **k: RetryPolicyFilter(retry_policy)),+ )+ monkeypatch.setattr(+ library_router,+ "DownloadService",+ lambda *a, **k: make_service(+ _NO_STORE, StubDownloader(), storage_mode="none", **service_options+ ),+ )+ return library_router.queue_all_undownloaded(None, username=USERNAME)+++def bulk_download(+ db, store, downloader, monkeypatch, research_ids, *, mode, **service_options+):+ """Run the download manager's bulk route and return its SSE events.++ "Get All Research PDFs" posts ``mode="pdf"``, "Extract Text to DB"+ posts ``mode="text_only"``. The route's session, the user's password+ and its ``DownloadService`` (a filesystem-mode service around+ ``downloader``, with ``service_options`` as for ``make_service``) are+ replaced. Which resources it queues, and the claim and download of+ each, are the production code.+ """+ monkeypatch.setattr(+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ monkeypatch.setattr(+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"+ )+ monkeypatch.setattr(+ library_router,+ "DownloadService",+ lambda *a, **k: make_service(+ store, downloader, storage_mode="filesystem", **service_options+ ),+ )+ body = json.dumps(+ {+ "research_ids": research_ids,+ "mode": mode,+ "collection_id": LIBRARY_COLLECTION_ID,+ }+ ).encode()++ async def receive():+ return {"type": "http.request", "body": body, "more_body": False}++ request = Request(+ {+ "type": "http",+ "method": "POST",+ "path": "/library/api/download-bulk",+ "headers": [(b"content-type", b"application/json")],+ "query_string": b"",+ "session": {"session_id": "sid"},+ },+ receive,+ )+ response = asyncio.run(+ library_router.download_bulk(request, username=USERNAME)+ )++ async def drain():+ chunks = []+ async for chunk in response.body_iterator:+ chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())+ return b"".join(chunks).decode()++ return [+ json.loads(line[len("data: ") :])+ for line in asyncio.run(drain()).split("\n")+ if line.startswith("data: ")+ ]+++# ===========================================================================+# Documents that still hold content are kept+# ===========================================================================++[email protected]("redownload_mode", ["none", "Database"])[email protected]("stored_mode", ["database", "filesystem"])+def test_redownload_that_stores_no_pdf_keeps_the_stored_one(+ db, store, world, filesystem_mode, monkeypatch, stored_mode, redownload_mode+):+ """R1: the PDF stored earlier keeps the document through a sync.++ No text is extracted, so the stored PDF is the document's only+ content. After the re-download the tracker has no path while the+ document still names its store. The sync kept a path-less document+ only when its label was "none", so it deleted this one, and with it+ the blob (database) or every reference to the file (filesystem).+ """+ url = "https://example.com/stored-then-redownloaded.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader()+ service = make_service(store, downloader, storage_mode=stored_mode)+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document_id = (+ other.query(Document).filter_by(resource_id=resource_id).one().id+ )+ stored_files = _tree(store.root)+ assert len(stored_files) == (1 if stored_mode == "filesystem" else 0)++ library = make_library_service(db, store, monkeypatch)+ assert library.mark_for_redownload([document_id]) == 1+ service.settings._values["research_library.pdf_storage_mode"] = (+ redownload_mode+ )+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2+ with db.fresh() as other:+ # The R1 state: the tracker lost its path, the document kept its PDF.+ assert find_tracker_by_url(other, url).file_path is None+ document = other.get(Document, document_id)+ assert document.status is DocumentStatus.COMPLETED+ assert document.storage_mode == stored_mode+ assert document.text_content is None++ stats = library.sync_library_with_filesystem()++ assert stats["files_missing"] == 0+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert (+ PDFStorageManager(store.root, "none").load_pdf(document, other)+ == PDF_BYTES+ )+ assert other.query(DocumentBlob).count() == (+ 1 if stored_mode == "database" else 0+ )+ assert _library_links(other, document_id) == 1+ assert _tree(store.root) == stored_files+ assert library.get_document_by_id(document_id)["has_pdf"] is True+ # The kept document still counts as downloaded: no third request.+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2+++ "tracker", [True, False], ids=["tracker-without-path", "no-tracker"]+)+def test_failed_placeholder_completed_with_text_is_kept(+ db, store, world, monkeypatch, tracker+):+ """R2: text saved into a failed-download placeholder survives a sync.++ The placeholder keeps the ``"database"`` column default and never gets+ a blob, so before the fix the sync treated it as a stored PDF that had+ gone missing and deleted the text with it. ``download_as_text`` creates+ no tracker; a resource whose PDF download was attempted first has one+ without a path.+ """+ url = "https://example.com/placeholder-then-text.pdf"+ resource_id = world.add_resource(url, tracker=tracker)+ service = make_service(store, StubDownloader(), storage_mode="database")+ resource = db.session.get(ResearchResource, resource_id)+ service._record_failed_text_extraction(+ db.session, resource, error="PDF download failed: HTTP 503"+ )+ db.session.commit()+ service._save_text_with_db(+ resource,+ SAVED_TEXT,+ db.session,+ extraction_method="native_api",+ extraction_source="arxiv_api",+ )+ db.session.commit()+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ assert document.status is DocumentStatus.COMPLETED+ assert document.storage_mode == "database"+ assert document.file_path is None+ assert other.query(DocumentBlob).count() == 0++ library = make_library_service(db, store, monkeypatch)+ stats = library.sync_library_with_filesystem()++ assert stats["files_missing"] == 0+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.status is DocumentStatus.COMPLETED+ assert document.text_content == SAVED_TEXT+ # Kept as it was: the sync does not rewrite the label.+ assert document.storage_mode == "database"+ view = library.get_document_by_id(document_id)+ assert view["has_text_db"] is True+ assert view["has_pdf"] is False++[email protected]("shape", ["tracker-without-path", "duplicate-link"])+def test_database_row_without_blob_keeps_its_text(+ db, store, world, monkeypatch, shape+):+ """R3: an existing ``"database"`` row with no blob keeps its text.++ A PDF that the size limit refused in database mode left this row: a+ ``"database"`` label, no blob, the extracted text, and a downloaded+ tracker without a path. Nothing repairs such rows, so the sync must+ tolerate them. In ``duplicate-link`` a second research cites the same+ URL and downloads identical bytes. The content-hash dedup links it to+ this row and copies the row's ``"database"`` label into the shared+ tracker. A ``"database"`` tracker describes a row labelled as stored+ in the database, without a path, so the sync takes its tracker-path+ branch for this row instead, which keeps the text and the tracker+ too.+ """+ url = "https://example.com/refused-earlier.pdf"+ resource_id = world.add_resource(url, downloaded=True)+ document_id = str(uuid.uuid4())+ db.session.add(+ Document(+ id=document_id,+ source_type_id=world.source_type_id,+ resource_id=resource_id,+ research_id=world.research_id,+ document_hash=PDF_HASH,+ original_url=url,+ file_size=len(PDF_BYTES),+ file_type="pdf",+ mime_type="application/pdf",+ title="Sync Contract Paper",+ status=DocumentStatus.COMPLETED,+ storage_mode="database",+ file_path=None,+ text_content=SAVED_TEXT,+ )+ )+ db.session.add(+ DocumentCollection(+ document_id=document_id, collection_id=LIBRARY_COLLECTION_ID+ )+ )+ db.session.commit()+ expected_tracker_path = None+ if shape == "duplicate-link":+ duplicate_id = world.add_resource(url, tracker=False)+ service = make_service(store, StubDownloader(), storage_mode="database")+ monkeypatch.setattr(+ service, "_extract_text_from_pdf", lambda _data: None+ )+ assert service.download_resource(duplicate_id) == (True, None)+ expected_tracker_path = "database"+ with db.fresh() as other:+ linked = other.get(ResearchResource, duplicate_id)+ assert linked.document_id == document_id+ assert other.query(Document).count() == 1+ assert other.query(DocumentBlob).count() == 0+ with db.fresh() as other:+ assert find_tracker_by_url(other, url).file_path == (+ expected_tracker_path+ )++ library = make_library_service(db, store, monkeypatch)+ stats = library.sync_library_with_filesystem()++ assert stats["files_missing"] == 0+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.text_content == SAVED_TEXT+ assert document.storage_mode == "database"+ assert _library_links(other, document_id) == 1+ # The tracker is left as it was, still marked downloaded.+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is True+ assert tracker.file_path == expected_tracker_path+ view = library.get_document_by_id(document_id)+ assert view["has_text_db"] is True+ assert view["has_pdf"] is False+++def test_zotero_import_with_a_source_url_is_kept(db, store, monkeypatch):+ """A Zotero-imported PDF has a source URL but no download tracker.++ ``ZoteroSyncService`` stores the PDF blob through ``save_pdf`` in+ "database" mode and copies the item's URL (or DOI link) into+ ``original_url``, which puts the document in the sync's scope. With no+ tracker it took the no-path branch, which kept only "none"-mode and+ sentinel documents.+ """+ source_type = SourceType(+ id=str(uuid.uuid4()), name="zotero", display_name="Zotero"+ )+ db.session.add(source_type)+ db.session.flush()+ document = Document(+ id=str(uuid.uuid4()),+ source_type_id=source_type.id,+ document_hash=PDF_HASH,+ original_url="https://doi.org/10.1234/zotero-item",+ filename="Zotero Item.pdf",+ file_size=len(PDF_BYTES),+ file_type="pdf",+ mime_type="application/pdf",+ title="Zotero Item",+ status=DocumentStatus.COMPLETED,+ text_content=SAVED_TEXT,+ extraction_source="zotero",+ )+ db.session.add(document)+ db.session.flush()+ PDFStorageManager(store.root, "database").save_pdf(+ pdf_content=PDF_BYTES,+ document=document,+ session=db.session,+ filename="Zotero Item.pdf",+ )+ db.session.commit()+ document_id = document.id++ stats = make_library_service(+ db, store, monkeypatch+ ).sync_library_with_filesystem()++ assert stats["files_missing"] == 0+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.storage_mode == "database"+ assert document.text_content == SAVED_TEXT+ assert (+ PDFStorageManager(store.root, "none").load_pdf(document, other)+ == PDF_BYTES+ )+++# ===========================================================================+# A removed PDF file whose text is left: keep the text, re-download the PDF+# ===========================================================================++[email protected]("via", ["research", "download-all-new"])[email protected]("tracker_path", ["kept", "lost"])+def test_removed_file_of_a_download_with_text_is_reported_and_downloaded_again(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ tracker_path,+ via,+):+ """The text stays, the missing PDF is reported and downloaded again.++ The usual filesystem-mode case: the download's real text extraction+ saved the PDF's text, then the user removed the PDF file from disk.+ The sync keeps the document, its text, its collection link and its+ chunks. It stores the document as text-only and resets the tracker, so+ the library no longer offers a PDF that is not there, and it reports+ the file missing. The download manager's research download and its+ "Download All New" both queue the document, and the download puts the+ PDF back into the same document. ``lost`` is the R1 sequence (a+ "none"-mode re-download cleared the tracker path) with the file removed+ afterwards, which reaches the sync's no-path branch.+ """+ url = "https://example.com/removed-but-extracted.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ own_path = document.file_path+ assert document.text_content == EXTRACTED_TEXT+ stored = store.root / own_path+ assert stored.is_file()+ library = make_library_service(db, store, monkeypatch)+ if tracker_path == "lost":+ assert library.mark_for_redownload([document_id]) == 1+ service.settings._values["research_library.pdf_storage_mode"] = "none"+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ assert find_tracker_by_url(other, url).file_path is None+ assert other.get(Document, document_id).storage_mode == (+ "filesystem"+ )+ chunk_id = _add_chunk(db, document_id)++ # While the file is on disk, text or no text, nothing is missing.+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.file_path == own_path+ assert library.get_document_by_id(document_id)["has_pdf"] is True++ stored.unlink()++ stats = library.sync_library_with_filesystem()++ assert stats["files_found"] == 0+ assert stats["files_missing"] == 1+ assert [entry["id"] for entry in stats["missing_files"]] == [document_id]+ assert stats["trackers_updated"] == 1+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.status is DocumentStatus.COMPLETED+ assert document.text_content == EXTRACTED_TEXT+ # Stored the way text-only documents are.+ assert document.storage_mode == "none"+ assert document.file_path == FILE_PATH_TEXT_ONLY+ assert _library_links(other, document_id) == 1+ assert other.get(DocumentChunk, chunk_id) is not None+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is False+ assert tracker.file_path is None+ view = library.get_document_by_id(document_id)+ assert view["has_pdf"] is False+ assert view["has_text_db"] is True++ # The download manager offers the document again...+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"+ if via == "research":+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ else:+ queued = download_all_new(db, monkeypatch)+ assert queued["queued"] == 1+ assert queued["research_ids"] == [world.research_id]+ # ...and its download stores the PDF in the same document.+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == (2 if tracker_path == "kept" else 3)+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.status is DocumentStatus.COMPLETED+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ assert other.get(DocumentChunk, chunk_id) is not None+ restored = store.root / document.file_path+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is True+ assert tracker.file_path == document.file_path+ assert restored.read_bytes() == TEXT_PDF_BYTES+ assert library.get_document_by_id(document_id)["has_pdf"] is True+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+++def test_pdf_still_stored_in_the_database_keeps_the_document_unchanged(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """A document whose PDF is still a blob is not missing its PDF.++ Downloaded in "database" mode and then again in "filesystem" mode,+ the document keeps its blob next to the new file, because the+ filesystem save does not remove it. When that file is removed the PDF+ is still stored, so the sync keeps the document, its text and its+ tracker unchanged. Making it text-only would hide the blob.+ """+ url = "https://example.com/blob-and-file.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="database")+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document_id = (+ other.query(Document).filter_by(resource_id=resource_id).one().id+ )+ library = make_library_service(db, store, monkeypatch)+ assert library.mark_for_redownload([document_id]) == 1+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ assert other.query(DocumentBlob).count() == 1+ own_path = document.file_path+ assert find_tracker_by_url(other, url).file_path == own_path+ (store.root / own_path).unlink()++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.file_path == own_path+ assert document.text_content == EXTRACTED_TEXT+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is True+ assert tracker.file_path == own_path+ assert (+ PDFStorageManager(store.root, "none").load_pdf(document, other)+ == TEXT_PDF_BYTES+ )+++def test_resource_linked_by_content_in_another_research_is_queued_again(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """A second research's resource linked to the document queues it.++ A second research cites the same paper. Its download stores no+ duplicate: the content-hash dedup links its resource to the existing+ document through ``document_id``. Once the sync has made that document+ text-only, the second research's own "Download PDFs" queues the linked+ resource, and its download stores the PDF in the shared document.+ """+ url = "https://example.com/cited-twice.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ second_research = ResearchHistory(+ id=str(uuid.uuid4()),+ query="second research citing the paper",+ mode="quick",+ status="completed",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(second_research)+ db.session.commit()+ second_research_id = second_research.id+ linked = ResearchResource(+ research_id=second_research_id,+ title="Sync Contract Paper",+ url=url,+ source_type="academic",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(linked)+ db.session.commit()+ linked_id = linked.id+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ assert (+ service.queue_research_downloads(+ second_research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ assert service.download_resource(linked_id) == (True, None)+ assert len(downloader.urls) == 3+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = store.root / document.file_path+ assert restored.read_bytes() == TEXT_PDF_BYTES+++def test_redownload_that_stores_no_pdf_stops_asking_for_it(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """A download in "none" mode settles a document that needed its PDF.++ The storage mode keeps no PDF, so the download can only refresh the+ text. The document must then count as downloaded, like any "none"-mode+ download. If it still asked for its PDF, every "Download All New"+ would fetch it again.+ """+ url = "https://example.com/removed-then-none-mode.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ (store.root / document.file_path).unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ service.settings._values["research_library.pdf_storage_mode"] = "none"+ assert download_all_new(db, monkeypatch)["queued"] == 1+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2++ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "none"+ assert document.file_path is None+ assert document.text_content == EXTRACTED_TEXT+ assert find_tracker_by_url(other, url).is_downloaded is True+ assert _tree(store.root) == set()+ assert download_all_new(db, monkeypatch)["queued"] == 0+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 0+ )+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2++[email protected]("label", ["written-now", "written-before-the-fix"])+def test_failed_placeholder_completed_by_a_pdf_download_is_downloaded_again(+ db, store, world, one_download_session, filesystem_mode, monkeypatch, label+):+ """A placeholder that a PDF download completed gets its PDF back.++ A failed download leaves a placeholder labelled ``file_type``+ "unknown", and a later PDF download completes that placeholder with the+ PDF and its text. The download kept the "unknown" label. Once the PDF+ file was removed, the sync made the document text-only and reported+ the file missing, but no download path asked for its PDF again: they+ ask only for documents labelled "pdf". Such a download now labels the+ document "pdf" (``written-now``). A row a download completed before+ that (``written-before-the-fix``) is labelled by the sync when it makes+ the document text-only.+ """+ url = "https://example.com/placeholder-then-pdf.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ resource = db.session.get(ResearchResource, resource_id)+ service._record_failed_text_extraction(+ db.session, resource, error="PDF download failed: HTTP 503"+ )+ db.session.commit()+ with db.fresh() as other:+ placeholder = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = placeholder.id+ assert placeholder.status is DocumentStatus.FAILED+ assert placeholder.file_type == "unknown"++ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.status is DocumentStatus.COMPLETED+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ stored = store.root / document.file_path+ assert stored.is_file()+ library = make_library_service(db, store, monkeypatch)+ if label == "written-now":+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert (document.file_type, document.mime_type) == (+ "pdf",+ "application/pdf",+ )+ assert library.get_document_by_id(document_id)["is_pdf"] is True+ else:+ # The labels such a download left before it labelled the document.+ db.session.query(Document).filter_by(id=document_id).update(+ {Document.file_type: "unknown", Document.mime_type: None},+ synchronize_session=False,+ )+ db.session.commit()+ chunk_id = _add_chunk(db, document_id)+ stored.unlink()++ stats = library.sync_library_with_filesystem()++ assert [entry["id"] for entry in stats["missing_files"]] == [document_id]+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.text_content == EXTRACTED_TEXT+ assert document.storage_mode == "none"+ assert document.file_path == FILE_PATH_TEXT_ONLY+ assert document.file_type == "pdf"+ # The sync reports only documents the download paths ask for.+ assert document_awaits_pdf(document)++ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ assert download_all_new(db, monkeypatch)["queued"] == 1+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 2+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ assert other.get(DocumentChunk, chunk_id) is not None+ restored = store.root / document.file_path+ assert restored.read_bytes() == TEXT_PDF_BYTES+ assert download_all_new(db, monkeypatch)["queued"] == 0+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+++def test_regenerated_pdf_goes_back_into_the_document_that_awaits_it(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """A PDF downloaded again with other bytes refills the same document.++ Publishers and arXiv regenerate PDFs, so the PDF downloaded for a+ document that the sync made text-only can differ from the removed one.+ It still belongs in that document: the document stores no other PDF,+ and its text, chunks and collection links are the paper's. A second+ document for the resource would leave this one waiting for its PDF+ for good, with two completed documents for one resource. The document+ takes the new PDF's hash, as a new download would, when no other+ document owns those bytes. A collision must preserve both documents;+ the following test covers that case and a later content dedup hit.+ """+ url = "https://example.com/regenerated.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ original_hash = hashlib.sha256(TEXT_PDF_BYTES).hexdigest()+ regenerated_hash = hashlib.sha256(REGENERATED_PDF_BYTES).hexdigest()+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ assert document.document_hash == original_hash+ stored = store.root / document.file_path+ chunk_id = _add_chunk(db, document_id)+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ downloader.content = REGENERATED_PDF_BYTES+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ assert service.download_resource(resource_id) == (True, None)++ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.status is DocumentStatus.COMPLETED+ assert document.storage_mode == "filesystem"+ assert document.file_size == len(REGENERATED_PDF_BYTES)+ assert document.text_content == REGENERATED_TEXT+ assert document.document_hash == regenerated_hash+ assert other.get(DocumentChunk, chunk_id) is not None+ assert _library_links(other, document_id) == 1+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is True+ assert tracker.file_path == document.file_path+ restored = store.root / document.file_path+ assert restored.read_bytes() == REGENERATED_PDF_BYTES+ assert download_all_new(db, monkeypatch)["queued"] == 0+ stats = library.sync_library_with_filesystem()+ assert stats["files_missing"] == 0++[email protected]("holder_storage", ["filesystem", "database"])+def test_colliding_refill_preserves_content_identity_for_later_downloads(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+ holder_storage,+):+ """Never put another document's bytes behind the missing PDF's hash.++ A failed refill must retain both documents and report failure. A later+ URL returning the original PDF must get those bytes, not the colliding+ PDF incorrectly stored under their hash. Use real downloads, sync,+ database rows, text extraction and PDF storage for the whole sequence.+ """+ url = "https://example.com/original.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ original_path = store.root / document.file_path+ chunk_id = _add_chunk(db, document_id)+ original_path.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ holder_url = "https://example.com/regenerated-elsewhere.pdf"+ holder_resource_id = world.add_resource(holder_url)+ downloader.content = REGENERATED_PDF_BYTES+ holder_service = make_service(+ store, downloader, storage_mode=holder_storage+ )+ assert holder_service.download_resource(holder_resource_id) == (True, None)+ with db.fresh() as other:+ holder = (+ other.query(Document)+ .filter_by(resource_id=holder_resource_id)+ .one()+ )+ holder_id = holder.id+ holder_path = holder.file_path+ tracker = find_tracker_by_url(other, url)+ tracker_before = (+ tracker.file_hash,+ tracker.file_path,+ tracker.file_size,+ tracker.downloaded_at,+ )+ holder_chunk_id = _add_chunk(db, holder_id)+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ expected_error = "PDF refill conflicts with another library document"+ assert service.download_resource(resource_id) == (False, expected_error)++ with db.fresh() as other:+ assert other.query(Document).count() == 2+ document = other.get(Document, document_id)+ assert document_awaits_pdf(document)+ assert (+ document.document_hash == hashlib.sha256(TEXT_PDF_BYTES).hexdigest()+ )+ assert document.text_content == EXTRACTED_TEXT+ assert document.file_size == len(TEXT_PDF_BYTES)+ assert not original_path.exists()+ # PDF text persistence now pins the resource to the exact stored+ # document. Refusing a collision must preserve that explicit link.+ assert (+ other.get(ResearchResource, resource_id).document_id == document_id+ )+ assert other.get(DocumentChunk, chunk_id) is not None+ assert _library_links(other, document_id) == 1+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is False+ assert (+ tracker.file_hash,+ tracker.file_path,+ tracker.file_size,+ tracker.downloaded_at,+ ) == tracker_before+ attempts = (+ other.query(DownloadAttempt)+ .filter_by(url_hash=tracker.url_hash)+ .order_by(DownloadAttempt.id)+ .all()+ )+ assert [attempt.succeeded for attempt in attempts] == [True, False]+ assert attempts[-1].error_message == expected_error+ assert (+ other.query(DownloadQueue)+ .filter_by(resource_id=resource_id)+ .one()+ .status+ is DocumentStatus.FAILED+ )+ assert (+ other.query(ResourceDownloadStatus)+ .filter_by(resource_id=resource_id)+ .one()+ .status+ == "temporarily_failed"+ )++ holder = other.get(Document, holder_id)+ assert (+ holder.document_hash+ == hashlib.sha256(REGENERATED_PDF_BYTES).hexdigest()+ )+ assert holder.text_content == REGENERATED_TEXT+ assert holder.file_path == holder_path+ assert holder.storage_mode == holder_storage+ assert (+ PDFStorageManager(+ store.root, holder_storage, username=USERNAME+ ).load_pdf(holder, other)+ == REGENERATED_PDF_BYTES+ )+ assert other.get(DocumentChunk, holder_chunk_id) is not None+ assert _library_links(other, holder_id) == 1+ assert find_tracker_by_url(other, holder_url).is_downloaded is True++ mirror_resource_id = world.add_resource(+ "https://example.com/original-mirror.pdf"+ )+ downloader.content = TEXT_PDF_BYTES+ assert service.download_resource(mirror_resource_id) == (True, None)+ with db.fresh() as other:+ mirror = other.get(ResearchResource, mirror_resource_id)+ assert mirror.document_id == document_id+ document = other.get(Document, mirror.document_id)+ stored = PDFStorageManager(+ store.root, "filesystem", username=USERNAME+ ).load_pdf(document, other)+ assert stored == TEXT_PDF_BYTES+ assert hashlib.sha256(stored).hexdigest() == document.document_hash+ assert document.text_content == EXTRACTED_TEXT+ assert other.get(DocumentChunk, chunk_id) is not None+ assert other.query(Document).count() == 2+++def test_text_extraction_leaves_a_document_that_awaits_its_pdf_alone(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """Text extraction does not queue a document that has its text.++ The sync made the document text-only after its PDF file was removed,+ and the PDF downloads ask for it again. A text extraction has nothing+ to add: the document holds its text, and the extraction never fetches+ the PDF that would settle it. Queued anyway, the document was reported+ as processed on every run, and a second research's resource linked to+ it went to the network-bound text extraction each time. The text mode+ skips both, as it did before the sync kept such documents, and the PDF+ mode still puts the PDF back.+ """+ url = "https://example.com/extract-text-after-removal.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ second_research = ResearchHistory(+ id=str(uuid.uuid4()),+ query="second research citing the paper",+ mode="quick",+ status="completed",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(second_research)+ db.session.commit()+ second_research_id = second_research.id+ linked = ResearchResource(+ research_id=second_research_id,+ title="Sync Contract Paper",+ url=url,+ source_type="academic",+ created_at="2026-08-02T00:00:00",+ )+ db.session.add(linked)+ db.session.commit()+ linked_id = linked.id+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ requests = len(downloader.urls)+ both = [world.research_id, second_research_id]++ for _ in range(2):+ events = bulk_download(+ db, store, downloader, monkeypatch, both, mode="text_only"+ )+ assert events[0]["total"] == 0, events+ assert events[-1]["complete"] is True+ assert events[-1]["error"] == (+ "No new papers to download for the selected research session(s)."+ )+ assert len(downloader.urls) == requests+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document_awaits_pdf(document)+ assert document.text_content == EXTRACTED_TEXT+ assert other.query(DownloadQueue).count() == 0++ events = bulk_download(+ db, store, downloader, monkeypatch, [world.research_id], mode="pdf"+ )+ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert events[-1]["complete"] is True+ assert "error" not in events[-1]+ assert len(downloader.urls) == requests + 1+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = store.root / document.file_path+ assert restored.read_bytes() == TEXT_PDF_BYTES+++ "via",+ [+ "owner-research-first",+ "linked-research-first",+ "download-all-new",+ "linked-queued-earlier",+ "linked-queued-earlier-run-over-both",+ "download-all-new-linked-queued-earlier",+ "owner-not-downloadable",+ ],+)+def test_paper_cited_by_two_researches_is_downloaded_once(+ db, store, world, one_download_session, filesystem_mode, monkeypatch, via+):+ """One download puts back the PDF of a document two researches share.++ A second research cites the same paper. Its download stored no+ duplicate: the content-hash dedup linked its resource to the first+ research's document (``document_id``). Once the sync has made that+ document text-only, either resource refills it. A bulk run over both+ researches, or "Download All New" and the bulk run it starts, queued+ both. The first download put the PDF back. The second fetched it+ again, saved it under its own resource's file name and left the first+ file on disk with nothing pointing to it.++ One resource per such document is queued now, the document's own one+ whichever research the bulk run queues first, and the other resource+ counts as downloaded once the PDF is back. A linked resource that an+ earlier run left queued does not keep a run over the first research+ from queueing the document's own resource+ (``linked-queued-earlier``), and a run over both researches downloads+ that queued resource rather than queue a second one+ (``linked-queued-earlier-run-over-both``), as "Download All New" does+ (``download-all-new-linked-queued-earlier``). When the document's own+ resource cannot be queued (``owner-not-downloadable``), the linked one+ refills it.+ """+ url = "https://example.com/cited-by-two-researches.pdf"+ owner_url = (+ "https://example.com/landing/cited-by-two-researches"+ if via == "owner-not-downloadable"+ else url+ )+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert document.resource_id == owner_id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ requests = len(downloader.urls)+ both = [world.research_id, second_research_id]++ refilled_by = owner_id+ if via == "download-all-new":+ queued = download_all_new(db, monkeypatch)+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [world.research_id]+ research_ids = queued["research_ids"]+ elif via == "owner-research-first":+ research_ids = both+ elif via == "linked-research-first":+ research_ids = both[::-1]+ elif "linked-queued-earlier" in via:+ # A run over the second research queued its resource and stopped+ # before downloading it.+ assert (+ service.queue_research_downloads(+ second_research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ if via == "linked-queued-earlier":+ research_ids = [world.research_id]+ elif via == "linked-queued-earlier-run-over-both":+ research_ids = both+ refilled_by = linked_id+ else:+ queued = download_all_new(db, monkeypatch)+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [second_research_id]+ research_ids = queued["research_ids"]+ refilled_by = linked_id+ else:+ research_ids = both[::-1]+ refilled_by = linked_id+ events = bulk_download(+ db, store, downloader, monkeypatch, research_ids, mode="pdf"+ )++ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert len(downloader.urls) == requests + 1+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ downloaded = {+ entry.resource_id+ for entry in other.query(DownloadQueue).filter_by(+ status=DocumentStatus.COMPLETED+ )+ }+ assert downloaded == {refilled_by}+ # The one file in the library is the one the document names.+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES++ # The other resource counts as downloaded: no request, no second file,+ # also when a run downloads the entry an earlier run left queued.+ events = bulk_download(db, store, downloader, monkeypatch, both, mode="pdf")+ left_queued = via == "linked-queued-earlier"+ assert events[0]["total"] == (1 if left_queued else 0), events+ other_id = linked_id if refilled_by == owner_id else owner_id+ assert service.download_resource(other_id) == (True, None)+ assert len(downloader.urls) == requests + 1+ assert _tree(store.root) == {restored}+ assert download_all_new(db, monkeypatch)["queued"] == 0+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+++ "own_link",+ [False, True],+ ids=["own-resource-unlinked", "own-resource-linked-to-its-document"],+)+def test_research_citing_a_paper_twice_queues_the_documents_own_resource(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ own_link,+):+ """Within one research, the document's own resource is queued.++ A research cites the paper under two URLs, and the second download was+ linked to the first one's document by content. The linked resource+ comes first in the research, so the queueing met it before the+ document's own resource, queued it, and skipped the own one. The+ document's own resource is considered first now, also by a "Download+ PDFs" that only queues (no run downloads the queue right away), and+ also when it points to its own document through ``document_id``+ (``own-resource-linked-to-its-document``): the order keys on the+ document's owner, not on ``document_id``.+ """+ # Created first, downloaded second: linked to the other's document.+ linked_id = world.add_resource("https://example.com/twice/abstract.pdf")+ owner_id = world.add_resource("https://example.com/twice/paper.pdf")+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner_id) == (True, None)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ owner_document_id = document.id+ assert document.resource_id == owner_id+ assert other.get(ResearchResource, linked_id).document_id == (+ document.id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ if own_link:+ owner = db.session.get(ResearchResource, owner_id)+ owner.document_id = owner_document_id+ db.session.commit()++ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ with db.fresh() as other:+ queued = [entry.resource_id for entry in other.query(DownloadQueue)]+ assert queued == [owner_id]+++def test_research_citing_a_paper_twice_queues_the_linked_one_if_the_own_failed(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+):+ """Within one research, a failing own resource leaves the PDF to the other.++ As in the case above, a research cites the paper under two URLs, and+ the second download was linked to the first one's document by content.+ Then the first URL stopped working, and a refill through it failed.+ "Download PDFs" for the research, which only queues, queued the failing+ resource again and skipped the linked one, which would have worked. It+ queues the linked resource now, and only that one.+ """+ linked_id = world.add_resource("https://example.com/twice/abstract.pdf")+ owner_url = "https://example.com/twice/paper.pdf"+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ assert service.download_resource(owner_id) == (True, None)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ assert document.resource_id == owner_id+ assert other.get(ResearchResource, linked_id).document_id == (+ document.id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ downloader.unreachable.add(owner_url)+ assert service.download_resource(owner_id) == (False, "Service unavailable")++ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ with db.fresh() as other:+ queued = [entry.resource_id for entry in other.query(DownloadQueue)]+ assert queued == [linked_id]+++ "via",+ ["owner-research-first", "linked-research-first", "download-all-new"],+)[email protected]("owner", ["unreachable", "refused-by-egress-policy"])+def test_owner_that_cannot_succeed_leaves_the_document_to_a_linked_resource(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+ owner,+ via,+):+ """A linked resource refills the document when its own one cannot.++ A second research cites the paper under another URL, and its download+ was linked to the document by content. After the sync made the+ document text-only, the URL of the document's own resource stopped+ working (``unreachable``: a refill through it failed, and the retry+ manager's cooldown has passed since), or the egress policy now refuses+ it (``refused-by-egress-policy``). The document's one download still+ went to its own resource. A bulk run over both researches queued that+ resource again on every run and skipped the linked one, which would+ have worked. "Download All New" kept the own resource without asking+ the egress policy, so the bulk run it started fetched nothing. A+ resource that cannot succeed no longer takes the download from a+ viable one: the linked resource refills the document, in either order+ of the researches and through "Download All New".+ """+ owner_url = "https://example.com/owner-copy/paper.pdf"+ linked_url = "https://mirror.example.org/paper.pdf"+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert document.resource_id == owner_id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ both = [world.research_id, second_research_id]+ options = {"retry_manager": retry_history}+ if owner == "refused-by-egress-policy":+ options["refused_urls"] = {owner_url}+ else:+ downloader.unreachable.add(owner_url)+ # A refill through the document's own resource fails...+ events = bulk_download(+ db,+ store,+ downloader,+ monkeypatch,+ [world.research_id],+ mode="pdf",+ **options,+ )+ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["skipped"]+ # ...and the retry manager's cooldown passes.+ with db.fresh() as other:+ status = (+ other.query(ResourceDownloadStatus)+ .filter_by(resource_id=owner_id)+ .one()+ )+ assert status.status == "temporarily_failed"+ status.retry_after_timestamp = datetime.now(UTC) - timedelta(+ minutes=1+ )+ other.commit()+ assert retry_history.should_retry_resource(owner_id).can_retry+ requests = len(downloader.urls)++ if via == "download-all-new":+ queued = download_all_new(+ db, monkeypatch, retry_policy=retry_history, **options+ )+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [second_research_id]+ research_ids = queued["research_ids"]+ elif via == "owner-research-first":+ research_ids = both+ else:+ research_ids = both[::-1]+ events = bulk_download(+ db, store, downloader, monkeypatch, research_ids, mode="pdf", **options+ )++ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert downloader.urls[requests:] == [linked_url]+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ downloaded = {+ entry.resource_id+ for entry in other.query(DownloadQueue).filter_by(+ status=DocumentStatus.COMPLETED+ )+ }+ assert downloaded == {linked_id}+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES++ # The document holds its PDF again, so nothing is queued or fetched.+ events = bulk_download(+ db, store, downloader, monkeypatch, both, mode="pdf", **options+ )+ assert events[0]["total"] == 0, events+ queued = download_all_new(+ db, monkeypatch, retry_policy=retry_history, **options+ )+ assert queued["queued"] == 0+ assert downloader.urls[requests:] == [linked_url]+++ "via", ["owner-research-first", "linked-research-first"]+)+def test_owner_the_retry_manager_holds_back_leaves_the_document_to_a_linked_one(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+ via,+):+ """A linked resource refills the document while its own one is held back.++ The URL of the document's own resource failed three times today before+ a download through it succeeded. The retry manager records the+ success, so the latest attempt did not fail, but it still counts+ today's three attempts, its daily limit, and holds the resource back+ until tomorrow. A second research cites the paper under another URL,+ and its download was linked to the document by content. After the+ sync made the document text-only, a bulk run over both researches+ downloads the PDF through the linked resource, in either order of the+ researches, as it does when the own resource's latest attempt failed.+ """+ owner_url = "https://example.com/owner-copy/held-back.pdf"+ linked_url = "https://mirror.example.org/held-back.pdf"+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES, unreachable={owner_url})+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ for _ in range(3):+ assert service.download_resource(owner_id) == (+ False,+ "Service unavailable",+ )+ downloader.unreachable.clear()+ assert service.download_resource(owner_id) == (True, None)+ assert (+ retry_history.status_tracker.get_resource_status(owner_id)["status"]+ == "completed"+ )+ assert not retry_history.should_retry_resource(owner_id).can_retry+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert document.resource_id == owner_id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ requests = len(downloader.urls)+ both = [world.research_id, second_research_id]++ events = bulk_download(+ db,+ store,+ downloader,+ monkeypatch,+ both if via == "owner-research-first" else both[::-1],+ mode="pdf",+ retry_manager=retry_history,+ )++ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert downloader.urls[requests:] == [linked_url]+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ downloaded = {+ entry.resource_id+ for entry in other.query(DownloadQueue).filter_by(+ status=DocumentStatus.COMPLETED+ )+ }+ assert downloaded == {linked_id}+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES+++@contextmanager+def logged_records():+ """The loguru records emitted inside the block, at every level.++ The package disables its loguru logging (``local_deep_research``'s+ ``__init__``), so it is enabled for the block and disabled again after.+ """+ records = []+ logger.enable("local_deep_research")+ sink = logger.add(lambda message: records.append(message.record))+ try:+ yield records+ finally:+ logger.remove(sink)+ logger.disable("local_deep_research")+++def test_ranking_records_no_denial_of_the_egress_policy():+ """Ranking a URL the egress policy refuses records no denied fetch.++ ``refill_rank`` asks the egress policy whether a resource's URL could+ be queued, only to rank the resources that could refill a document.+ Each URL the policy refused was recorded as a denied fetch although+ nothing was fetched: a ``policy_audit`` warning from the policy, one+ from ``DownloadService``, and a count toward the run's denied-fetch+ quota, which refuses every URL once it is used up. The rank still+ treats the URL as one that cannot be queued, and records nothing. The+ check before a fetch records the denial as before.+ """+ service = make_service(_NO_STORE, StubDownloader(), storage_mode="none")+ context = EgressContext(+ scope=EgressScope.PRIVATE_ONLY,+ primary_engine="library",+ require_local_llm=False,+ require_local_embeddings=False,+ )+ service._egress_context = context+ # A public IP literal: classified without a DNS lookup, and refused+ # under PRIVATE_ONLY.+ refused_url = "https://8.8.8.8/papers/refused.pdf"+ warning = logger.level("WARNING").no++ def recorded(records):+ return [+ record["message"]+ for record in records+ if record["level"].no >= warning+ or record["extra"].get("policy_audit")+ ]++ # No armed run context, whose counter would take the denial instead.+ clear_active_context()+ with logged_records() as records:+ rank = service.refill_rank(1, refused_url, 1)+ assert rank.cannot_queue+ assert recorded(records) == []+ assert context._fetch_denial_count["count"] == 0++ with logged_records() as records:+ assert service._check_url_against_policy(refused_url) == (+ False,+ "scope_mismatch_private_only",+ )+ assert len(recorded(records)) == 2, recorded(records)+ assert context._fetch_denial_count["count"] == 1++[email protected]("owner", ["unreachable", "refused-by-egress-policy"])+def test_queued_owner_that_cannot_succeed_does_not_hold_the_document(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+ owner,+):+ """A queued resource that cannot succeed does not hold the document.++ The document's own resource is already queued: "Download PDFs" for its+ research queued it again after a failed refill (``unreachable``), or it+ was queued before the egress policy came to refuse its URL+ (``refused-by-egress-policy``). A bulk run over both researches+ downloads that entry, and it skipped the second research's linked+ resource, so the document stayed without its PDF. The run queues the+ linked resource too now, and that one refills the document.+ """+ owner_url = "https://example.com/owner-copy/queued-paper.pdf"+ linked_url = "https://mirror.example.org/queued-paper.pdf"+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ options = {"retry_manager": retry_history}+ if owner == "unreachable":+ downloader.unreachable.add(owner_url)+ assert service.download_resource(owner_id) == (+ False,+ "Service unavailable",+ )+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ failed_requests = [owner_url]+ else:+ db.session.add(+ DownloadQueue(+ resource_id=owner_id,+ research_id=world.research_id,+ collection_id=LIBRARY_COLLECTION_ID,+ status=DocumentStatus.PENDING,+ )+ )+ db.session.commit()+ options["refused_urls"] = {owner_url}+ failed_requests = []+ requests = len(downloader.urls)++ events = bulk_download(+ db,+ store,+ downloader,+ monkeypatch,+ [world.research_id, second_research_id],+ mode="pdf",+ **options,+ )++ assert events[0]["total"] == 2, events+ assert [event.get("status") for event in events[1:-1]] == [+ "skipped",+ "success",+ ]+ assert downloader.urls[requests:] == [*failed_requests, linked_url]+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ entries = {+ entry.resource_id: entry.status+ for entry in other.query(DownloadQueue).all()+ }+ assert entries == {+ owner_id: DocumentStatus.FAILED,+ linked_id: DocumentStatus.COMPLETED,+ }+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES+++def test_download_all_new_does_not_keep_a_queued_owner_whose_last_attempt_failed(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ retry_history,+):+ """A queued owner that failed last loses "Download All New" to a viable one.++ The document's own resource is queued: a refill through it failed, and+ "Download PDFs" for its research queued it again. The retry manager's+ cooldown has passed since, so the retry-policy filter of "Download All+ New" lets it through, but its latest attempt failed. "Download All New"+ kept a resource whose queue entry is PENDING over the second research's+ linked resource, which would have worked, so the run it started fetched+ the failing URL again, and the document stayed without its PDF. It+ keeps the linked resource now, as a bulk run over both researches does+ (see ``test_queued_owner_that_cannot_succeed_does_not_hold_the_document``),+ and that one refills the document.+ """+ owner_url = "https://example.com/owner-copy/queued-again.pdf"+ linked_url = "https://mirror.example.org/queued-again.pdf"+ owner_id = world.add_resource(owner_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(+ store,+ downloader,+ storage_mode="filesystem",+ retry_manager=retry_history,+ )+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ # A refill through the document's own resource fails, and "Download+ # PDFs" for its research queues it again...+ downloader.unreachable.add(owner_url)+ assert service.download_resource(owner_id) == (False, "Service unavailable")+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ # ...and the retry manager's cooldown passes.+ with db.fresh() as other:+ status = (+ other.query(ResourceDownloadStatus)+ .filter_by(resource_id=owner_id)+ .one()+ )+ assert status.status == "temporarily_failed"+ status.retry_after_timestamp = datetime.now(UTC) - timedelta(minutes=1)+ other.commit()+ assert retry_history.should_retry_resource(owner_id).can_retry+ requests = len(downloader.urls)++ queued = download_all_new(+ db, monkeypatch, retry_policy=retry_history, retry_manager=retry_history+ )++ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [second_research_id]+ events = bulk_download(+ db,+ store,+ downloader,+ monkeypatch,+ queued["research_ids"],+ mode="pdf",+ retry_manager=retry_history,+ )+ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert downloader.urls[requests:] == [linked_url]+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ entries = {+ entry.resource_id: entry.status+ for entry in other.query(DownloadQueue).all()+ }+ # The owner's entry stays for a run over its research, which finds the+ # document holding its PDF again.+ assert entries == {+ owner_id: DocumentStatus.PENDING,+ linked_id: DocumentStatus.COMPLETED,+ }+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES++[email protected]("via", ["bulk-run", "download-all-new"])+def test_document_refilled_by_another_stream_is_not_queued_again(+ db, store, world, one_download_session, filesystem_mode, monkeypatch, via+):+ """A document another download stream is refilling is not queued.++ The second research's linked resource is PROCESSING: a bulk run over+ that research has claimed it, at the time the claim records, and is+ downloading the PDF into the shared document. A run over the first+ research would queue the document's own resource and fetch the same+ PDF at the same time, which the check in ``download_resource`` cannot+ stop, since neither download has stored the PDF yet. The run leaves+ the document to that stream, and so does "Download All New": it keeps+ the claimed resource, which the bulk run it starts does not download+ again.+ """+ url = "https://example.com/refilled-elsewhere.pdf"+ owner_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ stored = store.root / other.query(Document).one().file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ # The other stream's claim on the linked resource's queue entry.+ claim_queue_entry(db, linked_id, second_research_id)+ requests = len(downloader.urls)++ if via == "bulk-run":+ research_ids = [world.research_id]+ else:+ queued = download_all_new(db, monkeypatch)+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [second_research_id]+ research_ids = queued["research_ids"]+ events = bulk_download(+ db, store, downloader, monkeypatch, research_ids, mode="pdf"+ )++ assert events[0]["total"] == 0, events+ assert len(downloader.urls) == requests+ with db.fresh() as other:+ assert (+ other.query(DownloadQueue).filter_by(resource_id=owner_id).count()+ == 0+ )+ assert (+ other.query(DownloadQueue).filter_by(resource_id=linked_id).one()+ ).status is DocumentStatus.PROCESSING++[email protected]("via", ["bulk-run", "download-all-new"])[email protected]("claimed_at", ["before-the-restart", "not-recorded"])+def test_download_cut_off_by_a_restart_does_not_hold_the_document(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ claimed_at,+ via,+):+ """A queue entry a restart left PROCESSING does not hold the document.++ A bulk run claimed the document's own resource and the server+ restarted during the download. Nothing reclaims the entry: it stays+ PROCESSING, and no reset touches it. It held the document as one+ being downloaded, so no run and no "Download All New" ever queued the+ second research's linked resource, and the document never got its PDF+ back. The claim records its time now, and an entry claimed before this+ server process started (``before-the-restart``), or before claims+ recorded their time (``not-recorded``), stands for no download. The+ linked resource refills the document, and the stranded entry is left+ as it is. A claim this process made still holds the document, see+ ``test_document_refilled_by_another_stream_is_not_queued_again``.+ """+ url = "https://example.com/cut-off-by-a-restart.pdf"+ owner_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).one()+ document_id = document.id+ stored = store.root / document.file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ if claimed_at == "before-the-restart":+ claim_queue_entry(db, owner_id, world.research_id)+ # The server restarts: this process started after the claim.+ monkeypatch.setattr(+ library_utils,+ "_PROCESS_STARTED_AT",+ datetime.now(UTC) + timedelta(seconds=1),+ )+ else:+ # Claimed before claims recorded their time.+ db.session.add(+ DownloadQueue(+ resource_id=owner_id,+ research_id=world.research_id,+ collection_id=LIBRARY_COLLECTION_ID,+ status=DocumentStatus.PROCESSING,+ )+ )+ db.session.commit()+ requests = len(downloader.urls)+ both = [world.research_id, second_research_id]++ if via == "bulk-run":+ research_ids = both+ else:+ queued = download_all_new(db, monkeypatch)+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued+ assert queued["research_ids"] == [second_research_id]+ research_ids = queued["research_ids"]+ events = bulk_download(+ db, store, downloader, monkeypatch, research_ids, mode="pdf"+ )++ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert len(downloader.urls) == requests + 1+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == EXTRACTED_TEXT+ restored = document.file_path+ entries = {+ entry.resource_id: entry.status+ for entry in other.query(DownloadQueue).all()+ }+ assert entries == {+ owner_id: DocumentStatus.PROCESSING,+ linked_id: DocumentStatus.COMPLETED,+ }+ assert _tree(store.root) == {restored}+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES+++class _LookupOnlyRun(AbstractSet):+ """The research sessions of a bulk run, which may only be looked up."""++ def __init__(self, research_ids):+ self._ids = frozenset(research_ids)++ def __contains__(self, research_id):+ return research_id in self._ids++ def __len__(self):+ return len(self._ids)++ def __iter__(self):+ raise AssertionError(+ "queueing one research copied the research sessions of the run"+ )+++def test_bulk_queueing_looks_the_runs_research_sessions_up(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """Queueing a research of a bulk run does not copy the run's sessions.++ ``download_bulk`` queues each research of the run with the run's+ research sessions, which decide which resource refills a document that+ awaits its PDF. It passed its list, and each call copied it into a new+ set, so queueing the research sessions of a run took time quadratic in+ their number, which a bulk request may name up to 100,000 of. The+ route builds one set for the run now, and each call only looks it up.+ """+ url = "https://example.com/run-of-two-researches.pdf"+ owner_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ stored = store.root / other.query(Document).one().file_path+ stored.unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ both = [world.research_id, second_research_id]++ run = _LookupOnlyRun(both)+ queued = [+ service.queue_research_downloads(+ research_id, LIBRARY_COLLECTION_ID, run_research_ids=run+ )+ for research_id in both[::-1]+ ]+ assert queued == [0, 1]+ with db.fresh() as other:+ assert [entry.resource_id for entry in other.query(DownloadQueue)] == [+ owner_id+ ]++ passed = []+ queue_research_downloads = DownloadService.queue_research_downloads++ def record(self, research_id, collection_id=None, **kwargs):+ passed.append(kwargs["run_research_ids"])+ return queue_research_downloads(+ self, research_id, collection_id, **kwargs+ )++ monkeypatch.setattr(DownloadService, "queue_research_downloads", record)+ events = bulk_download(db, store, downloader, monkeypatch, both, mode="pdf")+ assert events[0]["total"] == 1, events+ assert [event.get("status") for event in events[1:-1]] == ["success"]+ assert len(passed) == 2+ assert passed[0] is passed[1]+ assert isinstance(passed[0], AbstractSet)+ assert passed[0] == set(both)+++def test_queueing_a_research_loads_no_text_of_the_documents_it_links_to(+ db, store, world, monkeypatch+):+ """Queueing reads the documents resources are linked to without text.++ A second research cites a paper the first one downloaded, and the+ content-hash dedup linked its resource to the first research's+ document. "Download PDFs" for the second research queues nothing for+ it, since the document holds its PDF. To consider each document's own+ resource first, the queueing loaded every document such a resource+ points to as a whole row, with its extracted text, which #4560 kept+ such scans from loading. It reads only the columns it compares now.+ This covers those lookups only: the check whether a resource already+ has a completed document of its own reads that document whole, as it+ did before.+ """+ url = "https://example.com/cited-again.pdf"+ owner_id = world.add_resource(url)+ service = make_service(+ store, StubDownloader(TEXT_PDF_BYTES), storage_mode="database"+ )+ assert service.download_resource(owner_id) == (True, None)+ second_research_id, linked_id = _cite_in_another_research(db, url)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ assert (+ other.get(ResearchResource, linked_id).document_id+ == other.query(Document).one().id+ )+ statements = []++ def record(conn, cursor, statement, parameters, context, executemany):+ statements.append(statement)++ event.listen(db.engine, "before_cursor_execute", record)+ try:+ queued = service.queue_research_downloads(+ second_research_id, LIBRARY_COLLECTION_ID+ )+ finally:+ event.remove(db.engine, "before_cursor_execute", record)++ assert queued == 0+ assert any(+ "JOIN documents" in s or "FROM documents" in s for s in statements+ )+ assert [s for s in statements if "text_content" in s] == []+++def render_download_manager(db, monkeypatch):+ """The Download Manager page, as the browser gets it.++ Runs the ``download_manager_page`` route and renders its template+ against this database. The route's session and settings sources are+ replaced; the ``LibraryService`` it builds reads through the session+ that ``make_library_service`` set up. Importing the app registers the+ template globals that the page's base template uses.+ """+ from local_deep_research.web import fastapi_app # noqa: F401++ monkeypatch.setattr(+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ monkeypatch.setattr(+ library_router,+ "get_settings_manager",+ lambda *a, **k: StubSettings(+ {"research_library.pdf_storage_mode": "filesystem"}+ ),+ )+ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": "/library/download-manager",+ "headers": [],+ "query_string": b"",+ "session": {"session_id": "sid", "username": USERNAME},+ }+ )+ response = library_router.download_manager_page(request, username=USERNAME)+ return response.body.decode()+++def test_download_manager_counts_a_document_awaiting_its_pdf_as_available(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """The Download Manager offers the PDF the sync reported missing.++ The sync's result sends the user to the Download Manager to download+ the missing files again. Its counts took every completed document as+ downloaded, so a document awaiting its PDF showed there as "Already+ Downloaded", and nothing was "Available to Download", while "Download+ All New" queued it. Until its PDF is back it counts as available now,+ in the totals and in its research's row, and its research's sources+ show it under "Sources by Domain" as not downloaded and without the+ "Downloaded" badge. Those two compared the document's status with the+ string "completed", which a loaded status never equals, so they showed+ no source as downloaded, also once its PDF was back.+ """+ url = "https://example.com/counted-as-available.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ library = make_library_service(db, store, monkeypatch)++ def counts():+ summary = library.get_download_manager_summary_stats()+ (research,) = library.get_research_list_with_stats()+ page = render_download_manager(db, monkeypatch)+ return {+ "already_downloaded": summary["already_downloaded"],+ "available_to_download": summary["available_to_download"],+ "research_downloaded": research["downloaded_count"],+ "research_downloadable": research["downloadable_count"],+ "sources_by_domain": [+ " ".join(stats.split())+ for stats in re.findall(+ r'<span class="ldr-domain-stats">(.*?)</span>', page, re.S+ )+ ],+ "downloaded_badges": page.count(+ '<span class="ldr-badge ldr-badge-success">Downloaded</span>'+ ),+ }++ downloaded = {+ "already_downloaded": 1,+ "available_to_download": 0,+ "research_downloaded": 1,+ "research_downloadable": 1,+ "sources_by_domain": ["1 total | 1 PDFs | 1 downloaded"],+ "downloaded_badges": 1,+ }+ assert counts() == downloaded+ with db.fresh() as other:+ stored = store.root / other.query(Document).one().file_path+ stored.unlink()+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ assert counts() == {+ "already_downloaded": 0,+ "available_to_download": 1,+ "research_downloaded": 0,+ "research_downloadable": 1,+ "sources_by_domain": ["1 total | 1 PDFs | 0 downloaded"],+ "downloaded_badges": 0,+ }+ assert download_all_new(db, monkeypatch)["queued"] == 1+ assert service.download_resource(resource_id) == (True, None)+ assert counts() == downloaded+++def _post(path, payload):+ """A POST request whose JSON body is ``payload``."""+ body = json.dumps(payload).encode()++ async def receive():+ return {"type": "http.request", "body": body, "more_body": False}++ return Request(+ {+ "type": "http",+ "method": "POST",+ "path": path,+ "headers": [(b"content-type", b"application/json")],+ "query_string": b"",+ "session": {"session_id": "sid"},+ },+ receive,+ )+++def source_routes(db, store, downloader, monkeypatch, research_id, url):+ """What the library's source routes report for ``url`` in a research.++ Runs ``get_research_sources`` and ``check_downloads``, and then+ ``download_source``, which downloads the resource unless it answers+ "Already downloaded", against this database. Their session and the+ user's password are replaced, and so is the ``DownloadService`` that+ ``download_source`` downloads with (a filesystem-mode service around+ ``downloader``). Which documents count as downloaded, and the queue+ entry, claim and download of the resource, are the production code.+ """+ monkeypatch.setattr(+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()+ )+ monkeypatch.setattr(+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"+ )+ monkeypatch.setattr(+ library_router,+ "DownloadService",+ lambda *a, **k: make_service(+ store, downloader, storage_mode="filesystem"+ ),+ )+ (source,) = library_router.get_research_sources(+ None, research_id, username=USERNAME+ )["sources"]+ checked = asyncio.run(+ library_router.check_downloads(+ _post(+ "/library/api/check-downloads",+ {"research_id": research_id, "urls": [url]},+ ),+ username=USERNAME,+ )+ )["download_status"][url]+ answer = asyncio.run(+ library_router.download_source(+ _post(+ "/library/api/download-source",+ {"research_id": research_id, "url": url},+ ),+ username=USERNAME,+ )+ )+ return {+ "research_sources": (source["downloaded"], source["document_id"]),+ "check_downloads": (+ checked["downloaded"],+ checked.get("document_id"),+ checked.get("resource_id"),+ ),+ "download_source": answer["message"],+ }+++def test_source_routes_report_a_download_as_the_download_manager_counts_it(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """The library's source routes follow the Download Manager's counts.++ ``get_research_sources`` and ``check_downloads`` report whether a+ research's source is downloaded, and ``download_source`` answers+ "Already downloaded" for a source that is and downloads any other.+ They compared the document's loaded status with the string+ "completed", which a ``DocumentStatus`` member never equals, so they+ reported every source as not downloaded, and ``download_source``+ queued and claimed a downloaded one again. They apply the Download+ Manager's rule now (``document_counts_as_downloaded``): a document that+ holds its PDF is downloaded, and one that awaits its PDF after the sync+ reported its file missing is not, so ``download_source`` puts the PDF+ back into it.+ """+ url = "https://example.com/reported-by-the-source-routes.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ stored = store.root / document.file_path+ library = make_library_service(db, store, monkeypatch)++ def routes():+ return source_routes(+ db, store, downloader, monkeypatch, world.research_id, url+ )++ downloaded = {+ "research_sources": (True, document_id),+ "check_downloads": (True, document_id, None),+ "download_source": "Already downloaded",+ }+ assert routes() == downloaded+ assert downloader.urls == [url]+ with db.fresh() as other:+ assert other.query(DownloadQueue).count() == 0++ stored.unlink()+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ with db.fresh() as other:+ assert document_awaits_pdf(other.get(Document, document_id))++ assert routes() == {+ "research_sources": (False, None),+ "check_downloads": (False, None, resource_id),+ "download_source": "Download completed",+ }+ # download_source put the PDF back into the same document, with one+ # request, and finished its queue entry.+ assert downloader.urls == [url, url]+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert not document_awaits_pdf(document)+ assert document.text_content == EXTRACTED_TEXT+ assert (store.root / document.file_path).read_bytes() == TEXT_PDF_BYTES+ (entry,) = other.query(DownloadQueue).all()+ assert (entry.resource_id, entry.status) == (+ resource_id,+ DocumentStatus.COMPLETED,+ )+ assert routes() == downloaded+ assert downloader.urls == [url, url]+++def render_document_details(db, monkeypatch, document_id):+ """The document details page, as the browser gets it.++ Runs the ``document_details_page`` route and renders its template. The+ ``LibraryService`` it builds reads through the session that+ ``make_library_service`` set up. Importing the app registers the+ template globals that the page's base template uses.+ """+ from local_deep_research.web import fastapi_app # noqa: F401++ request = Request(+ {+ "type": "http",+ "method": "GET",+ "path": f"/library/document/{document_id}",+ "headers": [],+ "query_string": b"",+ "session": {"session_id": "sid", "username": USERNAME},+ }+ )+ response = library_router.document_details_page(+ request, document_id, username=USERNAME+ )+ return response.body.decode()++[email protected]("status", ["completed", "failed"])+def test_document_details_page_shows_the_download_status(+ db, store, world, monkeypatch, status+):+ """The details page's "Download Status" badge shows the status.++ The page compares the status it gets from ``get_document_by_id`` with+ the string 'completed' and otherwise prints it in a warning badge. It+ got the loaded ``DocumentStatus`` member, which never equals that+ string, so every document showed a warning badge reading+ "Documentstatus.completed" or "Documentstatus.failed". It gets the+ status's value now: a completed download shows "Completed", and a+ failed download "Failed".+ """+ url = f"https://example.com/details-of-a-{status}-download.pdf"+ resource_id = world.add_resource(url)+ service = make_service(store, StubDownloader(), storage_mode="database")+ if status == "completed":+ assert service.download_resource(resource_id) == (True, None)+ else:+ resource = db.session.get(ResearchResource, resource_id)+ service._record_failed_text_extraction(+ db.session, resource, error="PDF download failed: HTTP 503"+ )+ db.session.commit()+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ assert document.status is DocumentStatus(status)+ make_library_service(db, store, monkeypatch)++ page = render_document_details(db, monkeypatch, document_id)++ badges = re.findall(+ r'Download Status</div>\s*<div class="ldr-metadata-value">\s*'+ r'<span class="ldr-badge (ldr-badge-[a-z]+)">([^<]*)</span>',+ page,+ )+ assert badges == [+ ("ldr-badge-success", "Completed")+ if status == "completed"+ else ("ldr-badge-warning", "Failed")+ ]+++def test_text_extraction_reads_the_stored_pdf_file_without_a_request(+ db, store, world, filesystem_mode, monkeypatch+):+ """Text extraction reads a downloaded PDF file before fetching it.++ Text extraction (``download_as_text``) tries the PDF file that a+ completed download stores (``_try_existing_pdf_extraction``) before it+ asks the source; only an arXiv paper's text is requested from arXiv+ before that. The step compared the document's loaded status with the+ string "completed", which a ``DocumentStatus`` member never equals, so+ it never read the file, and the text came from a request for the URL.+ A completed filesystem download without text, here one whose+ extraction found none when it was downloaded, now gets the text of its+ stored file, and no request is made.+ """+ url = "https://example.com/text-from-the-stored-file.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ # The PDF is stored under the library root that the step resolves its+ # path against.+ service.settings._values["research_library.storage_path"] = str(store.base)+ service._extract_text_from_pdf = lambda _data: None+ assert service.download_resource(resource_id) == (True, None)+ del service._extract_text_from_pdf+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ assert document.status is DocumentStatus.COMPLETED+ assert document.text_content is None+ assert (store.root / document.file_path).read_bytes() == TEXT_PDF_BYTES+ assert downloader.urls == [url]++ assert service.download_as_text(resource_id) == (True, None)++ assert downloader.urls == [url], "the text was fetched again"+ with db.fresh() as other:+ assert other.query(Document).count() == 1+ document = other.get(Document, document_id)+ assert document.text_content == EXTRACTED_TEXT+ assert (document.extraction_method, document.extraction_source) == (+ "pdf_extraction",+ "pdfplumber",+ )+++def test_cached_text_extraction_uses_explicit_document_over_retained_owner(+ db, store, world, one_download_session, filesystem_mode, monkeypatch+):+ """A retained older PDF must not supply text for a relinked resource."""+ linked_id = world.add_resource("https://example.com/current-version.pdf")+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ service.settings._values["research_library.storage_path"] = str(store.base)+ original_extract = service._extract_text_from_pdf+ service._extract_text_from_pdf = lambda _data: None+ assert service.download_resource(linked_id) == (True, None)+ _, mirror_id = _cite_in_another_research(+ db, "https://mirror.example.org/old-version.pdf"+ )+ assert service.download_resource(mirror_id) == (True, None)+ with db.fresh() as session:+ old_id = session.get(ResearchResource, linked_id).document_id+ assert session.get(ResearchResource, mirror_id).document_id == old_id++ _, owner_id = _cite_in_another_research(+ db, "https://example.org/new-version.pdf"+ )+ downloader.content = REGENERATED_PDF_BYTES+ assert service.download_resource(owner_id) == (True, None)+ with db.fresh() as session:+ canonical_id = session.get(ResearchResource, owner_id).document_id+ assert canonical_id != old_id+ library = make_library_service(db, store, monkeypatch)+ assert library.mark_for_redownload([old_id]) == 1+ assert service.download_resource(linked_id) == (True, None)+ downloader.content = TEXT_PDF_BYTES+ service._extract_text_from_pdf = original_extract+ assert service.download_resource(mirror_id) == (True, None)+ with db.fresh() as session:+ old = session.get(Document, old_id)+ canonical = session.get(Document, canonical_id)+ assert old.resource_id == linked_id+ assert old.status is DocumentStatus.COMPLETED+ assert canonical.status is DocumentStatus.COMPLETED+ assert old.text_content == EXTRACTED_TEXT+ assert canonical.text_content is None+ assert session.get(ResearchResource, linked_id).document_id == (+ canonical_id+ )+ assert (store.root / old.file_path).read_bytes() == TEXT_PDF_BYTES+ assert (store.root / canonical.file_path).read_bytes() == (+ REGENERATED_PDF_BYTES+ )+ requests_before = list(downloader.urls)++ assert service.download_as_text(linked_id) == (True, None)++ assert downloader.urls == requests_before, "the cached PDF was not used"+ with db.fresh() as session:+ assert session.get(ResearchResource, linked_id).document_id == (+ canonical_id+ )+ assert session.get(Document, canonical_id).text_content == (+ REGENERATED_TEXT+ )+ assert session.get(Document, old_id).text_content == EXTRACTED_TEXT++[email protected]("tracker_path", ["kept", "lost"])+def test_download_whose_resource_was_deleted_takes_its_pdf_back_by_content(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ tracker_path,+):+ """Without its resource, a removed PDF is cleared but not reported.++ A second research cites the same URL, and its download stores a+ regenerated PDF as a document of its own. Deleting that resource from+ its research keeps the document but clears its ``resource_id``. If the+ PDF file is then removed, no download path asks for the PDF again,+ since they start from a research's resources, so the sync does not+ report a file to download again. It keeps the document's text and+ makes it text-only, and resets the URL's tracker, as it does for a+ document that has its resource. Left naming the missing file, the+ document kept a later download of the same PDF bytes, which the+ content-hash dedup links to it, from storing the PDF: the dedup copied+ the dangling path into that download's tracker instead. ``lost`` is the+ R1 sequence (a "none"-mode re-download cleared the tracker path) before+ the resource was deleted, which reaches the sync's no-path branch.+ """+ url = "https://example.com/cited-twice-then-removed.pdf"+ first_id = world.add_resource(url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(first_id) == (True, None)+ _, second_id = _cite_in_another_research(db, url)+ downloader.content = REGENERATED_PDF_BYTES+ assert service.download_resource(second_id) == (True, None)+ with db.fresh() as other:+ document = other.query(Document).filter_by(resource_id=second_id).one()+ document_id = document.id+ own_path = document.file_path+ assert document.text_content == REGENERATED_TEXT+ assert find_tracker_by_url(other, url).file_path == own_path+ library = make_library_service(db, store, monkeypatch)+ if tracker_path == "lost":+ assert library.mark_for_redownload([document_id]) == 1+ service.settings._values["research_library.pdf_storage_mode"] = "none"+ assert service.download_resource(second_id) == (True, None)+ service.settings._values["research_library.pdf_storage_mode"] = (+ "filesystem"+ )+ with db.fresh() as other:+ assert find_tracker_by_url(other, url).file_path is None+ assert other.get(Document, document_id).file_path == own_path+ # As resource_service.delete_resource does: the row goes, its+ # documents stay with resource_id cleared.+ db.session.delete(db.session.get(ResearchResource, second_id))+ db.session.commit()+ with db.fresh() as other:+ assert other.get(Document, document_id).resource_id is None+ (store.root / own_path).unlink()+ chunk_id = _add_chunk(db, document_id)+ requests = len(downloader.urls)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (2, 0)+ assert stats["missing_files"] == []+ assert stats["trackers_updated"] == 1+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.text_content == REGENERATED_TEXT+ assert document.storage_mode == "none"+ assert document.file_path == FILE_PATH_TEXT_ONLY+ assert not document_awaits_pdf(document)+ assert other.get(DocumentChunk, chunk_id) is not None+ tracker = find_tracker_by_url(other, url)+ assert tracker.is_downloaded is False+ assert tracker.file_path is None+ view = library.get_document_by_id(document_id)+ assert view["has_pdf"] is False+ assert view["has_text_db"] is True+ # Nothing asks for its PDF...+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 0+ )+ assert download_all_new(db, monkeypatch)["queued"] == 0+ assert len(downloader.urls) == requests++ # ...but a later download of the same PDF for another resource stores+ # it in this document.+ _, third_id = _cite_in_another_research(+ db, url, query="third research citing the paper"+ )+ assert service.download_resource(third_id) == (True, None)+ assert len(downloader.urls) == requests + 1+ with db.fresh() as other:+ assert other.query(Document).count() == 2+ assert other.get(ResearchResource, third_id).document_id == (+ document_id+ )+ document = other.get(Document, document_id)+ assert document.storage_mode == "filesystem"+ assert document.text_content == REGENERATED_TEXT+ assert other.get(DocumentChunk, chunk_id) is not None+ restored = store.root / document.file_path+ assert find_tracker_by_url(other, url).file_path == document.file_path+ assert restored.read_bytes() == REGENERATED_PDF_BYTES+ assert library.get_document_by_id(document_id)["has_pdf"] is True+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)+++# ===========================================================================+# Documents that share their URL's tracker+# ===========================================================================++[email protected]("first", ["download", "placeholder"])+def test_placeholder_sharing_the_url_of_a_removed_file_is_left_as_it_is(+ db, store, world, one_download_session, filesystem_mode, monkeypatch, first+):+ """Only the document whose file was removed is reported missing.++ A URL has one tracker, however many documents have that URL. The first+ research downloaded the PDF into its document, and the tracker names+ that document's file. A second research's download of the same URL+ failed, and "Extract Text to DB" completed its placeholder with text:+ no PDF, no file path, the ``"database"`` column default and no blob.+ Once the first document's file was removed, a sync that reached the+ placeholder first judged it by the tracker's path. It labelled the+ placeholder a PDF, made it text-only and reported the other document's+ file missing for it, and the download manager then downloaded the PDF+ a second time, into the placeholder. The removed file was counted+ twice. The sync judges each document by what it stores itself, so the+ placeholder is left as it is whichever document the sync reaches first+ (``first``).+ """+ url = "https://example.com/shared-with-a-text-placeholder.pdf"+ owner_id = world.add_resource(url)+ _, placeholder_resource_id = _cite_in_another_research(db, url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ placeholder_resource = db.session.get(+ ResearchResource, placeholder_resource_id+ )++ def record_failed_download():+ service._record_failed_text_extraction(+ db.session,+ placeholder_resource,+ error="PDF download failed: HTTP 503",+ )+ db.session.commit()++ if first == "placeholder":+ record_failed_download()+ assert service.download_resource(owner_id) == (True, None)+ if first == "download":+ record_failed_download()+ service._save_text_with_db(+ placeholder_resource,+ SAVED_TEXT,+ db.session,+ extraction_method="native_api",+ extraction_source="arxiv_api",+ )+ db.session.commit()+ with db.fresh() as other:+ downloaded = other.query(Document).filter_by(resource_id=owner_id).one()+ downloaded_id = downloaded.id+ own_path = downloaded.file_path+ assert downloaded.text_content == EXTRACTED_TEXT+ placeholder = (+ other.query(Document)+ .filter_by(resource_id=placeholder_resource_id)+ .one()+ )+ placeholder_id = placeholder.id+ assert placeholder.status is DocumentStatus.COMPLETED+ assert placeholder.text_content == SAVED_TEXT+ placeholder_shape = (+ placeholder.file_type,+ placeholder.mime_type,+ placeholder.storage_mode,+ placeholder.file_path,+ )+ assert placeholder_shape == ("unknown", None, "database", None)+ assert other.query(DocumentBlob).count() == 0+ assert find_tracker_by_url(other, url).file_path == own_path+ # The order in which the sync reads the documents.+ assert [+ document.id+ for document in other.query(Document)+ .filter_by(status=DocumentStatus.COMPLETED)+ .filter(Document.original_url.isnot(None))+ ] == (+ [downloaded_id, placeholder_id]+ if first == "download"+ else [placeholder_id, downloaded_id]+ )+ (store.root / own_path).unlink()+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (1, 1)+ assert [+ (entry["id"], entry["path"]) for entry in stats["missing_files"]+ ] == [(downloaded_id, own_path)]+ assert stats["trackers_updated"] == 1+ with db.fresh() as other:+ placeholder = other.get(Document, placeholder_id)+ assert placeholder.text_content == SAVED_TEXT+ assert (+ placeholder.file_type,+ placeholder.mime_type,+ placeholder.storage_mode,+ placeholder.file_path,+ ) == placeholder_shape+ assert not document_awaits_pdf(placeholder)+ downloaded = other.get(Document, downloaded_id)+ assert downloaded.text_content == EXTRACTED_TEXT+ assert document_awaits_pdf(downloaded)++ # The download manager downloads the PDF once, into the document whose+ # file was removed.+ queued = download_all_new(db, monkeypatch)+ assert queued["queued"] == 1, queued+ assert queued["research_ids"] == [world.research_id]+ assert service.download_resource(owner_id) == (True, None)+ assert service.download_resource(placeholder_resource_id) == (True, None)+ assert len(downloader.urls) == 2+ with db.fresh() as other:+ restored = other.get(Document, downloaded_id).file_path+ placeholder = other.get(Document, placeholder_id)+ assert placeholder.file_path is None+ assert placeholder.text_content == SAVED_TEXT+ assert _tree(store.root) == {restored}+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)++[email protected]("text", [True, False], ids=["with-text", "no-text"])[email protected]("tracker_path", ["other-file", "sentinel"])+def test_removed_file_is_reported_although_the_url_tracker_names_another(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ tracker_path,+ text,+):+ """A document whose own file was removed is reported missing.++ Two researches cite the same URL, and each download stored its PDF in+ a document of its own: the second research got a regenerated PDF with+ other bytes, so the content-hash dedup did not link it to the first+ research's document. The URL's one tracker names the file stored+ last, the second research's (``other-file``). When the first+ research's file was removed, the sync judged the first document by+ the tracker's file, which is still there, and counted it found:+ nothing reported its PDF missing, and no download path asked for it+ again. A tracker that a content-hash duplicate download set to a+ sentinel such as ``blob_deleted``, before such downloads stopped+ copying sentinels, made the sync count every document of the URL found+ in the same way (``sentinel``). The sync judges the first document by+ its own file now. With its text left (``with-text``) it is made+ text-only and reported, and the download manager puts its PDF back.+ With nothing else left (``no-text``) it is deleted and reported, so+ that its URL can be downloaded again. The second research's document+ and the tracker are left as they are.+ """+ url = "https://example.com/two-copies-one-tracker.pdf"+ first_id = world.add_resource(url)+ content = TEXT_PDF_BYTES if text else PDF_BYTES+ downloader = StubDownloader(content)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(first_id) == (True, None)+ _, second_id = _cite_in_another_research(db, url)+ downloader.content = REGENERATED_PDF_BYTES+ assert service.download_resource(second_id) == (True, None)+ with db.fresh() as other:+ first = other.query(Document).filter_by(resource_id=first_id).one()+ first_document_id = first.id+ first_path = first.file_path+ assert first.text_content == (EXTRACTED_TEXT if text else None)+ second = other.query(Document).filter_by(resource_id=second_id).one()+ second_document_id = second.id+ second_path = second.file_path+ assert second_path != first_path+ assert find_tracker_by_url(other, url).file_path == second_path+ if tracker_path == "sentinel":+ db.session.query(DownloadTracker).filter_by(+ url_hash=get_url_hash(url)+ ).update(+ {DownloadTracker.file_path: FILE_PATH_BLOB_DELETED},+ synchronize_session=False,+ )+ db.session.commit()+ expected_tracker = (+ second_path if tracker_path == "other-file" else FILE_PATH_BLOB_DELETED,+ True,+ )+ (store.root / first_path).unlink()+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (1, 1)+ assert [+ (entry["id"], entry["path"]) for entry in stats["missing_files"]+ ] == [(first_document_id, first_path)]+ assert stats["trackers_updated"] == 0+ with db.fresh() as other:+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == expected_tracker+ second = other.get(Document, second_document_id)+ assert (second.storage_mode, second.file_path) == (+ "filesystem",+ second_path,+ )+ first = other.get(Document, first_document_id)+ if text:+ assert first.text_content == EXTRACTED_TEXT+ assert first.file_path == FILE_PATH_TEXT_ONLY+ assert document_awaits_pdf(first)+ else:+ assert first is None, "nothing of the document was left"+ assert (store.root / second_path).read_bytes() == REGENERATED_PDF_BYTES++ # The download manager downloads the first research's PDF again.+ downloader.content = content+ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 1+ )+ assert service.download_resource(first_id) == (True, None)+ with db.fresh() as other:+ first = other.query(Document).filter_by(resource_id=first_id).one()+ assert (first.id == first_document_id) is text+ restored = first.file_path+ assert (store.root / restored).read_bytes() == content+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)++[email protected]("first", ["deleted", "kept"])+def test_tracker_reset_for_two_documents_counts_once_in_either_order(+ db, store, world, one_download_session, filesystem_mode, monkeypatch, first+):+ """``trackers_updated`` counts a tracker once, whichever document is first.++ Two researches cite the same URL, and each download stored its PDF in+ a document of its own (other bytes, so the content-hash dedup did not+ link them). The URL's one tracker names the file of the first+ research's document, which has no text, and both files were removed.+ The sync deletes that document and resets the tracker, and makes the+ second research's document, which has its text, text-only and reports+ it. When the sync read the deleted document first (``deleted``), it+ reached the other one with the tracker reset already, reset it again+ and counted it twice. In the other order (``kept``) the tracker still+ named the first research's file when the sync reached the second+ document, so it did not describe that document and was left alone+ there, and the deletion counted it once. Either order counts it once+ now, and the documents and the tracker end the same.+ """+ url = "https://example.com/one-tracker-reset-twice.pdf"+ deleted_resource_id = world.add_resource(url)+ _, kept_resource_id = _cite_in_another_research(db, url)+ downloader = StubDownloader()+ service = make_service(store, downloader, storage_mode="filesystem")++ def download(resource_id, content):+ downloader.content = content+ assert service.download_resource(resource_id) == (True, None)++ # The sync reads the documents in the order they were stored.+ if first == "deleted":+ download(deleted_resource_id, PDF_BYTES)+ download(kept_resource_id, TEXT_PDF_BYTES)+ else:+ download(kept_resource_id, TEXT_PDF_BYTES)+ download(deleted_resource_id, PDF_BYTES)+ with db.fresh() as other:+ deleted = (+ other.query(Document)+ .filter_by(resource_id=deleted_resource_id)+ .one()+ )+ deleted_id, deleted_path = deleted.id, deleted.file_path+ assert deleted.text_content is None+ kept = (+ other.query(Document).filter_by(resource_id=kept_resource_id).one()+ )+ kept_id, kept_path = kept.id, kept.file_path+ assert kept.text_content == EXTRACTED_TEXT+ read_order = [+ document.id+ for document in other.query(Document)+ .filter_by(status=DocumentStatus.COMPLETED)+ .filter(Document.original_url.isnot(None))+ ]+ assert read_order == (+ [deleted_id, kept_id]+ if first == "deleted"+ else [kept_id, deleted_id]+ )+ # The tracker names the file written last. In both orders it names the+ # first research's file here, as when that file was written last.+ db.session.query(DownloadTracker).filter_by(+ url_hash=get_url_hash(url)+ ).update(+ {+ DownloadTracker.file_path: deleted_path,+ DownloadTracker.is_downloaded: True,+ },+ synchronize_session=False,+ )+ db.session.commit()+ (store.root / deleted_path).unlink()+ (store.root / kept_path).unlink()+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (0, 2)+ assert sorted(+ (entry["id"], entry["path"]) for entry in stats["missing_files"]+ ) == sorted([(deleted_id, deleted_path), (kept_id, kept_path)])+ assert stats["trackers_updated"] == 1+ with db.fresh() as other:+ assert other.get(Document, deleted_id) is None+ kept = other.get(Document, kept_id)+ assert kept.text_content == EXTRACTED_TEXT+ assert document_awaits_pdf(kept)+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == (None, False)++[email protected]("first", ["empty", "second"])[email protected]("holds", ["blob", "text", "nothing"])+def test_database_documents_of_one_url_are_each_judged_by_their_own_blob(+ db, store, world, monkeypatch, holds, first+):+ """Each database document of a URL is judged by what it stores itself.++ Three researches cite the same paper. The first research's document is+ a database-mode download that holds nothing (see+ ``_add_database_row_without_pdf``). The third research downloaded the+ same bytes in database mode, and the content-hash dedup linked its+ resource to that empty document and recorded ``"database"`` in the+ URL's one tracker. The second research's document is a second one: a+ database-mode download of a regenerated PDF with other bytes, which+ holds that PDF as a blob (``blob``) or holds nothing either, the size+ limit having refused that PDF too (``nothing``), or a failed-download+ placeholder that "Extract Text to DB" completed, which holds only text+ (``text``). Both documents are labelled as stored in the database,+ without a path, so the tracker describes both, and the sync judges+ each by its own blob and text, whichever it reads first (``first``):+ it deletes and reports, as ``"database"``, each document that holds+ nothing, and keeps the other one as it is. The tracker records that+ the URL's PDF is stored in the database. While the second document+ holds a blob, that is still true and the tracker is left as it is.+ Otherwise no document holds the PDF any more, and the tracker is reset+ once. When the tracker counted only for a document storing the same+ path, it described neither document: the documents that held nothing+ were deleted without being reported, and the tracker was never reset.+ """+ url = "https://example.com/two-database-documents.pdf"+ empty_resource_id = world.add_resource(url, downloaded=True)+ second_research_id, second_resource_id = _cite_in_another_research(db, url)+ _, linked_resource_id = _cite_in_another_research(+ db, url, query="third research citing the paper"+ )+ downloader = StubDownloader(REGENERATED_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="database")+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)++ def add_second_document():+ if holds == "blob":+ assert service.download_resource(second_resource_id) == (True, None)+ elif holds == "text":+ resource = db.session.get(ResearchResource, second_resource_id)+ service._record_failed_text_extraction(+ db.session, resource, error="PDF download failed: HTTP 503"+ )+ db.session.commit()+ service._save_text_with_db(+ resource,+ SAVED_TEXT,+ db.session,+ extraction_method="native_api",+ extraction_source="arxiv_api",+ )+ db.session.commit()+ else:+ _add_database_row_without_pdf(+ db,+ world,+ second_resource_id,+ second_research_id,+ url,+ REGENERATED_PDF_BYTES,+ )++ if first == "second":+ add_second_document()+ empty_id = _add_database_row_without_pdf(+ db, world, empty_resource_id, world.research_id, url, PDF_BYTES+ )+ if first == "empty":+ add_second_document()+ downloader.content = PDF_BYTES+ assert service.download_resource(linked_resource_id) == (True, None)+ with db.fresh() as other:+ linked = other.get(ResearchResource, linked_resource_id)+ assert linked.document_id == empty_id+ second = (+ other.query(Document)+ .filter_by(resource_id=second_resource_id)+ .one()+ )+ second_id = second.id+ second_shape = (+ second.file_type,+ second.storage_mode,+ second.file_path,+ second.text_content,+ )+ assert second_shape[1:] == (+ "database",+ None,+ SAVED_TEXT if holds == "text" else None,+ )+ assert [row.document_id for row in other.query(DocumentBlob)] == (+ [second_id] if holds == "blob" else []+ )+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)+ # The order in which the sync reads the documents.+ read_order = [+ document.id+ for document in other.query(Document)+ .filter_by(status=DocumentStatus.COMPLETED)+ .filter(Document.original_url.isnot(None))+ ]+ assert read_order == (+ [empty_id, second_id] if first == "empty" else [second_id, empty_id]+ )+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ second_kept = holds != "nothing"+ assert (stats["files_found"], stats["files_missing"]) == (+ (1, 1) if second_kept else (0, 2)+ )+ assert [+ (entry["id"], entry["path"]) for entry in stats["missing_files"]+ ] == [+ (document_id, "database")+ for document_id in read_order+ if document_id == empty_id or not second_kept+ ]+ assert stats["trackers_updated"] == (0 if holds == "blob" else 1)+ with db.fresh() as other:+ assert other.get(Document, empty_id) is None+ second = other.get(Document, second_id)+ if second_kept:+ assert (+ second.file_type,+ second.storage_mode,+ second.file_path,+ second.text_content,+ ) == second_shape+ else:+ assert second is None, "nothing of the document was left"+ if holds == "blob":+ assert (+ PDFStorageManager(store.root, "none").load_pdf(second, other)+ == REGENERATED_PDF_BYTES+ )+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == (+ ("database", True) if holds == "blob" else (None, False)+ )+++def test_document_stored_without_a_pdf_is_not_judged_by_a_database_tracker(+ db, store, world, monkeypatch+):+ """A ``"database"`` tracker does not describe a document without a PDF.++ The first research downloaded the paper with PDF storage set to+ "none", so its document stores no PDF, and the PDF had no text to+ extract. The second research got a regenerated PDF with other bytes in+ database mode, stored as a blob in a document of its own, and the+ URL's tracker records ``"database"``. That tracker describes only+ documents labelled as stored in the database. The first document is+ left alone, as every document stored without a PDF is, and not deleted+ for lacking a blob it never had.+ """+ url = "https://example.com/stored-without-a-pdf.pdf"+ first_id = world.add_resource(url)+ _, second_id = _cite_in_another_research(db, url)+ downloader = StubDownloader(PDF_BYTES)+ service = make_service(store, downloader, storage_mode="none")+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)+ assert service.download_resource(first_id) == (True, None)+ downloader.content = REGENERATED_PDF_BYTES+ service.settings._values["research_library.pdf_storage_mode"] = "database"+ assert service.download_resource(second_id) == (True, None)+ with db.fresh() as other:+ first = other.query(Document).filter_by(resource_id=first_id).one()+ first_document_id = first.id+ first_shape = (first.storage_mode, first.file_path, first.text_content)+ assert first_shape == ("none", None, None)+ second = other.query(Document).filter_by(resource_id=second_id).one()+ second_document_id = second.id+ assert (second.storage_mode, second.file_path) == ("database", None)+ assert [row.document_id for row in other.query(DocumentBlob)] == [+ second_document_id+ ]+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (2, 0)+ assert stats["missing_files"] == []+ assert stats["trackers_updated"] == 0+ with db.fresh() as other:+ first = other.get(Document, first_document_id)+ assert first is not None, "the library sync deleted the document"+ assert (+ first.storage_mode,+ first.file_path,+ first.text_content,+ ) == first_shape+ assert other.get(Document, second_document_id) is not None+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)+++# ===========================================================================+# "Remove PDF to save space" is not a missing file+# ===========================================================================++[email protected]("text", [True, False], ids=["with-text", "no-text"])[email protected]("stored_mode", ["database", "filesystem"])+def test_pdf_removed_to_save_space_is_kept_and_not_downloaded_again(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ stored_mode,+ text,+):+ """``delete_blob_only`` followed by a sync keeps the document as it is.++ "Remove PDF to save space" removes the stored PDF, blob or file, and+ marks the document ``blob_deleted``, but leaves the tracker naming the+ copy it removed. The sync resolved that name, found nothing, and+ deleted a document without text: its row, metadata and collection+ link were lost and the tracker was reset, so the download manager+ offered the PDF the user had removed for download again. A document+ with text must not be taken for one whose PDF went missing either.+ Neither the sync nor any download path brings the removed PDF back.+ """+ url = "https://example.com/removed-to-save-space.pdf"+ resource_id = world.add_resource(url)+ content = TEXT_PDF_BYTES if text else PDF_BYTES+ downloader = StubDownloader(content)+ service = make_service(store, downloader, storage_mode=stored_mode)+ if not text:+ monkeypatch.setattr(+ service, "_extract_text_from_pdf", lambda _data: None+ )+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document_id = (+ other.query(Document).filter_by(resource_id=resource_id).one().id+ )+ tracker_path = find_tracker_by_url(other, url).file_path+ assert tracker_path++ library = make_library_service(db, store, monkeypatch)+ # delete_blob_only resolves the file through the ambient settings.+ settings = StubSettings({"research_library.storage_path": str(store.base)})+ monkeypatch.setattr(+ db_utils,+ "get_settings_manager",+ lambda *a, **k: SimpleNamespace(+ get_setting=settings.get_setting, close=lambda: None+ ),+ )+ removed = document_deletion.DocumentDeletionService(+ USERNAME+ ).delete_blob_only(document_id)+ assert removed["deleted"] is True, removed+ assert removed["bytes_freed"] == len(content)+ assert _tree(store.root) == set()+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document.storage_mode == "none"+ assert document.file_path == FILE_PATH_BLOB_DELETED+ assert other.query(DocumentBlob).count() == 0+ # The shape the sync has to handle: the tracker still names the+ # removed copy.+ assert find_tracker_by_url(other, url).file_path == tracker_path++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+ assert stats["missing_files"] == []+ with db.fresh() as other:+ document = other.get(Document, document_id)+ assert document is not None, "the library sync deleted the document"+ assert document.storage_mode == "none"+ assert document.file_path == FILE_PATH_BLOB_DELETED+ assert document.text_content == (EXTRACTED_TEXT if text else None)+ assert _library_links(other, document_id) == 1+ assert find_tracker_by_url(other, url).is_downloaded is True+ assert library.get_document_by_id(document_id)["has_pdf"] is False++ assert (+ service.queue_research_downloads(+ world.research_id, LIBRARY_COLLECTION_ID+ )+ == 0+ )+ assert download_all_new(db, monkeypatch)["queued"] == 0+ assert service.download_resource(resource_id) == (True, None)+ assert len(downloader.urls) == 1+ assert _tree(store.root) == set()+++# ===========================================================================+# The deletion the sync exists for+# ===========================================================================++[email protected]("tracker_path", ["kept", "lost"])+def test_filesystem_download_whose_file_was_removed_is_deleted(+ db, store, world, filesystem_mode, monkeypatch, tracker_path+):+ """A download that had only its file is deleted once the file is gone.++ No text was extracted and there is no blob, so after the user removed+ the PDF from disk nothing of the document is left. The sync deletes it+ so that the URL can be downloaded again. ``lost`` is the R1 sequence+ (the tracker path cleared by a "none"-mode re-download) with the file+ removed afterwards, where the no-path branch must still delete.+ """+ url = "https://example.com/removed-from-disk.pdf"+ resource_id = world.add_resource(url)+ downloader = StubDownloader()+ service = make_service(store, downloader, storage_mode="filesystem")+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ document_id = document.id+ stored = store.root / document.file_path+ assert stored.is_file()+ library = make_library_service(db, store, monkeypatch)+ if tracker_path == "lost":+ assert library.mark_for_redownload([document_id]) == 1+ service.settings._values["research_library.pdf_storage_mode"] = "none"+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ assert find_tracker_by_url(other, url).file_path is None+ stored.unlink()++ stats = library.sync_library_with_filesystem()++ assert stats["files_missing"] == 1+ with db.fresh() as other:+ assert other.get(Document, document_id) is None+ assert _library_links(other, document_id) == 0+ tracker = find_tracker_by_url(other, url)+ assert tracker.file_path is None+ if tracker_path == "kept":+ assert tracker.is_downloaded is False+ assert stats["trackers_updated"] == 1+ assert [entry["id"] for entry in stats["missing_files"]] == [+ document_id+ ]++ # The point of the deletion: the URL downloads again.+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ redownloaded = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ assert redownloaded.id != document_id+ assert redownloaded.status is DocumentStatus.COMPLETED+ assert stored.is_file()+++def test_database_download_with_no_blob_and_no_text_is_deleted_and_reported(+ db, store, world, monkeypatch+):+ """A database-mode download with nothing left is deleted and reported.++ The document is a database-mode download that holds no blob and no+ text (see ``_add_database_row_without_pdf``). A second research+ downloaded the same bytes: the content-hash dedup linked its resource+ to the document and recorded ``"database"`` in the URL's tracker, as+ it does for a document labelled as stored in the database. Database+ storage leaves the document's ``file_path`` None, so while the tracker+ counted only for a document storing the same path it described no+ database document: the sync deleted this one, but neither reported it+ nor reset the tracker, which kept recording a PDF that no document+ holds. The tracker describes the document now. The sync deletes it,+ reports it as ``"database"`` and resets the tracker, and the URL+ downloads again. A file named ``database`` in the library root is not+ the document's PDF.+ """+ url = "https://example.com/database-row-with-nothing-left.pdf"+ resource_id = world.add_resource(url, downloaded=True)+ document_id = _add_database_row_without_pdf(+ db, world, resource_id, world.research_id, url, PDF_BYTES+ )+ _, linked_id = _cite_in_another_research(db, url)+ service = make_service(store, StubDownloader(), storage_mode="database")+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as other:+ assert other.get(ResearchResource, linked_id).document_id == (+ document_id+ )+ assert other.query(Document).count() == 1+ assert other.query(DocumentBlob).count() == 0+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)+ (store.root / "database").write_bytes(PDF_BYTES)+ library = make_library_service(db, store, monkeypatch)++ stats = library.sync_library_with_filesystem()++ assert (stats["files_found"], stats["files_missing"]) == (0, 1)+ assert stats["missing_files"] == [+ {+ "id": document_id,+ "title": "Sync Contract Paper",+ "path": "database",+ "url": url,+ }+ ]+ assert stats["trackers_updated"] == 1+ with db.fresh() as other:+ assert other.get(Document, document_id) is None+ assert _library_links(other, document_id) == 0+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == (None, False)++ # The point of the deletion: the URL downloads again.+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ redownloaded = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ assert redownloaded.id != document_id+ assert (+ other.query(DocumentBlob)+ .filter_by(document_id=redownloaded.id)+ .count()+ == 1+ )+ tracker = find_tracker_by_url(other, url)+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)+ stats = library.sync_library_with_filesystem()+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)+++# ===========================================================================+# What the sync returns+# ===========================================================================+++def test_sync_response_names_missing_files_by_their_stored_path(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ tmp_path,+):+ """The sync's response holds no absolute server path.++ ``POST /library/api/sync-library`` returns the sync's statistics, and+ each ``missing_files`` entry carried the absolute path its file+ resolved to on the server. That is the host's directory layout, which+ ``/library/api/check-downloads`` and the library listing withhold from+ the client (#6462). Every entry now gives the path stored for the file,+ relative to the library root: a download with text whose file is gone,+ reached through its tracker's path (``tracker-path``) or through its own+ after a "none"-mode re-download cleared the tracker's (``own-path``),+ and a download without text, which the sync deletes (``deleted``).+ """+ library = make_library_service(db, store, monkeypatch)+ service = make_service(store, StubDownloader(), storage_mode="filesystem")+ contents = {+ "tracker-path": TEXT_PDF_BYTES,+ "own-path": _text_pdf("Text of the paper downloaded again."),+ "deleted": PDF_BYTES,+ }+ stored = {}+ for case, content in contents.items():+ resource_id = world.add_resource(f"https://example.com/{case}.pdf")+ service.downloaders = [StubDownloader(content)]+ assert service.download_resource(resource_id) == (True, None)+ with db.fresh() as other:+ document = (+ other.query(Document).filter_by(resource_id=resource_id).one()+ )+ stored[document.id] = (case, document.file_path)+ if case == "own-path":+ assert library.mark_for_redownload([document.id]) == 1+ settings = service.settings._values+ settings["research_library.pdf_storage_mode"] = "none"+ assert service.download_resource(resource_id) == (True, None)+ settings["research_library.pdf_storage_mode"] = "filesystem"+ for _case, path in stored.values():+ (store.root / path).unlink()+ monkeypatch.setattr(+ library_router, "LibraryService", lambda username: library+ )++ stats = library_router.sync_library(None, username=USERNAME)++ body = json.dumps(stats)+ assert str(tmp_path) not in body, body+ assert stats["files_missing"] == 3+ assert {entry["id"]: entry["path"] for entry in stats["missing_files"]} == {+ document_id: path for document_id, (_case, path) in stored.items()+ }+ with db.fresh() as other:+ kept = {document.id for document in other.query(Document).all()}+ assert kept == {+ document_id+ for document_id, (case, _path) in stored.items()+ if case != "deleted"+ }+++ "entrypoint", ["research-queue", "all-new", "direct", "research-count"]+)+def test_canonical_refill_ignores_retained_old_owner(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ entrypoint,+):+ """A completed retained snapshot must not hide the resource's explicit target."""+ linked_url = "https://example.com/current-version.pdf"+ linked_id = world.add_resource(linked_url)+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ # First extraction fails; both references still retain the real old PDF.+ original_extract = service._extract_text_from_pdf+ service._extract_text_from_pdf = lambda _data: None+ assert service.download_resource(linked_id) == (True, None)+ _, mirror_id = _cite_in_another_research(+ db, "https://mirror.example.org/old-version.pdf", query="old version"+ )+ assert service.download_resource(mirror_id) == (True, None)+ with db.fresh() as session:+ old_id = session.get(ResearchResource, linked_id).document_id+ assert session.get(ResearchResource, mirror_id).document_id == old_id+ assert session.get(Document, old_id).text_content is None+ service._extract_text_from_pdf = original_extract+ owner_url = "https://example.org/new-version.pdf"+ _, owner_id = _cite_in_another_research(db, owner_url, query="new version")+ downloader.content = REGENERATED_PDF_BYTES+ assert service.download_resource(owner_id) == (True, None)+ with db.fresh() as session:+ canonical_id = session.get(ResearchResource, owner_id).document_id+ library = make_library_service(db, store, monkeypatch)+ assert library.mark_for_redownload([old_id]) == 1+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as session:+ assert (+ session.get(ResearchResource, linked_id).document_id == canonical_id+ )+ assert session.get(Document, old_id).status == DocumentStatus.PENDING+ # A later successful extraction through the old mirror completes the+ # retained snapshot, without changing the first resource's new link.+ downloader.content = TEXT_PDF_BYTES+ assert service.download_resource(mirror_id) == (True, None)+ with db.fresh() as session:+ old = session.get(Document, old_id)+ assert old.status == DocumentStatus.COMPLETED+ assert old.resource_id == linked_id+ assert (+ session.get(ResearchResource, linked_id).document_id == canonical_id+ )+ canonical = session.get(Document, canonical_id)+ assert canonical.text_content == REGENERATED_TEXT+ (store.root / canonical.file_path).unlink()+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ with db.fresh() as session:+ assert document_awaits_pdf(session.get(Document, canonical_id))+ assert (+ library_utils.get_document_for_resource(+ session, session.get(ResearchResource, linked_id)+ ).id+ == canonical_id+ )+ downloader.content = REGENERATED_PDF_BYTES+ if entrypoint == "research-queue":+ queued = service.queue_research_downloads(+ world.research_id,+ LIBRARY_COLLECTION_ID,+ run_research_ids=frozenset([world.research_id]),+ )+ assert queued == 1, (+ "old owned document hides explicit target awaiting refill"+ )+ elif entrypoint == "all-new":+ # The canonical owner's URL is denied now, but the linked URL works.+ answer = download_all_new(db, monkeypatch, refused_urls={owner_url})+ with db.fresh() as session:+ pending = [+ q.resource_id+ for q in session.query(DownloadQueue).filter_by(+ status=DocumentStatus.PENDING+ )+ ]+ assert pending == [linked_id], (answer, pending, linked_id)+ elif entrypoint == "research-count":+ row = next(+ row+ for row in library.get_research_list_with_stats()+ if row["id"] == world.research_id+ )+ assert row["downloaded_count"] == 0, row+ else:+ before = len(downloader.urls)+ assert service.download_resource(linked_id) == (True, None)+ with db.fresh() as session:+ assert not document_awaits_pdf(+ session.get(Document, canonical_id)+ ), (+ "success returned without refilling the canonical PDF",+ downloader.urls[before:],+ )diff --git a/tests/research_library/test_library_sync_integration.py b/tests/research_library/test_library_sync_integration.pynew file mode 100644index 0000000000..56f2f07681--- /dev/null+++ b/tests/research_library/test_library_sync_integration.py@@ -0,0 +1,212 @@+"""Library sync composes with canonical PDF reuse and text-index invalidation."""++import hashlib+from types import SimpleNamespace++import pytest++from local_deep_research.database.models.library import (+ Collection,+ Document,+ DocumentCollection,+ DocumentStatus,+ DownloadQueue,+ RagDocumentStatus,+)+from local_deep_research.database.models.research import ResearchResource+from local_deep_research.research_library.services import library_service+from local_deep_research.research_library.services.library_rag_service import (+ LibraryRAGService,+)+from local_deep_research.research_library.services.pdf_storage_manager import (+ PDFStorageManager,+)+from local_deep_research.research_library.utils import document_awaits_pdf+from tests.research_library.services.test_download_text_quality import (+ LONGER,+ extraction_case as extraction_case,+)+from tests.research_library.services.test_text_index_invalidation import (+ assert_pending,+ indexer_for,+ persist,+)+from tests.research_library.test_library_sync_contracts import (+ EXTRACTED_TEXT,+ REGENERATED_PDF_BYTES,+ REGENERATED_TEXT,+ TEXT_PDF_BYTES,+ StubDownloader,+ db as db,+ download_all_new,+ filesystem_mode as filesystem_mode,+ make_library_service,+ make_service,+ one_download_session as one_download_session,+ store as store,+ world as world,+)++[email protected]("changed", [False, True])+ "storage,limit", [("database", 3072), ("none", 3072), ("database", 0)]+)+def test_linked_refill_settles_storage_and_requested_collection(+ db,+ store,+ world,+ one_download_session,+ filesystem_mode,+ monkeypatch,+ changed,+ storage,+ limit,+):+ owner = world.add_resource("https://example.com/refill-owner.pdf")+ downloader = StubDownloader(TEXT_PDF_BYTES)+ service = make_service(store, downloader, storage_mode="filesystem")+ assert service.download_resource(owner) == (True, None)+ with db.fresh() as session:+ document = session.query(Document).filter_by(resource_id=owner).one()+ document_id = document.id+ (store.root / document.file_path).unlink()+ library = make_library_service(db, store, monkeypatch)+ assert library.sync_library_with_filesystem()["files_missing"] == 1++ linked = world.add_resource("https://example.com/refill-mirror.pdf")+ db.session.get(ResearchResource, linked).document_id = document_id+ db.session.add(Collection(id="requested", name="Requested"))+ db.session.flush()+ db.session.add(+ DownloadQueue(+ resource_id=linked,+ research_id=world.research_id,+ collection_id="requested",+ status=DocumentStatus.PENDING,+ )+ )+ db.session.commit()+ content = REGENERATED_PDF_BYTES if changed else TEXT_PDF_BYTES+ downloader.content = content+ service.settings._values.update(+ {+ "research_library.pdf_storage_mode": storage,+ "research_library.max_pdf_size_mb": limit,+ }+ )++ assert service.download_resource(linked) == (True, None)+ with db.fresh() as session:+ assert session.query(Document).count() == 1+ document = session.get(Document, document_id)+ assert document.document_hash == hashlib.sha256(content).hexdigest()+ assert document.text_content == (+ REGENERATED_TEXT if changed else EXTRACTED_TEXT+ )+ assert not document_awaits_pdf(document)+ assert (+ session.query(DocumentCollection)+ .filter_by(+ document_id=document_id,+ collection_id="requested",+ )+ .count()+ == 1+ )+ assert (+ session.query(DownloadQueue)+ .filter_by(+ resource_id=linked,+ )+ .one()+ .status+ is DocumentStatus.COMPLETED+ )+ manager = PDFStorageManager(store.root, storage)+ if storage == "database" and limit:+ assert manager.load_pdf(document, session) == content+ else:+ assert document.storage_mode == "none"+ assert document.file_path is None+ assert not manager.has_pdf(document, session)+ assert downloader.urls == [+ "https://example.com/refill-owner.pdf",+ "https://example.com/refill-mirror.pdf",+ ]+ assert download_all_new(db, monkeypatch)["queued"] == 0+ assert service.queue_research_downloads(world.research_id, "requested") == 0++[email protected]("mode", ["serial", "prepared"])+def test_regenerated_refill_keeps_old_vectors_invalid_until_reindexed(+ extraction_case,+ monkeypatch,+ tmp_path,+ mode,+):+ case = extraction_case(quality="low", incoming=LONGER, storage="database")+ base = tmp_path / "library"+ case.service.library_root = base / case.service.username+ with case.sessions() as session:+ document = session.get(Document, "canonical")+ document.original_url = "https://example.org/old-paper.pdf"+ document.document_hash = hashlib.sha256(+ b"previous PDF bytes"+ ).hexdigest()+ document.storage_mode = "filesystem"+ document.file_path = "pdfs/missing.pdf"+ session.get(+ ResearchResource, case.resource_id+ ).document_id = document.id+ session.commit()+ monkeypatch.setattr(+ library_service, "get_user_db_session", lambda *a, **k: case.sessions()+ )+ monkeypatch.setattr(+ library_service,+ "get_settings_manager",+ lambda **kw: SimpleNamespace(+ get_setting=lambda key, default=None: str(base),+ close=lambda: None,+ ),+ )+ library = library_service.LibraryService.__new__(+ library_service.LibraryService+ )+ library.username = case.service.username+ assert library.sync_library_with_filesystem()["files_missing"] == 1+ with case.sessions() as session:+ assert document_awaits_pdf(session.get(Document, "canonical"))++ indexer, backend = indexer_for(case)+ old_ids = backend._store.ids.copy()+ assert case.service.download_resource(case.resource_id) == (True, None)+ with case.sessions() as session:+ assert session.query(Document).count() == 1+ document = session.get(Document, "canonical")+ assert document.document_hash == case.pdf_hash+ assert not document_awaits_pdf(document)+ assert_pending(case)+ assert_pending(case, "selected")+ assert (+ LibraryRAGService.reconcile_collection_index(indexer, "library")[+ "indexed_documents"+ ]+ == 0+ )+ assert_pending(case)+ assert persist(indexer, mode)["status"] == "success"+ assert backend._store.removed == old_ids+ assert old_ids.isdisjoint(backend._store.ids)+ assert (+ LibraryRAGService.reconcile_collection_index(indexer, "library")[+ "indexed_documents"+ ]+ == 1+ )+ with case.sessions() as session:+ assert (+ session.get(RagDocumentStatus, ("canonical", "library")) is not None+ )+ assert_pending(case, "selected")diff --git a/tests/security/test_egress_policy.py b/tests/security/test_egress_policy.pyindex c4b285fb60..4acd261c93 100644--- a/tests/security/test_egress_policy.py+++ b/tests/security/test_egress_policy.py@@ -2807,6 +2807,43 @@ def test_security_denials_still_exhaust_fetch_quota():assert decision.reason == "denial_quota_exceeded"+ ("url", "reason"),+ [+ ("https://private.example/page", "scope_mismatch_public_only"),+ ("http://169.254.169.254/latest/meta-data/", "blocked_metadata_ip"),+ ],+)+def test_unrecorded_denial_is_the_same_decision_and_records_nothing(+ url, reason+):+ """``record_denial=False`` decides as a fetch's check does, recording nothing.++ For a caller that only ranks URLs and fetches none of them+ (``DownloadService.refill_rank``): the denial is neither counted toward+ the run's denied-fetch quota, however often it is given, nor written to+ the audit log. The default records the same denial.+ """+ from local_deep_research.security.egress import policy as policy_mod++ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)+ with (+ patch.object(policy_mod, "_classify_host", return_value=True),+ patch.object(policy_mod, "logger") as audit_log,+ ):+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 1):+ assert evaluate_url(url, ctx, record_denial=False) == Decision(+ False, reason+ )+ assert audit_log.mock_calls == []+ assert ctx._fetch_denial_count["count"] == 0++ assert evaluate_url(url, ctx) == Decision(False, reason)+ audit_log.bind.assert_called_once_with(policy_audit=True)+ audit_log.bind.return_value.warning.assert_called_once()+ assert ctx._fetch_denial_count["count"] == 1++def test_allowed_local_hostnames_accepts_unresolvable():"""A legitimate intranet/VPN hostname that does not resolve (DNS down /split-horizon) must be ACCEPTED on save (fail-open), not rejected — thediff --git a/tests/web/routers/test_download_queue_claim_routes_4691.py b/tests/web/routers/test_download_queue_claim_routes_4691.pyindex 3c34dc5ff2..a43309fb8b 100644--- a/tests/web/routers/test_download_queue_claim_routes_4691.py+++ b/tests/web/routers/test_download_queue_claim_routes_4691.py@@ -557,10 +557,11 @@ def _run_download_source(Session, ds_mock=None):def test_download_source_does_not_reset_an_in_flight_claim(tmp_path, request):"""``download_source`` resets the existing queue row to PENDING and thendownloads immediately in the request thread. Its only early return is- ``existing and existing.status == "completed"``, which is inert during the- race window: an in-flight bulk download has not written a completed- Document yet. So the reset itself must refuse a PROCESSING row -- the row- stays PROCESSING, no second download starts, and the caller gets 409.+ for a Document that counts as downloaded (``document_counts_as_downloaded``:+ completed, and not awaiting its PDF), which is inert during the race+ window: an in-flight bulk download has not written a completed Document+ yet. So the reset itself must refuse a PROCESSING row -- the row stays+ PROCESSING, no second download starts, and the caller gets 409.This is the reset-lost path. ``test_library_download_outcomes.py``'s``test_a_lost_claim_is_a_409_not_a_completion`` passes ``claim_result=0``diff --git a/tests/web/routers/test_library_api_response_shapes.py b/tests/web/routers/test_library_api_response_shapes.pyindex 605a1b51c2..0ae13dd3ee 100644--- a/tests/web/routers/test_library_api_response_shapes.py+++ b/tests/web/routers/test_library_api_response_shapes.py@@ -51,6 +51,11 @@ from unittest.mock import MagicMock, Mock, patchimport pytestfrom starlette.requests import Request+from local_deep_research.constants import FILE_PATH_TEXT_ONLY+from local_deep_research.database.models.library import (+ Document,+ DocumentStatus,+)from local_deep_research.web.routers import library as library_modulefrom local_deep_research.web.routers.library import (check_downloads,@@ -766,16 +771,38 @@ def _resource(rid=1, url="https://arxiv.org/abs/2301.00001", title="Paper"):return row-def _completed_document():- doc = Mock()- doc.id = "doc-1"- doc.status = "completed"- doc.file_type = "pdf"- doc.title = "Paper"- # The absolute server path. Present on the row the handler holds, so the- # only thing keeping it out of the response is the handler not copying it.- doc.file_path = "/srv/library/alice/pdfs/2301.00001.pdf"- return doc+def _completed_document(**columns):+ """A completed download of resource 1, as the handlers load it.++ A real ``Document``: its ``status`` is a ``DocumentStatus`` member, as on+ a loaded row, which the string ``"completed"`` never equals.+ """+ values = {+ "id": "doc-1",+ "resource_id": 1,+ "status": DocumentStatus.COMPLETED,+ "file_type": "pdf",+ "title": "Paper",+ "storage_mode": "filesystem",+ # The absolute server path. Present on the row the handler holds, so+ # the only thing keeping it out of the response is the handler not+ # copying it.+ "file_path": "/srv/library/alice/pdfs/2301.00001.pdf",+ }+ values.update(columns)+ return Document(**values)+++def _document_awaiting_its_pdf(**columns):+ """A completed download that "Sync library" left awaiting its PDF.++ Its PDF file was removed while its text was not, so the sync stored it+ the way text-only documents are and reported the file missing (see+ ``document_awaits_pdf``). It does not count as downloaded.+ """+ return _completed_document(+ storage_mode="none", file_path=FILE_PATH_TEXT_ONLY, **columns+ )def _run_check_downloads(session, document, payload):@@ -845,6 +872,27 @@ def test_check_downloads_reports_an_undownloaded_resource_by_id():}+def test_check_downloads_reports_a_document_awaiting_its_pdf_by_id():+ """A completed download that awaits its PDF is not downloaded.++ The Download Manager counts it as available to download, so the client+ gets the ``resource_id`` to download its PDF by, not the document.+ """+ resource = _resource()+ document = _document_awaiting_its_pdf()++ result = _run_check_downloads(+ _check_downloads_session(resource, document),+ document,+ {"research_id": "r1", "urls": [resource.url]},+ )++ assert result["download_status"][resource.url] == {+ "downloaded": False,+ "resource_id": 1,+ }++@pytest.mark.parametrize("payload",[@@ -928,11 +976,7 @@ def _source_resource(rid, url, title, preview="Preview text", score=0.95):def test_research_sources_carries_the_download_state_of_each_source():resource = _source_resource(1, "https://arxiv.org/abs/1", "Test Paper")- document = Mock()- document.id = "doc-1"- document.status = "completed"- document.file_type = "pdf"- document.created_at = datetime(2026, 5, 9)+ document = _completed_document(created_at=datetime(2026, 5, 9))with _patched(session=_sources_session([resource]),@@ -949,6 +993,26 @@ def test_research_sources_carries_the_download_state_of_each_source():assert source["downloaded"] is Trueassert source["document_id"] == "doc-1"assert source["file_type"] == "pdf"+ assert source["download_date"] == "2026-05-09T00:00:00"+++def test_research_sources_marks_a_document_awaiting_its_pdf_not_downloaded():+ """A completed download that awaits its PDF is not downloaded, as the+ Download Manager counts it, and the source names no document."""+ resource = _source_resource(1, "https://arxiv.org/abs/1", "Test Paper")+ document = _document_awaiting_its_pdf(created_at=datetime(2026, 5, 9))++ with _patched(+ session=_sources_session([resource]),+ get_document_for_resource=Mock(return_value=document),+ ):+ result = get_research_sources(_get_request(), "r1", username="alice")++ source = result["sources"][0]+ assert source["downloaded"] is False+ assert source["document_id"] is None+ assert source["file_type"] is None+ assert "download_date" not in sourcedef test_research_sources_marks_an_undownloaded_source():diff --git a/tests/web/routers/test_library_download_outcomes.py b/tests/web/routers/test_library_download_outcomes.pyindex 63323fd2ca..dac51411a8 100644--- a/tests/web/routers/test_library_download_outcomes.py+++ b/tests/web/routers/test_library_download_outcomes.py@@ -9,13 +9,18 @@ appeared in no test on the branch.Seven guarantees, all of them things a user reads or a template branches on:* ``"Already downloaded"`` -- ``/library/api/download-source`` short-circuits- only when a **completed** Document already exists for the resource. The- guard is ``existing and existing.status == "completed"``, and the status- half is load-bearing: a resource whose previous attempt is still ``pending``- or came back ``failed`` must be re-downloaded, not reported as already in- the library. Drop the status check and every failed download becomes- permanently unretryable through this endpoint, with the UI cheerfully- reporting success.+ only when the resource's Document counts as downloaded: it is+ **completed** and does not await its PDF (``document_counts_as_downloaded``).+ The status half is load-bearing: a resource whose previous attempt is still+ ``pending`` or came back ``failed`` must be re-downloaded, not reported as+ already in the library. Drop the status check and every failed download+ becomes permanently unretryable through this endpoint, with the UI+ cheerfully reporting success. A loaded status is a ``DocumentStatus``+ member, which the string ``"completed"`` never equals, so the documents+ here are real ``Document`` rows with member statuses: compared with that+ string, the guard never short-circuited. A completed document that awaits+ its PDF (the library sync kept its text after its PDF file went missing)+ is downloaded again.* ``"Download completed"`` / ``"Download failed"`` -- the two terminalmessages of the same endpoint. ``DownloadService.download_resource``returns ``(success, message)`` where ``message`` is an internal diagnostic@@ -62,6 +67,11 @@ import pytestfrom bs4 import BeautifulSoupfrom starlette.requests import Request+from local_deep_research.constants import FILE_PATH_TEXT_ONLY+from local_deep_research.database.models.library import (+ Document,+ DocumentStatus,+)from local_deep_research.research_library.services import (library_service as library_service_module,)@@ -213,11 +223,22 @@ def _queue_entry(queue_id=42):return row-def _document(document_id="doc-1", status="completed"):- doc = Mock()- doc.id = document_id- doc.status = status- return doc+def _document(document_id="doc-1", status=DocumentStatus.COMPLETED, **columns):+ """The resource's ``Document``, as ``get_document_for_resource`` loads it.++ A real row, whose ``status`` is a ``DocumentStatus`` member. By default+ a completed download of resource 1 with its PDF stored in the database.+ """+ values = {+ "id": document_id,+ "resource_id": 1,+ "status": status,+ "file_type": "pdf",+ "storage_mode": "database",+ "file_path": None,+ }+ values.update(columns)+ return Document(**values)def _download_service(result):@@ -780,7 +801,7 @@ class TestDocumentDetailsPositiveControl:class TestAlreadyDownloaded:def test_a_completed_document_short_circuits_with_its_id(self):response, session, service = _call_download_source(- existing_document=_document("doc-1", status="completed")+ existing_document=_document("doc-1"))assert response == {@@ -792,7 +813,7 @@ class TestAlreadyDownloaded:def test_the_short_circuit_does_no_work(self):"""The whole point of the branch: no second fetch, no queue row."""_response, session, service = _call_download_source(- existing_document=_document("doc-1", status="completed")+ existing_document=_document("doc-1"))service.download_resource.assert_not_called()@@ -800,12 +821,18 @@ class TestAlreadyDownloaded:assert session.queue_query.updates == []@pytest.mark.parametrize(- "status", ["pending", "processing", "failed", "in_progress", ""]+ "status",+ [+ DocumentStatus.PENDING,+ DocumentStatus.PROCESSING,+ DocumentStatus.FAILED,+ ],+ ids=lambda status: status.value,)def test_a_non_completed_document_is_re_downloaded(self, status):- """``existing.status == "completed"`` is the load-bearing half of the- guard. A previous attempt that failed or is stuck must not be- reported to the user as already in their library."""+ """The completed status is the load-bearing half of the guard. A+ previous attempt that failed or is stuck must not be reported to the+ user as already in their library."""response, _session, service = _call_download_source(existing_document=_document("doc-1", status=status),download_result=(True, "saved 1 file"),@@ -815,6 +842,21 @@ class TestAlreadyDownloaded:assert response == {"success": True, "message": "Download completed"}service.download_resource.assert_called_once_with(1)+ def test_a_document_that_awaits_its_pdf_is_downloaded_again(self):+ """A completed download that "Sync library" left awaiting its PDF+ does not count as downloaded: its PDF file was removed, the sync kept+ its text, stored it text-only and reported the file missing (see+ ``document_awaits_pdf``). The download puts its PDF back."""+ response, _session, service = _call_download_source(+ existing_document=_document(+ "doc-1", storage_mode="none", file_path=FILE_PATH_TEXT_ONLY+ ),+ download_result=(True, None),+ )++ assert response == {"success": True, "message": "Download completed"}+ service.download_resource.assert_called_once_with(1)+def test_no_existing_document_at_all_is_not_already_downloaded(self):response, _session, _service = _call_download_source(existing_document=Nonediff --git a/tests/web/routers/test_library_queue_all_undownloaded.py b/tests/web/routers/test_library_queue_all_undownloaded.pyindex 0fd16d39a8..3dc8b533c1 100644--- a/tests/web/routers/test_library_queue_all_undownloaded.py+++ b/tests/web/routers/test_library_queue_all_undownloaded.py@@ -72,6 +72,11 @@ class _ResourceQuery:def outerjoin(self, *args, **kwargs):return self+ def join(self, *args, **kwargs):+ # The lookup of resources whose document awaits its PDF+ # (``_one_resource_per_awaiting_document``) finds none here.+ return _ResourceQuery([])+def filter(self, *args, **kwargs):return self