Skip to content

~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.

fix(library): keep documents that still hold content during library sync (#7246)

LearningCircuit4d30f5205df4
diff --git a/changelog.d/+library-sync-keeps-content.bugfix.md b/changelog.d/+library-sync-keeps-content.bugfix.md
new file mode 100644
index 0000000000..396ee93b2a
--- /dev/null
+++ b/changelog.d/+library-sync-keeps-content.bugfix.md
@@ -0,0 +1,89 @@
+"Sync library" no longer deletes downloaded documents that still hold
+content. It deleted a document whose PDF it could not find even when the
+document still had its extracted text or a PDF stored in the database, and
+the text, stored PDF, collection memberships and search index entries went
+with it. This hit documents whose PDF had been removed to save space,
+documents re-downloaded with PDF storage set to "none", failed downloads
+later completed with text, documents labelled as stored in the database
+without a stored PDF, and Zotero imports with a source URL or DOI whose PDF
+is stored in the database.
+
+A document whose PDF file was removed from disk but whose extracted text
+was not now keeps its text, collection memberships and search index
+entries. If its research still has the resource it was downloaded for, the
+sync counts it under "Files Missing" and makes it a text-only document, so
+the library no longer offers a PDF that is not there, and the Download
+Manager counts it as available to download rather than as already
+downloaded. The Download Manager's PDF downloads then put its PDF back into
+the same document, also when the PDF downloaded again differs from the one
+removed. Each download run downloads it once, also when several of its
+researches cite the paper; two runs over different researches at the same
+time can each download it. The run downloads it through the document's
+own resource, or through another resource for the same paper when the own
+one cannot succeed: its URL is not downloadable or is refused by the
+egress policy, the retry manager holds it back, or its latest download
+attempt failed. A download that a restart of the server cut off does not
+keep the document's other resources from downloading its PDF. "Extract
+Text to DB" skips such a document, since it already has the text. If the
+resource is gone, the sync still makes the document text-only but does
+not count it under "Files Missing", because the Download Manager has no
+resource to download its PDF from. A later download of the identical PDF
+for another research then stores the PDF in it. A document whose PDF file
+was removed and that has no extracted text and no PDF stored elsewhere is
+still deleted, so it can be downloaded again. A document whose PDF was
+removed with "Remove PDF to save space" is kept as it is, and its PDF is
+not offered for download again.
+
+When a resource links to a newer document, an older retained document
+cannot hide the newer document's missing PDF. Download queues, direct
+refills and download counts follow the explicit link, while preserving
+the older document for resources that still use it. Cached PDF text
+extraction follows the same link, so it cannot report success while the
+linked document still lacks text.
+
+Refills preserve canonical document reuse and retry bookkeeping. A regenerated
+PDF updates the document whose file was lost and joins the requested collection;
+if another document owns those bytes, the refill reports a conflict and keeps
+both documents intact. Refills that cannot store a PDF because storage is disabled
+or its size limit is exceeded settle as text-only downloads. When accepted text
+changes, its old search entries stay invalid until reindexing finishes.
+
+The sync now checks each document against its own PDF: its file, or its
+blob when it stores the PDF in the database. Several documents can share
+a URL, such as the documents of two researches that cite the same paper.
+When another document with the same URL had stored its PDF later, a
+document whose file was removed counted as found as long as that other
+file was still there, so its missing PDF was never reported or downloaded
+again.
+
+A failed download that a later PDF download completes is now recorded as a
+PDF document, and so is a document that "Extract Text to DB" created once
+a later PDF download stores its PDF in it, so library search results link
+to their PDFs.
+
+In the Download Manager, the "downloaded" counts under a research's
+"Sources by Domain" and the "Downloaded" badges of its PDFs now show the
+PDFs that are downloaded. They compared the stored status with a string
+it never equals, so they showed none as downloaded. Five more places made
+the same comparison and now read the status correctly:
+
+- `GET /library/api/get-research-sources/{research_id}` and
+ `POST /library/api/check-downloads` reported every source as not
+ downloaded. They now report a source as downloaded when the Download
+ Manager counts its document as downloaded, which a document that awaits
+ its PDF is not.
+- `POST /library/api/download-source` never answered "Already
+ downloaded". For a downloaded source it queued the source again and
+ answered "Download completed". It now answers "Already downloaded" when
+ the Download Manager counts the source's document as downloaded, and
+ downloads the PDF of a document that awaits it.
+- The document details page showed the "Download Status" of every
+ document as a warning badge reading "Documentstatus.completed" or the
+ like. It now shows "Completed", or another status such as "Failed" in a
+ warning badge.
+- Text extraction, such as "Text Only (from PDFs)" or the document
+ scheduler's, never read the PDF file that a downloaded document without
+ text keeps on disk. It asked the source for the text or the PDF again,
+ or gave up when the retry manager held the source back. It now extracts
+ the text from that file before it asks the source. For an arXiv paper
+ the text is still requested from arXiv first, as before.
diff --git a/changelog.d/+library-sync-missing-file-paths.security.md b/changelog.d/+library-sync-missing-file-paths.security.md
new file mode 100644
index 0000000000..3270d407b8
--- /dev/null
+++ b/changelog.d/+library-sync-missing-file-paths.security.md
@@ -0,0 +1,10 @@
+"Sync library" (`POST /library/api/sync-library`) no longer returns the
+absolute server-side path of a missing file. Each `missing_files` entry
+carried the resolved location of the file on the server, which describes
+the host's directory layout rather than the document. It now gives the
+path stored for the file, relative to the library root, as
+`GET /library/api/documents` does with `file_path`, or `invalid` when that
+path does not resolve inside the library root. For a document whose PDF was
+stored in the database it gives `database`. This matches what
+`/library/api/check-downloads` and, since #6462, the library document API
+do.
diff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.py
index e4e24240e7..c86ddb37c5 100644
--- a/src/local_deep_research/research_library/services/download_service.py
+++ b/src/local_deep_research/research_library/services/download_service.py
@@ -16,14 +16,15 @@ import time
import uuid
from datetime import datetime, UTC
from pathlib import Path
-from typing import Any, Dict, Optional, Tuple, assert_never
+from collections.abc import Set as AbstractSet
+from typing import Any, Dict, NamedTuple, Optional, Tuple, assert_never
from urllib.parse import urlparse
import requests
from loguru import logger
from ...security import redact_url_for_log
-from sqlalchemy import func
-from sqlalchemy.orm import Session, load_only
+from sqlalchemy import and_, func, or_
+from sqlalchemy.orm import Session, defer, load_only
import pdfplumber
from pypdf import PdfReader
@@ -73,9 +74,14 @@ from ...database.library_init import get_source_type_id, get_default_library_id
from ...database.session_context import get_user_db_session, safe_rollback
from ...utilities.db_utils import get_settings_manager
from ...library.download_management import RetryManager
+from ...library.download_management.models import (
+ DownloadStatus as RetryStatus,
+)
from ...config.paths import get_library_directory
from ..utils import (
apply_user_subdir,
+ document_awaits_pdf,
+ download_claim_is_live,
ensure_in_collection,
find_tracker_by_url,
get_document_for_resource,
@@ -111,6 +117,41 @@ from ...utilities.pdf_extraction_limits import (
)
+class RefillRank(NamedTuple):
+ """Where a resource stands among the resources that refill a document.
+
+ A document that awaits its PDF (see ``document_awaits_pdf``) is
+ refilled by a download of its own resource (``Document.resource_id``)
+ or of a resource linked to it by content
+ (``ResearchResource.document_id``), and one download refills it. Unless
+ one of them is already being downloaded or queued, the queueing paths
+ give that download to the resource whose rank sorts first (see
+ ``DownloadService._refilled_by_another_resource``). The fields sort in
+ order, False before True.
+ """
+
+ #: Its URL cannot be queued: it is not downloadable, or the egress
+ #: policy refuses it.
+ cannot_queue: bool
+ #: The retry manager holds it back: a cooldown, a permanent failure or
+ #: a retry limit.
+ held_back: bool
+ #: Its latest download attempt failed.
+ failed_last: bool
+ #: It is not the document's own resource.
+ linked: bool
+ resource_id: int
+
+ @property
+ def viable(self) -> bool:
+ """Whether nothing known stands in the way of its download.
+
+ Its URL can be queued, the retry manager does not hold it back,
+ and its latest download attempt did not fail.
+ """
+ return not (self.cannot_queue or self.held_back or self.failed_last)
+
+
def _text_extraction_quality(method: str | None, source: str | None) -> str:
if method in {"native_api", "html_extraction"}:
return "high"
@@ -484,12 +525,21 @@ class DownloadService:
self._policy_locked = True
return None
- def _check_url_against_policy(self, url: str) -> Tuple[bool, str]:
+ def _check_url_against_policy(
+ self, url: str, *, record_denial: bool = True
+ ) -> Tuple[bool, str]:
"""Run evaluate_url against self._egress_context. Returns
(allowed, reason). When no context is configured (legacy
callers OR tests that mock __init__), returns (True, "no_context")
so back-compat callers aren't broken — but new callers passing
a snapshot DO get gated. Logged to policy_audit on denial.
+
+ ``record_denial=False`` returns the same decision without recording
+ a denial: it is not logged, here or by ``evaluate_url``, and does
+ not count toward the run's denied-fetch quota. Only ``refill_rank``
+ passes it, since it fetches nothing on the decision. Every other
+ caller checks a URL before it is queued or fetched, and records its
+ denials.
"""
# ``_policy_locked`` is set by _build_egress_context when a
# snapshot was supplied but the policy itself raised. Without
@@ -512,8 +562,10 @@ class DownloadService:
from ...security.egress.policy import evaluate_url
from ...security.ssrf_validator import redact_url_for_log
- decision = evaluate_url(url, egress_context)
- if not decision.allowed:
+ decision = evaluate_url(
+ url, egress_context, record_denial=record_denial
+ )
+ if not decision.allowed and record_denial:
logger.bind(policy_audit=True).warning(
"DownloadService URL denied by egress policy",
# Redact userinfo creds / query-string tokens from the log.
@@ -635,7 +687,12 @@ class DownloadService:
return None
def queue_research_downloads(
- self, research_id: str, collection_id: Optional[str] = None
+ self,
+ research_id: str,
+ collection_id: Optional[str] = None,
+ *,
+ text_only: bool = False,
+ run_research_ids: Optional[AbstractSet[str]] = None,
) -> int:
"""
Queue all downloadable PDFs from a research session.
@@ -643,6 +700,19 @@ class DownloadService:
Args:
research_id: The research session ID
collection_id: Optional target collection ID (defaults to Library if not provided)
+ text_only: True when the queued resources only get their text
+ extracted (``download_bulk``'s "text_only" mode). A completed
+ document that awaits its PDF (see ``document_awaits_pdf``)
+ already holds its text, so it then counts as done, like any
+ completed document. A text extraction would not fetch its PDF
+ either, so it would be queued and reported again on every run.
+ run_research_ids: The research sessions whose queued downloads
+ the same run downloads, as a set (``download_bulk`` builds
+ one from its ``research_ids`` for the whole run and passes it
+ for each research), or None when nothing downloads the queue
+ right away. It is only looked up, never copied, unless it
+ lacks ``research_id``. Decides which resource refills a
+ document that awaits its PDF, see the notes.
Returns:
Number of items queued
@@ -650,14 +720,44 @@ class DownloadService:
Notes:
Resets existing FAILED/COMPLETED queue entries for the research back
to PENDING, effectively retrying them. Resources that already have a
- PENDING queue entry or a COMPLETED Document are skipped. As of #4685,
+ PENDING queue entry or a COMPLETED Document are skipped, except a
+ COMPLETED Document that awaits its PDF when PDFs are queued. As of #4685,
``download_bulk`` calls this unconditionally on every bulk run, so
previously-failed downloads are automatically retried each time.
A PROCESSING entry is in flight in another ``download_bulk`` stream,
so it is left alone rather than reset: resetting it would let this
run re-claim and re-download it (#4691).
+
+ A document that awaits its PDF can be refilled through its own
+ resource or any resource linked to it by content
+ (``resource.document_id``), and one download refills it. So a
+ run leaves the document to one of them (see
+ ``_refilled_by_another_resource``): to one that a download holds,
+ else to one already queued for the run, unless that one is not
+ viable and another ranks before it, else to the one that ranks
+ first in ``refill_rank``. That is the document's own resource
+ unless it cannot succeed (its URL cannot be queued, the retry
+ manager holds it back, or its latest attempt failed) while a
+ linked one can. Once the document holds its PDF again,
+ ``download_resource`` reports the document's own resource as
+ already downloaded, and for a linked one whose URL the egress
+ policy allows, ``_download_pdf`` reuses that stored PDF instead
+ of fetching it. Of this research's resources the document's own
+ one is considered first.
"""
queued = 0
+ # The research sessions whose queue the same run downloads (see
+ # _refilled_by_another_resource). download_bulk passes one set for
+ # the whole run, so it is only looked up here: copying it for each
+ # research made the bulk pre-pass quadratic in the number of
+ # research sessions.
+ run = run_research_ids
+ if run is not None and research_id not in run:
+ run = run | {research_id}
+ # The refill_rank of each resource compared below, by resource and
+ # document owner, so that the egress policy and the retry manager
+ # are asked once per resource in this call.
+ ranks: Dict[Tuple[int, Optional[int]], RefillRank] = {}
# Get default library collection if no collection_id provided
if not collection_id:
@@ -673,11 +773,53 @@ class DownloadService:
.all()
)
- for resource in resources:
+ # A document's own resource before the resources linked to it
+ # by content, so that the own resource of a document awaiting
+ # its PDF is considered first when this research has both. A
+ # resource can also point to its own document through
+ # document_id. The sort is stable and leaves the other
+ # resources in order. It reads the owner of each document these
+ # resources point to, in one query of bare columns: loading the
+ # documents would load their text (#4560).
+ owner_of_linked_document = (
+ dict(
+ session.query(ResearchResource.id, Document.resource_id)
+ .join(Document, ResearchResource.document_id == Document.id)
+ .filter(ResearchResource.research_id == research_id)
+ .all()
+ )
+ if any(res.document_id for res in resources)
+ else {}
+ )
+
+ def linked_to_another_resources_document(res) -> bool:
+ return bool(res.document_id) and (
+ owner_of_linked_document.get(res.id) != res.id
+ )
+
+ for resource in sorted(
+ resources, key=linked_to_another_resources_document
+ ):
if self._is_downloadable(resource):
- # Library resources linked via document_id are already done
+ # The document awaiting its PDF that this resource would
+ # refill, if any (see document_awaits_pdf).
+ awaiting = None
+ # Library resources linked via document_id are already
+ # done, unless PDFs are queued and the linked document
+ # still needs its PDF (see document_awaits_pdf).
if resource.document_id:
- continue
+ # Not its text, which nothing here reads (#4560).
+ linked = (
+ None
+ if text_only
+ else session.query(Document)
+ .options(defer(Document.text_content))
+ .filter(Document.id == resource.document_id)
+ .first()
+ )
+ if linked is None or not document_awaits_pdf(linked):
+ continue
+ awaiting = linked
# Egress policy gate before queueing.
allowed, reason = self._check_url_against_policy(
@@ -700,18 +842,45 @@ class DownloadService:
.first()
)
- # Check if already downloaded (trust the database status)
+ # Check if already downloaded (trust the database status).
+ # When PDFs are queued, a completed document that still
+ # needs its PDF does not count (see document_awaits_pdf).
+ # An explicit link is authoritative; an older retained
+ # document can still name this resource as its owner.
existing_doc = (
- session.query(Document)
+ awaiting
+ if resource.document_id
+ else session.query(Document)
.filter_by(
resource_id=resource.id,
status=DocumentStatus.COMPLETED,
)
.first()
)
+ if (
+ not text_only
+ and existing_doc is not None
+ and document_awaits_pdf(existing_doc)
+ ):
+ awaiting = existing_doc
+ existing_doc = None
# Queue if not already queued and not marked as completed
if not existing_queue and not existing_doc:
+ # One download refills a document awaiting its PDF:
+ # leave it to the resource that does.
+ if (
+ awaiting is not None
+ and self._refilled_by_another_resource(
+ session,
+ awaiting,
+ resource,
+ run,
+ research_id,
+ ranks,
+ )
+ ):
+ continue
# Check one more time if ANY queue entry exists (regardless of status)
any_queue = (
session.query(LibraryDownloadQueue)
@@ -772,6 +941,166 @@ class DownloadService:
"""
return is_downloadable_url(resource.url)
+ def refill_rank(
+ self, resource_id: int, url: Optional[str], owner_id: Optional[int]
+ ) -> RefillRank:
+ """The rank of a resource that would refill a document awaiting its PDF.
+
+ ``owner_id`` is the document's own resource (``Document.resource_id``).
+ See ``RefillRank``: a viable resource ranks before one that is not,
+ and among viable resources the document's own one ranks first. Of
+ the others, one that can be queued ranks before one that cannot, and
+ one the retry manager would retry now before one it holds back. The
+ rank asks the egress policy and the retry manager, whose record of
+ attempts the "Download All New" retry-policy filter reads too.
+ ``queue_research_downloads`` and the "Download All New" route both
+ choose the resource that refills a document by this rank.
+
+ Ranking fetches nothing, so it records no denial of the egress
+ policy: a refused URL is neither written to the audit log nor
+ counted toward the run's denied-fetch quota (see
+ ``_check_url_against_policy``). Fetching the URL still checks it and
+ records the denial, and so does queueing it in
+ ``queue_research_downloads``.
+ """
+ cannot_queue = not (
+ url
+ and is_downloadable_url(url)
+ and self._check_url_against_policy(url, record_denial=False)[0]
+ )
+ held_back = failed_last = False
+ if self.retry_manager:
+ held_back = not self.retry_manager.should_retry_resource(
+ resource_id
+ ).can_retry
+ status = self.retry_manager.status_tracker.get_resource_status(
+ resource_id
+ )
+ failed_last = status is not None and status["status"] in (
+ RetryStatus.TEMPORARILY_FAILED.value,
+ RetryStatus.PERMANENTLY_FAILED.value,
+ )
+ return RefillRank(
+ cannot_queue=cannot_queue,
+ held_back=held_back,
+ failed_last=failed_last,
+ linked=resource_id != owner_id,
+ resource_id=resource_id,
+ )
+
+ def _refilled_by_another_resource(
+ self,
+ session: Session,
+ document: Document,
+ resource: ResearchResource,
+ run: Optional[AbstractSet[str]],
+ research_id: str,
+ ranks: Dict[Tuple[int, Optional[int]], RefillRank],
+ ) -> bool:
+ """Whether a resource other than ``resource`` refills ``document``.
+
+ A document that awaits its PDF (see ``document_awaits_pdf``) is
+ refilled by a download of its own resource (``Document.resource_id``,
+ unless that resource points to another document through
+ ``document_id``) or of any resource linked to it by content
+ (``ResearchResource.document_id``): each downloads into it, see
+ ``get_document_for_resource``. One such download refills it, so
+ ``queue_research_downloads`` queues ``resource`` only when this is
+ False. ``research_id`` is the research being queued, ``run`` the
+ research sessions of the run (see ``queue_research_downloads``),
+ and ``ranks`` holds the ``refill_rank`` of the resources compared
+ so far in that call. True when another of these resources
+
+ * has a PROCESSING queue entry that a download still holds (see
+ ``download_claim_is_live``): another stream is downloading it.
+ An entry that a restart left PROCESSING does not count, and that
+ resource is not queued either, since no reset touches a
+ PROCESSING entry;
+ * has a PENDING queue entry that the same run downloads, i.e. of a
+ research session in ``run``, and is viable or ranks before
+ ``resource`` (see ``RefillRank``). Without a run (``run`` None,
+ when nothing downloads the queue right away) any PENDING entry
+ counts: the run that downloads the queue later checks again with
+ its own research sessions. Within a run, a PENDING entry of a
+ research session outside it does not count, since the run would
+ not download it;
+ * ranks before ``resource`` and would be queued by the same call or
+ run: its research session is ``research_id`` or one in ``run``,
+ and it has no queue entry PENDING or PROCESSING. Ranking before
+ ``resource``, which can be queued, it can be queued too.
+
+ So, unless one of them is being downloaded, a resource that cannot
+ succeed does not keep a viable one from refilling the document, and
+ of the viable ones in the run the document's own resource refills
+ it, whichever research the run queues first, unless another viable
+ one is already queued for the run.
+ """
+ # Bare columns: the full rows would load each resource's
+ # content_preview (#4560).
+ others = (
+ session.query(
+ ResearchResource.id,
+ ResearchResource.url,
+ ResearchResource.research_id,
+ )
+ .filter(
+ ResearchResource.id != resource.id,
+ or_(
+ ResearchResource.document_id == document.id,
+ and_(
+ ResearchResource.id == document.resource_id,
+ ResearchResource.document_id.is_(None),
+ ),
+ ),
+ )
+ .all()
+ )
+ if not others:
+ return False
+ entries = {
+ entry.resource_id: entry
+ for entry in session.query(
+ LibraryDownloadQueue.resource_id,
+ LibraryDownloadQueue.status,
+ LibraryDownloadQueue.research_id,
+ LibraryDownloadQueue.last_attempt_at,
+ ).filter(
+ LibraryDownloadQueue.resource_id.in_(
+ [other.id for other in others]
+ ),
+ LibraryDownloadQueue.status.in_(
+ (DocumentStatus.PENDING, DocumentStatus.PROCESSING)
+ ),
+ )
+ }
+
+ def rank_of(res) -> RefillRank:
+ key = (res.id, document.resource_id)
+ if key not in ranks:
+ ranks[key] = self.refill_rank(
+ res.id, res.url, document.resource_id
+ )
+ return ranks[key]
+
+ own_rank = rank_of(resource)
+ queued_now = run if run is not None else {research_id}
+ for other in others:
+ entry = entries.get(other.id)
+ if entry is not None and entry.status == DocumentStatus.PROCESSING:
+ if download_claim_is_live(entry.last_attempt_at):
+ return True
+ continue
+ rank = rank_of(other)
+ if entry is not None:
+ if (run is None or entry.research_id in run) and (
+ rank.viable or rank < own_rank
+ ):
+ return True
+ continue
+ if rank < own_rank and other.research_id in queued_now:
+ return True
+ return False
+
def download_resource(self, resource_id: int) -> Tuple[bool, Optional[str]]:
"""
Download a specific resource.
@@ -785,16 +1114,22 @@ class DownloadService:
logger.error(f"Resource {resource_id} not found")
return False, "Resource not found"
- # Check if already downloaded (trust the database after sync)
- existing_doc = (
- session.query(Document)
- .filter_by(
- resource_id=resource_id, status=DocumentStatus.COMPLETED
- )
- .first()
+ # Check if already downloaded (trust the database after sync).
+ # A completed document that the sync made text-only because its
+ # PDF file was missing still needs its PDF, so it is downloaded
+ # again into that document (see document_awaits_pdf).
+ existing_query = session.query(Document).filter_by(
+ resource_id=resource_id, status=DocumentStatus.COMPLETED
)
-
- if existing_doc:
+ if resource.document_id:
+ # Never short-circuit on a retained older document. Keep
+ # linked-document retries on the normal path so they repair
+ # queue, tracker, and collection bookkeeping.
+ existing_query = existing_query.filter(
+ Document.id == resource.document_id
+ )
+ existing_doc = existing_query.first()
+ if existing_doc and not document_awaits_pdf(existing_doc):
logger.info(
"Resource already downloaded (according to database)"
)
@@ -1227,6 +1562,37 @@ class DownloadService:
file_size_val = len(pdf_content)
downloaded_at_val = datetime.now(UTC)
+ existing_doc = get_document_for_resource(sess, target_resource)
+ if (
+ existing_doc
+ and document_awaits_pdf(existing_doc)
+ and existing_doc.document_hash != file_hash_val
+ and sess.query(Document.id)
+ .filter(Document.document_hash == file_hash_val)
+ .first()
+ is not None
+ ):
+ # Keeping the old hash while storing different bytes
+ # poisons later content deduplication: a download of
+ # the original PDF would then serve the replacement.
+ # Preserve both documents until the caller can relink
+ # to the canonical owner, before changing any storage,
+ # text, or tracker state.
+ error_msg = (
+ "PDF refill conflicts with another library document"
+ )
+ sess.add(
+ DownloadAttempt(
+ url_hash=target_tracker.url_hash,
+ attempt_number=attempt_num,
+ attempted_at=attempted_at,
+ succeeded=False,
+ error_message=error_msg,
+ )
+ )
+ sess.commit()
+ return False, error_msg, status_code
+
target_tracker.file_hash = file_hash_val
target_tracker.file_size = file_size_val
target_tracker.is_downloaded = True
@@ -1254,14 +1620,13 @@ class DownloadService:
)
sess.add(attempt)
- # Check if library document already exists
- existing_doc = get_document_for_resource(sess, target_resource)
canonical_retry = False
# Older placeholder recovery could leave a nonempty PDF
# labelled "unknown"; that content still needs preservation.
if (
existing_doc
and existing_doc.document_hash != file_hash_val
+ and not document_awaits_pdf(existing_doc)
and (
existing_doc.status == DocumentStatus.COMPLETED
or (
@@ -1275,6 +1640,8 @@ class DownloadService:
# failed, or the link appeared while the download was in
# flight. Resolve changed bytes by hash without rewriting
# the document used by other resources.
+ # A sync-marked missing PDF may refill the same record;
+ # the collision guard above already protects other owners.
existing_doc = None
canonical_retry = True
if (
@@ -1309,6 +1676,7 @@ class DownloadService:
if existing_doc:
pdf_document_id = existing_doc.id
+ refills_missing_pdf = document_awaits_pdf(existing_doc)
# Update existing document. Only replace document_hash when
# transitioning from FAILED — that's the placeholder hash from
# _record_failed_text_extraction. For any other prior state
@@ -1322,6 +1690,28 @@ class DownloadService:
existing_doc.status = DocumentStatus.COMPLETED
existing_doc.processed_at = datetime.now(UTC)
+ # A document that the library sync made text-only after
+ # its PDF file went missing gets its PDF back here, in
+ # the same document, whatever the bytes hash to (see
+ # document_awaits_pdf). A publisher or arXiv can serve
+ # a regenerated PDF, so the bytes can differ from the
+ # lost ones. The document then takes their hash, as a
+ # new download would. A hash already owned by another
+ # document was refused before any writes above.
+ if (
+ refills_missing_pdf
+ and existing_doc.document_hash != file_hash_val
+ ):
+ existing_doc.document_hash = file_hash_val
+
+ if refills_missing_pdf and collection_id:
+ # A linked resource may request a different collection.
+ # Keep its membership when changed bytes refill this
+ # record instead of taking the canonical dedup path.
+ ensure_in_collection(
+ sess, existing_doc.id, collection_id
+ )
+
# Save PDF using storage manager (updates storage_mode and file_path)
file_path_result, _ = pdf_storage_manager.save_pdf(
pdf_content=pdf_content,
@@ -1331,6 +1721,23 @@ class DownloadService:
url=url,
resource_id=target_resource.id,
)
+ if file_path_result:
+ # The PDF is now stored in this document, so label
+ # it a PDF, as a new download is. A failed-download
+ # placeholder is labelled "unknown" and a text
+ # extraction "text". Either label hides the stored
+ # PDF from PDFStorageManager.has_pdf. Once the sync
+ # has made the document text-only, it also keeps
+ # document_awaits_pdf from asking for the PDF again.
+ existing_doc.file_type = "pdf"
+ existing_doc.mime_type = "application/pdf"
+ elif refills_missing_pdf:
+ # The PDF of a document that still needed one was
+ # downloaded but not stored, as in storage mode
+ # "none" or for a PDF over the size limit. Leave the
+ # document as a "none"-mode download leaves one, so
+ # that the download paths stop fetching it again.
+ existing_doc.file_path = None
# Update tracker
target_tracker.file_path = (
@@ -1447,6 +1854,14 @@ class DownloadService:
and file_path_result != "database"
else None
)
+ if file_path_result:
+ # The PDF is now stored in this document:
+ # label it a PDF, as the existing-document
+ # branch above does.
+ existing_by_hash.file_type = "pdf"
+ existing_by_hash.mime_type = (
+ "application/pdf"
+ )
# Keep sibling trackers for the same bytes
# consistent: the canonical URL's tracker still
# has file_path=None from its none-mode download,
@@ -1519,6 +1934,12 @@ class DownloadService:
# re-download, not data loss.
target_tracker.file_path = None
target_tracker.file_name = None
+ if document_awaits_pdf(existing_by_hash):
+ # Storage was disabled or refused this PDF's
+ # size. Settle a linked refill as a text-only
+ # download, just like the own-document path,
+ # so every bulk run does not fetch it again.
+ existing_by_hash.file_path = None
except Exception as exc:
log_exception_type(
"Failed to reuse/upgrade deduped PDF storage",
@@ -2410,12 +2831,24 @@ class DownloadService:
def _try_existing_pdf_extraction(
self, session, resource, resource_id: int
) -> Optional[Tuple[bool, Optional[str]]]:
- """Try extracting text from existing PDF in database."""
- pdf_document = (
- session.query(Document).filter_by(resource_id=resource_id).first()
- )
+ """Extract text from the PDF file of the resource's document.
+
+ Reads the file that the completed document's own ``file_path``
+ names (filesystem storage), without a request, and saves its text
+ into that document. A PDF stored in the database is not read here
+ (``_text_from_cached_pdf`` reads one for an arXiv paper). Returns
+ None, so that the caller tries its other sources, when the resource
+ has no document, the document is not completed or names no file,
+ its file is missing or yields no text, or the read or the save
+ fails.
+ """
+ # An older retained snapshot may still name this resource as its
+ # owner after the resource links to a newer document.
+ pdf_document = get_document_for_resource(session, resource)
- if not pdf_document or pdf_document.status != "completed":
+ # A loaded status is a DocumentStatus member, which never equals
+ # the string "completed".
+ if not pdf_document or pdf_document.status != DocumentStatus.COMPLETED:
return None
# Validate path to prevent path traversal attacks. Resolve against the
diff --git a/src/local_deep_research/research_library/services/library_service.py b/src/local_deep_research/research_library/services/library_service.py
index 09a1dff648..09be4153cd 100644
--- a/src/local_deep_research/research_library/services/library_service.py
+++ b/src/local_deep_research/research_library/services/library_service.py
@@ -13,10 +13,10 @@ from typing import Dict, List, Optional
from urllib.parse import urlparse
from loguru import logger
-from sqlalchemy import or_, func, case
+from sqlalchemy import and_, or_, func, case
from sqlalchemy.orm import aliased, defer
-from ...constants import FILE_PATH_SENTINELS
+from ...constants import FILE_PATH_SENTINELS, FILE_PATH_TEXT_ONLY
from ...utilities.arxiv import is_arxiv_paper_url
from ...database.models.download_tracker import DownloadTracker
from ...database.models.library import (
@@ -35,9 +35,12 @@ from ...utilities.resource_utils import safe_close
from ...security import url_authority_without_userinfo
from ...utilities.sql_utils import escape_like as _escape_like
from ..utils import (
+ document_awaits_pdf_clause,
+ document_counts_as_downloaded,
find_tracker_by_url,
get_absolute_path_from_settings,
open_file_location,
+ resource_document_clause,
)
# Filter dropdowns (library page + ``/api/research-list``) load research
@@ -55,6 +58,97 @@ _DROPDOWN_RESEARCH_LIMIT = 1000
_DOMAIN_SCAN_BATCH_SIZE = 1000
+def _names_a_file(file_path: Optional[str]) -> bool:
+ """Whether a ``file_path`` value names a file on disk.
+
+ ``None``, ``"database"`` and the sentinels (``blob_deleted``,
+ ``text_only_not_stored``, ``metadata_only``) name no file.
+ """
+ return (
+ bool(file_path)
+ and file_path != "database"
+ and file_path not in FILE_PATH_SENTINELS
+ )
+
+
+def _tracker_describes_storage(
+ tracker_path: Optional[str], doc: Document
+) -> bool:
+ """Whether a URL's tracker path names where ``doc`` stores its PDF.
+
+ Every document with the URL shares its tracker, and a download that
+ stores a PDF writes to the tracker where it stored it:
+
+ * filesystem storage: the file's path, which
+ ``PDFStorageManager.save_pdf`` also writes to the document, so the
+ two are equal (rows written before database storage was added,
+ 8d7bb38d1a, copied the tracker's path into the document);
+ * database storage: ``"database"``, while ``save_pdf`` labels the
+ document ``storage_mode`` "database" and leaves its ``file_path``
+ None, as it has since database storage was added (8d7bb38d1a).
+ ``upgrade_to_pdf`` labels a document it stores a PDF in the same
+ way, and the content-hash dedup of ``_download_pdf`` writes
+ ``"database"`` for a matched document with that label too.
+
+ So the tracker describes a document when it names the document's own
+ path, or when it is ``"database"`` and the document is labelled as
+ stored in the database, without a path. The label alone does not say
+ that a blob was stored (a failed-download placeholder keeps the
+ column's ``"database"`` default), so the sync still checks the
+ document's own blob. ``"database"`` can describe several documents of
+ the URL at once.
+ """
+ if tracker_path == doc.file_path:
+ return True
+ return (
+ tracker_path == "database"
+ and doc.storage_mode == "database"
+ and doc.file_path is None
+ )
+
+
+def _missing_file_entry(
+ doc: Document, stored_path: Optional[str], resolved_path
+) -> Dict:
+ """The sync's ``missing_files`` entry for a document whose file is gone.
+
+ ``POST /library/api/sync-library`` returns these entries to the client.
+ ``path`` is the stored path, which is relative to the user's library
+ root (the library listing returns the same value as ``file_path``), or
+ ``"invalid"`` when it does not resolve inside that root
+ (``resolved_path`` is ``None``). For a document whose PDF the URL's
+ tracker records as stored in the database, the stored value is
+ ``"database"``, which names no file. It is never ``resolved_path``
+ itself: the absolute path is the server's directory layout, which
+ ``/library/api/check-downloads`` and the library listing withhold from
+ the client (#6462).
+ """
+ return {
+ "id": doc.id,
+ "title": doc.title,
+ "path": stored_path if resolved_path and stored_path else "invalid",
+ "url": doc.original_url,
+ }
+
+
+def _counts_as_downloaded():
+ """SQL condition: a ``Document`` the Download Manager counts as downloaded.
+
+ A completed document, unless it awaits its PDF (see
+ ``document_awaits_pdf``): "Sync library" kept its text after its PDF
+ file went missing and reported the file missing, and "Download All New"
+ queues it, so the Download Manager counts it as available to download.
+ Without a matching document (the outer join's NULL row) the condition is
+ not true, so nothing is counted. ``document_counts_as_downloaded``
+ (``research_library.utils``) is the same rule for a loaded
+ ``Document``.
+ """
+ return and_(
+ Document.status == DocumentStatus.COMPLETED,
+ ~document_awaits_pdf_clause(),
+ )
+
+
class LibraryService:
"""Service for managing and querying the document library."""
@@ -71,6 +165,125 @@ class LibraryService:
is not None
)
+ def _remaining_content(
+ self, session, doc: Document, settings_manager
+ ) -> Optional[str]:
+ """What a download stores itself, whatever its URL's tracker names.
+
+ Returns ``"pdf"`` when its PDF is still stored (a blob in the
+ database, or a file at the document's own ``file_path``),
+ ``"text"`` when only its extracted text is left, and ``None`` when
+ nothing is.
+
+ The document's ``storage_mode`` and its tracker's ``file_path`` can
+ both be stale: a re-download in "none" mode clears the tracker path
+ but keeps the stored PDF, and a failed-download placeholder keeps
+ the ``"database"`` column default after text is saved into it. The
+ tracker, which every document with the URL shares, can also name
+ another document's file. So this checks what is actually stored.
+ Deleting a document that holds any of it would also delete that
+ content, its chunks and its vectors.
+ """
+ if self._has_blob_in_db(session, doc.id):
+ return "pdf"
+ if _names_a_file(doc.file_path):
+ resolved = get_absolute_path_from_settings(
+ doc.file_path, self.username, settings_manager=settings_manager
+ )
+ if resolved and resolved.is_file():
+ return "pdf"
+ has_text = (
+ session.query(Document.id)
+ .filter(
+ Document.id == doc.id,
+ Document.text_content.isnot(None),
+ Document.text_content != "",
+ )
+ .first()
+ is not None
+ )
+ return "text" if has_text else None
+
+ def _keep_text_report_missing_pdf(
+ self,
+ doc: Document,
+ tracker,
+ stored_path: Optional[str],
+ resolved_path,
+ stats: Dict,
+ ) -> None:
+ """Keep a document's text, drop its dangling PDF and report it.
+
+ For a research download whose PDF file is gone but whose extracted
+ text is not, and that still has its research resource
+ (``resource_id``), from which the download manager can download
+ the PDF again. The document keeps its row, text, collection links,
+ chunks and vectors. It is labelled a PDF and stored the way
+ text-only documents are (``file_type`` "pdf", ``storage_mode``
+ "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), so the library no
+ longer offers a PDF that is not there, and ``tracker`` is reset
+ to not downloaded unless it is None (see ``_clear_missing_pdf``).
+ It is counted in ``files_missing`` and listed in ``missing_files``
+ under ``stored_path``, the document's own path that named the
+ missing file (see ``_missing_file_entry``), and
+ ``document_awaits_pdf`` holds for it, so the download manager
+ downloads its PDF into it again.
+
+ The label is set here because what went missing was a PDF: the
+ sync calls this only for a missing file that the document's own
+ ``file_path`` names, and only PDF storage writes a file path into
+ a document. A
+ failed-download placeholder that a PDF download completed before
+ ``_download_pdf`` labelled such documents "pdf" is still labelled
+ "unknown", and would otherwise never be downloaded again.
+ """
+ doc.file_type = "pdf"
+ doc.mime_type = "application/pdf"
+ self._clear_missing_pdf(doc, tracker, stats)
+ stats["files_missing"] += 1
+ stats["missing_files"].append(
+ _missing_file_entry(doc, stored_path, resolved_path)
+ )
+
+ def _clear_missing_pdf(self, doc: Document, tracker, stats: Dict) -> None:
+ """Make a document whose PDF file is gone a text-only document.
+
+ It is stored the way text-only documents are (``storage_mode``
+ "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), and ``tracker``,
+ unless it is None, is reset to not downloaded, so neither names the
+ missing file any more. The sync passes the URL's tracker when it
+ names this document's path or no path, and None when it describes
+ where other documents store their PDF (another document's path, or
+ ``"database"`` for a document not stored there, see
+ ``_tracker_describes_storage``), which is left as it is. The reset
+ counts in ``trackers_updated`` only when it changes the tracker: a
+ tracker without a path can be reset already, for example by the
+ deletion of another document of the URL whose file it named, when
+ the sync read that document first. Counted again, that tracker
+ counted twice in that read order and once in the other.
+
+ The sync calls this directly, without reporting the document, for
+ a download that keeps its extracted text but no longer has its
+ research resource (``resource_id`` is NULL, e.g. after
+ ``delete_resource``). No download path asks for such a document's
+ PDF (``document_awaits_pdf`` needs the resource), and it counts in
+ ``files_found``, as every text-only document does. A later download
+ of the same PDF bytes for another resource still reaches it through
+ the content-hash match in ``DownloadService._download_pdf``. In a
+ storage mode that keeps PDFs, that match stores the PDF into a
+ matched document stored without one (``storage_mode`` "none").
+ Left naming the missing file, the document made that match copy the
+ dangling path into the new resource's tracker and store nothing.
+ """
+ doc.storage_mode = "none"
+ doc.file_path = FILE_PATH_TEXT_ONLY
+ if tracker is not None and (
+ tracker.is_downloaded or tracker.file_path is not None
+ ):
+ tracker.is_downloaded = False
+ tracker.file_path = None
+ stats["trackers_updated"] += 1
+
def _get_safe_absolute_path(self, file_path: str) -> Optional[str]:
"""
Get the absolute path for a file, safely handling invalid paths.
@@ -637,6 +850,9 @@ class LibraryService:
) -> List[Dict]:
"""Get research sessions with download statistics.
+ ``downloaded_count`` leaves out a document that awaits its PDF (see
+ ``_counts_as_downloaded``).
+
Args:
limit: Maximum number of results (0 = no limit, for backwards compat).
offset: Number of rows to skip. Only applied when limit > 0.
@@ -652,7 +868,7 @@ class LibraryService:
func.count(
func.distinct(
case(
- (Document.status == "completed", Document.id),
+ (_counts_as_downloaded(), Document.id),
else_=None,
)
)
@@ -679,8 +895,7 @@ class LibraryService:
)
.outerjoin(
Document,
- (ResearchResource.id == Document.resource_id)
- | (ResearchResource.document_id == Document.id),
+ resource_document_clause(),
)
.group_by(ResearchHistory.id)
.order_by(ResearchHistory.created_at.desc())
@@ -769,7 +984,10 @@ class LibraryService:
This is a lightweight query that only returns totals — used for
the download-manager header so stats remain accurate regardless
- of which page the user is viewing.
+ of which page the user is viewing. A document that awaits its PDF
+ does not count as downloaded (see ``_counts_as_downloaded``): it is
+ left out of ``already_downloaded``, so it is not subtracted from
+ ``available_to_download``.
"""
with get_user_db_session(self.username) as session:
row = (
@@ -783,7 +1001,7 @@ class LibraryService:
func.count(
func.distinct(
case(
- (Document.status == "completed", Document.id),
+ (_counts_as_downloaded(), Document.id),
else_=None,
)
)
@@ -811,8 +1029,7 @@ class LibraryService:
)
.outerjoin(
Document,
- (ResearchResource.id == Document.resource_id)
- | (ResearchResource.document_id == Document.id),
+ resource_document_clause(),
)
.one()
)
@@ -835,8 +1052,15 @@ class LibraryService:
"""Batch-fetch PDF documents and domain breakdowns for multiple research sessions.
Returns a dict keyed by research_id with:
- - "pdf_sources": list of document dicts (capped at limit_per_research)
+ - "pdf_sources": list of document dicts (capped at limit_per_research),
+ each with "document_title", "domain", "file_type" and
+ "downloaded" (whether the Download Manager shows it as
+ downloaded, with its "Downloaded" badge)
- "domains": dict of domain -> {total, pdfs, downloaded}
+
+ "downloaded" follows the rule of the page's other counts (see
+ ``document_counts_as_downloaded``): a completed document counts,
+ unless it awaits its PDF.
"""
if not research_ids:
return {}
@@ -895,7 +1119,8 @@ class LibraryService:
entry["domains"][domain]["total"] += 1
if doc.file_type == "pdf":
entry["domains"][domain]["pdfs"] += 1
- if doc.status == "completed":
+ downloaded = document_counts_as_downloaded(doc)
+ if downloaded:
entry["domains"][domain]["downloaded"] += 1
# PDF sources preview (capped)
@@ -911,7 +1136,7 @@ class LibraryService:
"document_title": title,
"domain": domain,
"file_type": doc.file_type,
- "download_status": doc.status or "unknown",
+ "downloaded": downloaded,
}
)
@@ -1002,7 +1227,10 @@ class LibraryService:
"domain": self._extract_domain(resource.url)
if resource
else "User Upload",
- "download_status": doc.status,
+ # The status's string value, which the details page
+ # compares with 'completed' and prints: a loaded
+ # DocumentStatus member never equals that string.
+ "download_status": doc.status.value,
"downloaded_at": doc.processed_at.isoformat()
if doc.processed_at
and hasattr(doc.processed_at, "isoformat")
@@ -1216,6 +1444,59 @@ class LibraryService:
Sync library database with filesystem.
Check which PDF files exist and update database accordingly.
+ Every document with the same URL shares that URL's tracker. The
+ tracker is used for a document only when it names where the
+ document stores its PDF (see ``_tracker_describes_storage``): the
+ document's own path, or ``"database"`` for a document labelled as
+ stored in the database. Each document is judged by what it stores
+ itself, and a document the tracker does not describe leaves the
+ tracker unchanged.
+
+ A download whose PDF file is missing is handled by what it still
+ stores (see ``_remaining_content``):
+
+ * its PDF, as a database blob or a file at its own ``file_path``:
+ kept unchanged;
+ * only its extracted text, while its own ``file_path`` names a
+ file, and it still has its research resource: kept with its text
+ but made text-only, the URL's tracker reset if it names that file
+ or no file, and counted in ``files_missing`` and
+ ``missing_files``, so that the download manager downloads its PDF
+ again (see ``_keep_text_report_missing_pdf``);
+ * only its extracted text, while its own ``file_path`` names a
+ file, with no research resource (e.g. a download whose resource
+ was deleted): kept with its text and made text-only, the URL's
+ tracker reset as above, and counted in ``files_found``. No
+ download path asks for its PDF, so it is not reported, but a
+ later download of the same PDF bytes stores the PDF in it (see
+ ``_clear_missing_pdf``);
+ * only its extracted text, with no file named (a "database" label
+ without a blob): kept unchanged;
+ * nothing: deleted, so that its URL can be downloaded again, and
+ counted in ``files_missing``. The URL's tracker is reset if it
+ names the document's file. If it is ``"database"`` and the
+ document is stored there, it is reset only when, once every
+ document has been checked, no other completed document under
+ that tracker still holds a PDF blob: until then the URL's PDF is
+ still stored in the database, as the tracker records.
+
+ Left alone before any of these checks: a document whose own
+ ``file_path`` is a sentinel, such as ``blob_deleted`` after "Remove
+ PDF to save space", and a document whose ``storage_mode`` is "none"
+ or unset (a text-only document), unless the URL's tracker names its
+ path.
+
+ ``missing_files`` lists every document counted in ``files_missing``
+ whose ``file_path`` names a file, by the path stored for it,
+ relative to the library root, never by the absolute path on the
+ server (see ``_missing_file_entry``). It also lists every document
+ deleted while the URL's tracker records it as stored in the
+ database, as ``"database"``.
+
+ ``trackers_updated`` counts the trackers the sync changed, each
+ once, whichever document of their URL it reads first (see
+ ``_clear_missing_pdf``).
+
Returns:
Statistics about the sync operation
"""
@@ -1246,6 +1527,12 @@ class LibraryService:
# Documents deleted below — their FAISS vectors + chunk rows are
# purged AFTER this transaction commits (see the post-commit block).
removed_doc_ids: List[str] = []
+ # The documents checked under each "database" tracker, by
+ # tracker id, and the "database" trackers whose document was
+ # deleted below. Whether such a tracker is reset is decided after
+ # the loop, once every document under it has been checked.
+ database_tracker_documents: Dict[int, List[str]] = {}
+ emptied_database_trackers: Dict[int, DownloadTracker] = {}
# Sync documents with filesystem.
#
@@ -1269,16 +1556,46 @@ class LibraryService:
# legacy-hash rows so pre-upgrade documents are not
# mistaken for tracker-less orphans and deleted below.
tracker = find_tracker_by_url(session, doc.original_url)
+ if tracker is not None and tracker.file_path == "database":
+ database_tracker_documents.setdefault(
+ tracker.id, []
+ ).append(doc.id)
+ # A URL has one tracker, shared by every document with
+ # that URL: the documents of two researches that cite
+ # the same paper, or a failed-download placeholder that
+ # "Extract Text to DB" completed with text next to the
+ # document that holds the PDF. The tracker's path is the
+ # one written last. It describes this document only when
+ # it names where the document stores its PDF: the
+ # document's own path, which every download that stores
+ # a PDF file writes to both, or "database" for a
+ # document labelled as stored in the database (see
+ # _tracker_describes_storage). Any other document is
+ # judged by what it stores itself, in the ``else``
+ # branch, and leaves the tracker alone. Judged by
+ # another document's file, it was counted found while
+ # its own file was missing, or labelled a PDF, made
+ # text-only and reported missing for a file it never
+ # had, which the download manager then downloaded into
+ # it as a second copy.
+ if (
+ tracker is not None
+ and tracker.file_path
+ and not _tracker_describes_storage(
+ tracker.file_path, doc
+ )
+ ):
+ tracker = None
if tracker and tracker.file_path:
- # Sentinel tracker paths (``blob_deleted``,
+ # The tracker names where this document stores its
+ # PDF: its own path, or the database.
+ # Sentinel paths (``blob_deleted``,
# ``text_only_not_stored``, ``metadata_only``) name
# no file by design — e.g. a blob purge keeps the
- # document complete with ``file_path="blob_deleted"``
- # while a dedup-mirrored duplicate tracker could
- # still carry that value. Resolving one as a
- # relative path always misses and would run
- # ``delete_document_completely`` on the
+ # document complete with ``file_path="blob_deleted"``.
+ # Resolving one as a relative path always misses and
+ # would run ``delete_document_completely`` on the
# text/chunks/vectors the sentinel was protecting
# (issue #6760 follow-up). Leave them alone like the
# text-only branch below.
@@ -1286,17 +1603,16 @@ class LibraryService:
stats["files_found"] += 1
continue
# Database-stored PDFs have no filesystem file to
- # sync — the blob is the source of truth. Treat a
- # present blob as found instead of falling into the
- # missing-file path that deletes the document.
+ # sync — the blob is the source of truth. A present
+ # blob counts as found. ``DocumentBlob`` is the
+ # module's import: importing it here would make it a
+ # local name of the whole function, which the
+ # post-loop check below could read only after this
+ # branch had run.
if (
tracker.file_path == "database"
or doc.storage_mode == "database"
):
- from ...database.models.library import (
- DocumentBlob,
- )
-
has_blob = (
session.query(DocumentBlob.document_id)
.filter_by(document_id=doc.id)
@@ -1319,25 +1635,69 @@ class LibraryService:
self.username,
settings_manager=settings_manager,
)
- if file_path and file_path.is_file():
+ # "database" names no file: a file of that name in
+ # the library root is not this document's PDF.
+ if (
+ _names_a_file(tracker.file_path)
+ and file_path
+ and file_path.is_file()
+ ):
+ stats["files_found"] += 1
+ continue
+ # The document's file is gone, or its blob is. What
+ # else it stores decides.
+ remaining = self._remaining_content(
+ session, doc, settings_manager
+ )
+ if remaining == "text" and _names_a_file(
+ tracker.file_path
+ ):
+ # Only the text is left of a PDF that was a
+ # file. Keep the text and clear the dangling
+ # PDF reference.
+ if doc.resource_id is not None:
+ # Its resource can download the PDF again:
+ # report the PDF missing, under the stored
+ # path that named the missing file.
+ self._keep_text_report_missing_pdf(
+ doc,
+ tracker,
+ tracker.file_path,
+ file_path,
+ stats,
+ )
+ else:
+ # No download path asks for its PDF, but a
+ # later download of the same bytes stores
+ # the PDF in it (see _clear_missing_pdf).
+ self._clear_missing_pdf(doc, tracker, stats)
+ stats["files_found"] += 1
+ elif remaining is not None:
+ # Its PDF is still stored as a blob, or its text
+ # is left and its path is the "database" label,
+ # which names no file. Keep it and its tracker
+ # unchanged.
stats["files_found"] += 1
else:
# File missing or path invalid - mark for re-download
stats["files_missing"] += 1
stats["missing_files"].append(
- {
- "id": doc.id,
- "title": doc.title,
- "path": str(file_path)
- if file_path
- else "invalid",
- "url": doc.original_url,
- }
+ _missing_file_entry(
+ doc, tracker.file_path, file_path
+ )
)
- # Reset tracker
- tracker.is_downloaded = False
- tracker.file_path = None
+ if tracker.file_path == "database":
+ # "database" describes every document of
+ # the URL stored there, and another one can
+ # still hold its PDF blob. Decided after the
+ # loop.
+ emptied_database_trackers[tracker.id] = tracker
+ else:
+ # Reset tracker
+ tracker.is_downloaded = False
+ tracker.file_path = None
+ stats["trackers_updated"] += 1
# Delete the document entry so it can be re-queued
from ..deletion.utils.cascade_helper import (
@@ -1348,19 +1708,86 @@ class LibraryService:
session, doc.id
)
removed_doc_ids.append(doc.id)
- stats["trackers_updated"] += 1
else:
- # No tracker or path. Text-only documents
- # (storage_mode "none") never have a file by design —
- # deleting them here would destroy text, chunks, and
- # vectors that are recoverable only by re-download
- # (dead links = permanent loss). Leave them alone.
+ # No tracker, a tracker without a path, or one that
+ # names where other documents store their PDF:
+ # another document's path, or "database" for a
+ # document not stored there. That tracker is not
+ # passed on (``tracker`` is None). Text-only
+ # documents (storage_mode "none") never have a file
+ # by design — deleting them here would destroy
+ # text, chunks, and vectors that are recoverable
+ # only by re-download (dead links = permanent loss).
+ # Leave them alone. A sentinel path on the document
+ # means the same: "Remove PDF to save space"
+ # (delete_blob_only) writes ``blob_deleted`` to the
+ # document but leaves the tracker naming the copy it
+ # removed. That PDF was removed on purpose, so it is
+ # neither missing nor to be downloaded again.
if (doc.storage_mode in (None, "none")) or (
doc.file_path in FILE_PATH_SENTINELS
):
stats["files_found"] += 1
continue
+ # Any other label can be stale, e.g. a lost tracker
+ # path after a "none"-mode re-download, or the
+ # "database" default on a placeholder that later got
+ # text. What the document still stores decides.
+ if settings_manager is None and doc.file_path:
+ settings_manager = get_settings_manager(
+ username=self.username
+ )
+ remaining = self._remaining_content(
+ session, doc, settings_manager
+ )
+ if remaining == "text" and _names_a_file(doc.file_path):
+ # Its own PDF file is gone and only the text is
+ # left. Keep the text and clear the dangling
+ # PDF reference.
+ if doc.resource_id is not None:
+ # Its resource can download the PDF again:
+ # report the PDF missing.
+ self._keep_text_report_missing_pdf(
+ doc,
+ tracker,
+ doc.file_path,
+ get_absolute_path_from_settings(
+ doc.file_path,
+ self.username,
+ settings_manager=settings_manager,
+ ),
+ stats,
+ )
+ else:
+ # No download path asks for its PDF, but a
+ # later download of the same bytes stores
+ # the PDF in it (see _clear_missing_pdf).
+ self._clear_missing_pdf(doc, tracker, stats)
+ stats["files_found"] += 1
+ continue
+ if remaining is not None:
+ # Its PDF is still stored, or its text is left
+ # and it names no file (e.g. the "database"
+ # default on a placeholder that later got text).
+ stats["files_found"] += 1
+ continue
+ # Nothing of it is left: delete it so that its URL
+ # can be downloaded again. A file its own path names
+ # is reported missing, as the branch above reports
+ # one that the tracker names.
stats["files_missing"] += 1
+ if _names_a_file(doc.file_path):
+ stats["missing_files"].append(
+ _missing_file_entry(
+ doc,
+ doc.file_path,
+ get_absolute_path_from_settings(
+ doc.file_path,
+ self.username,
+ settings_manager=settings_manager,
+ ),
+ )
+ )
from ..deletion.utils.cascade_helper import (
CascadeHelper,
)
@@ -1374,6 +1801,32 @@ class LibraryService:
settings_manager, "settings manager", allow_none=True
)
+ # A "database" tracker records that its URL's PDF is stored in
+ # the database, which can hold it for several documents of the
+ # URL. One of them was deleted above with nothing left. While
+ # another still holds a PDF blob, the record is true and the
+ # tracker is left as it is; otherwise it is reset. Every
+ # document under the tracker has been checked by now, so the
+ # order in which the sync read them does not change whether it
+ # is reset.
+ removed = set(removed_doc_ids)
+ for tracker_id, tracker in emptied_database_trackers.items():
+ others = [
+ document_id
+ for document_id in database_tracker_documents[tracker_id]
+ if document_id not in removed
+ ]
+ if others and (
+ session.query(DocumentBlob.document_id)
+ .filter(DocumentBlob.document_id.in_(others))
+ .first()
+ is not None
+ ):
+ continue
+ tracker.is_downloaded = False
+ tracker.file_path = None
+ stats["trackers_updated"] += 1
+
session.commit()
logger.info(
f"Library sync completed: {stats['files_found']} found, {stats['files_missing']} missing"
diff --git a/src/local_deep_research/research_library/utils/__init__.py b/src/local_deep_research/research_library/utils/__init__.py
index 97960aacdc..661b74fb9d 100644
--- a/src/local_deep_research/research_library/utils/__init__.py
+++ b/src/local_deep_research/research_library/utils/__init__.py
@@ -5,23 +5,31 @@ import os
import socket
import subprocess
import sys
+from datetime import UTC, datetime
from pathlib import Path
from typing import Optional
from urllib.parse import unquote, urlparse
from fastapi.responses import JSONResponse
from loguru import logger
+from sqlalchemy import and_, func, or_
from ...security import redact_url_for_log
from ...config.paths import get_library_directory
+from ...constants import FILE_PATH_TEXT_ONLY
+from ...database.models.research import ResearchResource
+from ...database.models.library import (
+ Document,
+ DocumentCollection,
+ DocumentStatus,
+)
from ...utilities.download_url import (
normalize_url_for_hash as normalize_url_for_hash,
get_url_hash as get_url_hash,
get_legacy_url_hashes as get_legacy_url_hashes,
get_url_hash_candidates as get_url_hash_candidates,
)
-from ...database.models.library import Document, DocumentCollection
from ...security.ip_ranges import PRIVATE_IP_RANGES
from ...security.network_utils import is_private_ip
from ...security.path_validator import PathValidator
@@ -262,6 +270,171 @@ def get_document_for_resource(session, resource):
return session.query(Document).filter_by(resource_id=resource.id).first()
+def resource_document_clause():
+ """Join resources to their canonical document, honoring explicit links.
+
+ Mirrors ``get_document_for_resource``: retained documents can still
+ name a resource as their owner after that resource links elsewhere.
+ """
+ return or_(
+ ResearchResource.document_id == Document.id,
+ and_(
+ ResearchResource.document_id.is_(None),
+ ResearchResource.id == Document.resource_id,
+ ),
+ )
+
+
+def document_awaits_pdf(document) -> bool:
+ """Whether a completed research download still needs its PDF.
+
+ "Sync library" keeps a downloaded document whose PDF file was removed
+ from disk if the document still has its extracted text and its own
+ research resource (``resource_id``). It labels the document a PDF and
+ stores it the way text-only documents are stored (``file_type`` "pdf",
+ ``storage_mode`` "none", ``file_path`` ``FILE_PATH_TEXT_ONLY``), resets
+ the URL's tracker unless it names where other documents store their
+ PDF, and reports the file missing, so every document it reports that
+ way matches here
+ (see ``LibraryService._keep_text_report_missing_pdf``). Everywhere else
+ a completed document counts as downloaded, so ``download_resource`` and
+ the PDF queueing paths (``queue_research_downloads`` and the "Download
+ All New" route, through ``document_awaits_pdf_clause``) ask this, for
+ the document's own resource and for the resources linked to it by
+ content (``ResearchResource.document_id``), and download the PDF again
+ into the same document. One download refills it, so the queueing paths
+ queue one of those resources (see ``DownloadService.refill_rank``).
+ Once the document holds its PDF again, ``download_resource`` reports
+ the document's own resource as already downloaded, and for a linked
+ one whose URL the egress policy allows, ``DownloadService._download_pdf``
+ reuses that stored PDF instead of fetching it. The
+ Download Manager asks too (``LibraryService``'s
+ ``get_research_list_with_stats``,
+ ``get_download_manager_summary_stats`` and ``get_pdf_previews_batch``):
+ such a document counts as available to download, not as downloaded,
+ and shows no "Downloaded" badge. So do the library routes
+ ``get_research_sources`` and ``check_downloads``, which report it as
+ not downloaded, and ``download_source``, which downloads it again
+ rather than answer "Already downloaded" (see
+ ``document_counts_as_downloaded``). Text extraction does not ask: such
+ a document already holds its text.
+
+ A document whose resource is gone (``resource_id`` NULL) never matches,
+ so nothing asks for its PDF. The sync stores such a document text-only
+ too, without reporting it, so that a later download of the same PDF
+ bytes for another resource stores the PDF in it (see
+ ``LibraryService._clear_missing_pdf``).
+
+ This checks ``file_type`` "pdf", not only that storage marker. Zotero
+ imports and uploads write the same marker for PDFs and differ only in
+ having no ``resource_id``. A text extraction (``file_type`` "text") has
+ the same ``file_path`` and differs only in the ``"database"`` default of
+ ``storage_mode``, so a writer that set it to "none", the value for a
+ document without a stored PDF, would make every extracted text ask for
+ a PDF. The sync can set the label itself because what it found missing
+ was a PDF: only ``PDFStorageManager.save_pdf`` writes a file path into a
+ document, and the sync converts a document only for a missing file
+ that its own ``file_path`` names. The URL's tracker, which every
+ document with that URL shares, counts for a document only when it
+ names that path, or when it is ``"database"`` and the document is
+ labelled as stored in the database, which names no file and so leads
+ to no conversion. So a document it converts with another label, such as a
+ failed-download placeholder that a PDF download completed before
+ ``_download_pdf`` labelled such documents "pdf", is picked up too.
+
+ No other writer leaves a research download in that shape: a
+ "none"-mode download keeps ``file_path=None``, "Remove PDF to save
+ space" writes ``blob_deleted``, and a text extraction is ``file_type``
+ "text". A download into the document that stores no PDF either, as in
+ storage mode "none" or for a PDF over the size limit, clears the mark,
+ see ``DownloadService._download_pdf``.
+ """
+ return (
+ document.status == DocumentStatus.COMPLETED
+ and document.file_type == "pdf"
+ and document.storage_mode == "none"
+ and document.file_path == FILE_PATH_TEXT_ONLY
+ and document.resource_id is not None
+ )
+
+
+def document_awaits_pdf_clause():
+ """:func:`document_awaits_pdf` as a SQL condition on ``Document``.
+
+ The nullable columns are compared through ``coalesce`` and ``status``
+ is NOT NULL, so on a ``Document`` row the condition is never NULL and
+ its negation is true for every document the function rejects.
+ """
+ return and_(
+ Document.status == DocumentStatus.COMPLETED,
+ func.coalesce(Document.file_type, "") == "pdf",
+ func.coalesce(Document.storage_mode, "") == "none",
+ func.coalesce(Document.file_path, "") == FILE_PATH_TEXT_ONLY,
+ Document.resource_id.isnot(None),
+ )
+
+
+def document_counts_as_downloaded(document) -> bool:
+ """Whether a loaded ``Document`` counts as downloaded.
+
+ A completed document counts, unless it awaits its PDF (see
+ ``document_awaits_pdf``), which the Download Manager offers for
+ download again. Its counts apply the same rule in SQL
+ (``LibraryService``'s ``_counts_as_downloaded``), and the source
+ previews under each research (``get_pdf_previews_batch``) apply this
+ function. So do the library routes that tell whether a research's
+ source is downloaded (``get_research_sources`` and ``check_downloads``)
+ and ``download_source``, which answers "Already downloaded" only for a
+ document that counts.
+
+ ``status`` loads as a ``DocumentStatus`` member. ``DocumentStatus`` is
+ a plain ``enum.Enum``, so a member never equals its string value
+ (``DocumentStatus.COMPLETED == "completed"`` is False), and the status
+ is compared with the member.
+ """
+ return document.status == DocumentStatus.COMPLETED and not (
+ document_awaits_pdf(document)
+ )
+
+
+# When this server process started. This module is imported while the app
+# starts, before any download can claim a queue row. See
+# download_claim_is_live.
+_PROCESS_STARTED_AT = datetime.now(UTC)
+
+
+def download_claim_is_live(claimed_at: Optional[datetime]) -> bool:
+ """Whether a PROCESSING download-queue row is still being downloaded.
+
+ ``claimed_at`` is the row's ``last_attempt_at``, which the download
+ that claims it sets (``_claim_download_queue_item`` in the library
+ routes). A download finishes or releases the row it claimed before it
+ ends, and it cannot outlive this server process, which serves with one
+ worker (see ``web/app.py``). So a row claimed before this process
+ started was cut off by a restart, as is one claimed before claims
+ recorded their time (``claimed_at`` None): nothing finishes or
+ releases it, so it does not stand for a download in progress. A row
+ whose release failed in this process (which is logged) still counts
+ as in progress until the next restart.
+
+ Both times are wall-clock readings, so a clock that goes back defeats
+ the rule. If this process started with its clock behind the time an
+ earlier process recorded for a claim, as on a device without a
+ real-time clock that starts with a stale time and has it corrected
+ later, that claim counts as in progress until the next restart, and no
+ other resource of its document refills the document until then. A
+ clock set back while this process runs can make a claim it made count
+ as cut off, so that another resource of the document can be queued
+ while that download runs. The queue row records nothing else that
+ tells the process that claimed it.
+ """
+ if claimed_at is None:
+ return False
+ if claimed_at.tzinfo is None:
+ claimed_at = claimed_at.replace(tzinfo=UTC)
+ return claimed_at >= _PROCESS_STARTED_AT
+
+
def find_tracker_by_url(session, url: str):
"""Find a ``DownloadTracker`` for ``url`` across hash schemes.
diff --git a/src/local_deep_research/security/egress/policy.py b/src/local_deep_research/security/egress/policy.py
index 2723810b65..cb473d8584 100644
--- a/src/local_deep_research/security/egress/policy.py
+++ b/src/local_deep_research/security/egress/policy.py
@@ -1587,13 +1587,22 @@ def _quota_ctx(ctx: EgressContext) -> EgressContext:
return active if active is not None else ctx
-def evaluate_url(url: str, ctx: EgressContext) -> Decision:
+def evaluate_url(
+ url: str, ctx: EgressContext, *, record_denial: bool = True
+) -> Decision:
"""Decide whether an arbitrary URL may be fetched (e.g., by ``fetch_content``).
Enforces the run's denied-fetch quota to prevent exhaustion attacks
via malicious indexed documents that loop the agent through hundreds
of denied fetches.
+
+ ``record_denial=False`` returns the same decision without recording a
+ denial: it is neither counted toward that quota nor written to the
+ audit log (an internal error is still logged). Only for a caller that
+ fetches nothing on the decision, such as one that ranks URLs; a fetch
+ must be decided with the default, so that its denials are counted.
"""
+ deny = _record_denial if record_denial else _unrecorded_denial
try:
# Quota check: hard fail after MAX_DENIED_FETCHES_PER_RUN. Anchor the
# counter to the run's active context (``_quota_ctx``) so the budget is
@@ -1605,19 +1614,19 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:
return Decision(False, "denial_quota_exceeded")
if not isinstance(url, str) or not url:
- return _record_denial(ctx, url, "url_malformed")
+ return deny(ctx, url, "url_malformed")
try:
parsed = urlsplit(url)
except Exception:
- return _record_denial(ctx, url, "url_malformed")
+ return deny(ctx, url, "url_malformed")
if parsed.scheme.lower() in _DANGEROUS_SCHEMES:
- return _record_denial(ctx, url, "dangerous_scheme")
+ return deny(ctx, url, "dangerous_scheme")
if parsed.scheme.lower() not in ("http", "https"):
- return _record_denial(ctx, url, "unsupported_scheme")
+ return deny(ctx, url, "unsupported_scheme")
if not parsed.hostname:
- return _record_denial(ctx, url, "no_hostname")
+ return deny(ctx, url, "no_hostname")
# HTTP client libraries (requests/urllib3) percent-DECODE the host
# in the netloc before the socket connect, but urlsplit().hostname
@@ -1650,7 +1659,7 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:
if is_ip_blocked(
host, allow_localhost=True, allow_private_ips=True
):
- return _record_denial(ctx, url, "blocked_metadata_ip")
+ return deny(ctx, url, "blocked_metadata_ip")
# is_ip_blocked only parses canonical IP notation, so an
# alternate-encoded metadata literal (octal/hex/integer) slips
# through above. Normalize it to the dotted-quad the resolver
@@ -1666,14 +1675,14 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:
alt, allow_localhost=True, allow_private_ips=True
)
):
- return _record_denial(ctx, url, "blocked_metadata_ip")
+ return deny(ctx, url, "blocked_metadata_ip")
# Metadata endpoints reachable by hostname (GCP) — is_ip_blocked
# can't see these because they aren't IP literals. Block by name so
# the invariant holds under PUBLIC_ONLY/BOTH too (the SSRF validator
# backstops the actual fetch, but evaluate_url's own guarantee must
# not depend on it).
if host.lower() in _METADATA_HOSTNAMES:
- return _record_denial(ctx, url, "blocked_metadata_ip")
+ return deny(ctx, url, "blocked_metadata_ip")
except Exception: # noqa: silent-exception - defensive, see below
# Non-IP host or helper error: fall through to normal
# classification (DNS path has its own handling).
@@ -1695,27 +1704,37 @@ def evaluate_url(url: str, ctx: EgressContext) -> Decision:
# bigger risk than false-positives on DOI redirects.
if classification is True:
return Decision(True, "allowed_private_host_under_strict")
- return _record_denial(ctx, url, "strict_public_host")
+ return deny(ctx, url, "strict_public_host")
if ctx.scope == EgressScope.PUBLIC_ONLY:
if classification is False:
return Decision(True, "allowed_public_host")
- return _record_denial(ctx, url, "scope_mismatch_public_only")
+ return deny(ctx, url, "scope_mismatch_public_only")
if ctx.scope == EgressScope.PRIVATE_ONLY:
if classification is True:
return Decision(True, "allowed_private_host")
- return _record_denial(ctx, url, "scope_mismatch_private_only")
+ return deny(ctx, url, "scope_mismatch_private_only")
# BOTH: any classified host is fine.
if classification is None:
- return _record_denial(ctx, url, "host_unclassified")
+ return deny(ctx, url, "host_unclassified")
return Decision(True, "allowed_both_scope")
except Exception: # pragma: no cover - defensive
logger.bind(policy_audit=True).exception("evaluate_url internal error")
return Decision(False, "internal_error")
+def _unrecorded_denial(
+ ctx: EgressContext, url: object, reason: str
+) -> Decision:
+ """A denial for ``evaluate_url(record_denial=False)``.
+
+ Neither counted toward the denied-fetch quota nor logged.
+ """
+ return Decision(False, reason)
+
+
def host_not_classified_public(url: str, ctx: EgressContext) -> bool:
"""True when ``url`` is an http(s) URL whose host the policy does NOT
classify public (private, or unclassifiable), decided WITHOUT recording
diff --git a/src/local_deep_research/web/routers/library.py b/src/local_deep_research/web/routers/library.py
index 7721deb0e8..d9ff7f8472 100644
--- a/src/local_deep_research/web/routers/library.py
+++ b/src/local_deep_research/web/routers/library.py
@@ -46,7 +46,11 @@ from ...research_library.services.pdf_storage_manager import PDFStorageManager
from ...security import failure_reason_for_log, redact_url_for_log
from ...research_library.utils import (
apply_user_subdir,
+ document_awaits_pdf_clause,
+ document_counts_as_downloaded,
+ download_claim_is_live,
get_document_for_resource,
+ resource_document_clause,
handle_api_error,
is_downloadable_domain,
is_downloadable_url,
@@ -1019,12 +1023,22 @@ def _claim_download_queue_item(db_session, queue_item_id):
for every row (issue #4691). The row-count guard makes the
PENDING -> PROCESSING transition the single point where exactly one stream
wins each row; the loser sees 0 updated rows and skips it.
+
+ The claim also records its time in ``last_attempt_at``. Nothing reclaims
+ a row left PROCESSING, e.g. by a restart during the download, and the
+ queueing of a document that awaits its PDF (see ``document_awaits_pdf``)
+ lets another resource refill it only while no download holds one of its
+ resources. ``download_claim_is_live`` tells a held row from one a restart
+ left behind by this time.
"""
claimed = (
db_session.query(LibraryDownloadQueue)
.filter_by(id=queue_item_id, status=DocumentStatus.PENDING)
.update(
- {LibraryDownloadQueue.status: DocumentStatus.PROCESSING},
+ {
+ LibraryDownloadQueue.status: DocumentStatus.PROCESSING,
+ LibraryDownloadQueue.last_attempt_at: datetime.now(UTC),
+ },
synchronize_session=False,
)
)
@@ -1155,11 +1169,20 @@ async def download_bulk(
# a COMPLETED Document, and re-queues any remaining row (e.g. a
# prior FAILED attempt) back to PENDING. Calling it BEFORE counting
# ensures the total reflects what will actually be processed
- # (issue #4660).
+ # (issue #4660). In PDF mode it also queues a COMPLETED Document
+ # that still needs its PDF (see document_awaits_pdf), through
+ # one resource per such document across all the research
+ # sessions of this run (run_research_ids); a text extraction
+ # already has that document's text, so text_only mode skips it.
+ # The set is built once for the run: each call only looks it up.
+ run_research_ids = frozenset(research_ids)
for research_id in research_ids:
try:
download_service.queue_research_downloads(
- research_id, collection_id
+ research_id,
+ collection_id,
+ text_only=mode == "text_only",
+ run_research_ids=run_research_ids,
)
except Exception:
queue_failures += 1
@@ -1486,6 +1509,110 @@ async def mark_for_redownload(
return {"success": True, "marked": count}
+def _one_resource_per_awaiting_document(
+ db_session, resources, username, password
+):
+ """Keep one of the resources that would refill the same document.
+
+ A document that awaits its PDF (see ``document_awaits_pdf``) does not
+ count as a download, so "Download All New" finds its own resource
+ (``Document.resource_id``) and every resource linked to it by content
+ (``ResearchResource.document_id``). Each of them downloads into that
+ document (``get_document_for_resource``), and one download refills it.
+ So of the ``resources`` that would refill the same document only one is
+ kept, the one ``queue_research_downloads`` would leave the document to
+ (see ``DownloadService._refilled_by_another_resource``): one that a
+ download holds (a PROCESSING queue entry, see
+ ``download_claim_is_live``), else a viable one that is already PENDING
+ in the download queue, else the one that ranks first in
+ ``DownloadService.refill_rank``. So a resource that cannot succeed (its
+ URL cannot be queued, the retry manager holds it back, or its latest
+ attempt failed) is kept only when it is being downloaded or no viable
+ one is there. A resource whose PROCESSING entry a restart left behind
+ comes last, since no reset touches that entry. The others count as
+ skipped. (Once the document holds its PDF again,
+ ``DownloadService.download_resource`` reports the document's own
+ resource as already downloaded, and for a linked one whose URL the
+ egress policy allows, ``DownloadService._download_pdf`` reuses that
+ stored PDF instead of fetching it.)
+
+ ``resources`` have passed the route's retry-policy and downloadability
+ checks. The ranking also asks the egress policy and the resources'
+ retry records, through a ``DownloadService`` for ``username``, which is
+ built only when some document has more than one of them.
+
+ Returns the kept resources in their order, and how many were left out.
+ """
+ awaiting = {
+ resource_id: (document_id, owner_id)
+ for resource_id, document_id, owner_id in (
+ db_session.query(
+ ResearchResource.id, Document.id, Document.resource_id
+ )
+ .join(
+ Document,
+ resource_document_clause(),
+ )
+ .filter(document_awaits_pdf_clause())
+ .all()
+ )
+ }
+ candidates = {}
+ for resource in resources:
+ if resource.id in awaiting:
+ document_id = awaiting[resource.id][0]
+ candidates.setdefault(document_id, []).append(resource)
+ shared = [rows for rows in candidates.values() if len(rows) > 1]
+ if not shared:
+ return resources, 0
+
+ entries = {
+ entry.resource_id: entry
+ for entry in db_session.query(
+ LibraryDownloadQueue.resource_id,
+ LibraryDownloadQueue.status,
+ LibraryDownloadQueue.last_attempt_at,
+ )
+ .filter(
+ LibraryDownloadQueue.resource_id.in_(
+ [row.id for rows in shared for row in rows]
+ ),
+ LibraryDownloadQueue.status.in_(
+ (DocumentStatus.PENDING, DocumentStatus.PROCESSING)
+ ),
+ )
+ .all()
+ }
+ left_out = set()
+ with DownloadService(username, password) as download_service:
+ for rows in shared:
+ owner_id = awaiting[rows[0].id][1]
+ choices = []
+ for row in rows:
+ rank = download_service.refill_rank(row.id, row.url, owner_id)
+ entry = entries.get(row.id)
+ if entry is None:
+ tier = 2
+ elif entry.status == DocumentStatus.PROCESSING:
+ tier = (
+ 0
+ if download_claim_is_live(entry.last_attempt_at)
+ else 3
+ )
+ else:
+ tier = 1 if rank.viable else 2
+ choices.append(((tier, rank), row.id))
+ kept_id = min(choices)[1]
+ left_out.update(row.id for row in rows if row.id != kept_id)
+ for resource_id in sorted(left_out):
+ logger.debug(
+ f"Skipping resource {resource_id}: another resource of its "
+ "document is queued to download its PDF"
+ )
+ kept = [resource for resource in resources if resource.id not in left_out]
+ return kept, len(left_out)
+
+
@router.post("/api/queue-all-undownloaded")
def queue_all_undownloaded(
request: Request, username: Annotated[str, Depends(require_auth)]
@@ -1499,6 +1626,11 @@ def queue_all_undownloaded(
# the columns the filter/loop use (id/url/research_id) instead of full
# ``ResearchResource`` entities so the ``content_preview`` Text column
# isn't materialized for every row on this whole-table scan (#4560).
+ # A completed document that still needs its PDF (the library sync
+ # kept its text after its PDF file went missing) is not counted as
+ # a download, see document_awaits_pdf. Only one of the resources
+ # that would refill such a document is queued, see
+ # _one_resource_per_awaiting_document.
undownloaded = (
db_session.query(
ResearchResource.id,
@@ -1507,11 +1639,9 @@ def queue_all_undownloaded(
)
.outerjoin(
Document,
- (
- (ResearchResource.id == Document.resource_id)
- | (ResearchResource.document_id == Document.id)
- )
- & (Document.status == "completed"),
+ resource_document_clause()
+ & (Document.status == "completed")
+ & ~document_awaits_pdf_clause(),
)
.filter(Document.id.is_(None))
.all()
@@ -1540,6 +1670,8 @@ def queue_all_undownloaded(
# Convert filter_results to dict for O(1) lookup instead of O(n²)
filter_results_by_id = {r.resource_id: r for r in filter_results}
+ # The resources that pass every check below, in scan order.
+ eligible = []
for resource in undownloaded:
# Check if resource passed the smart filter
filter_result = filter_results_by_id.get(resource.id)
@@ -1587,6 +1719,16 @@ def queue_all_undownloaded(
)
continue
+ eligible.append(resource)
+
+ # One download refills a document that awaits its PDF, so only one
+ # of the resources that would refill the same document is queued.
+ eligible, covered = _one_resource_per_awaiting_document(
+ db_session, eligible, username, user_password
+ )
+ skipped_count += covered
+
+ for resource in eligible:
# Check if already in queue (any status)
existing_queue = (
db_session.query(LibraryDownloadQueue)
@@ -1714,7 +1856,10 @@ def get_research_sources(
"file_type": None,
}
- if document and document.status == "completed":
+ # The Download Manager's rule: a completed document, unless it
+ # awaits its PDF (see document_counts_as_downloaded). A loaded
+ # status never equals the string "completed".
+ if document and document_counts_as_downloaded(document):
source_data.update(
{
"downloaded": True,
@@ -1760,7 +1905,10 @@ async def check_downloads(
)
for resource in resources:
document = get_document_for_resource(db_session, resource)
- if document and document.status == "completed":
+ # A document that awaits its PDF is reported not downloaded,
+ # with the resource_id to download it by (see
+ # document_counts_as_downloaded).
+ if document and document_counts_as_downloaded(document):
# Do NOT return document.file_path — it's the absolute
# server path, which leaks directory layout to any
# authenticated user. The client fetches via
@@ -1821,10 +1969,12 @@ async def download_source(
{"error": "Resource not found"}, status_code=404
)
- # Check if already downloaded
+ # Check if already downloaded, by the Download Manager's rule
+ # (see document_counts_as_downloaded). A document that awaits
+ # its PDF is downloaded again, into the same document.
existing = get_document_for_resource(db_session, resource)
- if existing and existing.status == "completed":
+ if existing and document_counts_as_downloaded(existing):
return {
"success": True,
"message": "Already downloaded",
diff --git a/src/local_deep_research/web/templates/pages/download_manager.html b/src/local_deep_research/web/templates/pages/download_manager.html
index d7d5225841..4fd6ae2f33 100644
--- a/src/local_deep_research/web/templates/pages/download_manager.html
+++ b/src/local_deep_research/web/templates/pages/download_manager.html
@@ -284,7 +284,7 @@
<i aria-hidden="true" class="fas fa-file-pdf"></i>
<span class="ldr-source-title">{{ source.document_title|truncate(60) }}</span>
<span class="ldr-source-domain">{{ source.domain }}</span>
- {% if source.download_status == 'completed' %}
+ {% if source.downloaded %}
<span class="ldr-badge ldr-badge-success">Downloaded</span>
{% endif %}
</div>
diff --git a/tests/research_library/services/test_download_service_arxiv_provenance.py b/tests/research_library/services/test_download_service_arxiv_provenance.py
index 0df0c9fc11..76e557be44 100644
--- a/tests/research_library/services/test_download_service_arxiv_provenance.py
+++ b/tests/research_library/services/test_download_service_arxiv_provenance.py
@@ -461,9 +461,9 @@ def test_arxiv_retry_denial_reads_a_database_stored_pdf(service):
session = MagicMock()
document = MagicMock()
document.id = "database-pdf-document"
- # The string form satisfies both stored-PDF readers' status checks, so
+ # The loaded member satisfies both stored-PDF readers' status checks, so
# the filesystem strategy is stopped by the missing file_path alone.
- document.status = "completed"
+ document.status = DocumentStatus.COMPLETED
document.file_type = "pdf"
document.file_path = None
# First lookup is the resource; any later one is its stored document.
diff --git a/tests/research_library/services/test_download_service_coverage.py b/tests/research_library/services/test_download_service_coverage.py
index 68ecaa0b9c..a3a251352f 100644
--- a/tests/research_library/services/test_download_service_coverage.py
+++ b/tests/research_library/services/test_download_service_coverage.py
@@ -962,7 +962,7 @@ class TestTryExistingPdfExtraction:
def test_not_completed(self, svc):
doc = MagicMock()
- doc.status = "pending"
+ doc.status = DocumentStatus.PENDING
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
doc
@@ -972,7 +972,7 @@ class TestTryExistingPdfExtraction:
def test_no_file_path(self, svc):
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = None
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -983,7 +983,7 @@ class TestTryExistingPdfExtraction:
def test_sentinel_file_path(self, svc):
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = "text_only_not_stored"
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -997,7 +997,7 @@ class TestTryExistingPdfExtraction:
def test_path_traversal_blocked(self, svc):
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = "../../etc/passwd"
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -1019,7 +1019,7 @@ class TestTryExistingPdfExtraction:
def test_file_not_found(self, svc, tmp_path):
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = str(tmp_path / "missing.pdf")
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -1043,7 +1043,7 @@ class TestTryExistingPdfExtraction:
pdf_file.write_bytes(b"fake pdf")
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = str(pdf_file)
doc.id = "doc-1"
session = MagicMock()
@@ -1071,7 +1071,7 @@ class TestTryExistingPdfExtraction:
pdf_file.write_bytes(b"bad pdf")
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = str(pdf_file)
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -1096,7 +1096,7 @@ class TestTryExistingPdfExtraction:
pdf_file.write_bytes(b"pdf")
doc = MagicMock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = str(pdf_file)
session = MagicMock()
session.query.return_value.filter_by.return_value.first.return_value = (
@@ -1919,6 +1919,10 @@ class TestDownloadResource:
session = MagicMock()
resource = MagicMock()
resource.id = 1
+ # No explicit document link (ResearchResource.document_id). A bare
+ # MagicMock attribute is truthy and would send download_resource
+ # down its explicit-link lookup, which these mocks do not stub.
+ resource.document_id = None
session.get.return_value = resource
existing_doc = MagicMock()
@@ -1940,6 +1944,7 @@ class TestDownloadResource:
resource = MagicMock()
resource.id = 1
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
# Return None for existing_doc check, then various other queries
@@ -1980,6 +1985,7 @@ class TestDownloadResource:
resource = MagicMock()
resource.id = 1
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
call_count = [0]
diff --git a/tests/research_library/services/test_download_service_deep_coverage.py b/tests/research_library/services/test_download_service_deep_coverage.py
index 3d96269fec..5f73350148 100644
--- a/tests/research_library/services/test_download_service_deep_coverage.py
+++ b/tests/research_library/services/test_download_service_deep_coverage.py
@@ -62,6 +62,10 @@ class TestDownloadResourceDeep:
resource = MagicMock()
resource.id = 5
resource.url = "https://example.com/paper.pdf"
+ # No explicit document link (ResearchResource.document_id). A bare
+ # MagicMock attribute is truthy and would send download_resource
+ # down its explicit-link lookup, which these mocks do not stub.
+ resource.document_id = None
session.get.return_value = resource
existing_tracker = MagicMock()
@@ -100,6 +104,7 @@ class TestDownloadResourceDeep:
resource = MagicMock()
resource.id = 2
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
tracker = MagicMock()
@@ -137,6 +142,7 @@ class TestDownloadResourceDeep:
resource = MagicMock()
resource.id = 3
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
tracker = MagicMock()
@@ -180,6 +186,7 @@ class TestDownloadResourceDeep:
resource = MagicMock()
resource.id = 4
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
tracker = MagicMock()
@@ -214,6 +221,7 @@ class TestDownloadResourceDeep:
resource = MagicMock()
resource.id = 6
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
tracker = MagicMock()
diff --git a/tests/research_library/services/test_download_service_logic.py b/tests/research_library/services/test_download_service_logic.py
index e956d2417f..3dc5a5f8e9 100644
--- a/tests/research_library/services/test_download_service_logic.py
+++ b/tests/research_library/services/test_download_service_logic.py
@@ -12,6 +12,8 @@ from unittest.mock import Mock, patch
import pytest
+from local_deep_research.database.models.library import DocumentStatus
+
# ---------------------------------------------------------------------------
# Helpers
@@ -235,11 +237,11 @@ class TestTryExistingPdfExtraction:
service.close()
def test_not_completed_returns_none(self, tmp_path):
- """Document with status != 'completed' -> returns None."""
+ """A Document that is not completed -> returns None."""
service = _create_service(tmp_path)
doc = Mock()
- doc.status = "pending"
+ doc.status = DocumentStatus.PENDING
mock_session = Mock()
mock_query = Mock()
@@ -256,7 +258,7 @@ class TestTryExistingPdfExtraction:
service = _create_service(tmp_path)
doc = Mock()
- doc.status = "completed"
+ doc.status = DocumentStatus.COMPLETED
doc.file_path = "pdfs/nonexistent.pdf"
mock_session = Mock()
diff --git a/tests/research_library/services/test_download_service_strategy_coverage.py b/tests/research_library/services/test_download_service_strategy_coverage.py
index 84f0db4b6b..f6e8f8bf86 100644
--- a/tests/research_library/services/test_download_service_strategy_coverage.py
+++ b/tests/research_library/services/test_download_service_strategy_coverage.py
@@ -17,6 +17,7 @@ from unittest.mock import MagicMock, patch
import pytest
+from local_deep_research.database.models.library import DocumentStatus
from local_deep_research.research_library.services.download_service import (
DownloadService,
)
@@ -81,6 +82,10 @@ class TestDownloadResourceArxivStrategy:
resource = MagicMock()
resource.id = 1
resource.url = "https://arxiv.org/abs/2301.00001"
+ # No explicit document link (ResearchResource.document_id). A bare
+ # MagicMock attribute is truthy and would send download_resource
+ # down its explicit-link lookup, which these mocks do not stub.
+ resource.document_id = None
session.get.return_value = resource
# Simulate: existing_doc=None, queue=None, tracker=None, post-download queue=None
@@ -279,6 +284,7 @@ class TestDownloadResourcePubmedStrategy:
resource = MagicMock()
resource.id = 2
resource.url = "https://pubmed.ncbi.nlm.nih.gov/12345678"
+ resource.document_id = None
session.get.return_value = resource
session.query.return_value.filter_by.return_value.first.return_value = (
None
@@ -507,6 +513,7 @@ class TestDownloadResourceDedupCheck:
resource = MagicMock()
resource.id = 10
resource.url = "https://arxiv.org/abs/2301.99999"
+ resource.document_id = None
session.get.return_value = resource
existing_doc = MagicMock()
@@ -531,6 +538,7 @@ class TestDownloadResourceDedupCheck:
resource = MagicMock()
resource.id = 11
resource.url = "https://arxiv.org/abs/2301.12345"
+ resource.document_id = None
session.get.return_value = resource
existing_tracker = MagicMock()
@@ -574,6 +582,7 @@ class TestDownloadResourceDedupCheck:
resource = MagicMock()
resource.id = 12
resource.url = "https://example.com/new-paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
# All filter_by queries return None (no existing doc, queue, or tracker)
@@ -671,7 +680,9 @@ class TestDownloadAsTextSuccess:
pdf_path.write_bytes(b"%PDF-stored")
pdf_document = MagicMock()
pdf_document.id = "stored-pdf-document"
- pdf_document.status = "completed"
+ # The member a loaded row holds, which the filesystem strategy
+ # compares with (the string "completed" never equals it).
+ pdf_document.status = DocumentStatus.COMPLETED
pdf_document.file_path = "stored.pdf"
session.query.return_value.filter_by.return_value.first.side_effect = [
resource,
@@ -903,6 +914,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 20
resource.url = "https://arxiv.org/abs/2301.11111"
+ resource.document_id = None
session.get.return_value = resource
session.query.return_value.filter_by.return_value.first.return_value = (
None
@@ -929,6 +941,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 21
resource.url = "https://example.com/paywalled.pdf"
+ resource.document_id = None
session.get.return_value = resource
session.query.return_value.filter_by.return_value.first.return_value = (
None
@@ -959,6 +972,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 22
resource.url = "https://pubmed.ncbi.nlm.nih.gov/77777777"
+ resource.document_id = None
session.get.return_value = resource
session.query.return_value.filter_by.return_value.first.return_value = (
None
@@ -983,6 +997,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 23
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
session.query.return_value.filter_by.return_value.first.return_value = (
None
@@ -1007,6 +1022,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 24
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
queue_entry = MagicMock()
@@ -1043,6 +1059,7 @@ class TestRetryFailedDownload:
resource = MagicMock()
resource.id = 25
resource.url = "https://example.com/paper.pdf"
+ resource.document_id = None
session.get.return_value = resource
queue_entry = MagicMock()
diff --git a/tests/research_library/services/test_library_service_collections_stats.py b/tests/research_library/services/test_library_service_collections_stats.py
index 072ded666d..813d97a01a 100644
--- a/tests/research_library/services/test_library_service_collections_stats.py
+++ b/tests/research_library/services/test_library_service_collections_stats.py
@@ -746,7 +746,16 @@ class TestGetPdfPreviewsBatch:
assert total_domain_docs == 5
def test_domain_breakdown_counts(self, mocker):
- """Domain breakdown correctly counts total/pdfs/downloaded."""
+ """Domain breakdown correctly counts total/pdfs/downloaded.
+
+ ``status`` is a ``DocumentStatus`` member, as the ORM loads it. Set
+ to the string "completed", it passed a comparison with that string
+ that a loaded document never passes.
+ """
+ from local_deep_research.database.models.library import (
+ DocumentStatus,
+ )
+
service = _make_service()
mock_session = MagicMock()
@@ -754,7 +763,7 @@ class TestGetPdfPreviewsBatch:
doc1.id = "d1"
doc1.research_id = "r1"
doc1.file_type = "pdf"
- doc1.status = "completed"
+ doc1.status = DocumentStatus.COMPLETED
doc1.original_url = None
doc1.filename = "a.pdf"
@@ -762,7 +771,7 @@ class TestGetPdfPreviewsBatch:
doc2.id = "d2"
doc2.research_id = "r1"
doc2.file_type = "pdf"
- doc2.status = "pending"
+ doc2.status = DocumentStatus.PENDING
doc2.original_url = None
doc2.filename = "b.pdf"
@@ -788,6 +797,10 @@ class TestGetPdfPreviewsBatch:
assert arxiv["total"] == 2
assert arxiv["pdfs"] == 2
assert arxiv["downloaded"] == 1 # only doc1 is completed
+ # The "Downloaded" badge of each source.
+ assert [
+ source["downloaded"] for source in result["r1"]["pdf_sources"]
+ ] == [True, False]
def test_resource_none_falls_back_to_doc_filename(self, mocker):
"""When resource is None, title falls back to doc.filename."""
diff --git a/tests/research_library/services/test_library_service_coverage.py b/tests/research_library/services/test_library_service_coverage.py
index 87f0c7b37c..1b9f363084 100644
--- a/tests/research_library/services/test_library_service_coverage.py
+++ b/tests/research_library/services/test_library_service_coverage.py
@@ -14,6 +14,8 @@ Focuses narrowly on uncovered branches:
from contextlib import contextmanager
from unittest.mock import Mock, MagicMock, patch
+from local_deep_research.database.models.library import DocumentStatus
+
MODULE = "local_deep_research.research_library.services.library_service"
@@ -69,7 +71,7 @@ def _setup_get_document_by_id(
mock_doc.file_type = "pdf"
mock_doc.mime_type = "application/pdf"
mock_doc.text_content = None
- mock_doc.status = "completed"
+ mock_doc.status = DocumentStatus.COMPLETED
mock_doc.processed_at = None
mock_doc.favorite = False
mock_doc.tags = []
@@ -285,7 +287,7 @@ class TestGetDocumentByIdResearchCreatedAt:
mock_doc.file_type = "pdf"
mock_doc.mime_type = "application/pdf"
mock_doc.text_content = None
- mock_doc.status = "completed"
+ mock_doc.status = DocumentStatus.COMPLETED
mock_doc.processed_at = None
mock_doc.favorite = False
mock_doc.tags = []
diff --git a/tests/research_library/services/test_library_service_crud_deep.py b/tests/research_library/services/test_library_service_crud_deep.py
index 27bb0079e3..34eb8dbe54 100644
--- a/tests/research_library/services/test_library_service_crud_deep.py
+++ b/tests/research_library/services/test_library_service_crud_deep.py
@@ -16,6 +16,7 @@ from local_deep_research.constants import (
FILE_PATH_METADATA_ONLY,
FILE_PATH_TEXT_ONLY,
)
+from local_deep_research.database.models.library import DocumentStatus
# ============== Helper ==============
@@ -302,7 +303,7 @@ class TestGetDocumentByIdHappyPaths:
mock_doc.text_content = doc_overrides.get(
"text_content", "one two three"
)
- mock_doc.status = "completed"
+ mock_doc.status = DocumentStatus.COMPLETED
mock_doc.processed_at = None
mock_doc.favorite = False
mock_doc.tags = []
@@ -365,6 +366,8 @@ class TestGetDocumentByIdHappyPaths:
assert result is not None
assert result["id"] == "doc-123"
assert result["document_title"] == "Test Paper"
+ # The details page compares the status's string value.
+ assert result["download_status"] == "completed"
assert "has_pdf" in result
assert "word_count" in result
assert "collections" in result
@@ -517,6 +520,22 @@ class TestGetLibraryStatsValues:
# ============== sync_library_with_filesystem ==============
+# The sync deletes a missing-file download only when the document holds
+# nothing else (no text, no blob, no file of its own). A Mock document has
+# no real columns or rows for that check to read, so the deletion tests
+# below pin it to "holds nothing else" (``None``). The rule itself is
+# covered against a real database in
+# tests/research_library/test_library_sync_contracts.py.
+#
+# The sync uses a tracker's file path for a document only when the
+# document stores the same path, as every download that stores a PDF file
+# writes it. The Mock documents below whose tracker names a file store
+# that path.
+_REMAINING_CONTENT = (
+ "local_deep_research.research_library.services.library_service."
+ "LibraryService._remaining_content"
+)
+
class TestSyncLibraryWithFilesystem:
"""Tests for sync_library_with_filesystem logic."""
@@ -543,6 +562,7 @@ class TestSyncLibraryWithFilesystem:
mock_doc.id = "doc-1"
mock_doc.title = "Test"
mock_doc.original_url = "https://example.com/doc.pdf"
+ mock_doc.file_path = "pdfs/doc.pdf"
mock_tracker = Mock()
mock_tracker.file_path = "pdfs/doc.pdf"
@@ -583,7 +603,7 @@ class TestSyncLibraryWithFilesystem:
assert result["files_found"] == 1
def test_doc_with_tracker_file_missing(self, mocker):
- """Doc with tracker but missing file → files_missing + cascade delete."""
+ """Doc with tracker, missing file, nothing else → cascade delete."""
service = _make_service()
mock_session = MagicMock()
@@ -591,6 +611,7 @@ class TestSyncLibraryWithFilesystem:
mock_doc.id = "doc-1"
mock_doc.title = "Test"
mock_doc.original_url = "https://example.com/doc.pdf"
+ mock_doc.file_path = "pdfs/gone.pdf"
mock_tracker = Mock()
mock_tracker.file_path = "pdfs/gone.pdf"
@@ -621,14 +642,19 @@ class TestSyncLibraryWithFilesystem:
mocker.patch(
"local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely"
)
+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)
result = service.sync_library_with_filesystem()
assert result["files_missing"] == 1
assert result["trackers_updated"] == 1
+ remaining.assert_called_once()
def test_doc_without_tracker(self, mocker):
- """Doc with no tracker → files_missing + cascade delete."""
+ """Doc with no tracker that holds nothing else → cascade delete.
+
+ The missing file its own path names is reported under that path.
+ """
service = _make_service()
mock_session = MagicMock()
@@ -636,6 +662,7 @@ class TestSyncLibraryWithFilesystem:
mock_doc.id = "doc-1"
mock_doc.title = "Test"
mock_doc.original_url = "https://example.com/doc.pdf"
+ mock_doc.file_path = "pdfs/gone.pdf"
call_count = {"n": 0}
@@ -654,16 +681,27 @@ class TestSyncLibraryWithFilesystem:
mock_session.query.side_effect = query_router
_mock_session_cm(mocker, mock_session)
+ mock_path = MagicMock()
+ mock_path.is_file.return_value = False
+ mocker.patch(
+ "local_deep_research.research_library.services.library_service.get_absolute_path_from_settings",
+ return_value=mock_path,
+ )
mocker.patch(
"local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely"
)
+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)
result = service.sync_library_with_filesystem()
assert result["files_missing"] == 1
+ assert [
+ (entry["id"], entry["path"]) for entry in result["missing_files"]
+ ] == [("doc-1", "pdfs/gone.pdf")]
+ remaining.assert_called_once()
def test_multiple_mixed_docs(self, mocker):
- """2 found + 1 missing → correct totals."""
+ """2 found + 1 missing that holds nothing else → correct totals."""
service = _make_service()
mock_session = MagicMock()
@@ -673,6 +711,7 @@ class TestSyncLibraryWithFilesystem:
d.id = f"doc-{i}"
d.title = f"Doc {i}"
d.original_url = f"https://example.com/doc{i}.pdf"
+ d.file_path = f"pdfs/doc{i}.pdf"
docs.append(d)
trackers = [Mock(file_path=f"pdfs/doc{i}.pdf") for i in range(3)]
@@ -718,12 +757,15 @@ class TestSyncLibraryWithFilesystem:
mocker.patch(
"local_deep_research.research_library.deletion.utils.cascade_helper.CascadeHelper.delete_document_completely"
)
+ remaining = mocker.patch(_REMAINING_CONTENT, return_value=None)
result = service.sync_library_with_filesystem()
assert result["total_documents"] == 3
assert result["files_found"] == 2
assert result["files_missing"] == 1
+ # Asked only for the document whose file is missing.
+ remaining.assert_called_once()
def test_settings_manager_resolved_once_not_per_document(self, mocker):
"""Regression test for the #6095 follow-up (issue 1).
@@ -749,6 +791,7 @@ class TestSyncLibraryWithFilesystem:
d.id = f"doc-{i}"
d.title = f"Doc {i}"
d.original_url = f"https://example.com/doc{i}.pdf"
+ d.file_path = f"pdfs/doc{i}.pdf"
docs.append(d)
trackers = [Mock(file_path=f"pdfs/doc{i}.pdf") for i in range(3)]
diff --git a/tests/research_library/test_library_sync_contracts.py b/tests/research_library/test_library_sync_contracts.py
new file mode 100644
index 0000000000..bfeae4c341
--- /dev/null
+++ b/tests/research_library/test_library_sync_contracts.py
@@ -0,0 +1,4238 @@
+"""Contracts for what "Sync library" does with downloads whose PDF is missing.
+
+``LibraryService.sync_library_with_filesystem`` is there to deal with a
+downloaded document whose PDF file was removed from disk, so that its URL
+can be downloaded again. Before PDF and text documents were merged into
+one ``Document`` row, the sync checked PDF records only and left text
+documents alone. Deleting the merged row also deletes its text, its PDF
+blob, its collection links and, after the commit, its chunks and vectors.
+
+The contract pinned here: the sync deletes a download whose file is
+missing only when nothing else of it is stored (no extracted text, no PDF
+blob in the database, no file at the document's own path). The
+``storage_mode`` label and the tracker's ``file_path`` cannot decide this
+on their own, because production writers leave both stale:
+
+ * R1: re-downloading an existing document in "none" mode, or in an
+ unrecognised mode that ``PDFStorageManager`` runs as "none", clears
+ the tracker path. The document keeps its stored PDF and its label.
+ * R2: ``_record_failed_text_extraction`` creates its placeholder with
+ the ``"database"`` column default. ``_save_text_with_db`` later
+ completes it with text and never stores a PDF.
+ * R3: rows already written with ``storage_mode="database"`` and no blob
+ keep that shape; nothing repairs them. A content-hash duplicate
+ download copies ``"database"`` into the shared tracker.
+ * A Zotero import stores its PDF blob and the item's source URL but
+ creates no tracker at all.
+ * A URL has one tracker, which every document with that URL shares: the
+ documents of two researches that cite the same paper, or a
+ failed-download placeholder that "Extract Text to DB" completed next
+ to the document holding the PDF. Its path names the file written
+ last, so it describes a document only when it is the document's own
+ path, or when it is ``"database"`` and the document is labelled as
+ stored in the database, which can be several documents. The sync
+ judges every other document by what it stores itself and leaves the
+ tracker alone. A ``"database"`` tracker is reset only when no
+ document under it still holds a PDF blob.
+
+A download whose file was removed while its extracted text was not keeps
+the text. If it still has its research resource, the sync makes it
+text-only, resets its tracker and reports the file missing, and the
+download manager's PDF downloads put the PDF back into the same document,
+also when the PDF downloaded again has other bytes. A run downloads it
+once, also when a second research's resource is linked to the document:
+through the document's own resource, unless that one cannot succeed or a
+restart left its queue entry PROCESSING, and the linked one can.
+Choosing that resource records no denial of the egress policy and loads
+no document's text. The Download Manager counts such a document as
+available to download, in its totals and in its research's sources, and
+so do the library's source routes, through which ``download_source``
+puts its PDF back. Every document the sync reports this way is one
+``document_awaits_pdf`` asks for, including a failed-download placeholder
+that a PDF download completed. Text extraction leaves such a document
+alone: it already has the text. Without its resource no download path
+asks for the PDF, so the sync makes the document text-only without
+reporting it, and a later download of the same PDF bytes for another
+resource stores the PDF in it. A PDF removed with "Remove PDF to save
+space" is not a missing file: the document is kept as it is, and its PDF
+is not downloaded again. The sync's response names each missing file by
+its stored path, relative to the library root, never by its absolute
+path on the server, and counts each tracker it changes once in
+``trackers_updated``, whichever document of the tracker's URL it reads
+first.
+
+A loaded ``Document.status`` is a ``DocumentStatus`` member, which never
+equals its string value, so it is compared with the member: text
+extraction reads the PDF file that a completed download stores before it
+asks the source, and the document details page shows the document's
+status.
+
+The deletion the sync is for still happens. A filesystem-mode download
+with no text whose file was removed is deleted and can be downloaded
+again, and so can a database-mode download with no blob and no text,
+which the sync reports as ``"database"``.
+
+The harness follows ``test_download_service_contracts.py``: a real
+on-disk SQLite file with foreign keys on, a real ``tmp_path`` library
+store, the production ``DownloadService`` writers (built with ``__new__``
+around a stub downloader, so no request is ever made), the production
+sync, ``delete_blob_only``, download queueing, the bulk download route,
+the Download Manager page, the library's source routes and the document
+details page. Only the session, settings and password sources are
+replaced, plus the retry-policy filter of the "Download All New" route
+and the routes' ``DownloadService`` factory. The service's retry manager
+has no attempts on record, except where a case uses the production one
+over the test database, and its egress policy allows every URL, except
+where a case refuses one, through a stand-in or, in one case, the
+production policy. Rows are read back through a fresh session.
+"""
+
+import asyncio
+import hashlib
+import json
+import re
+import uuid
+from collections.abc import Set as AbstractSet
+from contextlib import contextmanager, nullcontext
+from datetime import UTC, datetime, timedelta
+from types import SimpleNamespace
+
+import pytest
+from loguru import logger
+from sqlalchemy import create_engine, event
+from sqlalchemy.orm import sessionmaker
+from starlette.requests import Request
+
+from local_deep_research.constants import (
+ FILE_PATH_BLOB_DELETED,
+ FILE_PATH_TEXT_ONLY,
+)
+from local_deep_research.database.models import Base
+from local_deep_research.database.models.download_tracker import (
+ DownloadAttempt,
+ DownloadTracker,
+)
+from local_deep_research.database.models.library import (
+ Collection,
+ Document,
+ DocumentBlob,
+ DocumentChunk,
+ DocumentCollection,
+ DocumentStatus,
+ DownloadQueue,
+ EmbeddingProvider,
+ SourceType,
+)
+from local_deep_research.database.models.research import (
+ ResearchHistory,
+ ResearchResource,
+)
+from local_deep_research.library.download_management import (
+ FailureClassifier,
+ ResourceDownloadStatus,
+ ResourceFilter,
+ ResourceStatusTracker,
+ RetryDecision,
+ RetryManager,
+)
+from local_deep_research.library.download_management import (
+ models as retry_models,
+)
+from local_deep_research.research_library import utils as library_utils
+from local_deep_research.research_library.deletion.services import (
+ document_deletion,
+)
+from local_deep_research.research_library.downloaders.base import (
+ ContentType,
+ DownloadResult,
+)
+from local_deep_research.research_library.services import (
+ download_service,
+ library_service,
+)
+from local_deep_research.research_library.services.download_service import (
+ DownloadService,
+)
+from local_deep_research.research_library.services.pdf_storage_manager import (
+ PDFStorageManager,
+ resolve_pdf_storage_mode,
+)
+from local_deep_research.research_library.utils import (
+ document_awaits_pdf,
+ find_tracker_by_url,
+ get_url_hash,
+)
+from local_deep_research.security.egress.audit_hook import (
+ clear_active_context,
+)
+from local_deep_research.security.egress.policy import (
+ EgressContext,
+ EgressScope,
+)
+from local_deep_research.utilities import db_utils
+from local_deep_research.web.routers import library as library_router
+
+USERNAME = "library_sync_user"
+LIBRARY_COLLECTION_ID = "library-sync-contracts"
+PDF_BYTES = (
+ b"%PDF-1.4\n1 0 obj<</Type/Catalog>>endobj\ntrailer<</Root 1 0 R>>\n%%EOF\n"
+)
+PDF_HASH = hashlib.sha256(PDF_BYTES).hexdigest()
+SAVED_TEXT = "Text that only this document holds."
+EXTRACTED_TEXT = "Text extracted from the downloaded PDF."
+
+
+def _text_pdf(text):
+ """A one-page PDF with a real text layer.
+
+ ``PDF_BYTES`` has no pages, so the download's text extraction finds no
+ text in it. This one goes through the real extraction and yields
+ ``text``, the usual case for a downloaded paper.
+ """
+ objects = [
+ b"<< /Type /Catalog /Pages 2 0 R >>",
+ b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+ b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 400 200] "
+ b"/Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
+ b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>",
+ ]
+ stream = f"BT /F1 12 Tf 10 100 Td ({text}) Tj ET".encode("latin-1")
+ objects.append(
+ b"<< /Length "
+ + str(len(stream)).encode()
+ + b" >>\nstream\n"
+ + stream
+ + b"\nendstream"
+ )
+ out = bytearray(b"%PDF-1.4\n")
+ offsets = []
+ for number, body in enumerate(objects, start=1):
+ offsets.append(len(out))
+ out += f"{number} 0 obj\n".encode() + body + b"\nendobj\n"
+ xref_offset = len(out)
+ out += f"xref\n0 {len(objects) + 1}\n0000000000 65535 f \n".encode()
+ for offset in offsets:
+ out += f"{offset:010d} 00000 n \n".encode()
+ out += (
+ f"trailer\n<< /Size {len(objects) + 1} /Root 1 0 R >>\n"
+ f"startxref\n{xref_offset}\n%%EOF\n"
+ ).encode()
+ return bytes(out)
+
+
+TEXT_PDF_BYTES = _text_pdf(EXTRACTED_TEXT)
+# The same paper served again as a regenerated PDF: other bytes, other text.
+REGENERATED_TEXT = "Text extracted from the regenerated PDF."
+REGENERATED_PDF_BYTES = _text_pdf(REGENERATED_TEXT)
+# Hard-coded for the same reason as in test_download_service_contracts.py:
+# a renamed gate must fail the filesystem cases, not turn them into
+# database-mode cases.
+FILESYSTEM_GATE_ENV = "LDR_RESEARCH_LIBRARY_ALLOW_FILESYSTEM_PDF_STORAGE"
+
+
+class StubSettings:
+ """Minimal stand-in for SettingsManager.get_setting."""
+
+ def __init__(self, values):
+ self._values = values
+
+ def get_setting(self, key, default=None):
+ return self._values.get(key, default)
+
+
+class StubDownloader:
+ """Returns ``content`` for every URL and records the calls.
+
+ A URL in ``unreachable`` fails the way a server that is down does
+ (HTTP 503).
+ """
+
+ def __init__(self, content=PDF_BYTES, unreachable=()):
+ self.content = content
+ self.unreachable = set(unreachable)
+ self.urls = []
+
+ def can_handle(self, url):
+ return True
+
+ def download_with_result(self, url, content_type=ContentType.PDF):
+ self.urls.append(url)
+ if url in self.unreachable:
+ return DownloadResult(
+ skip_reason="Service unavailable", status_code=503
+ )
+ return DownloadResult(
+ content=self.content, is_success=True, status_code=200
+ )
+
+
+class NoRetryHistory:
+ """A retry manager with no attempts on record: it holds nothing back."""
+
+ status_tracker = SimpleNamespace(get_resource_status=lambda _id: None)
+
+ def record_attempt(self, **kwargs):
+ pass
+
+ def should_retry_resource(self, resource_id):
+ return RetryDecision(can_retry=True)
+
+
+def store(tmp_path):
+ """The per-user library store the download and the sync both use."""
+ base = tmp_path / "library"
+ root = base / USERNAME
+ (root / "pdfs").mkdir(parents=True)
+ return SimpleNamespace(base=base, root=root)
+
+
+def _tree(root):
+ """Every file under ``root``, as posix-relative strings."""
+ return {
+ p.relative_to(root).as_posix() for p in root.rglob("*") if p.is_file()
+ }
+
+
+def db(tmp_path):
+ """Real on-disk SQLite, FK enforcement on, plus a fresh-session reader."""
+ db_dir = tmp_path / "db"
+ db_dir.mkdir()
+ engine = create_engine(f"sqlite:///{db_dir / 'library.db'}")
+
+ @event.listens_for(engine, "connect")
+ def _fk_on(dbapi_connection, _record):
+ cur = dbapi_connection.cursor()
+ cur.execute("PRAGMA foreign_keys = ON")
+ cur.close()
+
+ Base.metadata.create_all(engine)
+ Session = sessionmaker(bind=engine)
+
+ @contextmanager
+ def fresh():
+ """A brand-new session on the same file, never the mutated one."""
+ with Session() as other:
+ yield other
+
+ with Session() as session:
+ try:
+ yield SimpleNamespace(engine=engine, session=session, fresh=fresh)
+ finally:
+ engine.dispose()
+
+
+def world(db, monkeypatch):
+ """A research session, the Library collection and resource factory.
+
+ The download service's own session, source-type and default-collection
+ lookups are pointed at this database.
+ """
+ session = db.session
+ source_type = SourceType(
+ id=str(uuid.uuid4()),
+ name="research_download",
+ display_name="Research Download",
+ )
+ research = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="library sync contract",
+ mode="quick",
+ status="completed",
+ created_at="2026-08-01T00:00:00",
+ )
+ session.add_all(
+ [
+ source_type,
+ research,
+ Collection(id=LIBRARY_COLLECTION_ID, name="Library"),
+ ]
+ )
+ session.commit()
+ source_type_id = source_type.id
+ research_id = research.id
+
+ monkeypatch.setattr(
+ download_service, "get_source_type_id", lambda *a, **k: source_type_id
+ )
+ monkeypatch.setattr(
+ download_service,
+ "get_default_library_id",
+ lambda *a, **k: LIBRARY_COLLECTION_ID,
+ )
+ monkeypatch.setattr(
+ download_service, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+
+ def add_resource(url, *, tracker=True, downloaded=False):
+ """A cited resource, with the URL's tracker unless one exists."""
+ resource = ResearchResource(
+ research_id=research_id,
+ title="Sync Contract Paper",
+ url=url,
+ source_type="academic",
+ created_at="2026-08-01T00:00:00",
+ )
+ session.add(resource)
+ session.commit()
+ if tracker:
+ session.add(
+ DownloadTracker(
+ url=url,
+ url_hash=get_url_hash(url),
+ first_resource_id=resource.id,
+ is_downloaded=downloaded,
+ file_hash=PDF_HASH if downloaded else None,
+ )
+ )
+ session.commit()
+ return resource.id
+
+ return SimpleNamespace(
+ source_type_id=source_type_id,
+ research_id=research_id,
+ add_resource=add_resource,
+ )
+
+
+def one_download_session(db, world, monkeypatch):
+ """Give the download service one reused session, as production does.
+
+ ``get_user_db_session`` hands every scope on a thread the same
+ thread-local session and does not close it at scope exit. The
+ download saves the extracted text in a later scope than the one that
+ stored the PDF, through the resource instance that scope loaded. With
+ the fresh session per scope that ``world`` sets up, that instance is
+ detached by then and the text is never saved. The cases whose
+ documents must hold the text of a real extraction use this.
+ """
+ session = sessionmaker(bind=db.engine)()
+ monkeypatch.setattr(
+ download_service,
+ "get_user_db_session",
+ lambda *a, **k: nullcontext(session),
+ )
+ try:
+ yield session
+ finally:
+ session.close()
+
+
+def filesystem_mode(monkeypatch):
+ """Open the operator gate for unencrypted filesystem PDF storage."""
+ monkeypatch.setenv(FILESYSTEM_GATE_ENV, "true")
+ assert resolve_pdf_storage_mode("filesystem") == "filesystem", (
+ "the filesystem-storage operator gate did not open; the filesystem "
+ "cases would silently run in database mode"
+ )
+
+
+def retry_history(db):
+ """The production retry manager, keeping its record in this database.
+
+ ``download_resource`` records each attempt with it, and the refill
+ ranking and the "Download All New" retry-policy filter read it back.
+ Only its database source is replaced.
+ """
+ retry_models.Base.metadata.create_all(db.engine)
+ tracker = ResourceStatusTracker.__new__(ResourceStatusTracker)
+ tracker.username = USERNAME
+ tracker.password = None
+ tracker.engine = db.engine
+ tracker.Session = sessionmaker(bind=db.engine)
+ manager = RetryManager.__new__(RetryManager)
+ manager.username = USERNAME
+ manager.failure_classifier = FailureClassifier()
+ manager.status_tracker = tracker
+ return manager
+
+
+def make_service(
+ store, downloader, *, storage_mode, retry_manager=None, refused_urls=()
+):
+ """A ``DownloadService`` reduced to what ``download_resource`` uses.
+
+ ``retry_manager`` defaults to one without attempts on record. The
+ egress policy refuses ``refused_urls`` and allows every other URL, at
+ queueing and at the download alike.
+ """
+ service = DownloadService.__new__(DownloadService)
+ service.username = USERNAME
+ service.password = None
+ service._closed = False
+ service.settings = StubSettings(
+ {
+ "research_library.pdf_storage_mode": storage_mode,
+ "research_library.max_pdf_size_mb": 3072,
+ }
+ )
+ service.library_root = str(store.root)
+ service.downloaders = [downloader]
+ service.retry_manager = retry_manager or NoRetryHistory()
+ service._egress_context = None
+ if refused_urls:
+ refused = set(refused_urls)
+ service._check_url_against_policy = lambda url, record_denial=True: (
+ (False, "refused_by_test_policy")
+ if url in refused
+ else (True, "no_context")
+ )
+ return service
+
+
+def make_library_service(db, store, monkeypatch):
+ """A ``LibraryService`` whose real sync runs against this database.
+
+ Only the session and settings sources are replaced. The keep/delete
+ rule, tracker lookup, blob check, path resolution against ``store``
+ and the post-commit purge are the production code.
+ """
+ monkeypatch.setattr(
+ library_service, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ monkeypatch.setattr(
+ document_deletion, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ settings = StubSettings({"research_library.storage_path": str(store.base)})
+ monkeypatch.setattr(
+ library_service,
+ "get_settings_manager",
+ lambda **_kwargs: SimpleNamespace(
+ get_setting=settings.get_setting, close=lambda: None
+ ),
+ )
+ service = library_service.LibraryService.__new__(
+ library_service.LibraryService
+ )
+ service.username = USERNAME
+ return service
+
+
+def _library_links(session, document_id):
+ return (
+ session.query(DocumentCollection)
+ .filter_by(document_id=document_id)
+ .count()
+ )
+
+
+def _add_chunk(db, document_id):
+ """A search-index chunk row of the document, as indexing writes one."""
+ text = f"chunk of {document_id}"
+ chunk = DocumentChunk(
+ chunk_hash=hashlib.sha256(text.encode()).hexdigest(),
+ source_type="document",
+ source_id=document_id,
+ collection_name=f"collection_{LIBRARY_COLLECTION_ID}",
+ chunk_text=text,
+ chunk_index=0,
+ start_char=0,
+ end_char=len(text),
+ word_count=len(text.split()),
+ embedding_id=str(uuid.uuid4()),
+ embedding_model="fake-model",
+ embedding_model_type=EmbeddingProvider.SENTENCE_TRANSFORMERS,
+ embedding_dimension=8,
+ )
+ db.session.add(chunk)
+ db.session.commit()
+ return chunk.id
+
+
+def _cite_in_another_research(
+ db, url, query="second research citing the paper"
+):
+ """A further research session citing ``url``, and its resource."""
+ research = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query=query,
+ mode="quick",
+ status="completed",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(research)
+ db.session.commit()
+ resource = ResearchResource(
+ research_id=research.id,
+ title="Sync Contract Paper",
+ url=url,
+ source_type="academic",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(resource)
+ db.session.commit()
+ return research.id, resource.id
+
+
+def _add_database_row_without_pdf(
+ db, world, resource_id, research_id, url, content
+):
+ """An existing database-mode download that holds no PDF and no text.
+
+ The R3 shape without text: a database-mode download whose PDF the size
+ limit refused, and that had no text to extract, left a completed row
+ labelled ``"database"``, with no ``file_path``, no blob and no text.
+ Nothing repairs such rows. ``content`` is the refused PDF, whose hash
+ the row keeps, so a later download of the same bytes is linked to it.
+ """
+ document_id = str(uuid.uuid4())
+ db.session.add(
+ Document(
+ id=document_id,
+ source_type_id=world.source_type_id,
+ resource_id=resource_id,
+ research_id=research_id,
+ document_hash=hashlib.sha256(content).hexdigest(),
+ original_url=url,
+ file_size=len(content),
+ file_type="pdf",
+ mime_type="application/pdf",
+ title="Sync Contract Paper",
+ status=DocumentStatus.COMPLETED,
+ storage_mode="database",
+ file_path=None,
+ )
+ )
+ db.session.add(
+ DocumentCollection(
+ document_id=document_id, collection_id=LIBRARY_COLLECTION_ID
+ )
+ )
+ db.session.commit()
+ return document_id
+
+
+class PassingResourceFilter:
+ """The retry-policy filter, letting every resource through."""
+
+ def __init__(self, *args, **kwargs):
+ pass
+
+ def filter_downloadable_resources(self, resources):
+ return [
+ SimpleNamespace(resource_id=row.id, can_retry=True, reason="ok")
+ for row in resources
+ ]
+
+ def get_filter_summary(self, resources):
+ return SimpleNamespace(
+ to_dict=lambda: {},
+ permanently_failed_count=0,
+ temporarily_failed_count=0,
+ )
+
+ def get_skipped_resources_info(self, resources):
+ return []
+
+
+def claim_queue_entry(db, resource_id, research_id):
+ """Queue ``resource_id`` and claim it as a bulk download stream does.
+
+ The claim is the production one (``_claim_download_queue_item``), so
+ the entry is PROCESSING and carries the time the claim records.
+ """
+ entry = DownloadQueue(
+ resource_id=resource_id,
+ research_id=research_id,
+ collection_id=LIBRARY_COLLECTION_ID,
+ status=DocumentStatus.PENDING,
+ )
+ db.session.add(entry)
+ db.session.commit()
+ assert library_router._claim_download_queue_item(db.session, entry.id)
+ with db.fresh() as other:
+ claimed = other.get(DownloadQueue, entry.id)
+ assert claimed.status is DocumentStatus.PROCESSING
+ assert claimed.last_attempt_at is not None
+ return entry.id
+
+
+class RetryPolicyFilter(ResourceFilter):
+ """The production retry-policy filter, over the given retry manager."""
+
+ def __init__(self, retry_manager):
+ self.username = USERNAME
+ self.retry_manager = retry_manager
+
+
+# The ranking-only DownloadService of "Download All New" stores nothing.
+_NO_STORE = SimpleNamespace(root="unused-library-root", base="unused-base")
+
+
+def download_all_new(db, monkeypatch, *, retry_policy=None, **service_options):
+ """Queue what the download manager's "Download All New" would.
+
+ Runs the ``queue_all_undownloaded`` route against this database. Its
+ session, the user's password and the retry-policy filter are replaced:
+ the filter lets every resource through, or is the production filter
+ over ``retry_policy``. So is the ``DownloadService`` the route ranks a
+ document's resources with (``service_options`` as for
+ ``make_service``). Which resources count as not downloaded, and which
+ of a document's resources is queued, are the production code.
+ """
+ monkeypatch.setattr(
+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ monkeypatch.setattr(
+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"
+ )
+ monkeypatch.setattr(
+ library_router,
+ "ResourceFilter",
+ PassingResourceFilter
+ if retry_policy is None
+ else (lambda *a, **k: RetryPolicyFilter(retry_policy)),
+ )
+ monkeypatch.setattr(
+ library_router,
+ "DownloadService",
+ lambda *a, **k: make_service(
+ _NO_STORE, StubDownloader(), storage_mode="none", **service_options
+ ),
+ )
+ return library_router.queue_all_undownloaded(None, username=USERNAME)
+
+
+def bulk_download(
+ db, store, downloader, monkeypatch, research_ids, *, mode, **service_options
+):
+ """Run the download manager's bulk route and return its SSE events.
+
+ "Get All Research PDFs" posts ``mode="pdf"``, "Extract Text to DB"
+ posts ``mode="text_only"``. The route's session, the user's password
+ and its ``DownloadService`` (a filesystem-mode service around
+ ``downloader``, with ``service_options`` as for ``make_service``) are
+ replaced. Which resources it queues, and the claim and download of
+ each, are the production code.
+ """
+ monkeypatch.setattr(
+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ monkeypatch.setattr(
+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"
+ )
+ monkeypatch.setattr(
+ library_router,
+ "DownloadService",
+ lambda *a, **k: make_service(
+ store, downloader, storage_mode="filesystem", **service_options
+ ),
+ )
+ body = json.dumps(
+ {
+ "research_ids": research_ids,
+ "mode": mode,
+ "collection_id": LIBRARY_COLLECTION_ID,
+ }
+ ).encode()
+
+ async def receive():
+ return {"type": "http.request", "body": body, "more_body": False}
+
+ request = Request(
+ {
+ "type": "http",
+ "method": "POST",
+ "path": "/library/api/download-bulk",
+ "headers": [(b"content-type", b"application/json")],
+ "query_string": b"",
+ "session": {"session_id": "sid"},
+ },
+ receive,
+ )
+ response = asyncio.run(
+ library_router.download_bulk(request, username=USERNAME)
+ )
+
+ async def drain():
+ chunks = []
+ async for chunk in response.body_iterator:
+ chunks.append(chunk if isinstance(chunk, bytes) else chunk.encode())
+ return b"".join(chunks).decode()
+
+ return [
+ json.loads(line[len("data: ") :])
+ for line in asyncio.run(drain()).split("\n")
+ if line.startswith("data: ")
+ ]
+
+
+# ===========================================================================
+# Documents that still hold content are kept
+# ===========================================================================
+
+
[email protected]("redownload_mode", ["none", "Database"])
[email protected]("stored_mode", ["database", "filesystem"])
+def test_redownload_that_stores_no_pdf_keeps_the_stored_one(
+ db, store, world, filesystem_mode, monkeypatch, stored_mode, redownload_mode
+):
+ """R1: the PDF stored earlier keeps the document through a sync.
+
+ No text is extracted, so the stored PDF is the document's only
+ content. After the re-download the tracker has no path while the
+ document still names its store. The sync kept a path-less document
+ only when its label was "none", so it deleted this one, and with it
+ the blob (database) or every reference to the file (filesystem).
+ """
+ url = "https://example.com/stored-then-redownloaded.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader()
+ service = make_service(store, downloader, storage_mode=stored_mode)
+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document_id = (
+ other.query(Document).filter_by(resource_id=resource_id).one().id
+ )
+ stored_files = _tree(store.root)
+ assert len(stored_files) == (1 if stored_mode == "filesystem" else 0)
+
+ library = make_library_service(db, store, monkeypatch)
+ assert library.mark_for_redownload([document_id]) == 1
+ service.settings._values["research_library.pdf_storage_mode"] = (
+ redownload_mode
+ )
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+ with db.fresh() as other:
+ # The R1 state: the tracker lost its path, the document kept its PDF.
+ assert find_tracker_by_url(other, url).file_path is None
+ document = other.get(Document, document_id)
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.storage_mode == stored_mode
+ assert document.text_content is None
+
+ stats = library.sync_library_with_filesystem()
+
+ assert stats["files_missing"] == 0
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert (
+ PDFStorageManager(store.root, "none").load_pdf(document, other)
+ == PDF_BYTES
+ )
+ assert other.query(DocumentBlob).count() == (
+ 1 if stored_mode == "database" else 0
+ )
+ assert _library_links(other, document_id) == 1
+ assert _tree(store.root) == stored_files
+ assert library.get_document_by_id(document_id)["has_pdf"] is True
+ # The kept document still counts as downloaded: no third request.
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+
+
+ "tracker", [True, False], ids=["tracker-without-path", "no-tracker"]
+)
+def test_failed_placeholder_completed_with_text_is_kept(
+ db, store, world, monkeypatch, tracker
+):
+ """R2: text saved into a failed-download placeholder survives a sync.
+
+ The placeholder keeps the ``"database"`` column default and never gets
+ a blob, so before the fix the sync treated it as a stored PDF that had
+ gone missing and deleted the text with it. ``download_as_text`` creates
+ no tracker; a resource whose PDF download was attempted first has one
+ without a path.
+ """
+ url = "https://example.com/placeholder-then-text.pdf"
+ resource_id = world.add_resource(url, tracker=tracker)
+ service = make_service(store, StubDownloader(), storage_mode="database")
+ resource = db.session.get(ResearchResource, resource_id)
+ service._record_failed_text_extraction(
+ db.session, resource, error="PDF download failed: HTTP 503"
+ )
+ db.session.commit()
+ service._save_text_with_db(
+ resource,
+ SAVED_TEXT,
+ db.session,
+ extraction_method="native_api",
+ extraction_source="arxiv_api",
+ )
+ db.session.commit()
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.storage_mode == "database"
+ assert document.file_path is None
+ assert other.query(DocumentBlob).count() == 0
+
+ library = make_library_service(db, store, monkeypatch)
+ stats = library.sync_library_with_filesystem()
+
+ assert stats["files_missing"] == 0
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.text_content == SAVED_TEXT
+ # Kept as it was: the sync does not rewrite the label.
+ assert document.storage_mode == "database"
+ view = library.get_document_by_id(document_id)
+ assert view["has_text_db"] is True
+ assert view["has_pdf"] is False
+
+
[email protected]("shape", ["tracker-without-path", "duplicate-link"])
+def test_database_row_without_blob_keeps_its_text(
+ db, store, world, monkeypatch, shape
+):
+ """R3: an existing ``"database"`` row with no blob keeps its text.
+
+ A PDF that the size limit refused in database mode left this row: a
+ ``"database"`` label, no blob, the extracted text, and a downloaded
+ tracker without a path. Nothing repairs such rows, so the sync must
+ tolerate them. In ``duplicate-link`` a second research cites the same
+ URL and downloads identical bytes. The content-hash dedup links it to
+ this row and copies the row's ``"database"`` label into the shared
+ tracker. A ``"database"`` tracker describes a row labelled as stored
+ in the database, without a path, so the sync takes its tracker-path
+ branch for this row instead, which keeps the text and the tracker
+ too.
+ """
+ url = "https://example.com/refused-earlier.pdf"
+ resource_id = world.add_resource(url, downloaded=True)
+ document_id = str(uuid.uuid4())
+ db.session.add(
+ Document(
+ id=document_id,
+ source_type_id=world.source_type_id,
+ resource_id=resource_id,
+ research_id=world.research_id,
+ document_hash=PDF_HASH,
+ original_url=url,
+ file_size=len(PDF_BYTES),
+ file_type="pdf",
+ mime_type="application/pdf",
+ title="Sync Contract Paper",
+ status=DocumentStatus.COMPLETED,
+ storage_mode="database",
+ file_path=None,
+ text_content=SAVED_TEXT,
+ )
+ )
+ db.session.add(
+ DocumentCollection(
+ document_id=document_id, collection_id=LIBRARY_COLLECTION_ID
+ )
+ )
+ db.session.commit()
+ expected_tracker_path = None
+ if shape == "duplicate-link":
+ duplicate_id = world.add_resource(url, tracker=False)
+ service = make_service(store, StubDownloader(), storage_mode="database")
+ monkeypatch.setattr(
+ service, "_extract_text_from_pdf", lambda _data: None
+ )
+ assert service.download_resource(duplicate_id) == (True, None)
+ expected_tracker_path = "database"
+ with db.fresh() as other:
+ linked = other.get(ResearchResource, duplicate_id)
+ assert linked.document_id == document_id
+ assert other.query(Document).count() == 1
+ assert other.query(DocumentBlob).count() == 0
+ with db.fresh() as other:
+ assert find_tracker_by_url(other, url).file_path == (
+ expected_tracker_path
+ )
+
+ library = make_library_service(db, store, monkeypatch)
+ stats = library.sync_library_with_filesystem()
+
+ assert stats["files_missing"] == 0
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.text_content == SAVED_TEXT
+ assert document.storage_mode == "database"
+ assert _library_links(other, document_id) == 1
+ # The tracker is left as it was, still marked downloaded.
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is True
+ assert tracker.file_path == expected_tracker_path
+ view = library.get_document_by_id(document_id)
+ assert view["has_text_db"] is True
+ assert view["has_pdf"] is False
+
+
+def test_zotero_import_with_a_source_url_is_kept(db, store, monkeypatch):
+ """A Zotero-imported PDF has a source URL but no download tracker.
+
+ ``ZoteroSyncService`` stores the PDF blob through ``save_pdf`` in
+ "database" mode and copies the item's URL (or DOI link) into
+ ``original_url``, which puts the document in the sync's scope. With no
+ tracker it took the no-path branch, which kept only "none"-mode and
+ sentinel documents.
+ """
+ source_type = SourceType(
+ id=str(uuid.uuid4()), name="zotero", display_name="Zotero"
+ )
+ db.session.add(source_type)
+ db.session.flush()
+ document = Document(
+ id=str(uuid.uuid4()),
+ source_type_id=source_type.id,
+ document_hash=PDF_HASH,
+ original_url="https://doi.org/10.1234/zotero-item",
+ filename="Zotero Item.pdf",
+ file_size=len(PDF_BYTES),
+ file_type="pdf",
+ mime_type="application/pdf",
+ title="Zotero Item",
+ status=DocumentStatus.COMPLETED,
+ text_content=SAVED_TEXT,
+ extraction_source="zotero",
+ )
+ db.session.add(document)
+ db.session.flush()
+ PDFStorageManager(store.root, "database").save_pdf(
+ pdf_content=PDF_BYTES,
+ document=document,
+ session=db.session,
+ filename="Zotero Item.pdf",
+ )
+ db.session.commit()
+ document_id = document.id
+
+ stats = make_library_service(
+ db, store, monkeypatch
+ ).sync_library_with_filesystem()
+
+ assert stats["files_missing"] == 0
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.storage_mode == "database"
+ assert document.text_content == SAVED_TEXT
+ assert (
+ PDFStorageManager(store.root, "none").load_pdf(document, other)
+ == PDF_BYTES
+ )
+
+
+# ===========================================================================
+# A removed PDF file whose text is left: keep the text, re-download the PDF
+# ===========================================================================
+
+
[email protected]("via", ["research", "download-all-new"])
[email protected]("tracker_path", ["kept", "lost"])
+def test_removed_file_of_a_download_with_text_is_reported_and_downloaded_again(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ tracker_path,
+ via,
+):
+ """The text stays, the missing PDF is reported and downloaded again.
+
+ The usual filesystem-mode case: the download's real text extraction
+ saved the PDF's text, then the user removed the PDF file from disk.
+ The sync keeps the document, its text, its collection link and its
+ chunks. It stores the document as text-only and resets the tracker, so
+ the library no longer offers a PDF that is not there, and it reports
+ the file missing. The download manager's research download and its
+ "Download All New" both queue the document, and the download puts the
+ PDF back into the same document. ``lost`` is the R1 sequence (a
+ "none"-mode re-download cleared the tracker path) with the file removed
+ afterwards, which reaches the sync's no-path branch.
+ """
+ url = "https://example.com/removed-but-extracted.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ own_path = document.file_path
+ assert document.text_content == EXTRACTED_TEXT
+ stored = store.root / own_path
+ assert stored.is_file()
+ library = make_library_service(db, store, monkeypatch)
+ if tracker_path == "lost":
+ assert library.mark_for_redownload([document_id]) == 1
+ service.settings._values["research_library.pdf_storage_mode"] = "none"
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ assert find_tracker_by_url(other, url).file_path is None
+ assert other.get(Document, document_id).storage_mode == (
+ "filesystem"
+ )
+ chunk_id = _add_chunk(db, document_id)
+
+ # While the file is on disk, text or no text, nothing is missing.
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.file_path == own_path
+ assert library.get_document_by_id(document_id)["has_pdf"] is True
+
+ stored.unlink()
+
+ stats = library.sync_library_with_filesystem()
+
+ assert stats["files_found"] == 0
+ assert stats["files_missing"] == 1
+ assert [entry["id"] for entry in stats["missing_files"]] == [document_id]
+ assert stats["trackers_updated"] == 1
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.text_content == EXTRACTED_TEXT
+ # Stored the way text-only documents are.
+ assert document.storage_mode == "none"
+ assert document.file_path == FILE_PATH_TEXT_ONLY
+ assert _library_links(other, document_id) == 1
+ assert other.get(DocumentChunk, chunk_id) is not None
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is False
+ assert tracker.file_path is None
+ view = library.get_document_by_id(document_id)
+ assert view["has_pdf"] is False
+ assert view["has_text_db"] is True
+
+ # The download manager offers the document again...
+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"
+ if via == "research":
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ else:
+ queued = download_all_new(db, monkeypatch)
+ assert queued["queued"] == 1
+ assert queued["research_ids"] == [world.research_id]
+ # ...and its download stores the PDF in the same document.
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == (2 if tracker_path == "kept" else 3)
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ assert other.get(DocumentChunk, chunk_id) is not None
+ restored = store.root / document.file_path
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is True
+ assert tracker.file_path == document.file_path
+ assert restored.read_bytes() == TEXT_PDF_BYTES
+ assert library.get_document_by_id(document_id)["has_pdf"] is True
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+
+
+def test_pdf_still_stored_in_the_database_keeps_the_document_unchanged(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """A document whose PDF is still a blob is not missing its PDF.
+
+ Downloaded in "database" mode and then again in "filesystem" mode,
+ the document keeps its blob next to the new file, because the
+ filesystem save does not remove it. When that file is removed the PDF
+ is still stored, so the sync keeps the document, its text and its
+ tracker unchanged. Making it text-only would hide the blob.
+ """
+ url = "https://example.com/blob-and-file.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="database")
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document_id = (
+ other.query(Document).filter_by(resource_id=resource_id).one().id
+ )
+ library = make_library_service(db, store, monkeypatch)
+ assert library.mark_for_redownload([document_id]) == 1
+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ assert other.query(DocumentBlob).count() == 1
+ own_path = document.file_path
+ assert find_tracker_by_url(other, url).file_path == own_path
+ (store.root / own_path).unlink()
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.file_path == own_path
+ assert document.text_content == EXTRACTED_TEXT
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is True
+ assert tracker.file_path == own_path
+ assert (
+ PDFStorageManager(store.root, "none").load_pdf(document, other)
+ == TEXT_PDF_BYTES
+ )
+
+
+def test_resource_linked_by_content_in_another_research_is_queued_again(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """A second research's resource linked to the document queues it.
+
+ A second research cites the same paper. Its download stores no
+ duplicate: the content-hash dedup links its resource to the existing
+ document through ``document_id``. Once the sync has made that document
+ text-only, the second research's own "Download PDFs" queues the linked
+ resource, and its download stores the PDF in the shared document.
+ """
+ url = "https://example.com/cited-twice.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ second_research = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="second research citing the paper",
+ mode="quick",
+ status="completed",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(second_research)
+ db.session.commit()
+ second_research_id = second_research.id
+ linked = ResearchResource(
+ research_id=second_research_id,
+ title="Sync Contract Paper",
+ url=url,
+ source_type="academic",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(linked)
+ db.session.commit()
+ linked_id = linked.id
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ assert (
+ service.queue_research_downloads(
+ second_research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ assert service.download_resource(linked_id) == (True, None)
+ assert len(downloader.urls) == 3
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = store.root / document.file_path
+ assert restored.read_bytes() == TEXT_PDF_BYTES
+
+
+def test_redownload_that_stores_no_pdf_stops_asking_for_it(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """A download in "none" mode settles a document that needed its PDF.
+
+ The storage mode keeps no PDF, so the download can only refresh the
+ text. The document must then count as downloaded, like any "none"-mode
+ download. If it still asked for its PDF, every "Download All New"
+ would fetch it again.
+ """
+ url = "https://example.com/removed-then-none-mode.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ (store.root / document.file_path).unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ service.settings._values["research_library.pdf_storage_mode"] = "none"
+ assert download_all_new(db, monkeypatch)["queued"] == 1
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "none"
+ assert document.file_path is None
+ assert document.text_content == EXTRACTED_TEXT
+ assert find_tracker_by_url(other, url).is_downloaded is True
+ assert _tree(store.root) == set()
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 0
+ )
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+
+
[email protected]("label", ["written-now", "written-before-the-fix"])
+def test_failed_placeholder_completed_by_a_pdf_download_is_downloaded_again(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch, label
+):
+ """A placeholder that a PDF download completed gets its PDF back.
+
+ A failed download leaves a placeholder labelled ``file_type``
+ "unknown", and a later PDF download completes that placeholder with the
+ PDF and its text. The download kept the "unknown" label. Once the PDF
+ file was removed, the sync made the document text-only and reported
+ the file missing, but no download path asked for its PDF again: they
+ ask only for documents labelled "pdf". Such a download now labels the
+ document "pdf" (``written-now``). A row a download completed before
+ that (``written-before-the-fix``) is labelled by the sync when it makes
+ the document text-only.
+ """
+ url = "https://example.com/placeholder-then-pdf.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ resource = db.session.get(ResearchResource, resource_id)
+ service._record_failed_text_extraction(
+ db.session, resource, error="PDF download failed: HTTP 503"
+ )
+ db.session.commit()
+ with db.fresh() as other:
+ placeholder = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = placeholder.id
+ assert placeholder.status is DocumentStatus.FAILED
+ assert placeholder.file_type == "unknown"
+
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ stored = store.root / document.file_path
+ assert stored.is_file()
+ library = make_library_service(db, store, monkeypatch)
+ if label == "written-now":
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert (document.file_type, document.mime_type) == (
+ "pdf",
+ "application/pdf",
+ )
+ assert library.get_document_by_id(document_id)["is_pdf"] is True
+ else:
+ # The labels such a download left before it labelled the document.
+ db.session.query(Document).filter_by(id=document_id).update(
+ {Document.file_type: "unknown", Document.mime_type: None},
+ synchronize_session=False,
+ )
+ db.session.commit()
+ chunk_id = _add_chunk(db, document_id)
+ stored.unlink()
+
+ stats = library.sync_library_with_filesystem()
+
+ assert [entry["id"] for entry in stats["missing_files"]] == [document_id]
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.text_content == EXTRACTED_TEXT
+ assert document.storage_mode == "none"
+ assert document.file_path == FILE_PATH_TEXT_ONLY
+ assert document.file_type == "pdf"
+ # The sync reports only documents the download paths ask for.
+ assert document_awaits_pdf(document)
+
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ assert download_all_new(db, monkeypatch)["queued"] == 1
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ assert other.get(DocumentChunk, chunk_id) is not None
+ restored = store.root / document.file_path
+ assert restored.read_bytes() == TEXT_PDF_BYTES
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+
+
+def test_regenerated_pdf_goes_back_into_the_document_that_awaits_it(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """A PDF downloaded again with other bytes refills the same document.
+
+ Publishers and arXiv regenerate PDFs, so the PDF downloaded for a
+ document that the sync made text-only can differ from the removed one.
+ It still belongs in that document: the document stores no other PDF,
+ and its text, chunks and collection links are the paper's. A second
+ document for the resource would leave this one waiting for its PDF
+ for good, with two completed documents for one resource. The document
+ takes the new PDF's hash, as a new download would, when no other
+ document owns those bytes. A collision must preserve both documents;
+ the following test covers that case and a later content dedup hit.
+ """
+ url = "https://example.com/regenerated.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ original_hash = hashlib.sha256(TEXT_PDF_BYTES).hexdigest()
+ regenerated_hash = hashlib.sha256(REGENERATED_PDF_BYTES).hexdigest()
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ assert document.document_hash == original_hash
+ stored = store.root / document.file_path
+ chunk_id = _add_chunk(db, document_id)
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ downloader.content = REGENERATED_PDF_BYTES
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ assert service.download_resource(resource_id) == (True, None)
+
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.storage_mode == "filesystem"
+ assert document.file_size == len(REGENERATED_PDF_BYTES)
+ assert document.text_content == REGENERATED_TEXT
+ assert document.document_hash == regenerated_hash
+ assert other.get(DocumentChunk, chunk_id) is not None
+ assert _library_links(other, document_id) == 1
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is True
+ assert tracker.file_path == document.file_path
+ restored = store.root / document.file_path
+ assert restored.read_bytes() == REGENERATED_PDF_BYTES
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ stats = library.sync_library_with_filesystem()
+ assert stats["files_missing"] == 0
+
+
[email protected]("holder_storage", ["filesystem", "database"])
+def test_colliding_refill_preserves_content_identity_for_later_downloads(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+ holder_storage,
+):
+ """Never put another document's bytes behind the missing PDF's hash.
+
+ A failed refill must retain both documents and report failure. A later
+ URL returning the original PDF must get those bytes, not the colliding
+ PDF incorrectly stored under their hash. Use real downloads, sync,
+ database rows, text extraction and PDF storage for the whole sequence.
+ """
+ url = "https://example.com/original.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ original_path = store.root / document.file_path
+ chunk_id = _add_chunk(db, document_id)
+ original_path.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ holder_url = "https://example.com/regenerated-elsewhere.pdf"
+ holder_resource_id = world.add_resource(holder_url)
+ downloader.content = REGENERATED_PDF_BYTES
+ holder_service = make_service(
+ store, downloader, storage_mode=holder_storage
+ )
+ assert holder_service.download_resource(holder_resource_id) == (True, None)
+ with db.fresh() as other:
+ holder = (
+ other.query(Document)
+ .filter_by(resource_id=holder_resource_id)
+ .one()
+ )
+ holder_id = holder.id
+ holder_path = holder.file_path
+ tracker = find_tracker_by_url(other, url)
+ tracker_before = (
+ tracker.file_hash,
+ tracker.file_path,
+ tracker.file_size,
+ tracker.downloaded_at,
+ )
+ holder_chunk_id = _add_chunk(db, holder_id)
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ expected_error = "PDF refill conflicts with another library document"
+ assert service.download_resource(resource_id) == (False, expected_error)
+
+ with db.fresh() as other:
+ assert other.query(Document).count() == 2
+ document = other.get(Document, document_id)
+ assert document_awaits_pdf(document)
+ assert (
+ document.document_hash == hashlib.sha256(TEXT_PDF_BYTES).hexdigest()
+ )
+ assert document.text_content == EXTRACTED_TEXT
+ assert document.file_size == len(TEXT_PDF_BYTES)
+ assert not original_path.exists()
+ # PDF text persistence now pins the resource to the exact stored
+ # document. Refusing a collision must preserve that explicit link.
+ assert (
+ other.get(ResearchResource, resource_id).document_id == document_id
+ )
+ assert other.get(DocumentChunk, chunk_id) is not None
+ assert _library_links(other, document_id) == 1
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is False
+ assert (
+ tracker.file_hash,
+ tracker.file_path,
+ tracker.file_size,
+ tracker.downloaded_at,
+ ) == tracker_before
+ attempts = (
+ other.query(DownloadAttempt)
+ .filter_by(url_hash=tracker.url_hash)
+ .order_by(DownloadAttempt.id)
+ .all()
+ )
+ assert [attempt.succeeded for attempt in attempts] == [True, False]
+ assert attempts[-1].error_message == expected_error
+ assert (
+ other.query(DownloadQueue)
+ .filter_by(resource_id=resource_id)
+ .one()
+ .status
+ is DocumentStatus.FAILED
+ )
+ assert (
+ other.query(ResourceDownloadStatus)
+ .filter_by(resource_id=resource_id)
+ .one()
+ .status
+ == "temporarily_failed"
+ )
+
+ holder = other.get(Document, holder_id)
+ assert (
+ holder.document_hash
+ == hashlib.sha256(REGENERATED_PDF_BYTES).hexdigest()
+ )
+ assert holder.text_content == REGENERATED_TEXT
+ assert holder.file_path == holder_path
+ assert holder.storage_mode == holder_storage
+ assert (
+ PDFStorageManager(
+ store.root, holder_storage, username=USERNAME
+ ).load_pdf(holder, other)
+ == REGENERATED_PDF_BYTES
+ )
+ assert other.get(DocumentChunk, holder_chunk_id) is not None
+ assert _library_links(other, holder_id) == 1
+ assert find_tracker_by_url(other, holder_url).is_downloaded is True
+
+ mirror_resource_id = world.add_resource(
+ "https://example.com/original-mirror.pdf"
+ )
+ downloader.content = TEXT_PDF_BYTES
+ assert service.download_resource(mirror_resource_id) == (True, None)
+ with db.fresh() as other:
+ mirror = other.get(ResearchResource, mirror_resource_id)
+ assert mirror.document_id == document_id
+ document = other.get(Document, mirror.document_id)
+ stored = PDFStorageManager(
+ store.root, "filesystem", username=USERNAME
+ ).load_pdf(document, other)
+ assert stored == TEXT_PDF_BYTES
+ assert hashlib.sha256(stored).hexdigest() == document.document_hash
+ assert document.text_content == EXTRACTED_TEXT
+ assert other.get(DocumentChunk, chunk_id) is not None
+ assert other.query(Document).count() == 2
+
+
+def test_text_extraction_leaves_a_document_that_awaits_its_pdf_alone(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """Text extraction does not queue a document that has its text.
+
+ The sync made the document text-only after its PDF file was removed,
+ and the PDF downloads ask for it again. A text extraction has nothing
+ to add: the document holds its text, and the extraction never fetches
+ the PDF that would settle it. Queued anyway, the document was reported
+ as processed on every run, and a second research's resource linked to
+ it went to the network-bound text extraction each time. The text mode
+ skips both, as it did before the sync kept such documents, and the PDF
+ mode still puts the PDF back.
+ """
+ url = "https://example.com/extract-text-after-removal.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ second_research = ResearchHistory(
+ id=str(uuid.uuid4()),
+ query="second research citing the paper",
+ mode="quick",
+ status="completed",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(second_research)
+ db.session.commit()
+ second_research_id = second_research.id
+ linked = ResearchResource(
+ research_id=second_research_id,
+ title="Sync Contract Paper",
+ url=url,
+ source_type="academic",
+ created_at="2026-08-02T00:00:00",
+ )
+ db.session.add(linked)
+ db.session.commit()
+ linked_id = linked.id
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ requests = len(downloader.urls)
+ both = [world.research_id, second_research_id]
+
+ for _ in range(2):
+ events = bulk_download(
+ db, store, downloader, monkeypatch, both, mode="text_only"
+ )
+ assert events[0]["total"] == 0, events
+ assert events[-1]["complete"] is True
+ assert events[-1]["error"] == (
+ "No new papers to download for the selected research session(s)."
+ )
+ assert len(downloader.urls) == requests
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document_awaits_pdf(document)
+ assert document.text_content == EXTRACTED_TEXT
+ assert other.query(DownloadQueue).count() == 0
+
+ events = bulk_download(
+ db, store, downloader, monkeypatch, [world.research_id], mode="pdf"
+ )
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert events[-1]["complete"] is True
+ assert "error" not in events[-1]
+ assert len(downloader.urls) == requests + 1
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = store.root / document.file_path
+ assert restored.read_bytes() == TEXT_PDF_BYTES
+
+
+ "via",
+ [
+ "owner-research-first",
+ "linked-research-first",
+ "download-all-new",
+ "linked-queued-earlier",
+ "linked-queued-earlier-run-over-both",
+ "download-all-new-linked-queued-earlier",
+ "owner-not-downloadable",
+ ],
+)
+def test_paper_cited_by_two_researches_is_downloaded_once(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch, via
+):
+ """One download puts back the PDF of a document two researches share.
+
+ A second research cites the same paper. Its download stored no
+ duplicate: the content-hash dedup linked its resource to the first
+ research's document (``document_id``). Once the sync has made that
+ document text-only, either resource refills it. A bulk run over both
+ researches, or "Download All New" and the bulk run it starts, queued
+ both. The first download put the PDF back. The second fetched it
+ again, saved it under its own resource's file name and left the first
+ file on disk with nothing pointing to it.
+
+ One resource per such document is queued now, the document's own one
+ whichever research the bulk run queues first, and the other resource
+ counts as downloaded once the PDF is back. A linked resource that an
+ earlier run left queued does not keep a run over the first research
+ from queueing the document's own resource
+ (``linked-queued-earlier``), and a run over both researches downloads
+ that queued resource rather than queue a second one
+ (``linked-queued-earlier-run-over-both``), as "Download All New" does
+ (``download-all-new-linked-queued-earlier``). When the document's own
+ resource cannot be queued (``owner-not-downloadable``), the linked one
+ refills it.
+ """
+ url = "https://example.com/cited-by-two-researches.pdf"
+ owner_url = (
+ "https://example.com/landing/cited-by-two-researches"
+ if via == "owner-not-downloadable"
+ else url
+ )
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert document.resource_id == owner_id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ requests = len(downloader.urls)
+ both = [world.research_id, second_research_id]
+
+ refilled_by = owner_id
+ if via == "download-all-new":
+ queued = download_all_new(db, monkeypatch)
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [world.research_id]
+ research_ids = queued["research_ids"]
+ elif via == "owner-research-first":
+ research_ids = both
+ elif via == "linked-research-first":
+ research_ids = both[::-1]
+ elif "linked-queued-earlier" in via:
+ # A run over the second research queued its resource and stopped
+ # before downloading it.
+ assert (
+ service.queue_research_downloads(
+ second_research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ if via == "linked-queued-earlier":
+ research_ids = [world.research_id]
+ elif via == "linked-queued-earlier-run-over-both":
+ research_ids = both
+ refilled_by = linked_id
+ else:
+ queued = download_all_new(db, monkeypatch)
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [second_research_id]
+ research_ids = queued["research_ids"]
+ refilled_by = linked_id
+ else:
+ research_ids = both[::-1]
+ refilled_by = linked_id
+ events = bulk_download(
+ db, store, downloader, monkeypatch, research_ids, mode="pdf"
+ )
+
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert len(downloader.urls) == requests + 1
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ downloaded = {
+ entry.resource_id
+ for entry in other.query(DownloadQueue).filter_by(
+ status=DocumentStatus.COMPLETED
+ )
+ }
+ assert downloaded == {refilled_by}
+ # The one file in the library is the one the document names.
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+ # The other resource counts as downloaded: no request, no second file,
+ # also when a run downloads the entry an earlier run left queued.
+ events = bulk_download(db, store, downloader, monkeypatch, both, mode="pdf")
+ left_queued = via == "linked-queued-earlier"
+ assert events[0]["total"] == (1 if left_queued else 0), events
+ other_id = linked_id if refilled_by == owner_id else owner_id
+ assert service.download_resource(other_id) == (True, None)
+ assert len(downloader.urls) == requests + 1
+ assert _tree(store.root) == {restored}
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+
+
+ "own_link",
+ [False, True],
+ ids=["own-resource-unlinked", "own-resource-linked-to-its-document"],
+)
+def test_research_citing_a_paper_twice_queues_the_documents_own_resource(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ own_link,
+):
+ """Within one research, the document's own resource is queued.
+
+ A research cites the paper under two URLs, and the second download was
+ linked to the first one's document by content. The linked resource
+ comes first in the research, so the queueing met it before the
+ document's own resource, queued it, and skipped the own one. The
+ document's own resource is considered first now, also by a "Download
+ PDFs" that only queues (no run downloads the queue right away), and
+ also when it points to its own document through ``document_id``
+ (``own-resource-linked-to-its-document``): the order keys on the
+ document's owner, not on ``document_id``.
+ """
+ # Created first, downloaded second: linked to the other's document.
+ linked_id = world.add_resource("https://example.com/twice/abstract.pdf")
+ owner_id = world.add_resource("https://example.com/twice/paper.pdf")
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner_id) == (True, None)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ owner_document_id = document.id
+ assert document.resource_id == owner_id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document.id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ if own_link:
+ owner = db.session.get(ResearchResource, owner_id)
+ owner.document_id = owner_document_id
+ db.session.commit()
+
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ with db.fresh() as other:
+ queued = [entry.resource_id for entry in other.query(DownloadQueue)]
+ assert queued == [owner_id]
+
+
+def test_research_citing_a_paper_twice_queues_the_linked_one_if_the_own_failed(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+):
+ """Within one research, a failing own resource leaves the PDF to the other.
+
+ As in the case above, a research cites the paper under two URLs, and
+ the second download was linked to the first one's document by content.
+ Then the first URL stopped working, and a refill through it failed.
+ "Download PDFs" for the research, which only queues, queued the failing
+ resource again and skipped the linked one, which would have worked. It
+ queues the linked resource now, and only that one.
+ """
+ linked_id = world.add_resource("https://example.com/twice/abstract.pdf")
+ owner_url = "https://example.com/twice/paper.pdf"
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ assert service.download_resource(owner_id) == (True, None)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ assert document.resource_id == owner_id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document.id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ downloader.unreachable.add(owner_url)
+ assert service.download_resource(owner_id) == (False, "Service unavailable")
+
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ with db.fresh() as other:
+ queued = [entry.resource_id for entry in other.query(DownloadQueue)]
+ assert queued == [linked_id]
+
+
+ "via",
+ ["owner-research-first", "linked-research-first", "download-all-new"],
+)
[email protected]("owner", ["unreachable", "refused-by-egress-policy"])
+def test_owner_that_cannot_succeed_leaves_the_document_to_a_linked_resource(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+ owner,
+ via,
+):
+ """A linked resource refills the document when its own one cannot.
+
+ A second research cites the paper under another URL, and its download
+ was linked to the document by content. After the sync made the
+ document text-only, the URL of the document's own resource stopped
+ working (``unreachable``: a refill through it failed, and the retry
+ manager's cooldown has passed since), or the egress policy now refuses
+ it (``refused-by-egress-policy``). The document's one download still
+ went to its own resource. A bulk run over both researches queued that
+ resource again on every run and skipped the linked one, which would
+ have worked. "Download All New" kept the own resource without asking
+ the egress policy, so the bulk run it started fetched nothing. A
+ resource that cannot succeed no longer takes the download from a
+ viable one: the linked resource refills the document, in either order
+ of the researches and through "Download All New".
+ """
+ owner_url = "https://example.com/owner-copy/paper.pdf"
+ linked_url = "https://mirror.example.org/paper.pdf"
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert document.resource_id == owner_id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ both = [world.research_id, second_research_id]
+ options = {"retry_manager": retry_history}
+ if owner == "refused-by-egress-policy":
+ options["refused_urls"] = {owner_url}
+ else:
+ downloader.unreachable.add(owner_url)
+ # A refill through the document's own resource fails...
+ events = bulk_download(
+ db,
+ store,
+ downloader,
+ monkeypatch,
+ [world.research_id],
+ mode="pdf",
+ **options,
+ )
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["skipped"]
+ # ...and the retry manager's cooldown passes.
+ with db.fresh() as other:
+ status = (
+ other.query(ResourceDownloadStatus)
+ .filter_by(resource_id=owner_id)
+ .one()
+ )
+ assert status.status == "temporarily_failed"
+ status.retry_after_timestamp = datetime.now(UTC) - timedelta(
+ minutes=1
+ )
+ other.commit()
+ assert retry_history.should_retry_resource(owner_id).can_retry
+ requests = len(downloader.urls)
+
+ if via == "download-all-new":
+ queued = download_all_new(
+ db, monkeypatch, retry_policy=retry_history, **options
+ )
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [second_research_id]
+ research_ids = queued["research_ids"]
+ elif via == "owner-research-first":
+ research_ids = both
+ else:
+ research_ids = both[::-1]
+ events = bulk_download(
+ db, store, downloader, monkeypatch, research_ids, mode="pdf", **options
+ )
+
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert downloader.urls[requests:] == [linked_url]
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ downloaded = {
+ entry.resource_id
+ for entry in other.query(DownloadQueue).filter_by(
+ status=DocumentStatus.COMPLETED
+ )
+ }
+ assert downloaded == {linked_id}
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+ # The document holds its PDF again, so nothing is queued or fetched.
+ events = bulk_download(
+ db, store, downloader, monkeypatch, both, mode="pdf", **options
+ )
+ assert events[0]["total"] == 0, events
+ queued = download_all_new(
+ db, monkeypatch, retry_policy=retry_history, **options
+ )
+ assert queued["queued"] == 0
+ assert downloader.urls[requests:] == [linked_url]
+
+
+ "via", ["owner-research-first", "linked-research-first"]
+)
+def test_owner_the_retry_manager_holds_back_leaves_the_document_to_a_linked_one(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+ via,
+):
+ """A linked resource refills the document while its own one is held back.
+
+ The URL of the document's own resource failed three times today before
+ a download through it succeeded. The retry manager records the
+ success, so the latest attempt did not fail, but it still counts
+ today's three attempts, its daily limit, and holds the resource back
+ until tomorrow. A second research cites the paper under another URL,
+ and its download was linked to the document by content. After the
+ sync made the document text-only, a bulk run over both researches
+ downloads the PDF through the linked resource, in either order of the
+ researches, as it does when the own resource's latest attempt failed.
+ """
+ owner_url = "https://example.com/owner-copy/held-back.pdf"
+ linked_url = "https://mirror.example.org/held-back.pdf"
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES, unreachable={owner_url})
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ for _ in range(3):
+ assert service.download_resource(owner_id) == (
+ False,
+ "Service unavailable",
+ )
+ downloader.unreachable.clear()
+ assert service.download_resource(owner_id) == (True, None)
+ assert (
+ retry_history.status_tracker.get_resource_status(owner_id)["status"]
+ == "completed"
+ )
+ assert not retry_history.should_retry_resource(owner_id).can_retry
+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert document.resource_id == owner_id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ requests = len(downloader.urls)
+ both = [world.research_id, second_research_id]
+
+ events = bulk_download(
+ db,
+ store,
+ downloader,
+ monkeypatch,
+ both if via == "owner-research-first" else both[::-1],
+ mode="pdf",
+ retry_manager=retry_history,
+ )
+
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert downloader.urls[requests:] == [linked_url]
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ downloaded = {
+ entry.resource_id
+ for entry in other.query(DownloadQueue).filter_by(
+ status=DocumentStatus.COMPLETED
+ )
+ }
+ assert downloaded == {linked_id}
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+
+@contextmanager
+def logged_records():
+ """The loguru records emitted inside the block, at every level.
+
+ The package disables its loguru logging (``local_deep_research``'s
+ ``__init__``), so it is enabled for the block and disabled again after.
+ """
+ records = []
+ logger.enable("local_deep_research")
+ sink = logger.add(lambda message: records.append(message.record))
+ try:
+ yield records
+ finally:
+ logger.remove(sink)
+ logger.disable("local_deep_research")
+
+
+def test_ranking_records_no_denial_of_the_egress_policy():
+ """Ranking a URL the egress policy refuses records no denied fetch.
+
+ ``refill_rank`` asks the egress policy whether a resource's URL could
+ be queued, only to rank the resources that could refill a document.
+ Each URL the policy refused was recorded as a denied fetch although
+ nothing was fetched: a ``policy_audit`` warning from the policy, one
+ from ``DownloadService``, and a count toward the run's denied-fetch
+ quota, which refuses every URL once it is used up. The rank still
+ treats the URL as one that cannot be queued, and records nothing. The
+ check before a fetch records the denial as before.
+ """
+ service = make_service(_NO_STORE, StubDownloader(), storage_mode="none")
+ context = EgressContext(
+ scope=EgressScope.PRIVATE_ONLY,
+ primary_engine="library",
+ require_local_llm=False,
+ require_local_embeddings=False,
+ )
+ service._egress_context = context
+ # A public IP literal: classified without a DNS lookup, and refused
+ # under PRIVATE_ONLY.
+ refused_url = "https://8.8.8.8/papers/refused.pdf"
+ warning = logger.level("WARNING").no
+
+ def recorded(records):
+ return [
+ record["message"]
+ for record in records
+ if record["level"].no >= warning
+ or record["extra"].get("policy_audit")
+ ]
+
+ # No armed run context, whose counter would take the denial instead.
+ clear_active_context()
+ with logged_records() as records:
+ rank = service.refill_rank(1, refused_url, 1)
+ assert rank.cannot_queue
+ assert recorded(records) == []
+ assert context._fetch_denial_count["count"] == 0
+
+ with logged_records() as records:
+ assert service._check_url_against_policy(refused_url) == (
+ False,
+ "scope_mismatch_private_only",
+ )
+ assert len(recorded(records)) == 2, recorded(records)
+ assert context._fetch_denial_count["count"] == 1
+
+
[email protected]("owner", ["unreachable", "refused-by-egress-policy"])
+def test_queued_owner_that_cannot_succeed_does_not_hold_the_document(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+ owner,
+):
+ """A queued resource that cannot succeed does not hold the document.
+
+ The document's own resource is already queued: "Download PDFs" for its
+ research queued it again after a failed refill (``unreachable``), or it
+ was queued before the egress policy came to refuse its URL
+ (``refused-by-egress-policy``). A bulk run over both researches
+ downloads that entry, and it skipped the second research's linked
+ resource, so the document stayed without its PDF. The run queues the
+ linked resource too now, and that one refills the document.
+ """
+ owner_url = "https://example.com/owner-copy/queued-paper.pdf"
+ linked_url = "https://mirror.example.org/queued-paper.pdf"
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ options = {"retry_manager": retry_history}
+ if owner == "unreachable":
+ downloader.unreachable.add(owner_url)
+ assert service.download_resource(owner_id) == (
+ False,
+ "Service unavailable",
+ )
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ failed_requests = [owner_url]
+ else:
+ db.session.add(
+ DownloadQueue(
+ resource_id=owner_id,
+ research_id=world.research_id,
+ collection_id=LIBRARY_COLLECTION_ID,
+ status=DocumentStatus.PENDING,
+ )
+ )
+ db.session.commit()
+ options["refused_urls"] = {owner_url}
+ failed_requests = []
+ requests = len(downloader.urls)
+
+ events = bulk_download(
+ db,
+ store,
+ downloader,
+ monkeypatch,
+ [world.research_id, second_research_id],
+ mode="pdf",
+ **options,
+ )
+
+ assert events[0]["total"] == 2, events
+ assert [event.get("status") for event in events[1:-1]] == [
+ "skipped",
+ "success",
+ ]
+ assert downloader.urls[requests:] == [*failed_requests, linked_url]
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ entries = {
+ entry.resource_id: entry.status
+ for entry in other.query(DownloadQueue).all()
+ }
+ assert entries == {
+ owner_id: DocumentStatus.FAILED,
+ linked_id: DocumentStatus.COMPLETED,
+ }
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+
+def test_download_all_new_does_not_keep_a_queued_owner_whose_last_attempt_failed(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ retry_history,
+):
+ """A queued owner that failed last loses "Download All New" to a viable one.
+
+ The document's own resource is queued: a refill through it failed, and
+ "Download PDFs" for its research queued it again. The retry manager's
+ cooldown has passed since, so the retry-policy filter of "Download All
+ New" lets it through, but its latest attempt failed. "Download All New"
+ kept a resource whose queue entry is PENDING over the second research's
+ linked resource, which would have worked, so the run it started fetched
+ the failing URL again, and the document stayed without its PDF. It
+ keeps the linked resource now, as a bulk run over both researches does
+ (see ``test_queued_owner_that_cannot_succeed_does_not_hold_the_document``),
+ and that one refills the document.
+ """
+ owner_url = "https://example.com/owner-copy/queued-again.pdf"
+ linked_url = "https://mirror.example.org/queued-again.pdf"
+ owner_id = world.add_resource(owner_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(
+ store,
+ downloader,
+ storage_mode="filesystem",
+ retry_manager=retry_history,
+ )
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, linked_url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ # A refill through the document's own resource fails, and "Download
+ # PDFs" for its research queues it again...
+ downloader.unreachable.add(owner_url)
+ assert service.download_resource(owner_id) == (False, "Service unavailable")
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ # ...and the retry manager's cooldown passes.
+ with db.fresh() as other:
+ status = (
+ other.query(ResourceDownloadStatus)
+ .filter_by(resource_id=owner_id)
+ .one()
+ )
+ assert status.status == "temporarily_failed"
+ status.retry_after_timestamp = datetime.now(UTC) - timedelta(minutes=1)
+ other.commit()
+ assert retry_history.should_retry_resource(owner_id).can_retry
+ requests = len(downloader.urls)
+
+ queued = download_all_new(
+ db, monkeypatch, retry_policy=retry_history, retry_manager=retry_history
+ )
+
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [second_research_id]
+ events = bulk_download(
+ db,
+ store,
+ downloader,
+ monkeypatch,
+ queued["research_ids"],
+ mode="pdf",
+ retry_manager=retry_history,
+ )
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert downloader.urls[requests:] == [linked_url]
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ entries = {
+ entry.resource_id: entry.status
+ for entry in other.query(DownloadQueue).all()
+ }
+ # The owner's entry stays for a run over its research, which finds the
+ # document holding its PDF again.
+ assert entries == {
+ owner_id: DocumentStatus.PENDING,
+ linked_id: DocumentStatus.COMPLETED,
+ }
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+
[email protected]("via", ["bulk-run", "download-all-new"])
+def test_document_refilled_by_another_stream_is_not_queued_again(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch, via
+):
+ """A document another download stream is refilling is not queued.
+
+ The second research's linked resource is PROCESSING: a bulk run over
+ that research has claimed it, at the time the claim records, and is
+ downloading the PDF into the shared document. A run over the first
+ research would queue the document's own resource and fetch the same
+ PDF at the same time, which the check in ``download_resource`` cannot
+ stop, since neither download has stored the PDF yet. The run leaves
+ the document to that stream, and so does "Download All New": it keeps
+ the claimed resource, which the bulk run it starts does not download
+ again.
+ """
+ url = "https://example.com/refilled-elsewhere.pdf"
+ owner_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ stored = store.root / other.query(Document).one().file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ # The other stream's claim on the linked resource's queue entry.
+ claim_queue_entry(db, linked_id, second_research_id)
+ requests = len(downloader.urls)
+
+ if via == "bulk-run":
+ research_ids = [world.research_id]
+ else:
+ queued = download_all_new(db, monkeypatch)
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [second_research_id]
+ research_ids = queued["research_ids"]
+ events = bulk_download(
+ db, store, downloader, monkeypatch, research_ids, mode="pdf"
+ )
+
+ assert events[0]["total"] == 0, events
+ assert len(downloader.urls) == requests
+ with db.fresh() as other:
+ assert (
+ other.query(DownloadQueue).filter_by(resource_id=owner_id).count()
+ == 0
+ )
+ assert (
+ other.query(DownloadQueue).filter_by(resource_id=linked_id).one()
+ ).status is DocumentStatus.PROCESSING
+
+
[email protected]("via", ["bulk-run", "download-all-new"])
[email protected]("claimed_at", ["before-the-restart", "not-recorded"])
+def test_download_cut_off_by_a_restart_does_not_hold_the_document(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ claimed_at,
+ via,
+):
+ """A queue entry a restart left PROCESSING does not hold the document.
+
+ A bulk run claimed the document's own resource and the server
+ restarted during the download. Nothing reclaims the entry: it stays
+ PROCESSING, and no reset touches it. It held the document as one
+ being downloaded, so no run and no "Download All New" ever queued the
+ second research's linked resource, and the document never got its PDF
+ back. The claim records its time now, and an entry claimed before this
+ server process started (``before-the-restart``), or before claims
+ recorded their time (``not-recorded``), stands for no download. The
+ linked resource refills the document, and the stranded entry is left
+ as it is. A claim this process made still holds the document, see
+ ``test_document_refilled_by_another_stream_is_not_queued_again``.
+ """
+ url = "https://example.com/cut-off-by-a-restart.pdf"
+ owner_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).one()
+ document_id = document.id
+ stored = store.root / document.file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ if claimed_at == "before-the-restart":
+ claim_queue_entry(db, owner_id, world.research_id)
+ # The server restarts: this process started after the claim.
+ monkeypatch.setattr(
+ library_utils,
+ "_PROCESS_STARTED_AT",
+ datetime.now(UTC) + timedelta(seconds=1),
+ )
+ else:
+ # Claimed before claims recorded their time.
+ db.session.add(
+ DownloadQueue(
+ resource_id=owner_id,
+ research_id=world.research_id,
+ collection_id=LIBRARY_COLLECTION_ID,
+ status=DocumentStatus.PROCESSING,
+ )
+ )
+ db.session.commit()
+ requests = len(downloader.urls)
+ both = [world.research_id, second_research_id]
+
+ if via == "bulk-run":
+ research_ids = both
+ else:
+ queued = download_all_new(db, monkeypatch)
+ assert (queued["queued"], queued["skipped"]) == (1, 1), queued
+ assert queued["research_ids"] == [second_research_id]
+ research_ids = queued["research_ids"]
+ events = bulk_download(
+ db, store, downloader, monkeypatch, research_ids, mode="pdf"
+ )
+
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert len(downloader.urls) == requests + 1
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == EXTRACTED_TEXT
+ restored = document.file_path
+ entries = {
+ entry.resource_id: entry.status
+ for entry in other.query(DownloadQueue).all()
+ }
+ assert entries == {
+ owner_id: DocumentStatus.PROCESSING,
+ linked_id: DocumentStatus.COMPLETED,
+ }
+ assert _tree(store.root) == {restored}
+ assert (store.root / restored).read_bytes() == TEXT_PDF_BYTES
+
+
+class _LookupOnlyRun(AbstractSet):
+ """The research sessions of a bulk run, which may only be looked up."""
+
+ def __init__(self, research_ids):
+ self._ids = frozenset(research_ids)
+
+ def __contains__(self, research_id):
+ return research_id in self._ids
+
+ def __len__(self):
+ return len(self._ids)
+
+ def __iter__(self):
+ raise AssertionError(
+ "queueing one research copied the research sessions of the run"
+ )
+
+
+def test_bulk_queueing_looks_the_runs_research_sessions_up(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """Queueing a research of a bulk run does not copy the run's sessions.
+
+ ``download_bulk`` queues each research of the run with the run's
+ research sessions, which decide which resource refills a document that
+ awaits its PDF. It passed its list, and each call copied it into a new
+ set, so queueing the research sessions of a run took time quadratic in
+ their number, which a bulk request may name up to 100,000 of. The
+ route builds one set for the run now, and each call only looks it up.
+ """
+ url = "https://example.com/run-of-two-researches.pdf"
+ owner_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ stored = store.root / other.query(Document).one().file_path
+ stored.unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ both = [world.research_id, second_research_id]
+
+ run = _LookupOnlyRun(both)
+ queued = [
+ service.queue_research_downloads(
+ research_id, LIBRARY_COLLECTION_ID, run_research_ids=run
+ )
+ for research_id in both[::-1]
+ ]
+ assert queued == [0, 1]
+ with db.fresh() as other:
+ assert [entry.resource_id for entry in other.query(DownloadQueue)] == [
+ owner_id
+ ]
+
+ passed = []
+ queue_research_downloads = DownloadService.queue_research_downloads
+
+ def record(self, research_id, collection_id=None, **kwargs):
+ passed.append(kwargs["run_research_ids"])
+ return queue_research_downloads(
+ self, research_id, collection_id, **kwargs
+ )
+
+ monkeypatch.setattr(DownloadService, "queue_research_downloads", record)
+ events = bulk_download(db, store, downloader, monkeypatch, both, mode="pdf")
+ assert events[0]["total"] == 1, events
+ assert [event.get("status") for event in events[1:-1]] == ["success"]
+ assert len(passed) == 2
+ assert passed[0] is passed[1]
+ assert isinstance(passed[0], AbstractSet)
+ assert passed[0] == set(both)
+
+
+def test_queueing_a_research_loads_no_text_of_the_documents_it_links_to(
+ db, store, world, monkeypatch
+):
+ """Queueing reads the documents resources are linked to without text.
+
+ A second research cites a paper the first one downloaded, and the
+ content-hash dedup linked its resource to the first research's
+ document. "Download PDFs" for the second research queues nothing for
+ it, since the document holds its PDF. To consider each document's own
+ resource first, the queueing loaded every document such a resource
+ points to as a whole row, with its extracted text, which #4560 kept
+ such scans from loading. It reads only the columns it compares now.
+ This covers those lookups only: the check whether a resource already
+ has a completed document of its own reads that document whole, as it
+ did before.
+ """
+ url = "https://example.com/cited-again.pdf"
+ owner_id = world.add_resource(url)
+ service = make_service(
+ store, StubDownloader(TEXT_PDF_BYTES), storage_mode="database"
+ )
+ assert service.download_resource(owner_id) == (True, None)
+ second_research_id, linked_id = _cite_in_another_research(db, url)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ assert (
+ other.get(ResearchResource, linked_id).document_id
+ == other.query(Document).one().id
+ )
+ statements = []
+
+ def record(conn, cursor, statement, parameters, context, executemany):
+ statements.append(statement)
+
+ event.listen(db.engine, "before_cursor_execute", record)
+ try:
+ queued = service.queue_research_downloads(
+ second_research_id, LIBRARY_COLLECTION_ID
+ )
+ finally:
+ event.remove(db.engine, "before_cursor_execute", record)
+
+ assert queued == 0
+ assert any(
+ "JOIN documents" in s or "FROM documents" in s for s in statements
+ )
+ assert [s for s in statements if "text_content" in s] == []
+
+
+def render_download_manager(db, monkeypatch):
+ """The Download Manager page, as the browser gets it.
+
+ Runs the ``download_manager_page`` route and renders its template
+ against this database. The route's session and settings sources are
+ replaced; the ``LibraryService`` it builds reads through the session
+ that ``make_library_service`` set up. Importing the app registers the
+ template globals that the page's base template uses.
+ """
+ from local_deep_research.web import fastapi_app # noqa: F401
+
+ monkeypatch.setattr(
+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ monkeypatch.setattr(
+ library_router,
+ "get_settings_manager",
+ lambda *a, **k: StubSettings(
+ {"research_library.pdf_storage_mode": "filesystem"}
+ ),
+ )
+ request = Request(
+ {
+ "type": "http",
+ "method": "GET",
+ "path": "/library/download-manager",
+ "headers": [],
+ "query_string": b"",
+ "session": {"session_id": "sid", "username": USERNAME},
+ }
+ )
+ response = library_router.download_manager_page(request, username=USERNAME)
+ return response.body.decode()
+
+
+def test_download_manager_counts_a_document_awaiting_its_pdf_as_available(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """The Download Manager offers the PDF the sync reported missing.
+
+ The sync's result sends the user to the Download Manager to download
+ the missing files again. Its counts took every completed document as
+ downloaded, so a document awaiting its PDF showed there as "Already
+ Downloaded", and nothing was "Available to Download", while "Download
+ All New" queued it. Until its PDF is back it counts as available now,
+ in the totals and in its research's row, and its research's sources
+ show it under "Sources by Domain" as not downloaded and without the
+ "Downloaded" badge. Those two compared the document's status with the
+ string "completed", which a loaded status never equals, so they showed
+ no source as downloaded, also once its PDF was back.
+ """
+ url = "https://example.com/counted-as-available.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ library = make_library_service(db, store, monkeypatch)
+
+ def counts():
+ summary = library.get_download_manager_summary_stats()
+ (research,) = library.get_research_list_with_stats()
+ page = render_download_manager(db, monkeypatch)
+ return {
+ "already_downloaded": summary["already_downloaded"],
+ "available_to_download": summary["available_to_download"],
+ "research_downloaded": research["downloaded_count"],
+ "research_downloadable": research["downloadable_count"],
+ "sources_by_domain": [
+ " ".join(stats.split())
+ for stats in re.findall(
+ r'<span class="ldr-domain-stats">(.*?)</span>', page, re.S
+ )
+ ],
+ "downloaded_badges": page.count(
+ '<span class="ldr-badge ldr-badge-success">Downloaded</span>'
+ ),
+ }
+
+ downloaded = {
+ "already_downloaded": 1,
+ "available_to_download": 0,
+ "research_downloaded": 1,
+ "research_downloadable": 1,
+ "sources_by_domain": ["1 total | 1 PDFs | 1 downloaded"],
+ "downloaded_badges": 1,
+ }
+ assert counts() == downloaded
+ with db.fresh() as other:
+ stored = store.root / other.query(Document).one().file_path
+ stored.unlink()
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ assert counts() == {
+ "already_downloaded": 0,
+ "available_to_download": 1,
+ "research_downloaded": 0,
+ "research_downloadable": 1,
+ "sources_by_domain": ["1 total | 1 PDFs | 0 downloaded"],
+ "downloaded_badges": 0,
+ }
+ assert download_all_new(db, monkeypatch)["queued"] == 1
+ assert service.download_resource(resource_id) == (True, None)
+ assert counts() == downloaded
+
+
+def _post(path, payload):
+ """A POST request whose JSON body is ``payload``."""
+ body = json.dumps(payload).encode()
+
+ async def receive():
+ return {"type": "http.request", "body": body, "more_body": False}
+
+ return Request(
+ {
+ "type": "http",
+ "method": "POST",
+ "path": path,
+ "headers": [(b"content-type", b"application/json")],
+ "query_string": b"",
+ "session": {"session_id": "sid"},
+ },
+ receive,
+ )
+
+
+def source_routes(db, store, downloader, monkeypatch, research_id, url):
+ """What the library's source routes report for ``url`` in a research.
+
+ Runs ``get_research_sources`` and ``check_downloads``, and then
+ ``download_source``, which downloads the resource unless it answers
+ "Already downloaded", against this database. Their session and the
+ user's password are replaced, and so is the ``DownloadService`` that
+ ``download_source`` downloads with (a filesystem-mode service around
+ ``downloader``). Which documents count as downloaded, and the queue
+ entry, claim and download of the resource, are the production code.
+ """
+ monkeypatch.setattr(
+ library_router, "get_user_db_session", lambda *a, **k: db.fresh()
+ )
+ monkeypatch.setattr(
+ library_router, "get_authenticated_user_password", lambda *a, **k: "pw"
+ )
+ monkeypatch.setattr(
+ library_router,
+ "DownloadService",
+ lambda *a, **k: make_service(
+ store, downloader, storage_mode="filesystem"
+ ),
+ )
+ (source,) = library_router.get_research_sources(
+ None, research_id, username=USERNAME
+ )["sources"]
+ checked = asyncio.run(
+ library_router.check_downloads(
+ _post(
+ "/library/api/check-downloads",
+ {"research_id": research_id, "urls": [url]},
+ ),
+ username=USERNAME,
+ )
+ )["download_status"][url]
+ answer = asyncio.run(
+ library_router.download_source(
+ _post(
+ "/library/api/download-source",
+ {"research_id": research_id, "url": url},
+ ),
+ username=USERNAME,
+ )
+ )
+ return {
+ "research_sources": (source["downloaded"], source["document_id"]),
+ "check_downloads": (
+ checked["downloaded"],
+ checked.get("document_id"),
+ checked.get("resource_id"),
+ ),
+ "download_source": answer["message"],
+ }
+
+
+def test_source_routes_report_a_download_as_the_download_manager_counts_it(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """The library's source routes follow the Download Manager's counts.
+
+ ``get_research_sources`` and ``check_downloads`` report whether a
+ research's source is downloaded, and ``download_source`` answers
+ "Already downloaded" for a source that is and downloads any other.
+ They compared the document's loaded status with the string
+ "completed", which a ``DocumentStatus`` member never equals, so they
+ reported every source as not downloaded, and ``download_source``
+ queued and claimed a downloaded one again. They apply the Download
+ Manager's rule now (``document_counts_as_downloaded``): a document that
+ holds its PDF is downloaded, and one that awaits its PDF after the sync
+ reported its file missing is not, so ``download_source`` puts the PDF
+ back into it.
+ """
+ url = "https://example.com/reported-by-the-source-routes.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ stored = store.root / document.file_path
+ library = make_library_service(db, store, monkeypatch)
+
+ def routes():
+ return source_routes(
+ db, store, downloader, monkeypatch, world.research_id, url
+ )
+
+ downloaded = {
+ "research_sources": (True, document_id),
+ "check_downloads": (True, document_id, None),
+ "download_source": "Already downloaded",
+ }
+ assert routes() == downloaded
+ assert downloader.urls == [url]
+ with db.fresh() as other:
+ assert other.query(DownloadQueue).count() == 0
+
+ stored.unlink()
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ with db.fresh() as other:
+ assert document_awaits_pdf(other.get(Document, document_id))
+
+ assert routes() == {
+ "research_sources": (False, None),
+ "check_downloads": (False, None, resource_id),
+ "download_source": "Download completed",
+ }
+ # download_source put the PDF back into the same document, with one
+ # request, and finished its queue entry.
+ assert downloader.urls == [url, url]
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert not document_awaits_pdf(document)
+ assert document.text_content == EXTRACTED_TEXT
+ assert (store.root / document.file_path).read_bytes() == TEXT_PDF_BYTES
+ (entry,) = other.query(DownloadQueue).all()
+ assert (entry.resource_id, entry.status) == (
+ resource_id,
+ DocumentStatus.COMPLETED,
+ )
+ assert routes() == downloaded
+ assert downloader.urls == [url, url]
+
+
+def render_document_details(db, monkeypatch, document_id):
+ """The document details page, as the browser gets it.
+
+ Runs the ``document_details_page`` route and renders its template. The
+ ``LibraryService`` it builds reads through the session that
+ ``make_library_service`` set up. Importing the app registers the
+ template globals that the page's base template uses.
+ """
+ from local_deep_research.web import fastapi_app # noqa: F401
+
+ request = Request(
+ {
+ "type": "http",
+ "method": "GET",
+ "path": f"/library/document/{document_id}",
+ "headers": [],
+ "query_string": b"",
+ "session": {"session_id": "sid", "username": USERNAME},
+ }
+ )
+ response = library_router.document_details_page(
+ request, document_id, username=USERNAME
+ )
+ return response.body.decode()
+
+
[email protected]("status", ["completed", "failed"])
+def test_document_details_page_shows_the_download_status(
+ db, store, world, monkeypatch, status
+):
+ """The details page's "Download Status" badge shows the status.
+
+ The page compares the status it gets from ``get_document_by_id`` with
+ the string 'completed' and otherwise prints it in a warning badge. It
+ got the loaded ``DocumentStatus`` member, which never equals that
+ string, so every document showed a warning badge reading
+ "Documentstatus.completed" or "Documentstatus.failed". It gets the
+ status's value now: a completed download shows "Completed", and a
+ failed download "Failed".
+ """
+ url = f"https://example.com/details-of-a-{status}-download.pdf"
+ resource_id = world.add_resource(url)
+ service = make_service(store, StubDownloader(), storage_mode="database")
+ if status == "completed":
+ assert service.download_resource(resource_id) == (True, None)
+ else:
+ resource = db.session.get(ResearchResource, resource_id)
+ service._record_failed_text_extraction(
+ db.session, resource, error="PDF download failed: HTTP 503"
+ )
+ db.session.commit()
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ assert document.status is DocumentStatus(status)
+ make_library_service(db, store, monkeypatch)
+
+ page = render_document_details(db, monkeypatch, document_id)
+
+ badges = re.findall(
+ r'Download Status</div>\s*<div class="ldr-metadata-value">\s*'
+ r'<span class="ldr-badge (ldr-badge-[a-z]+)">([^<]*)</span>',
+ page,
+ )
+ assert badges == [
+ ("ldr-badge-success", "Completed")
+ if status == "completed"
+ else ("ldr-badge-warning", "Failed")
+ ]
+
+
+def test_text_extraction_reads_the_stored_pdf_file_without_a_request(
+ db, store, world, filesystem_mode, monkeypatch
+):
+ """Text extraction reads a downloaded PDF file before fetching it.
+
+ Text extraction (``download_as_text``) tries the PDF file that a
+ completed download stores (``_try_existing_pdf_extraction``) before it
+ asks the source; only an arXiv paper's text is requested from arXiv
+ before that. The step compared the document's loaded status with the
+ string "completed", which a ``DocumentStatus`` member never equals, so
+ it never read the file, and the text came from a request for the URL.
+ A completed filesystem download without text, here one whose
+ extraction found none when it was downloaded, now gets the text of its
+ stored file, and no request is made.
+ """
+ url = "https://example.com/text-from-the-stored-file.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ # The PDF is stored under the library root that the step resolves its
+ # path against.
+ service.settings._values["research_library.storage_path"] = str(store.base)
+ service._extract_text_from_pdf = lambda _data: None
+ assert service.download_resource(resource_id) == (True, None)
+ del service._extract_text_from_pdf
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ assert document.status is DocumentStatus.COMPLETED
+ assert document.text_content is None
+ assert (store.root / document.file_path).read_bytes() == TEXT_PDF_BYTES
+ assert downloader.urls == [url]
+
+ assert service.download_as_text(resource_id) == (True, None)
+
+ assert downloader.urls == [url], "the text was fetched again"
+ with db.fresh() as other:
+ assert other.query(Document).count() == 1
+ document = other.get(Document, document_id)
+ assert document.text_content == EXTRACTED_TEXT
+ assert (document.extraction_method, document.extraction_source) == (
+ "pdf_extraction",
+ "pdfplumber",
+ )
+
+
+def test_cached_text_extraction_uses_explicit_document_over_retained_owner(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch
+):
+ """A retained older PDF must not supply text for a relinked resource."""
+ linked_id = world.add_resource("https://example.com/current-version.pdf")
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ service.settings._values["research_library.storage_path"] = str(store.base)
+ original_extract = service._extract_text_from_pdf
+ service._extract_text_from_pdf = lambda _data: None
+ assert service.download_resource(linked_id) == (True, None)
+ _, mirror_id = _cite_in_another_research(
+ db, "https://mirror.example.org/old-version.pdf"
+ )
+ assert service.download_resource(mirror_id) == (True, None)
+ with db.fresh() as session:
+ old_id = session.get(ResearchResource, linked_id).document_id
+ assert session.get(ResearchResource, mirror_id).document_id == old_id
+
+ _, owner_id = _cite_in_another_research(
+ db, "https://example.org/new-version.pdf"
+ )
+ downloader.content = REGENERATED_PDF_BYTES
+ assert service.download_resource(owner_id) == (True, None)
+ with db.fresh() as session:
+ canonical_id = session.get(ResearchResource, owner_id).document_id
+ assert canonical_id != old_id
+ library = make_library_service(db, store, monkeypatch)
+ assert library.mark_for_redownload([old_id]) == 1
+ assert service.download_resource(linked_id) == (True, None)
+ downloader.content = TEXT_PDF_BYTES
+ service._extract_text_from_pdf = original_extract
+ assert service.download_resource(mirror_id) == (True, None)
+ with db.fresh() as session:
+ old = session.get(Document, old_id)
+ canonical = session.get(Document, canonical_id)
+ assert old.resource_id == linked_id
+ assert old.status is DocumentStatus.COMPLETED
+ assert canonical.status is DocumentStatus.COMPLETED
+ assert old.text_content == EXTRACTED_TEXT
+ assert canonical.text_content is None
+ assert session.get(ResearchResource, linked_id).document_id == (
+ canonical_id
+ )
+ assert (store.root / old.file_path).read_bytes() == TEXT_PDF_BYTES
+ assert (store.root / canonical.file_path).read_bytes() == (
+ REGENERATED_PDF_BYTES
+ )
+ requests_before = list(downloader.urls)
+
+ assert service.download_as_text(linked_id) == (True, None)
+
+ assert downloader.urls == requests_before, "the cached PDF was not used"
+ with db.fresh() as session:
+ assert session.get(ResearchResource, linked_id).document_id == (
+ canonical_id
+ )
+ assert session.get(Document, canonical_id).text_content == (
+ REGENERATED_TEXT
+ )
+ assert session.get(Document, old_id).text_content == EXTRACTED_TEXT
+
+
[email protected]("tracker_path", ["kept", "lost"])
+def test_download_whose_resource_was_deleted_takes_its_pdf_back_by_content(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ tracker_path,
+):
+ """Without its resource, a removed PDF is cleared but not reported.
+
+ A second research cites the same URL, and its download stores a
+ regenerated PDF as a document of its own. Deleting that resource from
+ its research keeps the document but clears its ``resource_id``. If the
+ PDF file is then removed, no download path asks for the PDF again,
+ since they start from a research's resources, so the sync does not
+ report a file to download again. It keeps the document's text and
+ makes it text-only, and resets the URL's tracker, as it does for a
+ document that has its resource. Left naming the missing file, the
+ document kept a later download of the same PDF bytes, which the
+ content-hash dedup links to it, from storing the PDF: the dedup copied
+ the dangling path into that download's tracker instead. ``lost`` is the
+ R1 sequence (a "none"-mode re-download cleared the tracker path) before
+ the resource was deleted, which reaches the sync's no-path branch.
+ """
+ url = "https://example.com/cited-twice-then-removed.pdf"
+ first_id = world.add_resource(url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(first_id) == (True, None)
+ _, second_id = _cite_in_another_research(db, url)
+ downloader.content = REGENERATED_PDF_BYTES
+ assert service.download_resource(second_id) == (True, None)
+ with db.fresh() as other:
+ document = other.query(Document).filter_by(resource_id=second_id).one()
+ document_id = document.id
+ own_path = document.file_path
+ assert document.text_content == REGENERATED_TEXT
+ assert find_tracker_by_url(other, url).file_path == own_path
+ library = make_library_service(db, store, monkeypatch)
+ if tracker_path == "lost":
+ assert library.mark_for_redownload([document_id]) == 1
+ service.settings._values["research_library.pdf_storage_mode"] = "none"
+ assert service.download_resource(second_id) == (True, None)
+ service.settings._values["research_library.pdf_storage_mode"] = (
+ "filesystem"
+ )
+ with db.fresh() as other:
+ assert find_tracker_by_url(other, url).file_path is None
+ assert other.get(Document, document_id).file_path == own_path
+ # As resource_service.delete_resource does: the row goes, its
+ # documents stay with resource_id cleared.
+ db.session.delete(db.session.get(ResearchResource, second_id))
+ db.session.commit()
+ with db.fresh() as other:
+ assert other.get(Document, document_id).resource_id is None
+ (store.root / own_path).unlink()
+ chunk_id = _add_chunk(db, document_id)
+ requests = len(downloader.urls)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)
+ assert stats["missing_files"] == []
+ assert stats["trackers_updated"] == 1
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.text_content == REGENERATED_TEXT
+ assert document.storage_mode == "none"
+ assert document.file_path == FILE_PATH_TEXT_ONLY
+ assert not document_awaits_pdf(document)
+ assert other.get(DocumentChunk, chunk_id) is not None
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.is_downloaded is False
+ assert tracker.file_path is None
+ view = library.get_document_by_id(document_id)
+ assert view["has_pdf"] is False
+ assert view["has_text_db"] is True
+ # Nothing asks for its PDF...
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 0
+ )
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ assert len(downloader.urls) == requests
+
+ # ...but a later download of the same PDF for another resource stores
+ # it in this document.
+ _, third_id = _cite_in_another_research(
+ db, url, query="third research citing the paper"
+ )
+ assert service.download_resource(third_id) == (True, None)
+ assert len(downloader.urls) == requests + 1
+ with db.fresh() as other:
+ assert other.query(Document).count() == 2
+ assert other.get(ResearchResource, third_id).document_id == (
+ document_id
+ )
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "filesystem"
+ assert document.text_content == REGENERATED_TEXT
+ assert other.get(DocumentChunk, chunk_id) is not None
+ restored = store.root / document.file_path
+ assert find_tracker_by_url(other, url).file_path == document.file_path
+ assert restored.read_bytes() == REGENERATED_PDF_BYTES
+ assert library.get_document_by_id(document_id)["has_pdf"] is True
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)
+
+
+# ===========================================================================
+# Documents that share their URL's tracker
+# ===========================================================================
+
+
[email protected]("first", ["download", "placeholder"])
+def test_placeholder_sharing_the_url_of_a_removed_file_is_left_as_it_is(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch, first
+):
+ """Only the document whose file was removed is reported missing.
+
+ A URL has one tracker, however many documents have that URL. The first
+ research downloaded the PDF into its document, and the tracker names
+ that document's file. A second research's download of the same URL
+ failed, and "Extract Text to DB" completed its placeholder with text:
+ no PDF, no file path, the ``"database"`` column default and no blob.
+ Once the first document's file was removed, a sync that reached the
+ placeholder first judged it by the tracker's path. It labelled the
+ placeholder a PDF, made it text-only and reported the other document's
+ file missing for it, and the download manager then downloaded the PDF
+ a second time, into the placeholder. The removed file was counted
+ twice. The sync judges each document by what it stores itself, so the
+ placeholder is left as it is whichever document the sync reaches first
+ (``first``).
+ """
+ url = "https://example.com/shared-with-a-text-placeholder.pdf"
+ owner_id = world.add_resource(url)
+ _, placeholder_resource_id = _cite_in_another_research(db, url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ placeholder_resource = db.session.get(
+ ResearchResource, placeholder_resource_id
+ )
+
+ def record_failed_download():
+ service._record_failed_text_extraction(
+ db.session,
+ placeholder_resource,
+ error="PDF download failed: HTTP 503",
+ )
+ db.session.commit()
+
+ if first == "placeholder":
+ record_failed_download()
+ assert service.download_resource(owner_id) == (True, None)
+ if first == "download":
+ record_failed_download()
+ service._save_text_with_db(
+ placeholder_resource,
+ SAVED_TEXT,
+ db.session,
+ extraction_method="native_api",
+ extraction_source="arxiv_api",
+ )
+ db.session.commit()
+ with db.fresh() as other:
+ downloaded = other.query(Document).filter_by(resource_id=owner_id).one()
+ downloaded_id = downloaded.id
+ own_path = downloaded.file_path
+ assert downloaded.text_content == EXTRACTED_TEXT
+ placeholder = (
+ other.query(Document)
+ .filter_by(resource_id=placeholder_resource_id)
+ .one()
+ )
+ placeholder_id = placeholder.id
+ assert placeholder.status is DocumentStatus.COMPLETED
+ assert placeholder.text_content == SAVED_TEXT
+ placeholder_shape = (
+ placeholder.file_type,
+ placeholder.mime_type,
+ placeholder.storage_mode,
+ placeholder.file_path,
+ )
+ assert placeholder_shape == ("unknown", None, "database", None)
+ assert other.query(DocumentBlob).count() == 0
+ assert find_tracker_by_url(other, url).file_path == own_path
+ # The order in which the sync reads the documents.
+ assert [
+ document.id
+ for document in other.query(Document)
+ .filter_by(status=DocumentStatus.COMPLETED)
+ .filter(Document.original_url.isnot(None))
+ ] == (
+ [downloaded_id, placeholder_id]
+ if first == "download"
+ else [placeholder_id, downloaded_id]
+ )
+ (store.root / own_path).unlink()
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (1, 1)
+ assert [
+ (entry["id"], entry["path"]) for entry in stats["missing_files"]
+ ] == [(downloaded_id, own_path)]
+ assert stats["trackers_updated"] == 1
+ with db.fresh() as other:
+ placeholder = other.get(Document, placeholder_id)
+ assert placeholder.text_content == SAVED_TEXT
+ assert (
+ placeholder.file_type,
+ placeholder.mime_type,
+ placeholder.storage_mode,
+ placeholder.file_path,
+ ) == placeholder_shape
+ assert not document_awaits_pdf(placeholder)
+ downloaded = other.get(Document, downloaded_id)
+ assert downloaded.text_content == EXTRACTED_TEXT
+ assert document_awaits_pdf(downloaded)
+
+ # The download manager downloads the PDF once, into the document whose
+ # file was removed.
+ queued = download_all_new(db, monkeypatch)
+ assert queued["queued"] == 1, queued
+ assert queued["research_ids"] == [world.research_id]
+ assert service.download_resource(owner_id) == (True, None)
+ assert service.download_resource(placeholder_resource_id) == (True, None)
+ assert len(downloader.urls) == 2
+ with db.fresh() as other:
+ restored = other.get(Document, downloaded_id).file_path
+ placeholder = other.get(Document, placeholder_id)
+ assert placeholder.file_path is None
+ assert placeholder.text_content == SAVED_TEXT
+ assert _tree(store.root) == {restored}
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)
+
+
[email protected]("text", [True, False], ids=["with-text", "no-text"])
[email protected]("tracker_path", ["other-file", "sentinel"])
+def test_removed_file_is_reported_although_the_url_tracker_names_another(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ tracker_path,
+ text,
+):
+ """A document whose own file was removed is reported missing.
+
+ Two researches cite the same URL, and each download stored its PDF in
+ a document of its own: the second research got a regenerated PDF with
+ other bytes, so the content-hash dedup did not link it to the first
+ research's document. The URL's one tracker names the file stored
+ last, the second research's (``other-file``). When the first
+ research's file was removed, the sync judged the first document by
+ the tracker's file, which is still there, and counted it found:
+ nothing reported its PDF missing, and no download path asked for it
+ again. A tracker that a content-hash duplicate download set to a
+ sentinel such as ``blob_deleted``, before such downloads stopped
+ copying sentinels, made the sync count every document of the URL found
+ in the same way (``sentinel``). The sync judges the first document by
+ its own file now. With its text left (``with-text``) it is made
+ text-only and reported, and the download manager puts its PDF back.
+ With nothing else left (``no-text``) it is deleted and reported, so
+ that its URL can be downloaded again. The second research's document
+ and the tracker are left as they are.
+ """
+ url = "https://example.com/two-copies-one-tracker.pdf"
+ first_id = world.add_resource(url)
+ content = TEXT_PDF_BYTES if text else PDF_BYTES
+ downloader = StubDownloader(content)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(first_id) == (True, None)
+ _, second_id = _cite_in_another_research(db, url)
+ downloader.content = REGENERATED_PDF_BYTES
+ assert service.download_resource(second_id) == (True, None)
+ with db.fresh() as other:
+ first = other.query(Document).filter_by(resource_id=first_id).one()
+ first_document_id = first.id
+ first_path = first.file_path
+ assert first.text_content == (EXTRACTED_TEXT if text else None)
+ second = other.query(Document).filter_by(resource_id=second_id).one()
+ second_document_id = second.id
+ second_path = second.file_path
+ assert second_path != first_path
+ assert find_tracker_by_url(other, url).file_path == second_path
+ if tracker_path == "sentinel":
+ db.session.query(DownloadTracker).filter_by(
+ url_hash=get_url_hash(url)
+ ).update(
+ {DownloadTracker.file_path: FILE_PATH_BLOB_DELETED},
+ synchronize_session=False,
+ )
+ db.session.commit()
+ expected_tracker = (
+ second_path if tracker_path == "other-file" else FILE_PATH_BLOB_DELETED,
+ True,
+ )
+ (store.root / first_path).unlink()
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (1, 1)
+ assert [
+ (entry["id"], entry["path"]) for entry in stats["missing_files"]
+ ] == [(first_document_id, first_path)]
+ assert stats["trackers_updated"] == 0
+ with db.fresh() as other:
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == expected_tracker
+ second = other.get(Document, second_document_id)
+ assert (second.storage_mode, second.file_path) == (
+ "filesystem",
+ second_path,
+ )
+ first = other.get(Document, first_document_id)
+ if text:
+ assert first.text_content == EXTRACTED_TEXT
+ assert first.file_path == FILE_PATH_TEXT_ONLY
+ assert document_awaits_pdf(first)
+ else:
+ assert first is None, "nothing of the document was left"
+ assert (store.root / second_path).read_bytes() == REGENERATED_PDF_BYTES
+
+ # The download manager downloads the first research's PDF again.
+ downloader.content = content
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 1
+ )
+ assert service.download_resource(first_id) == (True, None)
+ with db.fresh() as other:
+ first = other.query(Document).filter_by(resource_id=first_id).one()
+ assert (first.id == first_document_id) is text
+ restored = first.file_path
+ assert (store.root / restored).read_bytes() == content
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)
+
+
[email protected]("first", ["deleted", "kept"])
+def test_tracker_reset_for_two_documents_counts_once_in_either_order(
+ db, store, world, one_download_session, filesystem_mode, monkeypatch, first
+):
+ """``trackers_updated`` counts a tracker once, whichever document is first.
+
+ Two researches cite the same URL, and each download stored its PDF in
+ a document of its own (other bytes, so the content-hash dedup did not
+ link them). The URL's one tracker names the file of the first
+ research's document, which has no text, and both files were removed.
+ The sync deletes that document and resets the tracker, and makes the
+ second research's document, which has its text, text-only and reports
+ it. When the sync read the deleted document first (``deleted``), it
+ reached the other one with the tracker reset already, reset it again
+ and counted it twice. In the other order (``kept``) the tracker still
+ named the first research's file when the sync reached the second
+ document, so it did not describe that document and was left alone
+ there, and the deletion counted it once. Either order counts it once
+ now, and the documents and the tracker end the same.
+ """
+ url = "https://example.com/one-tracker-reset-twice.pdf"
+ deleted_resource_id = world.add_resource(url)
+ _, kept_resource_id = _cite_in_another_research(db, url)
+ downloader = StubDownloader()
+ service = make_service(store, downloader, storage_mode="filesystem")
+
+ def download(resource_id, content):
+ downloader.content = content
+ assert service.download_resource(resource_id) == (True, None)
+
+ # The sync reads the documents in the order they were stored.
+ if first == "deleted":
+ download(deleted_resource_id, PDF_BYTES)
+ download(kept_resource_id, TEXT_PDF_BYTES)
+ else:
+ download(kept_resource_id, TEXT_PDF_BYTES)
+ download(deleted_resource_id, PDF_BYTES)
+ with db.fresh() as other:
+ deleted = (
+ other.query(Document)
+ .filter_by(resource_id=deleted_resource_id)
+ .one()
+ )
+ deleted_id, deleted_path = deleted.id, deleted.file_path
+ assert deleted.text_content is None
+ kept = (
+ other.query(Document).filter_by(resource_id=kept_resource_id).one()
+ )
+ kept_id, kept_path = kept.id, kept.file_path
+ assert kept.text_content == EXTRACTED_TEXT
+ read_order = [
+ document.id
+ for document in other.query(Document)
+ .filter_by(status=DocumentStatus.COMPLETED)
+ .filter(Document.original_url.isnot(None))
+ ]
+ assert read_order == (
+ [deleted_id, kept_id]
+ if first == "deleted"
+ else [kept_id, deleted_id]
+ )
+ # The tracker names the file written last. In both orders it names the
+ # first research's file here, as when that file was written last.
+ db.session.query(DownloadTracker).filter_by(
+ url_hash=get_url_hash(url)
+ ).update(
+ {
+ DownloadTracker.file_path: deleted_path,
+ DownloadTracker.is_downloaded: True,
+ },
+ synchronize_session=False,
+ )
+ db.session.commit()
+ (store.root / deleted_path).unlink()
+ (store.root / kept_path).unlink()
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (0, 2)
+ assert sorted(
+ (entry["id"], entry["path"]) for entry in stats["missing_files"]
+ ) == sorted([(deleted_id, deleted_path), (kept_id, kept_path)])
+ assert stats["trackers_updated"] == 1
+ with db.fresh() as other:
+ assert other.get(Document, deleted_id) is None
+ kept = other.get(Document, kept_id)
+ assert kept.text_content == EXTRACTED_TEXT
+ assert document_awaits_pdf(kept)
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == (None, False)
+
+
[email protected]("first", ["empty", "second"])
[email protected]("holds", ["blob", "text", "nothing"])
+def test_database_documents_of_one_url_are_each_judged_by_their_own_blob(
+ db, store, world, monkeypatch, holds, first
+):
+ """Each database document of a URL is judged by what it stores itself.
+
+ Three researches cite the same paper. The first research's document is
+ a database-mode download that holds nothing (see
+ ``_add_database_row_without_pdf``). The third research downloaded the
+ same bytes in database mode, and the content-hash dedup linked its
+ resource to that empty document and recorded ``"database"`` in the
+ URL's one tracker. The second research's document is a second one: a
+ database-mode download of a regenerated PDF with other bytes, which
+ holds that PDF as a blob (``blob``) or holds nothing either, the size
+ limit having refused that PDF too (``nothing``), or a failed-download
+ placeholder that "Extract Text to DB" completed, which holds only text
+ (``text``). Both documents are labelled as stored in the database,
+ without a path, so the tracker describes both, and the sync judges
+ each by its own blob and text, whichever it reads first (``first``):
+ it deletes and reports, as ``"database"``, each document that holds
+ nothing, and keeps the other one as it is. The tracker records that
+ the URL's PDF is stored in the database. While the second document
+ holds a blob, that is still true and the tracker is left as it is.
+ Otherwise no document holds the PDF any more, and the tracker is reset
+ once. When the tracker counted only for a document storing the same
+ path, it described neither document: the documents that held nothing
+ were deleted without being reported, and the tracker was never reset.
+ """
+ url = "https://example.com/two-database-documents.pdf"
+ empty_resource_id = world.add_resource(url, downloaded=True)
+ second_research_id, second_resource_id = _cite_in_another_research(db, url)
+ _, linked_resource_id = _cite_in_another_research(
+ db, url, query="third research citing the paper"
+ )
+ downloader = StubDownloader(REGENERATED_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="database")
+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)
+
+ def add_second_document():
+ if holds == "blob":
+ assert service.download_resource(second_resource_id) == (True, None)
+ elif holds == "text":
+ resource = db.session.get(ResearchResource, second_resource_id)
+ service._record_failed_text_extraction(
+ db.session, resource, error="PDF download failed: HTTP 503"
+ )
+ db.session.commit()
+ service._save_text_with_db(
+ resource,
+ SAVED_TEXT,
+ db.session,
+ extraction_method="native_api",
+ extraction_source="arxiv_api",
+ )
+ db.session.commit()
+ else:
+ _add_database_row_without_pdf(
+ db,
+ world,
+ second_resource_id,
+ second_research_id,
+ url,
+ REGENERATED_PDF_BYTES,
+ )
+
+ if first == "second":
+ add_second_document()
+ empty_id = _add_database_row_without_pdf(
+ db, world, empty_resource_id, world.research_id, url, PDF_BYTES
+ )
+ if first == "empty":
+ add_second_document()
+ downloader.content = PDF_BYTES
+ assert service.download_resource(linked_resource_id) == (True, None)
+ with db.fresh() as other:
+ linked = other.get(ResearchResource, linked_resource_id)
+ assert linked.document_id == empty_id
+ second = (
+ other.query(Document)
+ .filter_by(resource_id=second_resource_id)
+ .one()
+ )
+ second_id = second.id
+ second_shape = (
+ second.file_type,
+ second.storage_mode,
+ second.file_path,
+ second.text_content,
+ )
+ assert second_shape[1:] == (
+ "database",
+ None,
+ SAVED_TEXT if holds == "text" else None,
+ )
+ assert [row.document_id for row in other.query(DocumentBlob)] == (
+ [second_id] if holds == "blob" else []
+ )
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)
+ # The order in which the sync reads the documents.
+ read_order = [
+ document.id
+ for document in other.query(Document)
+ .filter_by(status=DocumentStatus.COMPLETED)
+ .filter(Document.original_url.isnot(None))
+ ]
+ assert read_order == (
+ [empty_id, second_id] if first == "empty" else [second_id, empty_id]
+ )
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ second_kept = holds != "nothing"
+ assert (stats["files_found"], stats["files_missing"]) == (
+ (1, 1) if second_kept else (0, 2)
+ )
+ assert [
+ (entry["id"], entry["path"]) for entry in stats["missing_files"]
+ ] == [
+ (document_id, "database")
+ for document_id in read_order
+ if document_id == empty_id or not second_kept
+ ]
+ assert stats["trackers_updated"] == (0 if holds == "blob" else 1)
+ with db.fresh() as other:
+ assert other.get(Document, empty_id) is None
+ second = other.get(Document, second_id)
+ if second_kept:
+ assert (
+ second.file_type,
+ second.storage_mode,
+ second.file_path,
+ second.text_content,
+ ) == second_shape
+ else:
+ assert second is None, "nothing of the document was left"
+ if holds == "blob":
+ assert (
+ PDFStorageManager(store.root, "none").load_pdf(second, other)
+ == REGENERATED_PDF_BYTES
+ )
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == (
+ ("database", True) if holds == "blob" else (None, False)
+ )
+
+
+def test_document_stored_without_a_pdf_is_not_judged_by_a_database_tracker(
+ db, store, world, monkeypatch
+):
+ """A ``"database"`` tracker does not describe a document without a PDF.
+
+ The first research downloaded the paper with PDF storage set to
+ "none", so its document stores no PDF, and the PDF had no text to
+ extract. The second research got a regenerated PDF with other bytes in
+ database mode, stored as a blob in a document of its own, and the
+ URL's tracker records ``"database"``. That tracker describes only
+ documents labelled as stored in the database. The first document is
+ left alone, as every document stored without a PDF is, and not deleted
+ for lacking a blob it never had.
+ """
+ url = "https://example.com/stored-without-a-pdf.pdf"
+ first_id = world.add_resource(url)
+ _, second_id = _cite_in_another_research(db, url)
+ downloader = StubDownloader(PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="none")
+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)
+ assert service.download_resource(first_id) == (True, None)
+ downloader.content = REGENERATED_PDF_BYTES
+ service.settings._values["research_library.pdf_storage_mode"] = "database"
+ assert service.download_resource(second_id) == (True, None)
+ with db.fresh() as other:
+ first = other.query(Document).filter_by(resource_id=first_id).one()
+ first_document_id = first.id
+ first_shape = (first.storage_mode, first.file_path, first.text_content)
+ assert first_shape == ("none", None, None)
+ second = other.query(Document).filter_by(resource_id=second_id).one()
+ second_document_id = second.id
+ assert (second.storage_mode, second.file_path) == ("database", None)
+ assert [row.document_id for row in other.query(DocumentBlob)] == [
+ second_document_id
+ ]
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (2, 0)
+ assert stats["missing_files"] == []
+ assert stats["trackers_updated"] == 0
+ with db.fresh() as other:
+ first = other.get(Document, first_document_id)
+ assert first is not None, "the library sync deleted the document"
+ assert (
+ first.storage_mode,
+ first.file_path,
+ first.text_content,
+ ) == first_shape
+ assert other.get(Document, second_document_id) is not None
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)
+
+
+# ===========================================================================
+# "Remove PDF to save space" is not a missing file
+# ===========================================================================
+
+
[email protected]("text", [True, False], ids=["with-text", "no-text"])
[email protected]("stored_mode", ["database", "filesystem"])
+def test_pdf_removed_to_save_space_is_kept_and_not_downloaded_again(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ stored_mode,
+ text,
+):
+ """``delete_blob_only`` followed by a sync keeps the document as it is.
+
+ "Remove PDF to save space" removes the stored PDF, blob or file, and
+ marks the document ``blob_deleted``, but leaves the tracker naming the
+ copy it removed. The sync resolved that name, found nothing, and
+ deleted a document without text: its row, metadata and collection
+ link were lost and the tracker was reset, so the download manager
+ offered the PDF the user had removed for download again. A document
+ with text must not be taken for one whose PDF went missing either.
+ Neither the sync nor any download path brings the removed PDF back.
+ """
+ url = "https://example.com/removed-to-save-space.pdf"
+ resource_id = world.add_resource(url)
+ content = TEXT_PDF_BYTES if text else PDF_BYTES
+ downloader = StubDownloader(content)
+ service = make_service(store, downloader, storage_mode=stored_mode)
+ if not text:
+ monkeypatch.setattr(
+ service, "_extract_text_from_pdf", lambda _data: None
+ )
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document_id = (
+ other.query(Document).filter_by(resource_id=resource_id).one().id
+ )
+ tracker_path = find_tracker_by_url(other, url).file_path
+ assert tracker_path
+
+ library = make_library_service(db, store, monkeypatch)
+ # delete_blob_only resolves the file through the ambient settings.
+ settings = StubSettings({"research_library.storage_path": str(store.base)})
+ monkeypatch.setattr(
+ db_utils,
+ "get_settings_manager",
+ lambda *a, **k: SimpleNamespace(
+ get_setting=settings.get_setting, close=lambda: None
+ ),
+ )
+ removed = document_deletion.DocumentDeletionService(
+ USERNAME
+ ).delete_blob_only(document_id)
+ assert removed["deleted"] is True, removed
+ assert removed["bytes_freed"] == len(content)
+ assert _tree(store.root) == set()
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document.storage_mode == "none"
+ assert document.file_path == FILE_PATH_BLOB_DELETED
+ assert other.query(DocumentBlob).count() == 0
+ # The shape the sync has to handle: the tracker still names the
+ # removed copy.
+ assert find_tracker_by_url(other, url).file_path == tracker_path
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+ assert stats["missing_files"] == []
+ with db.fresh() as other:
+ document = other.get(Document, document_id)
+ assert document is not None, "the library sync deleted the document"
+ assert document.storage_mode == "none"
+ assert document.file_path == FILE_PATH_BLOB_DELETED
+ assert document.text_content == (EXTRACTED_TEXT if text else None)
+ assert _library_links(other, document_id) == 1
+ assert find_tracker_by_url(other, url).is_downloaded is True
+ assert library.get_document_by_id(document_id)["has_pdf"] is False
+
+ assert (
+ service.queue_research_downloads(
+ world.research_id, LIBRARY_COLLECTION_ID
+ )
+ == 0
+ )
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ assert service.download_resource(resource_id) == (True, None)
+ assert len(downloader.urls) == 1
+ assert _tree(store.root) == set()
+
+
+# ===========================================================================
+# The deletion the sync exists for
+# ===========================================================================
+
+
[email protected]("tracker_path", ["kept", "lost"])
+def test_filesystem_download_whose_file_was_removed_is_deleted(
+ db, store, world, filesystem_mode, monkeypatch, tracker_path
+):
+ """A download that had only its file is deleted once the file is gone.
+
+ No text was extracted and there is no blob, so after the user removed
+ the PDF from disk nothing of the document is left. The sync deletes it
+ so that the URL can be downloaded again. ``lost`` is the R1 sequence
+ (the tracker path cleared by a "none"-mode re-download) with the file
+ removed afterwards, where the no-path branch must still delete.
+ """
+ url = "https://example.com/removed-from-disk.pdf"
+ resource_id = world.add_resource(url)
+ downloader = StubDownloader()
+ service = make_service(store, downloader, storage_mode="filesystem")
+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ document_id = document.id
+ stored = store.root / document.file_path
+ assert stored.is_file()
+ library = make_library_service(db, store, monkeypatch)
+ if tracker_path == "lost":
+ assert library.mark_for_redownload([document_id]) == 1
+ service.settings._values["research_library.pdf_storage_mode"] = "none"
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ assert find_tracker_by_url(other, url).file_path is None
+ stored.unlink()
+
+ stats = library.sync_library_with_filesystem()
+
+ assert stats["files_missing"] == 1
+ with db.fresh() as other:
+ assert other.get(Document, document_id) is None
+ assert _library_links(other, document_id) == 0
+ tracker = find_tracker_by_url(other, url)
+ assert tracker.file_path is None
+ if tracker_path == "kept":
+ assert tracker.is_downloaded is False
+ assert stats["trackers_updated"] == 1
+ assert [entry["id"] for entry in stats["missing_files"]] == [
+ document_id
+ ]
+
+ # The point of the deletion: the URL downloads again.
+ service.settings._values["research_library.pdf_storage_mode"] = "filesystem"
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ redownloaded = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ assert redownloaded.id != document_id
+ assert redownloaded.status is DocumentStatus.COMPLETED
+ assert stored.is_file()
+
+
+def test_database_download_with_no_blob_and_no_text_is_deleted_and_reported(
+ db, store, world, monkeypatch
+):
+ """A database-mode download with nothing left is deleted and reported.
+
+ The document is a database-mode download that holds no blob and no
+ text (see ``_add_database_row_without_pdf``). A second research
+ downloaded the same bytes: the content-hash dedup linked its resource
+ to the document and recorded ``"database"`` in the URL's tracker, as
+ it does for a document labelled as stored in the database. Database
+ storage leaves the document's ``file_path`` None, so while the tracker
+ counted only for a document storing the same path it described no
+ database document: the sync deleted this one, but neither reported it
+ nor reset the tracker, which kept recording a PDF that no document
+ holds. The tracker describes the document now. The sync deletes it,
+ reports it as ``"database"`` and resets the tracker, and the URL
+ downloads again. A file named ``database`` in the library root is not
+ the document's PDF.
+ """
+ url = "https://example.com/database-row-with-nothing-left.pdf"
+ resource_id = world.add_resource(url, downloaded=True)
+ document_id = _add_database_row_without_pdf(
+ db, world, resource_id, world.research_id, url, PDF_BYTES
+ )
+ _, linked_id = _cite_in_another_research(db, url)
+ service = make_service(store, StubDownloader(), storage_mode="database")
+ monkeypatch.setattr(service, "_extract_text_from_pdf", lambda _data: None)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as other:
+ assert other.get(ResearchResource, linked_id).document_id == (
+ document_id
+ )
+ assert other.query(Document).count() == 1
+ assert other.query(DocumentBlob).count() == 0
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)
+ (store.root / "database").write_bytes(PDF_BYTES)
+ library = make_library_service(db, store, monkeypatch)
+
+ stats = library.sync_library_with_filesystem()
+
+ assert (stats["files_found"], stats["files_missing"]) == (0, 1)
+ assert stats["missing_files"] == [
+ {
+ "id": document_id,
+ "title": "Sync Contract Paper",
+ "path": "database",
+ "url": url,
+ }
+ ]
+ assert stats["trackers_updated"] == 1
+ with db.fresh() as other:
+ assert other.get(Document, document_id) is None
+ assert _library_links(other, document_id) == 0
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == (None, False)
+
+ # The point of the deletion: the URL downloads again.
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ redownloaded = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ assert redownloaded.id != document_id
+ assert (
+ other.query(DocumentBlob)
+ .filter_by(document_id=redownloaded.id)
+ .count()
+ == 1
+ )
+ tracker = find_tracker_by_url(other, url)
+ assert (tracker.file_path, tracker.is_downloaded) == ("database", True)
+ stats = library.sync_library_with_filesystem()
+ assert (stats["files_found"], stats["files_missing"]) == (1, 0)
+
+
+# ===========================================================================
+# What the sync returns
+# ===========================================================================
+
+
+def test_sync_response_names_missing_files_by_their_stored_path(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ tmp_path,
+):
+ """The sync's response holds no absolute server path.
+
+ ``POST /library/api/sync-library`` returns the sync's statistics, and
+ each ``missing_files`` entry carried the absolute path its file
+ resolved to on the server. That is the host's directory layout, which
+ ``/library/api/check-downloads`` and the library listing withhold from
+ the client (#6462). Every entry now gives the path stored for the file,
+ relative to the library root: a download with text whose file is gone,
+ reached through its tracker's path (``tracker-path``) or through its own
+ after a "none"-mode re-download cleared the tracker's (``own-path``),
+ and a download without text, which the sync deletes (``deleted``).
+ """
+ library = make_library_service(db, store, monkeypatch)
+ service = make_service(store, StubDownloader(), storage_mode="filesystem")
+ contents = {
+ "tracker-path": TEXT_PDF_BYTES,
+ "own-path": _text_pdf("Text of the paper downloaded again."),
+ "deleted": PDF_BYTES,
+ }
+ stored = {}
+ for case, content in contents.items():
+ resource_id = world.add_resource(f"https://example.com/{case}.pdf")
+ service.downloaders = [StubDownloader(content)]
+ assert service.download_resource(resource_id) == (True, None)
+ with db.fresh() as other:
+ document = (
+ other.query(Document).filter_by(resource_id=resource_id).one()
+ )
+ stored[document.id] = (case, document.file_path)
+ if case == "own-path":
+ assert library.mark_for_redownload([document.id]) == 1
+ settings = service.settings._values
+ settings["research_library.pdf_storage_mode"] = "none"
+ assert service.download_resource(resource_id) == (True, None)
+ settings["research_library.pdf_storage_mode"] = "filesystem"
+ for _case, path in stored.values():
+ (store.root / path).unlink()
+ monkeypatch.setattr(
+ library_router, "LibraryService", lambda username: library
+ )
+
+ stats = library_router.sync_library(None, username=USERNAME)
+
+ body = json.dumps(stats)
+ assert str(tmp_path) not in body, body
+ assert stats["files_missing"] == 3
+ assert {entry["id"]: entry["path"] for entry in stats["missing_files"]} == {
+ document_id: path for document_id, (_case, path) in stored.items()
+ }
+ with db.fresh() as other:
+ kept = {document.id for document in other.query(Document).all()}
+ assert kept == {
+ document_id
+ for document_id, (case, _path) in stored.items()
+ if case != "deleted"
+ }
+
+
+ "entrypoint", ["research-queue", "all-new", "direct", "research-count"]
+)
+def test_canonical_refill_ignores_retained_old_owner(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ entrypoint,
+):
+ """A completed retained snapshot must not hide the resource's explicit target."""
+ linked_url = "https://example.com/current-version.pdf"
+ linked_id = world.add_resource(linked_url)
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ # First extraction fails; both references still retain the real old PDF.
+ original_extract = service._extract_text_from_pdf
+ service._extract_text_from_pdf = lambda _data: None
+ assert service.download_resource(linked_id) == (True, None)
+ _, mirror_id = _cite_in_another_research(
+ db, "https://mirror.example.org/old-version.pdf", query="old version"
+ )
+ assert service.download_resource(mirror_id) == (True, None)
+ with db.fresh() as session:
+ old_id = session.get(ResearchResource, linked_id).document_id
+ assert session.get(ResearchResource, mirror_id).document_id == old_id
+ assert session.get(Document, old_id).text_content is None
+ service._extract_text_from_pdf = original_extract
+ owner_url = "https://example.org/new-version.pdf"
+ _, owner_id = _cite_in_another_research(db, owner_url, query="new version")
+ downloader.content = REGENERATED_PDF_BYTES
+ assert service.download_resource(owner_id) == (True, None)
+ with db.fresh() as session:
+ canonical_id = session.get(ResearchResource, owner_id).document_id
+ library = make_library_service(db, store, monkeypatch)
+ assert library.mark_for_redownload([old_id]) == 1
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as session:
+ assert (
+ session.get(ResearchResource, linked_id).document_id == canonical_id
+ )
+ assert session.get(Document, old_id).status == DocumentStatus.PENDING
+ # A later successful extraction through the old mirror completes the
+ # retained snapshot, without changing the first resource's new link.
+ downloader.content = TEXT_PDF_BYTES
+ assert service.download_resource(mirror_id) == (True, None)
+ with db.fresh() as session:
+ old = session.get(Document, old_id)
+ assert old.status == DocumentStatus.COMPLETED
+ assert old.resource_id == linked_id
+ assert (
+ session.get(ResearchResource, linked_id).document_id == canonical_id
+ )
+ canonical = session.get(Document, canonical_id)
+ assert canonical.text_content == REGENERATED_TEXT
+ (store.root / canonical.file_path).unlink()
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ with db.fresh() as session:
+ assert document_awaits_pdf(session.get(Document, canonical_id))
+ assert (
+ library_utils.get_document_for_resource(
+ session, session.get(ResearchResource, linked_id)
+ ).id
+ == canonical_id
+ )
+ downloader.content = REGENERATED_PDF_BYTES
+ if entrypoint == "research-queue":
+ queued = service.queue_research_downloads(
+ world.research_id,
+ LIBRARY_COLLECTION_ID,
+ run_research_ids=frozenset([world.research_id]),
+ )
+ assert queued == 1, (
+ "old owned document hides explicit target awaiting refill"
+ )
+ elif entrypoint == "all-new":
+ # The canonical owner's URL is denied now, but the linked URL works.
+ answer = download_all_new(db, monkeypatch, refused_urls={owner_url})
+ with db.fresh() as session:
+ pending = [
+ q.resource_id
+ for q in session.query(DownloadQueue).filter_by(
+ status=DocumentStatus.PENDING
+ )
+ ]
+ assert pending == [linked_id], (answer, pending, linked_id)
+ elif entrypoint == "research-count":
+ row = next(
+ row
+ for row in library.get_research_list_with_stats()
+ if row["id"] == world.research_id
+ )
+ assert row["downloaded_count"] == 0, row
+ else:
+ before = len(downloader.urls)
+ assert service.download_resource(linked_id) == (True, None)
+ with db.fresh() as session:
+ assert not document_awaits_pdf(
+ session.get(Document, canonical_id)
+ ), (
+ "success returned without refilling the canonical PDF",
+ downloader.urls[before:],
+ )
diff --git a/tests/research_library/test_library_sync_integration.py b/tests/research_library/test_library_sync_integration.py
new file mode 100644
index 0000000000..56f2f07681
--- /dev/null
+++ b/tests/research_library/test_library_sync_integration.py
@@ -0,0 +1,212 @@
+"""Library sync composes with canonical PDF reuse and text-index invalidation."""
+
+import hashlib
+from types import SimpleNamespace
+
+import pytest
+
+from local_deep_research.database.models.library import (
+ Collection,
+ Document,
+ DocumentCollection,
+ DocumentStatus,
+ DownloadQueue,
+ RagDocumentStatus,
+)
+from local_deep_research.database.models.research import ResearchResource
+from local_deep_research.research_library.services import library_service
+from local_deep_research.research_library.services.library_rag_service import (
+ LibraryRAGService,
+)
+from local_deep_research.research_library.services.pdf_storage_manager import (
+ PDFStorageManager,
+)
+from local_deep_research.research_library.utils import document_awaits_pdf
+from tests.research_library.services.test_download_text_quality import (
+ LONGER,
+ extraction_case as extraction_case,
+)
+from tests.research_library.services.test_text_index_invalidation import (
+ assert_pending,
+ indexer_for,
+ persist,
+)
+from tests.research_library.test_library_sync_contracts import (
+ EXTRACTED_TEXT,
+ REGENERATED_PDF_BYTES,
+ REGENERATED_TEXT,
+ TEXT_PDF_BYTES,
+ StubDownloader,
+ db as db,
+ download_all_new,
+ filesystem_mode as filesystem_mode,
+ make_library_service,
+ make_service,
+ one_download_session as one_download_session,
+ store as store,
+ world as world,
+)
+
+
[email protected]("changed", [False, True])
+ "storage,limit", [("database", 3072), ("none", 3072), ("database", 0)]
+)
+def test_linked_refill_settles_storage_and_requested_collection(
+ db,
+ store,
+ world,
+ one_download_session,
+ filesystem_mode,
+ monkeypatch,
+ changed,
+ storage,
+ limit,
+):
+ owner = world.add_resource("https://example.com/refill-owner.pdf")
+ downloader = StubDownloader(TEXT_PDF_BYTES)
+ service = make_service(store, downloader, storage_mode="filesystem")
+ assert service.download_resource(owner) == (True, None)
+ with db.fresh() as session:
+ document = session.query(Document).filter_by(resource_id=owner).one()
+ document_id = document.id
+ (store.root / document.file_path).unlink()
+ library = make_library_service(db, store, monkeypatch)
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+
+ linked = world.add_resource("https://example.com/refill-mirror.pdf")
+ db.session.get(ResearchResource, linked).document_id = document_id
+ db.session.add(Collection(id="requested", name="Requested"))
+ db.session.flush()
+ db.session.add(
+ DownloadQueue(
+ resource_id=linked,
+ research_id=world.research_id,
+ collection_id="requested",
+ status=DocumentStatus.PENDING,
+ )
+ )
+ db.session.commit()
+ content = REGENERATED_PDF_BYTES if changed else TEXT_PDF_BYTES
+ downloader.content = content
+ service.settings._values.update(
+ {
+ "research_library.pdf_storage_mode": storage,
+ "research_library.max_pdf_size_mb": limit,
+ }
+ )
+
+ assert service.download_resource(linked) == (True, None)
+ with db.fresh() as session:
+ assert session.query(Document).count() == 1
+ document = session.get(Document, document_id)
+ assert document.document_hash == hashlib.sha256(content).hexdigest()
+ assert document.text_content == (
+ REGENERATED_TEXT if changed else EXTRACTED_TEXT
+ )
+ assert not document_awaits_pdf(document)
+ assert (
+ session.query(DocumentCollection)
+ .filter_by(
+ document_id=document_id,
+ collection_id="requested",
+ )
+ .count()
+ == 1
+ )
+ assert (
+ session.query(DownloadQueue)
+ .filter_by(
+ resource_id=linked,
+ )
+ .one()
+ .status
+ is DocumentStatus.COMPLETED
+ )
+ manager = PDFStorageManager(store.root, storage)
+ if storage == "database" and limit:
+ assert manager.load_pdf(document, session) == content
+ else:
+ assert document.storage_mode == "none"
+ assert document.file_path is None
+ assert not manager.has_pdf(document, session)
+ assert downloader.urls == [
+ "https://example.com/refill-owner.pdf",
+ "https://example.com/refill-mirror.pdf",
+ ]
+ assert download_all_new(db, monkeypatch)["queued"] == 0
+ assert service.queue_research_downloads(world.research_id, "requested") == 0
+
+
[email protected]("mode", ["serial", "prepared"])
+def test_regenerated_refill_keeps_old_vectors_invalid_until_reindexed(
+ extraction_case,
+ monkeypatch,
+ tmp_path,
+ mode,
+):
+ case = extraction_case(quality="low", incoming=LONGER, storage="database")
+ base = tmp_path / "library"
+ case.service.library_root = base / case.service.username
+ with case.sessions() as session:
+ document = session.get(Document, "canonical")
+ document.original_url = "https://example.org/old-paper.pdf"
+ document.document_hash = hashlib.sha256(
+ b"previous PDF bytes"
+ ).hexdigest()
+ document.storage_mode = "filesystem"
+ document.file_path = "pdfs/missing.pdf"
+ session.get(
+ ResearchResource, case.resource_id
+ ).document_id = document.id
+ session.commit()
+ monkeypatch.setattr(
+ library_service, "get_user_db_session", lambda *a, **k: case.sessions()
+ )
+ monkeypatch.setattr(
+ library_service,
+ "get_settings_manager",
+ lambda **kw: SimpleNamespace(
+ get_setting=lambda key, default=None: str(base),
+ close=lambda: None,
+ ),
+ )
+ library = library_service.LibraryService.__new__(
+ library_service.LibraryService
+ )
+ library.username = case.service.username
+ assert library.sync_library_with_filesystem()["files_missing"] == 1
+ with case.sessions() as session:
+ assert document_awaits_pdf(session.get(Document, "canonical"))
+
+ indexer, backend = indexer_for(case)
+ old_ids = backend._store.ids.copy()
+ assert case.service.download_resource(case.resource_id) == (True, None)
+ with case.sessions() as session:
+ assert session.query(Document).count() == 1
+ document = session.get(Document, "canonical")
+ assert document.document_hash == case.pdf_hash
+ assert not document_awaits_pdf(document)
+ assert_pending(case)
+ assert_pending(case, "selected")
+ assert (
+ LibraryRAGService.reconcile_collection_index(indexer, "library")[
+ "indexed_documents"
+ ]
+ == 0
+ )
+ assert_pending(case)
+ assert persist(indexer, mode)["status"] == "success"
+ assert backend._store.removed == old_ids
+ assert old_ids.isdisjoint(backend._store.ids)
+ assert (
+ LibraryRAGService.reconcile_collection_index(indexer, "library")[
+ "indexed_documents"
+ ]
+ == 1
+ )
+ with case.sessions() as session:
+ assert (
+ session.get(RagDocumentStatus, ("canonical", "library")) is not None
+ )
+ assert_pending(case, "selected")
diff --git a/tests/security/test_egress_policy.py b/tests/security/test_egress_policy.py
index c4b285fb60..4acd261c93 100644
--- a/tests/security/test_egress_policy.py
+++ b/tests/security/test_egress_policy.py
@@ -2807,6 +2807,43 @@ def test_security_denials_still_exhaust_fetch_quota():
assert decision.reason == "denial_quota_exceeded"
+ ("url", "reason"),
+ [
+ ("https://private.example/page", "scope_mismatch_public_only"),
+ ("http://169.254.169.254/latest/meta-data/", "blocked_metadata_ip"),
+ ],
+)
+def test_unrecorded_denial_is_the_same_decision_and_records_nothing(
+ url, reason
+):
+ """``record_denial=False`` decides as a fetch's check does, recording nothing.
+
+ For a caller that only ranks URLs and fetches none of them
+ (``DownloadService.refill_rank``): the denial is neither counted toward
+ the run's denied-fetch quota, however often it is given, nor written to
+ the audit log. The default records the same denial.
+ """
+ from local_deep_research.security.egress import policy as policy_mod
+
+ ctx = make_ctx(scope=EgressScope.PUBLIC_ONLY)
+ with (
+ patch.object(policy_mod, "_classify_host", return_value=True),
+ patch.object(policy_mod, "logger") as audit_log,
+ ):
+ for _ in range(MAX_DENIED_FETCHES_PER_RUN + 1):
+ assert evaluate_url(url, ctx, record_denial=False) == Decision(
+ False, reason
+ )
+ assert audit_log.mock_calls == []
+ assert ctx._fetch_denial_count["count"] == 0
+
+ assert evaluate_url(url, ctx) == Decision(False, reason)
+ audit_log.bind.assert_called_once_with(policy_audit=True)
+ audit_log.bind.return_value.warning.assert_called_once()
+ assert ctx._fetch_denial_count["count"] == 1
+
+
def test_allowed_local_hostnames_accepts_unresolvable():
"""A legitimate intranet/VPN hostname that does not resolve (DNS down /
split-horizon) must be ACCEPTED on save (fail-open), not rejected — the
diff --git a/tests/web/routers/test_download_queue_claim_routes_4691.py b/tests/web/routers/test_download_queue_claim_routes_4691.py
index 3c34dc5ff2..a43309fb8b 100644
--- a/tests/web/routers/test_download_queue_claim_routes_4691.py
+++ b/tests/web/routers/test_download_queue_claim_routes_4691.py
@@ -557,10 +557,11 @@ def _run_download_source(Session, ds_mock=None):
def test_download_source_does_not_reset_an_in_flight_claim(tmp_path, request):
"""``download_source`` resets the existing queue row to PENDING and then
downloads immediately in the request thread. Its only early return is
- ``existing and existing.status == "completed"``, which is inert during the
- race window: an in-flight bulk download has not written a completed
- Document yet. So the reset itself must refuse a PROCESSING row -- the row
- stays PROCESSING, no second download starts, and the caller gets 409.
+ for a Document that counts as downloaded (``document_counts_as_downloaded``:
+ completed, and not awaiting its PDF), which is inert during the race
+ window: an in-flight bulk download has not written a completed Document
+ yet. So the reset itself must refuse a PROCESSING row -- the row stays
+ PROCESSING, no second download starts, and the caller gets 409.
This is the reset-lost path. ``test_library_download_outcomes.py``'s
``test_a_lost_claim_is_a_409_not_a_completion`` passes ``claim_result=0``
diff --git a/tests/web/routers/test_library_api_response_shapes.py b/tests/web/routers/test_library_api_response_shapes.py
index 605a1b51c2..0ae13dd3ee 100644
--- a/tests/web/routers/test_library_api_response_shapes.py
+++ b/tests/web/routers/test_library_api_response_shapes.py
@@ -51,6 +51,11 @@ from unittest.mock import MagicMock, Mock, patch
import pytest
from starlette.requests import Request
+from local_deep_research.constants import FILE_PATH_TEXT_ONLY
+from local_deep_research.database.models.library import (
+ Document,
+ DocumentStatus,
+)
from local_deep_research.web.routers import library as library_module
from local_deep_research.web.routers.library import (
check_downloads,
@@ -766,16 +771,38 @@ def _resource(rid=1, url="https://arxiv.org/abs/2301.00001", title="Paper"):
return row
-def _completed_document():
- doc = Mock()
- doc.id = "doc-1"
- doc.status = "completed"
- doc.file_type = "pdf"
- doc.title = "Paper"
- # The absolute server path. Present on the row the handler holds, so the
- # only thing keeping it out of the response is the handler not copying it.
- doc.file_path = "/srv/library/alice/pdfs/2301.00001.pdf"
- return doc
+def _completed_document(**columns):
+ """A completed download of resource 1, as the handlers load it.
+
+ A real ``Document``: its ``status`` is a ``DocumentStatus`` member, as on
+ a loaded row, which the string ``"completed"`` never equals.
+ """
+ values = {
+ "id": "doc-1",
+ "resource_id": 1,
+ "status": DocumentStatus.COMPLETED,
+ "file_type": "pdf",
+ "title": "Paper",
+ "storage_mode": "filesystem",
+ # The absolute server path. Present on the row the handler holds, so
+ # the only thing keeping it out of the response is the handler not
+ # copying it.
+ "file_path": "/srv/library/alice/pdfs/2301.00001.pdf",
+ }
+ values.update(columns)
+ return Document(**values)
+
+
+def _document_awaiting_its_pdf(**columns):
+ """A completed download that "Sync library" left awaiting its PDF.
+
+ Its PDF file was removed while its text was not, so the sync stored it
+ the way text-only documents are and reported the file missing (see
+ ``document_awaits_pdf``). It does not count as downloaded.
+ """
+ return _completed_document(
+ storage_mode="none", file_path=FILE_PATH_TEXT_ONLY, **columns
+ )
def _run_check_downloads(session, document, payload):
@@ -845,6 +872,27 @@ def test_check_downloads_reports_an_undownloaded_resource_by_id():
}
+def test_check_downloads_reports_a_document_awaiting_its_pdf_by_id():
+ """A completed download that awaits its PDF is not downloaded.
+
+ The Download Manager counts it as available to download, so the client
+ gets the ``resource_id`` to download its PDF by, not the document.
+ """
+ resource = _resource()
+ document = _document_awaiting_its_pdf()
+
+ result = _run_check_downloads(
+ _check_downloads_session(resource, document),
+ document,
+ {"research_id": "r1", "urls": [resource.url]},
+ )
+
+ assert result["download_status"][resource.url] == {
+ "downloaded": False,
+ "resource_id": 1,
+ }
+
+
@pytest.mark.parametrize(
"payload",
[
@@ -928,11 +976,7 @@ def _source_resource(rid, url, title, preview="Preview text", score=0.95):
def test_research_sources_carries_the_download_state_of_each_source():
resource = _source_resource(1, "https://arxiv.org/abs/1", "Test Paper")
- document = Mock()
- document.id = "doc-1"
- document.status = "completed"
- document.file_type = "pdf"
- document.created_at = datetime(2026, 5, 9)
+ document = _completed_document(created_at=datetime(2026, 5, 9))
with _patched(
session=_sources_session([resource]),
@@ -949,6 +993,26 @@ def test_research_sources_carries_the_download_state_of_each_source():
assert source["downloaded"] is True
assert source["document_id"] == "doc-1"
assert source["file_type"] == "pdf"
+ assert source["download_date"] == "2026-05-09T00:00:00"
+
+
+def test_research_sources_marks_a_document_awaiting_its_pdf_not_downloaded():
+ """A completed download that awaits its PDF is not downloaded, as the
+ Download Manager counts it, and the source names no document."""
+ resource = _source_resource(1, "https://arxiv.org/abs/1", "Test Paper")
+ document = _document_awaiting_its_pdf(created_at=datetime(2026, 5, 9))
+
+ with _patched(
+ session=_sources_session([resource]),
+ get_document_for_resource=Mock(return_value=document),
+ ):
+ result = get_research_sources(_get_request(), "r1", username="alice")
+
+ source = result["sources"][0]
+ assert source["downloaded"] is False
+ assert source["document_id"] is None
+ assert source["file_type"] is None
+ assert "download_date" not in source
def test_research_sources_marks_an_undownloaded_source():
diff --git a/tests/web/routers/test_library_download_outcomes.py b/tests/web/routers/test_library_download_outcomes.py
index 63323fd2ca..dac51411a8 100644
--- a/tests/web/routers/test_library_download_outcomes.py
+++ b/tests/web/routers/test_library_download_outcomes.py
@@ -9,13 +9,18 @@ appeared in no test on the branch.
Seven guarantees, all of them things a user reads or a template branches on:
* ``"Already downloaded"`` -- ``/library/api/download-source`` short-circuits
- only when a **completed** Document already exists for the resource. The
- guard is ``existing and existing.status == "completed"``, and the status
- half is load-bearing: a resource whose previous attempt is still ``pending``
- or came back ``failed`` must be re-downloaded, not reported as already in
- the library. Drop the status check and every failed download becomes
- permanently unretryable through this endpoint, with the UI cheerfully
- reporting success.
+ only when the resource's Document counts as downloaded: it is
+ **completed** and does not await its PDF (``document_counts_as_downloaded``).
+ The status half is load-bearing: a resource whose previous attempt is still
+ ``pending`` or came back ``failed`` must be re-downloaded, not reported as
+ already in the library. Drop the status check and every failed download
+ becomes permanently unretryable through this endpoint, with the UI
+ cheerfully reporting success. A loaded status is a ``DocumentStatus``
+ member, which the string ``"completed"`` never equals, so the documents
+ here are real ``Document`` rows with member statuses: compared with that
+ string, the guard never short-circuited. A completed document that awaits
+ its PDF (the library sync kept its text after its PDF file went missing)
+ is downloaded again.
* ``"Download completed"`` / ``"Download failed"`` -- the two terminal
messages of the same endpoint. ``DownloadService.download_resource``
returns ``(success, message)`` where ``message`` is an internal diagnostic
@@ -62,6 +67,11 @@ import pytest
from bs4 import BeautifulSoup
from starlette.requests import Request
+from local_deep_research.constants import FILE_PATH_TEXT_ONLY
+from local_deep_research.database.models.library import (
+ Document,
+ DocumentStatus,
+)
from local_deep_research.research_library.services import (
library_service as library_service_module,
)
@@ -213,11 +223,22 @@ def _queue_entry(queue_id=42):
return row
-def _document(document_id="doc-1", status="completed"):
- doc = Mock()
- doc.id = document_id
- doc.status = status
- return doc
+def _document(document_id="doc-1", status=DocumentStatus.COMPLETED, **columns):
+ """The resource's ``Document``, as ``get_document_for_resource`` loads it.
+
+ A real row, whose ``status`` is a ``DocumentStatus`` member. By default
+ a completed download of resource 1 with its PDF stored in the database.
+ """
+ values = {
+ "id": document_id,
+ "resource_id": 1,
+ "status": status,
+ "file_type": "pdf",
+ "storage_mode": "database",
+ "file_path": None,
+ }
+ values.update(columns)
+ return Document(**values)
def _download_service(result):
@@ -780,7 +801,7 @@ class TestDocumentDetailsPositiveControl:
class TestAlreadyDownloaded:
def test_a_completed_document_short_circuits_with_its_id(self):
response, session, service = _call_download_source(
- existing_document=_document("doc-1", status="completed")
+ existing_document=_document("doc-1")
)
assert response == {
@@ -792,7 +813,7 @@ class TestAlreadyDownloaded:
def test_the_short_circuit_does_no_work(self):
"""The whole point of the branch: no second fetch, no queue row."""
_response, session, service = _call_download_source(
- existing_document=_document("doc-1", status="completed")
+ existing_document=_document("doc-1")
)
service.download_resource.assert_not_called()
@@ -800,12 +821,18 @@ class TestAlreadyDownloaded:
assert session.queue_query.updates == []
@pytest.mark.parametrize(
- "status", ["pending", "processing", "failed", "in_progress", ""]
+ "status",
+ [
+ DocumentStatus.PENDING,
+ DocumentStatus.PROCESSING,
+ DocumentStatus.FAILED,
+ ],
+ ids=lambda status: status.value,
)
def test_a_non_completed_document_is_re_downloaded(self, status):
- """``existing.status == "completed"`` is the load-bearing half of the
- guard. A previous attempt that failed or is stuck must not be
- reported to the user as already in their library."""
+ """The completed status is the load-bearing half of the guard. A
+ previous attempt that failed or is stuck must not be reported to the
+ user as already in their library."""
response, _session, service = _call_download_source(
existing_document=_document("doc-1", status=status),
download_result=(True, "saved 1 file"),
@@ -815,6 +842,21 @@ class TestAlreadyDownloaded:
assert response == {"success": True, "message": "Download completed"}
service.download_resource.assert_called_once_with(1)
+ def test_a_document_that_awaits_its_pdf_is_downloaded_again(self):
+ """A completed download that "Sync library" left awaiting its PDF
+ does not count as downloaded: its PDF file was removed, the sync kept
+ its text, stored it text-only and reported the file missing (see
+ ``document_awaits_pdf``). The download puts its PDF back."""
+ response, _session, service = _call_download_source(
+ existing_document=_document(
+ "doc-1", storage_mode="none", file_path=FILE_PATH_TEXT_ONLY
+ ),
+ download_result=(True, None),
+ )
+
+ assert response == {"success": True, "message": "Download completed"}
+ service.download_resource.assert_called_once_with(1)
+
def test_no_existing_document_at_all_is_not_already_downloaded(self):
response, _session, _service = _call_download_source(
existing_document=None
diff --git a/tests/web/routers/test_library_queue_all_undownloaded.py b/tests/web/routers/test_library_queue_all_undownloaded.py
index 0fd16d39a8..3dc8b533c1 100644
--- a/tests/web/routers/test_library_queue_all_undownloaded.py
+++ b/tests/web/routers/test_library_queue_all_undownloaded.py
@@ -72,6 +72,11 @@ class _ResourceQuery:
def outerjoin(self, *args, **kwargs):
return self
+ def join(self, *args, **kwargs):
+ # The lookup of resources whose document awaits its PDF
+ # (``_one_resource_per_awaiting_document``) finds none here.
+ return _ResourceQuery([])
+
def filter(self, *args, **kwargs):
return self