~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(library): bound PDF text extraction by pages and total characters (#6459)
LearningCircuit112c5ec0cbdf
diff --git a/changelog.d/6459.security.md b/changelog.d/6459.security.mdnew file mode 100644index 0000000000..36fb3c9927--- /dev/null+++ b/changelog.d/6459.security.md@@ -0,0 +1 @@+Library download/ingestion PDF text extraction is now bounded by three ceilings, defined as module constants in ``utilities/pdf_extraction_limits``: pages walked (``MAX_PDF_EXTRACTION_PAGES``, 500 per attempt — the download service's PyPDF fallback runs only when pdfplumber produced no text, so one extraction there can walk up to 1,000 pages in total), characters kept (``MAX_PDF_EXTRACTED_CHARS``, 10,000,000 characters including the separators that join pages; the page that crosses the ceiling contributes the prefix that fits instead of being dropped), and CPU time between pages (``MAX_PDF_EXTRACTION_CPU_SECONDS``, 600 — one budget per whole extraction, shared by the pdfplumber attempt and its PyPDF fallback, which is not started once the budget is spent). The CPU budget counts only the extracting thread's own CPU time (``time.thread_time()``), not wall-clock time, so waiting for the GIL or for other users' extractions does not spend it; the switching overhead of GIL contention does count, and the budget is sized not to cut an honest document short under load: a dense 500-page document measured about 146 s of thread CPU alone and about 250-330 s with 4 to 16 extractions contending, leaving about 1.8x margin or more. The budget is checked between pages, so it is not a hard CPU cap: the page-tree parse, the page in progress when it runs out and, when pdfplumber found no text with budget left, the PyPDF fallback's parse all run to completion. A document longer than 500 pages therefore yields at most the text of its first 500 pages — fewer if the character ceiling, or on a much slower host the CPU budget, stops the walk first — and the stored text is not marked as partial. The download service's pdfplumber walk also closes every page it reaches, releasing that page's layout and text-map caches (about 25 MB for a dense page). Closing does not release what the PDF libraries decode and parse for a page: pdfminer and pypdf keep every decoded stream and parsed object for the life of the document, so a crafted page of 20 MB of spaces (about 20 KB compressed, no characters, about 0.17 s of CPU) added about 20 MB per page — about 10 GB at the page ceiling from a 10 MB file. All three walks (pdfplumber, its PyPDF fallback, the base helper) now release those caches once the decoded stream bytes plus an estimate of the parsed objects reach ``MAX_PDF_RETAINED_DECODED_BYTES`` (64 MiB): decoded streams go back to their encoded state and the parsed objects the walk added are dropped, to be read again from the file if a later page needs them. What a walk holds between pages therefore stays near that threshold — crafted PDFs that added 20-60 MB per page peaked about 100-250 MB above the starting memory — and honest documents, which stay far below it, are left alone. The release reaches into library internals, pinned to pdfplumber 0.11.10 / pdfminer.six 20260107 and pypdf 6.16.1; if an upgrade removes them the release is skipped, extraction carries on, and canary tests fail. The ceilings are enforced in the download service extractor and in the shared base-downloader ``extract_text_from_pdf`` that the library's downloader text branches and the search-time PDF fetcher route through. What is NOT bounded: page-tree materialisation — ``pdfplumber`` and ``pypdf`` build every page of the document before the first page is yielded, so a file with many pages costs O(all pages) time and memory before any ceiling is consulted, whatever page count it declares; and the cost of a single page — one page's ``extract_text()`` runs to completion inside the library with no check point available to the caller, and a few kilobytes of compressed content streams can take minutes or exhaust gigabytes of address space there, and what that page decodes and parses is held until the release after it. Bounding either would require a separate process. Nor is the memory held between releases bounded exactly: parsed objects are estimated per array item or dictionary entry, so it can exceed the threshold by a small multiple. The arXiv search engine's inline PDF text extraction, the web upload route's extraction service and the ``document_loaders``-based extraction used by RAG uploads and Zotero sync are not bounded by this change; a companion change (#6472) proposes the same page and character ceilings for the first two.diff --git a/src/local_deep_research/research_library/downloaders/base.py b/src/local_deep_research/research_library/downloaders/base.pyindex bf7952a5d5..347d96cef8 100644--- a/src/local_deep_research/research_library/downloaders/base.py+++ b/src/local_deep_research/research_library/downloaders/base.py@@ -5,6 +5,7 @@ Base Academic Content Downloader Abstract Classfrom abc import ABC, abstractmethodfrom typing import Optional, Dict, Any, NamedTuplefrom enum import Enum+import timeimport requestsfrom urllib.parse import urlparsefrom loguru import logger@@ -14,6 +15,12 @@ from ...web_search_engines.rate_limiting import (AdaptiveRateLimitTracker,)from ...security import SafeSession+from ...utilities.pdf_extraction_limits import (+ MAX_PDF_EXTRACTED_CHARS,+ MAX_PDF_EXTRACTION_CPU_SECONDS,+ MAX_PDF_EXTRACTION_PAGES,+)+from ...utilities.pdf_stream_release import PypdfWalkReleaserfrom ...security.log_sanitizer import scrub_errorfrom ...security.ssrf_validator import redact_url_for_log@@ -414,12 +421,37 @@ class BaseDownloader(ABC):This is part of the public API and can be used by other modules.+ What is bounded (``utilities/pdf_extraction_limits``): at most+ ``MAX_PDF_EXTRACTION_PAGES`` pages are extracted, at most+ ``MAX_PDF_EXTRACTED_CHARS`` characters are returned (separators+ included; the page that crosses the ceiling is truncated to the+ budget left), and no new page is started once the calling thread+ has spent ``MAX_PDF_EXTRACTION_CPU_SECONDS`` of its own CPU time+ (``time.thread_time()``, so GIL waits and other requests' load do+ not count) since entry. pypdf keeps every stream it decodes and+ every object it parses on the reader; a releaser+ (``utilities/pdf_stream_release``) releases them once they reach+ ``MAX_PDF_RETAINED_DECODED_BYTES``, so what the walk holds+ between pages stays near that threshold.++ What is NOT bounded: the budget is checked between pages only,+ so neither a single page's ``extract_text()`` nor pypdf's+ page-tree construction (``PdfReader`` and the first access to its+ pages, O(all pages) in time and memory for a file with many pages)+ can be interrupted, and what one page decodes and parses is held+ until the release after it.+Args:pdf_content: PDF file content as bytesReturns:Extracted text, or None if extraction failed"""+ # Started at entry so the budget covers the whole call, parse+ # included, as in the download service's extractor. Per-thread CPU+ # time, not wall time, so time spent waiting on other requests is+ # not charged (GIL-switch overhead still is; see the constant).+ deadline = time.thread_time() + MAX_PDF_EXTRACTION_CPU_SECONDStry:import io@@ -430,9 +462,48 @@ class BaseDownloader(ABC):pdf_reader = PdfReader(pdf_file)text_content = []- for page in pdf_reader.pages:+ extracted_chars = 0+ releaser = PypdfWalkReleaser(pdf_reader)+ # The CPU budget is checked BETWEEN pages only — the sole check+ # point available without running the extractor in its own+ # process — so a single slow page still runs to completion+ # inside pypdf.+ for page_number, page in enumerate(pdf_reader.pages):+ if page_number >= MAX_PDF_EXTRACTION_PAGES:+ logger.warning(+ "PDF extraction stopped at page ceiling "+ f"({MAX_PDF_EXTRACTION_PAGES}); document has "+ f"{len(pdf_reader.pages)} pages"+ )+ break+ if time.thread_time() >= deadline:+ logger.warning(+ "PDF extraction stopped at the CPU-time ceiling "+ f"({MAX_PDF_EXTRACTION_CPU_SECONDS}s) after "+ f"{page_number} pages"+ )+ break+ releaser.before_page()text = page.extract_text()+ # pypdf keeps what it decodes and parses cached on the+ # reader; release it once it reaches+ # MAX_PDF_RETAINED_DECODED_BYTES.+ releaser.after_page()if text:+ remaining = MAX_PDF_EXTRACTED_CHARS - extracted_chars+ if len(text) > remaining:+ # Keep the slice that fits rather than dropping+ # the tripping page whole.+ if remaining > 0:+ text_content.append(text[:remaining])+ extracted_chars += remaining+ logger.warning(+ "PDF extraction truncated at character ceiling "+ f"({MAX_PDF_EXTRACTED_CHARS})"+ )+ break+ # + 1 for the "\n" separator the join adds.+ extracted_chars += len(text) + 1text_content.append(text)full_text = "\n".join(text_content)diff --git a/src/local_deep_research/research_library/services/download_service.py b/src/local_deep_research/research_library/services/download_service.pyindex 9ca0be43ba..af42b01ad8 100644--- a/src/local_deep_research/research_library/services/download_service.py+++ b/src/local_deep_research/research_library/services/download_service.py@@ -84,6 +84,14 @@ from ..extraction_catalog import ()from ...constants import DEFAULT_SEARCH_TOOL+# Extraction ceilings live in ``utilities.pdf_extraction_limits`` so that+# the base downloader shares one definition without importing this module.+from ...utilities.pdf_extraction_limits import (+ MAX_PDF_EXTRACTED_CHARS,+ MAX_PDF_EXTRACTION_CPU_SECONDS,+ MAX_PDF_EXTRACTION_PAGES,+)+def _arxiv_text_provenance(source: ArxivTextSource) -> tuple[str, str]:match source:@@ -1114,32 +1122,170 @@ class DownloadService:"""Extract text from PDF content using multiple methods for best results.+ pdfplumber is tried first; PyPDF runs only when pdfplumber produced+ no text. What is bounded (``utilities/pdf_extraction_limits``): at+ most ``MAX_PDF_EXTRACTION_PAGES`` pages are extracted per attempt,+ at most ``MAX_PDF_EXTRACTED_CHARS`` characters are returned+ (separators included; the page that crosses the ceiling is+ truncated to the budget left), and ONE CPU-time budget of+ ``MAX_PDF_EXTRACTION_CPU_SECONDS`` covers the whole extraction,+ both attempts included. It is measured with+ ``time.thread_time()``, the calling thread's own CPU time, so+ waiting for the GIL or for other extractions does not spend it.+ If the budget is spent when pdfplumber finishes without text,+ the PyPDF fallback is not started. Every pdfplumber page the walk+ reaches is closed once done with, the break paths included, which+ releases its per-page caches (parsed layout objects, text map).+ Closing does not release what pdfminer decoded and parsed for the+ page -- decoded streams, parsed objects, fonts -- which it keeps+ on the document, nor does pypdf release its own; both walks hand+ each page to a releaser (``utilities/pdf_stream_release``) that+ releases those caches once they reach+ ``MAX_PDF_RETAINED_DECODED_BYTES``, so what the walk holds+ between pages stays near that threshold.++ What is NOT bounded: the budget is checked between pages only,+ so neither a single page's ``extract_text()`` nor the parser's+ page-tree construction (``pdfplumber.open``/``PdfReader`` and the+ first access to their pages, O(all pages) in time and memory for+ a file with many pages) can be interrupted, and what one page+ decodes and parses is held until the release after it.+Args:pdf_content: Raw PDF bytesReturns:Extracted text or None if extraction fails"""+ # One CPU budget for the whole extraction, shared by both attempts:+ # a per-attempt budget would let a document that exhausts the+ # pdfplumber budget buy a second full budget (and a second+ # page-tree parse) in the PyPDF fallback. thread_time() counts+ # only this thread's CPU, not wall time, so GIL waits and other+ # requests' extractions do not spend it (GIL-switch overhead+ # does; the budget is sized for that, see the constant).+ deadline = time.thread_time() + MAX_PDF_EXTRACTION_CPU_SECONDStry:# First try with pdfplumber (better for complex layouts)import io+ from ...utilities.pdf_stream_release import (+ PdfplumberWalkReleaser,+ PypdfWalkReleaser,+ )+ from ...utilities.resource_utils import safe_close++ # Extraction ceilings: the storage cap bounds file SIZE, not+ # extraction COMPLEXITY. These stop the walk after+ # MAX_PDF_EXTRACTION_PAGES pages, MAX_PDF_EXTRACTED_CHARS+ # characters or the shared CPU budget, whichever comes first.+ # They are consulted only once the parser has materialised+ # the page tree, and only between pages.with pdfplumber.open(io.BytesIO(pdf_content)) as pdf:text_parts = []- for page in pdf.pages:- page_text = page.extract_text()- if page_text:- text_parts.append(page_text)+ extracted_chars = 0+ releaser = PdfplumberWalkReleaser(pdf)+ for page_number, page in enumerate(pdf.pages):+ # pdfplumber keeps each page's parsed layout objects+ # and text map cached on the Page until close(): about+ # 25 MB per dense page, so walking to the page ceiling+ # without closing would hold gigabytes. Close every+ # page this loop reaches, on the break paths too.+ try:+ if page_number >= MAX_PDF_EXTRACTION_PAGES:+ logger.warning(+ "PDF extraction stopped at page ceiling "+ f"({MAX_PDF_EXTRACTION_PAGES}); document has "+ f"{len(pdf.pages)} pages"+ )+ break+ if time.thread_time() >= deadline:+ logger.warning(+ "PDF extraction stopped at the CPU-time ceiling "+ f"({MAX_PDF_EXTRACTION_CPU_SECONDS}s) after "+ f"{page_number} pages"+ )+ break+ releaser.before_page()+ page_text = page.extract_text()+ if page_text:+ remaining = (+ MAX_PDF_EXTRACTED_CHARS - extracted_chars+ )+ if len(page_text) > remaining:+ # Keep the slice that fits instead of dropping+ # the tripping page: a first page over the+ # ceiling must still yield the budget's worth+ # of text, not nothing.+ if remaining > 0:+ text_parts.append(page_text[:remaining])+ extracted_chars += remaining+ logger.warning(+ "PDF extraction truncated at character "+ f"ceiling ({MAX_PDF_EXTRACTED_CHARS})"+ )+ break+ # + 2 for the "\n\n" separator the join adds.+ extracted_chars += len(page_text) + 2+ text_parts.append(page_text)+ finally:+ # safe_close: a failing close() must not mask the+ # page's own exception or discard text kept so far.+ safe_close(page, "pdfplumber page")+ # close() leaves what pdfminer decoded and parsed+ # for the page cached on the document; release it+ # once it reaches MAX_PDF_RETAINED_DECODED_BYTES.+ releaser.after_page()if text_parts:return "\n\n".join(text_parts)- # Fallback to PyPDF if pdfplumber fails+ # Fallback to PyPDF if pdfplumber produced no text -- but only+ # within the same budget: do not start a second parse once the+ # CPU budget is spent.+ if time.thread_time() >= deadline:+ logger.warning(+ "PDF extraction CPU-time ceiling "+ f"({MAX_PDF_EXTRACTION_CPU_SECONDS}s) reached before any "+ "text was extracted; not starting the PyPDF fallback"+ )+ return None+reader = PdfReader(io.BytesIO(pdf_content))text_parts = []- for page in reader.pages:+ extracted_chars = 0+ releaser = PypdfWalkReleaser(reader)+ for page_number, page in enumerate(reader.pages):+ if page_number >= MAX_PDF_EXTRACTION_PAGES:+ logger.warning(+ "PDF extraction (PyPDF fallback) stopped at page "+ f"ceiling ({MAX_PDF_EXTRACTION_PAGES})"+ )+ break+ if time.thread_time() >= deadline:+ logger.warning(+ "PDF extraction (PyPDF fallback) stopped at the "+ f"CPU-time ceiling ({MAX_PDF_EXTRACTION_CPU_SECONDS}s) "+ f"after {page_number} pages"+ )+ break+ releaser.before_page()text = page.extract_text()+ # pypdf caches what it decodes and parses on the reader too.+ releaser.after_page()if text:+ remaining = MAX_PDF_EXTRACTED_CHARS - extracted_chars+ if len(text) > remaining:+ if remaining > 0:+ text_parts.append(text[:remaining])+ extracted_chars += remaining+ logger.warning(+ "PDF extraction (PyPDF fallback) truncated at "+ f"character ceiling ({MAX_PDF_EXTRACTED_CHARS})"+ )+ break+ # + 2 for the "\n\n" separator the join adds.+ extracted_chars += len(text) + 2text_parts.append(text)if text_parts:@@ -1320,6 +1466,26 @@ class DownloadService:doc.extraction_method = "pdf_extraction"doc.extraction_source = "pdfplumber"doc.extraction_quality = "medium"+ # A truncated extraction (page, character or CPU-time ceiling) is+ # only signalled by the warnings logged in+ # ``_extract_text_from_pdf``: this row records the text but not+ # that it is partial, so a 700-page book yields the text of its+ # first 500 pages and is indexed as if that were the whole book.+ # The ``Document`` model (``database/models/library.py``) has no+ # free-form metadata column to hang ``extraction_truncated`` on —+ # its JSON columns are ``authors`` and ``tags``, and+ # ``document_metadata`` belongs to ``DocumentChunk``. Persisting+ # the flag needs a schema change (a nullable ``Boolean`` beside+ # ``has_formatting_issues``/``has_encoding_issues``, or a JSON+ # ``extraction_metadata``), which is left to the owner. It would+ # be set here and wherever else ``_extract_text_from_pdf`` output+ # is saved: the three ``_save_text_with_db`` calls fed by it (in+ # ``_download_pdf``, ``_try_existing_pdf_extraction`` and+ # ``_fallback_pdf_extraction``), plus the ``_save_text_with_db``+ # calls in ``_try_arxiv_text_extraction`` and+ # ``_try_api_text_extraction`` for downloader text (arXiv PDF text+ # among it) that ``BaseDownloader.extract_text_from_pdf`` bounds+ # the same way.resource.document_id = doc.idsession.commit()diff --git a/src/local_deep_research/utilities/pdf_extraction_limits.py b/src/local_deep_research/utilities/pdf_extraction_limits.pynew file mode 100644index 0000000000..83564e4014--- /dev/null+++ b/src/local_deep_research/utilities/pdf_extraction_limits.py@@ -0,0 +1,133 @@+"""Shared ceilings for PDF text extraction.++The storage/upload caps bound a document's file SIZE, not the COST of+extracting text from it: a PDF built from millions of cheap pages, or one+whose streams decompress into huge text, can still burn unbounded CPU+and memory inside the server process while a page-walking extractor+chews through it.++At this tree these ceilings bound two extraction paths: the download+service's ``_extract_text_from_pdf``+(``research_library/services/download_service.py``, pdfplumber and its+PyPDF fallback) and the downloader base helper ``extract_text_from_pdf``+(``research_library/downloaders/base.py``). On those paths they bound+exactly four things:++- pages walked: text is extracted from at most+ ``MAX_PDF_EXTRACTION_PAGES`` (500) pages per attempt. The download+ service's PyPDF fallback runs only when pdfplumber produced no text,+ so one extraction there can walk up to twice that many pages (500+ empty pdfplumber pages, then up to 500 PyPDF pages). A longer+ document yields the text of at most its first 500 pages, and fewer+ if another ceiling stops the walk first;+- characters kept: at most ``MAX_PDF_EXTRACTED_CHARS`` (10,000,000)+ characters -- Python ``str`` characters, i.e. code points, not bytes+ -- including the separators that join pages; the page that crosses+ the ceiling contributes the prefix that fits;+- CPU time between pages: one whole extraction, every attempt of it+ included, stops starting new pages once the extracting thread has+ spent ``MAX_PDF_EXTRACTION_CPU_SECONDS`` of its OWN CPU time+ (``time.thread_time()``) since it began. Wall-clock time is+ deliberately not used: time spent waiting for the GIL or for other+ requests' extractions is not work done on this document, and counting+ it would let server load cut honest documents short;+- what the PDF library keeps between pages: pdfminer and pypdf keep+ every stream they decode and every object they parse for the life of+ the document, and closing a pdfplumber page does not release them.+ Once the decoded stream bytes plus an estimate of the parsed objects+ the walk has cached reach ``MAX_PDF_RETAINED_DECODED_BYTES`` (64 MiB),+ ``utilities/pdf_stream_release`` returns the decoded streams to their+ encoded state and drops the parsed objects (and pdfminer's fonts) the+ walk added. What the walk holds between pages therefore stays near+ that threshold instead of growing with the pages walked.++What they do NOT bound:++- page-tree materialisation. ``pdfplumber`` builds every page object,+ and ``pypdf`` flattens the whole page tree, before the first page is+ yielded -- O(all pages) time and memory for a file with many pages,+ whatever page count the file declares. None of the ceilings above is+ consulted until that has finished.+- the cost of a single page. One page's ``extract_text()`` runs to+ completion inside the library with no check point available to us; a+ few kilobytes of compressed content streams can take minutes or+ exhaust gigabytes of address space there, and what that page decodes+ and parses is held until the release after it.+- the exact size of what is held between releases. Parsed objects are+ estimated at a fixed number of bytes per array item or dictionary+ entry, and pdfminer's fonts keep their own copy of their widths, so+ the memory held can exceed the threshold by a small multiple: walks+ that held ~20 MB (decoded streams) or ~60 MB (parsed ``/Widths``+ arrays) more per page peaked ~100-250 MB above their starting RSS.+ Objects cached before the walk began (the page tree) are kept, and a+ released stream that a later page uses again is decoded again, at the+ CPU budget's expense.++Bounding the first two would need a separate process, which these constants+deliberately do not introduce. The other extraction sites in the tree+(the upload-path extraction service, the arXiv engine's inline+extraction, and the ``document_loaders``-based upload and Zotero-sync+routes) are not bounded by this module at this tree; a companion change+(#6472) proposes the same page and character ceilings for the first+two.++Text cut short by any of these ceilings is returned, and stored, like a+complete extraction; only a logged warning says it is partial.+"""++from __future__ import annotations++# 500 pages comfortably covers book-length documents; real research papers+# are two orders of magnitude below it.+MAX_PDF_EXTRACTION_PAGES = 500++# 10 MB of text is ~5 million words — far beyond any real paper, while+# capping decompression-bomb output.+MAX_PDF_EXTRACTED_CHARS = 10_000_000++# CPU budget, in seconds of the extracting thread's own CPU time+# (``time.thread_time()``), for one whole extraction: every attempt of+# it, from entry to return, shares this one budget. It is checked+# BETWEEN pages only, so it cannot interrupt a page already inside+# pdfplumber/pypdf, nor the parser's page-tree construction before the+# first page.+#+# Sized so an honest document at the page ceiling is not cut. Measured+# with pdfplumber 0.11.10 on CPython 3.14: a synthetic 500-page PDF of+# very dense text (~12,700 characters per page, about twice a dense book+# page) costs ~146 s of the thread's CPU when extracted alone, and+# ~0.5-0.66 s per page (~250-330 s for 500 pages) with 4 to 16+# extractions contending for the GIL, whose switching overhead+# thread_time() does count; an ordinary text page (~5,700 characters)+# costs ~0.14 s alone. 600 s leaves 2x margin over the dense document+# with 4 extractions contending and ~1.8x with 16. Per character that is+# ~23 us alone, ~39 us at 4-way and ~52 us at 16-way contention, so the+# character ceiling (10,000,000 characters: ~230-520 s) binds before+# this budget on honest text. Heavier contention, or a host several times+# slower, can still reach it on a dense 500-page document; the text is+# then cut the same way the page ceiling cuts it.+#+# The budget still stops a document whose individual pages are+# pathologically expensive, but it is not a hard cap on the extraction's+# CPU. Unchecked are: the page-tree parse before the first check+# (``pdfplumber.open``/``PdfReader``), the page already running when the+# budget runs out, and -- in the download service, when pdfplumber+# produced no text with budget still left -- the PyPDF fallback's+# ``PdfReader`` parse, which starts after the last check and runs to+# completion before its first page is checked.+MAX_PDF_EXTRACTION_CPU_SECONDS = 600.0++# What the PDF library may keep cached between pages, in (estimated)+# bytes. pdfminer and pypdf keep every stream they decode, and every+# object they parse, for the life of the document, and page.close() does+# not release them: a page whose content stream is 20 MB of spaces+# compresses to ~20 KB, yields no characters and costs ~0.17 s of CPU,+# so a walk to the page ceiling held ~10 GB from a ~10 MB file; a page+# whose font points at a 500,000-entry /Widths array held ~60 MB more per+# page. Once the decoded stream bytes plus an estimate of the parsed+# objects the walk has cached reach this many bytes,+# ``utilities/pdf_stream_release`` releases them (checked between pages,+# like the other ceilings). Honest documents -- content streams of tens+# to hundreds of KB per page, a few embedded fonts -- stay far below it:+# no release ran on any of the real and synthetic documents measured.+MAX_PDF_RETAINED_DECODED_BYTES = 64 * 1024 * 1024diff --git a/src/local_deep_research/utilities/pdf_stream_release.py b/src/local_deep_research/utilities/pdf_stream_release.pynew file mode 100644index 0000000000..3ffb3e0cdb--- /dev/null+++ b/src/local_deep_research/utilities/pdf_stream_release.py@@ -0,0 +1,278 @@+"""Release what the PDF libraries cache between the pages of a text walk.++Both PDF libraries keep, for the life of the document, everything they+decode or parse while extracting a page, and closing a pdfplumber page+does not release it:++- decoded streams. pdfminer (under pdfplumber) stores a stream's+ decompressed bytes on the ``PDFStream`` object (``.data``), which its+ object cache and the page objects keep reachable; pypdf stores a+ decoded copy on each ``EncodedStreamObject`` (``.decoded_self``),+ which its reader's object cache keeps reachable. A crafted page whose+ content stream is 20 MB of spaces compresses to ~20 KB, yields no+ characters and costs ~0.17 s of CPU, so a walk to the page ceiling+ held ~10 GB from a ~10 MB file.+- parsed objects. Every dictionary and array a page resolves (fonts,+ their ``/Widths``, resources) stays in the object cache, and a+ pdfminer font object keeps its own copy of the widths. A page whose+ font points at a 500,000-entry ``/Widths`` array held ~60 MB more per+ page, at ~2 s of CPU per page.++A walker creates one releaser per document and calls ``before_page()``+just before each page's ``extract_text()`` and ``after_page()`` once the+page is done. ``after_page()`` estimates what the object cache holds+since the last release -- decoded stream bytes, plus+``_BYTES_PER_PARSED_ELEMENT`` for every element of the objects the walk+has parsed -- and, once that reaches ``MAX_PDF_RETAINED_DECODED_BYTES``,+releases it: decoded streams go back to their encoded state (the next+access decodes them again from their raw bytes, so a stream a later+page shares still extracts correctly), and every object the walk added+to the cache is dropped, to be parsed again from the file if a later+page needs it. Objects cached before the walk began (the page tree)+are kept. Honest documents stay far below the threshold and are left+alone.++Each ``after_page()`` scans the document's object cache, so it costs+time in proportion to that cache: well under a second over a whole+500-page honest document, ~0.16 s per page for a cache of a million+objects, which takes a page tree of about a million pages (itself an+O(all pages) cost paid before the walk). That time is spent between+the CPU budget's checks and counts against it.++These are library internals, pinned to pdfplumber 0.11.10 /+pdfminer.six 20260107 and pypdf 6.16.1. Every access is guarded: if an+attribute is missing or anything raises, the releaser stops releasing+and extraction carries on unchanged. The canary tests in+``tests/research_library/services/test_pdf_extraction_bounds.py``+(``TestLibraryInternalsTheReleaseReliesOn``) fail if a library upgrade+removes what the releasers rely on, so an upgrade cannot undo them+silently.+"""++from __future__ import annotations++from itertools import islice+from typing import Any++from loguru import logger++from . import pdf_extraction_limits++# Estimated bytes a parsed object element (an array item, a dictionary+# entry) keeps alive. Measured on a 500,000-entry /Widths array: ~124+# bytes per entry in pypdf (a boxed number object plus its list slot),+# and in pdfminer the cached array plus the font's own widths table.+_BYTES_PER_PARSED_ELEMENT = 128+++def _limit() -> int:+ return pdf_extraction_limits.MAX_PDF_RETAINED_DECODED_BYTES+++def _count_elements(obj: object, skip: tuple[type, ...]) -> int:+ """Number of items in the lists and dicts nested in *obj*.++ Objects of the *skip* types (streams, accounted by their decoded+ bytes instead) are not descended into. Parsed PDF objects are trees,+ so the walk is linear in their size -- no more than parsing them cost.+ """+ count = 0+ stack = [obj]+ while stack:+ item = stack.pop()+ if isinstance(item, skip):+ continue+ if isinstance(item, dict):+ count += len(item)+ stack.extend(item.values())+ elif isinstance(item, (list, tuple)):+ count += len(item)+ stack.extend(item)+ return count+++class PdfplumberWalkReleaser:+ """Releases pdfminer's caches while pdfplumber walks a document."""++ def __init__(self, pdf: Any) -> None:+ self._pdf = pdf+ self._enabled = True+ self._pre_walk_keys: set | None = None+ self._page_mark: int | None = None+ self._parsed_elements = 0++ def _parts(self) -> tuple[Any, dict, dict, dict]:+ doc = self._pdf.doc+ cache = doc._cached_objs+ parsed_objstms = doc._parsed_objs+ fonts = self._pdf.rsrcmgr._cached_fonts+ if not (+ isinstance(cache, dict)+ and isinstance(parsed_objstms, dict)+ and isinstance(fonts, dict)+ ):+ raise TypeError("unexpected pdfminer cache layout")+ return doc, cache, parsed_objstms, fonts++ def before_page(self) -> None:+ """Call just before a page's ``extract_text()``. Never raises."""+ if not self._enabled:+ return+ try:+ _doc, cache, _parsed, _fonts = self._parts()+ if self._pre_walk_keys is None:+ # The page tree is materialised by now: what is cached+ # here was not added by the walk and is never dropped.+ self._pre_walk_keys = set(cache)+ self._page_mark = len(cache)+ except Exception:+ self._disable()++ def after_page(self) -> bool:+ """Call once the page is done (after ``close()``). Returns True+ when a release ran. Never raises."""+ if not self._enabled or self._page_mark is None:+ return False+ try:+ from pdfminer.pdftypes import PDFStream++ doc, cache, parsed_objstms, fonts = self._parts()+ # Objects the getobj() calls of this page appended to the cache.+ for obj, _genno in islice(cache.values(), self._page_mark, None):+ self._parsed_elements += _count_elements(obj, (PDFStream,))+ self._page_mark = None+ held = self._parsed_elements * _BYTES_PER_PARSED_ELEMENT+ for obj, _genno in list(cache.values()):+ if isinstance(obj, PDFStream) and obj.data is not None:+ held += len(obj.data)+ if held < _limit():+ return False+ # Rewound in place rather than dropped: pages hold the streams+ # cached before the walk directly (PDFPage.contents), and a+ # finished page's content parser, which pdfminer leaves in a+ # reference cycle, still holds the streams it read.+ self._rewind_streams(doc, cache, PDFStream)+ # Everything the walk added is parsed again from the file if a+ # later page needs it.+ pre_walk = self._pre_walk_keys or set()+ for objid in list(cache):+ if objid not in pre_walk:+ del cache[objid]+ # Objects parsed out of object streams are cached separately.+ parsed_objstms.clear()+ # Fonts keep their widths and embedded font program (a+ # TrueType font holds its decoded FontFile2); rebuilt on demand.+ fonts.clear()+ self._parsed_elements = 0+ return True+ except Exception:+ self._disable()+ return False++ @staticmethod+ def _rewind_streams(doc: Any, cache: dict, stream_type: type) -> None:+ """Return every decoded, cached ``PDFStream`` to its encoded state.++ pdfminer drops a stream's raw bytes once it has decoded them+ (``rawdata`` becomes ``None``), so they are read back by parsing+ the object from the file again. The cached object keeps its+ identity -- pages that hold it directly (``PDFPage.contents``)+ see the rewound object -- and only ``rawdata``/``data`` change.+ """+ for objid, entry in list(cache.items()):+ obj = entry[0]+ if not isinstance(obj, stream_type) or obj.data is None:+ continue+ if obj.rawdata is not None:+ continue # not a state decode() leaves behind+ del cache[objid]+ try:+ fresh = doc.getobj(objid)+ except Exception:+ fresh = None+ finally:+ cache[objid] = entry+ if (+ fresh is not obj+ and isinstance(fresh, stream_type)+ and fresh.rawdata is not None+ and fresh.data is None+ ):+ obj.rawdata = fresh.rawdata+ obj.data = None++ def _disable(self) -> None:+ self._enabled = False+ logger.debug("pdfminer cache release disabled for this document")+++class PypdfWalkReleaser:+ """Releases pypdf's caches while a ``PdfReader`` is walked."""++ def __init__(self, reader: Any) -> None:+ self._reader = reader+ self._enabled = True+ self._pre_walk_keys: set | None = None+ self._page_mark: int | None = None+ self._parsed_elements = 0++ def _cache(self) -> dict:+ cache = self._reader.resolved_objects+ if not isinstance(cache, dict):+ raise TypeError("unexpected pypdf cache layout")+ return cache++ def before_page(self) -> None:+ """Call just before a page's ``extract_text()``. Never raises."""+ if not self._enabled:+ return+ try:+ cache = self._cache()+ if self._pre_walk_keys is None:+ # The page tree was flattened to yield this page.+ self._pre_walk_keys = set(cache)+ self._page_mark = len(cache)+ except Exception:+ self._disable()++ def after_page(self) -> bool:+ """Call once the page is done. Returns True when a release ran.+ Never raises."""+ if not self._enabled or self._page_mark is None:+ return False+ try:+ from pypdf.generic import EncodedStreamObject++ cache = self._cache()+ for obj in islice(cache.values(), self._page_mark, None):+ self._parsed_elements += _count_elements(+ obj, (EncodedStreamObject,)+ )+ self._page_mark = None+ held = self._parsed_elements * _BYTES_PER_PARSED_ELEMENT+ decoded = []+ for obj in list(cache.values()):+ if isinstance(obj, EncodedStreamObject):+ copy = obj.decoded_self+ if copy is not None:+ held += len(copy.get_data())+ decoded.append(obj)+ if held < _limit():+ return False+ # EncodedStreamObject keeps its raw bytes and decodes them+ # again when decoded_self is None.+ for obj in decoded:+ obj.decoded_self = None+ pre_walk = self._pre_walk_keys or set()+ for key in list(cache):+ if key not in pre_walk:+ del cache[key]+ self._parsed_elements = 0+ return True+ except Exception:+ self._disable()+ return False++ def _disable(self) -> None:+ self._enabled = False+ logger.debug("pypdf cache release disabled for this document")diff --git a/tests/research_library/services/test_pdf_extraction_bounds.py b/tests/research_library/services/test_pdf_extraction_bounds.pynew file mode 100644index 0000000000..d2921b6029--- /dev/null+++ b/tests/research_library/services/test_pdf_extraction_bounds.py@@ -0,0 +1,1629 @@+"""Extraction-bound contracts for ``DownloadService._extract_text_from_pdf``.++Downloaded PDFs are external content: the storage layer caps their size+(default 3 GB), but extraction complexity is a different axis — a PDF+of millions of cheap pages, or one whose pages decompress into huge+text, burns unbounded CPU and memory inside the server process while+``pdfplumber`` walks every page. Extraction must stop at a page ceiling+and cap the total extracted characters.+"""++from __future__ import annotations++from unittest.mock import MagicMock, Mock, patch++import pytest++from local_deep_research.utilities.pdf_extraction_limits import (+ MAX_PDF_EXTRACTED_CHARS,+ MAX_PDF_EXTRACTION_CPU_SECONDS,+ MAX_PDF_EXTRACTION_PAGES,+)+++def service(mocker):+ """DownloadService with its constructor dependencies mocked."""+ mock_settings = Mock()+ mock_settings.get_setting.return_value = "/tmp/test_library"+ mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "get_settings_manager",+ return_value=mock_settings,+ )+ mocker.patch("pathlib.Path.mkdir")+ mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "RetryManager"+ )+ from local_deep_research.research_library.services.download_service import (+ DownloadService,+ )++ return DownloadService(username="test_user")+++def _pdf_with_pages(mocker, pages):+ """Patch pdfplumber.open with a PDF holding *pages* mock pages."""+ mock_pdf = MagicMock()+ mock_pdf.pages = pages+ mock_pdf.__enter__ = Mock(return_value=mock_pdf)+ mock_pdf.__exit__ = Mock(return_value=False)+ mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "pdfplumber.open",+ return_value=mock_pdf,+ )+ return mock_pdf+++class TestPageCeiling:+ def test_pathological_page_count_is_not_fully_walked(self, service, mocker):+ """A 600-page document must not have every page extracted."""+ pages = []+ for _ in range(600):+ page = MagicMock()+ page.extract_text.return_value = "page text"+ pages.append(page)+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ # Extraction still produced text ...+ assert text+ # ... and stopped exactly AT the page ceiling, not merely+ # "somewhere before 600": a loop that broke early (at page 1, or+ # one page short) would also satisfy `walked < len(pages)`, and+ # would silently cut honest documents shorter than the ceiling.+ walked = sum(1 for page in pages if page.extract_text.called)+ assert walked == MAX_PDF_EXTRACTION_PAGES, (+ f"extraction walked {walked} of {len(pages)} pages — expected "+ f"exactly the page ceiling ({MAX_PDF_EXTRACTION_PAGES})"+ )+++class TestCharacterCeiling:+ def test_decompression_sized_text_is_capped(self, service, mocker):+ """20 MB of extracted text must not be returned in full."""+ per_page = "x" * 200_000+ pages = []+ for _ in range(100):+ page = MagicMock()+ page.extract_text.return_value = per_page+ pages.append(page)+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ total = len(per_page) * len(pages)+ assert text+ assert len(text) < total, (+ f"extraction returned all {len(text)} chars ({total} expected "+ "total) — no extracted-character ceiling is in force"+ )+++class TestPyPDFFallbackPageCeiling:+ def test_fallback_page_count_is_not_fully_walked(self, service, mocker):+ """The PyPDF fallback loop must honor the page ceiling too.++ pdfplumber yields no text, so extraction falls through to the+ ``PdfReader`` path; a 600-page mock document must not have every+ page extracted there.+ """+ empty_pages = [MagicMock()]+ empty_pages[0].extract_text.return_value = ""+ _pdf_with_pages(mocker, empty_pages)++ pypdf_pages = []+ for _ in range(600):+ page = MagicMock()+ page.extract_text.return_value = "page text"+ pypdf_pages.append(page)+ mock_reader = MagicMock()+ mock_reader.pages = pypdf_pages+ mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "PdfReader",+ return_value=mock_reader,+ )++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text+ walked = sum(1 for page in pypdf_pages if page.extract_text.called)+ assert walked == MAX_PDF_EXTRACTION_PAGES, (+ f"fallback extraction walked {walked} of {len(pypdf_pages)} "+ f"pages — expected exactly the page ceiling "+ f"({MAX_PDF_EXTRACTION_PAGES}) on the PyPDF fallback"+ )+++class TestPyPDFFallbackCharacterCeiling:+ def test_fallback_decompression_sized_text_is_capped(self, service, mocker):+ """The PyPDF fallback loop must honor the character ceiling too.++ pdfplumber yields no text, so extraction falls through to the+ ``PdfReader`` path; three pages that decompress to ~4 M chars+ each trip the 10 M character ceiling mid-document, so the+ trailing short page must never be walked at all.+ """+ empty_pages = [MagicMock()]+ empty_pages[0].extract_text.return_value = ""+ _pdf_with_pages(mocker, empty_pages)++ per_page = "x" * 4_000_000+ pypdf_pages = []+ for _ in range(3):+ page = MagicMock()+ page.extract_text.return_value = per_page+ pypdf_pages.append(page)+ trailing_page = MagicMock()+ trailing_page.extract_text.return_value = "trailing short page"+ pypdf_pages.append(trailing_page)+ mock_reader = MagicMock()+ mock_reader.pages = pypdf_pages+ mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "PdfReader",+ return_value=mock_reader,+ )++ text = service._extract_text_from_pdf(b"%PDF-fake")++ total = len(per_page) * 3 + len("trailing short page")+ assert text+ assert len(text) < total, (+ f"fallback extraction returned all {len(text)} chars "+ f"({total} expected total) — no extracted-character ceiling "+ "is in force on the PyPDF fallback"+ )+ walked = sum(1 for page in pypdf_pages if page.extract_text.called)+ assert 0 < walked < len(pypdf_pages), (+ f"fallback extraction walked {walked} of {len(pypdf_pages)} "+ "pages — no extracted-character ceiling is in force on the "+ "PyPDF fallback"+ )+ assert not trailing_page.extract_text.called, (+ "the trailing short page was walked despite the character "+ "ceiling tripping earlier"+ )+++class TestBaseDownloaderBounds:+ """The shared base-downloader choke point must enforce both ceilings.++ Every downloader text branch and the search-time PDF fetcher route+ through ``BaseDownloader.extract_text_from_pdf``, so its loop is a+ bound on its own — not merely a mirror of the download service's.+ """++ def test_page_count_is_not_fully_walked(self, mocker):+ """A 600-page document must not have every page extracted."""+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ pages = []+ for _ in range(600):+ page = MagicMock()+ page.extract_text.return_value = "page text"+ pages.append(page)+ mock_reader = MagicMock()+ mock_reader.pages = pages++ # Patch pypdf.PdfReader since it's imported inside the function+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert text+ walked = sum(1 for page in pages if page.extract_text.called)+ assert walked == MAX_PDF_EXTRACTION_PAGES, (+ f"base-downloader extraction walked {walked} of {len(pages)} "+ f"pages — expected exactly the page ceiling "+ f"({MAX_PDF_EXTRACTION_PAGES}) on the base downloader"+ )++ def test_decompression_sized_text_is_capped(self, mocker):+ """Pages that decompress to ~4 M chars each must be capped."""+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ per_page = "x" * 4_000_000+ pages = []+ for _ in range(3):+ page = MagicMock()+ page.extract_text.return_value = per_page+ pages.append(page)+ trailing_page = MagicMock()+ trailing_page.extract_text.return_value = "trailing short page"+ pages.append(trailing_page)+ mock_reader = MagicMock()+ mock_reader.pages = pages++ # Patch pypdf.PdfReader since it's imported inside the function+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ total = len(per_page) * 3 + len("trailing short page")+ assert text+ assert len(text) < total, (+ f"base-downloader extraction returned all {len(text)} chars "+ f"({total} expected total) — no extracted-character ceiling "+ "is in force on the base downloader"+ )+ walked = sum(1 for page in pages if page.extract_text.called)+ assert 0 < walked < len(pages), (+ f"base-downloader extraction walked {walked} of {len(pages)} "+ "pages — no extracted-character ceiling is in force on the "+ "base downloader"+ )+ assert not trailing_page.extract_text.called, (+ "the trailing short page was walked despite the character "+ "ceiling tripping earlier"+ )+++def _pages_of(texts):+ """Mock pages yielding *texts*, one string per page."""+ pages = []+ for text in texts:+ page = MagicMock()+ page.extract_text.return_value = text+ pages.append(page)+ return pages+++class TestTruncationRatherThanDropping:+ """The page that crosses the character ceiling must be truncated.++ Dropping it whole loses up to a full page of text that fitted the+ budget, and — when the very first page is over the ceiling — turns a+ bounded extraction into a reported extraction FAILURE, which the+ download service records as an error and which sends+ ``_extract_text_from_pdf`` on to the PyPDF branch to repeat the whole+ unbounded parse a second time.+ """++ def test_tripping_page_is_sliced_to_the_remaining_budget(+ self, service, mocker+ ):+ pages = _pages_of(+ [+ "a" * (MAX_PDF_EXTRACTED_CHARS - 10),+ "b" * 1_000,+ "c" * 10,+ ]+ )+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ # The "\n\n" separator counts against the budget: 10 left after+ # the first page is 2 for the separator and 8 for the tripping+ # page's prefix.+ # Compared by length and tail: a failing ``==`` on 10 M-character+ # strings would make pytest diff them.+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned — the page that crossed the "+ "ceiling was not sliced to the remaining budget"+ )+ assert text[-12:] == "aa\n\n" + "b" * 8, (+ "the tripping page did not contribute its prefix"+ )+ assert not pages[2].extract_text.called++ def test_single_over_ceiling_page_yields_the_budget_not_none(+ self, service, mocker+ ):+ pages = _pages_of(["x" * (MAX_PDF_EXTRACTED_CHARS + 1)])+ _pdf_with_pages(mocker, pages)+ pypdf = mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "PdfReader"+ )++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None, (+ "a first page over the character ceiling returned None — "+ "downstream records this as an extraction failure"+ )+ assert len(text) == MAX_PDF_EXTRACTED_CHARS+ assert not pypdf.called, (+ "extraction fell through to the PyPDF branch and repeated the "+ "whole parse a second time"+ )++ def test_exactly_the_ceiling_is_kept_in_full(self, service, mocker):+ pages = _pages_of(["x" * MAX_PDF_EXTRACTED_CHARS])+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS++ def test_one_over_the_ceiling_is_truncated_to_the_ceiling(+ self, service, mocker+ ):+ pages = _pages_of(["x" * (MAX_PDF_EXTRACTED_CHARS + 1)])+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS++ def test_base_downloader_truncates_rather_than_drops(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ pages = _pages_of(["x" * (MAX_PDF_EXTRACTED_CHARS + 1)])+ mock_reader = MagicMock()+ mock_reader.pages = pages+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert text is not None, (+ "a first page over the character ceiling returned None from "+ "the base-downloader choke point"+ )+ assert len(text) == MAX_PDF_EXTRACTED_CHARS+++class TestCpuTimeBudget:+ """Extraction stops when the CPU-time budget is spent between pages.++ Page and character ceilings do not bind on a document whose pages are+ individually slow but few and terse; the deadline does. It can only+ be checked BETWEEN pages, so a single page already inside+ pdfplumber/pypdf still runs to completion.+ """++ def test_deadline_stops_the_walk_and_keeps_what_was_extracted(+ self, service, mocker+ ):+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])++ def slow_first_page():+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return "first page text"++ pages = _pages_of([None, "second page text"])+ pages[0].extract_text.side_effect = slow_first_page+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text == "first page text", (+ "the deadline either never fired or discarded the text "+ "already extracted"+ )+ assert not pages[1].extract_text.called, (+ "the walk continued past the CPU-time budget"+ )++ def test_base_downloader_honours_the_deadline(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])++ def slow_first_page():+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return "first page text"++ pages = _pages_of([None, "second page text"])+ pages[0].extract_text.side_effect = slow_first_page+ mock_reader = MagicMock()+ mock_reader.pages = pages+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert text == "first page text"+ assert not pages[1].extract_text.called, (+ "the base-downloader walk continued past the CPU-time budget"+ )++ def test_base_downloader_deadline_covers_pdfreader_construction(+ self, mocker+ ):+ """A slow ``PdfReader()`` parse must also count against the+ deadline — not just a slow page inside the loop."""+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])++ pages = _pages_of(["first page text", "second page text"])+ mock_reader = MagicMock()+ mock_reader.pages = pages++ def slow_construction(_file):+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return mock_reader++ mocker.patch("pypdf.PdfReader", side_effect=slow_construction)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert not any(page.extract_text.called for page in pages), (+ "a page was walked even though PdfReader() construction alone "+ "spent the whole deadline"+ )+ assert text is None, (+ "the deadline did not cover PdfReader() construction — a slow "+ "parse still returned extracted text"+ )++ def test_download_service_deadline_covers_pdfplumber_open(+ self, service, mocker+ ):+ """A slow ``pdfplumber.open()`` parse must also count against the+ shared deadline — not just a slow page inside the loop."""+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])++ pages = _pages_of(["first page text", "second page text"])+ mock_pdf = MagicMock()+ mock_pdf.pages = pages+ mock_pdf.__enter__ = Mock(return_value=mock_pdf)+ mock_pdf.__exit__ = Mock(return_value=False)++ def slow_open(_file):+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return mock_pdf++ mocker.patch(+ "local_deep_research.research_library.services."+ "download_service.pdfplumber.open",+ side_effect=slow_open,+ )+ reader = _reader_with_pages(mocker, _pages_of(["fallback text"]))++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert not any(page.extract_text.called for page in pages), (+ "a page was walked even though pdfplumber.open() construction "+ "alone spent the whole deadline"+ )+ assert not reader.called, (+ "pdfplumber.open() construction alone spent the deadline, yet "+ "the PyPDF fallback was started"+ )+ assert text is None, (+ "the deadline did not cover pdfplumber.open() construction — "+ "a slow parse still returned extracted text"+ )+++def _reader_with_pages(mocker, pages):+ """Patch the download service's PdfReader to yield *pages*."""+ mock_reader = MagicMock()+ mock_reader.pages = pages+ return mocker.patch(+ "local_deep_research.research_library.services.download_service."+ "PdfReader",+ return_value=mock_reader,+ )+++class TestOneDeadlinePerExtraction:+ """``_extract_text_from_pdf`` has ONE budget for both attempts.++ A per-attempt deadline let a document that exhausted the pdfplumber+ budget without text buy a second full budget — and a second full+ page-tree parse — in the PyPDF fallback.+ """++ def test_pdfplumber_deadline_stop_does_not_start_the_fallback(+ self, service, mocker+ ):+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])++ def slow_empty_page():+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return ""++ pages = _pages_of([None, "second page text"])+ pages[0].extract_text.side_effect = slow_empty_page+ _pdf_with_pages(mocker, pages)+ reader = _reader_with_pages(mocker, _pages_of(["fallback text"]))++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is None+ assert not pages[1].extract_text.called+ assert not reader.called, (+ "pdfplumber stopped on the deadline, yet the PyPDF fallback "+ "was started — a second parse on a spent budget"+ )++ def test_fallback_honours_the_deadline(self, service, mocker):+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])+ _pdf_with_pages(mocker, _pages_of([""]))++ def slow_first_page():+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return "first fallback page"++ fallback_pages = _pages_of([None, "second fallback page"])+ fallback_pages[0].extract_text.side_effect = slow_first_page+ _reader_with_pages(mocker, fallback_pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text == "first fallback page"+ assert not fallback_pages[1].extract_text.called, (+ "the PyPDF fallback walk continued past the CPU-time budget"+ )++ def test_fallback_shares_the_budget_pdfplumber_spent(self, service, mocker):+ """Each attempt alone stays under the budget; together they don't."""+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])+ step = MAX_PDF_EXTRACTION_CPU_SECONDS * 0.6++ def slow(result):+ def page_text():+ clock["now"] += step+ return result++ return page_text++ plumber_pages = _pages_of([None])+ plumber_pages[0].extract_text.side_effect = slow("")+ _pdf_with_pages(mocker, plumber_pages)+ fallback_pages = _pages_of([None, "second fallback page"])+ fallback_pages[0].extract_text.side_effect = slow("first fallback page")+ _reader_with_pages(mocker, fallback_pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text == "first fallback page"+ assert not fallback_pages[1].extract_text.called, (+ "the PyPDF fallback started a fresh budget instead of sharing "+ "the one pdfplumber had already spent"+ )+++class TestPyPDFFallbackTruncation:+ def test_fallback_slices_the_tripping_page(self, service, mocker):+ _pdf_with_pages(mocker, _pages_of([""]))+ fallback_pages = _pages_of(+ [+ "a" * (MAX_PDF_EXTRACTED_CHARS - 10),+ "b" * 1_000,+ "c" * 10,+ ]+ )+ _reader_with_pages(mocker, fallback_pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned — the PyPDF fallback did not "+ "slice the page that crossed the character ceiling to the "+ "remaining budget"+ )+ assert text[-12:] == "aa\n\n" + "b" * 8, (+ "the PyPDF fallback's tripping page did not contribute its prefix"+ )+ assert not fallback_pages[2].extract_text.called++ def test_fallback_single_over_ceiling_page_yields_the_budget(+ self, service, mocker+ ):+ _pdf_with_pages(mocker, _pages_of([""]))+ _reader_with_pages(+ mocker, _pages_of(["x" * (MAX_PDF_EXTRACTED_CHARS + 1)])+ )++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None, (+ "a first fallback page over the character ceiling returned None"+ )+ assert len(text) == MAX_PDF_EXTRACTED_CHARS+++class TestSeparatorsCountAgainstTheCeiling:+ """The returned string, join separators included, fits the ceiling."""++ def test_download_service_output_fits_the_ceiling(self, service, mocker):+ half = MAX_PDF_EXTRACTED_CHARS // 2+ pages = _pages_of(["a" * half, "b" * half, "c" * 10])+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned for a "+ f"{MAX_PDF_EXTRACTED_CHARS}-character ceiling"+ )++ def test_base_downloader_output_fits_the_ceiling(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ half = MAX_PDF_EXTRACTED_CHARS // 2+ mock_reader = MagicMock()+ mock_reader.pages = _pages_of(["a" * half, "b" * half, "c" * 10])+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned for a "+ f"{MAX_PDF_EXTRACTED_CHARS}-character ceiling"+ )+++class TestExactFitThenOneMorePage:+ """A page that fills the budget exactly, then one more page.++ After an exact-fit page the running count includes that page's join+ separator, so the budget left for the next page is NEGATIVE (-2 in the+ download service, -1 in the base helper). Slicing with a negative+ ``remaining`` would keep all but the last one or two characters of+ the next page, overshooting the ceiling; the ``remaining > 0`` guard+ must append nothing instead.+ """++ def test_download_service_pdfplumber_loop(self, service, mocker):+ pages = _pages_of(["x" * MAX_PDF_EXTRACTED_CHARS, "y" * 100])+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned — the page after an exact "+ "fit was sliced with a negative remaining budget"+ )+ assert "y" not in text[-200:]++ def test_download_service_pypdf_fallback_loop(self, service, mocker):+ _pdf_with_pages(mocker, _pages_of([""]))+ _reader_with_pages(+ mocker, _pages_of(["x" * MAX_PDF_EXTRACTED_CHARS, "y" * 100])+ )++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned — the PyPDF fallback sliced "+ "the page after an exact fit with a negative remaining budget"+ )+ assert "y" not in text[-200:]++ def test_base_downloader_loop(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ mock_reader = MagicMock()+ mock_reader.pages = _pages_of(+ ["x" * MAX_PDF_EXTRACTED_CHARS, "y" * 100]+ )+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert text is not None+ assert len(text) == MAX_PDF_EXTRACTED_CHARS, (+ f"{len(text)} characters returned — the base downloader sliced "+ "the page after an exact fit with a negative remaining budget"+ )+ assert "y" not in text[-200:]+++def _wall_clock_racing_ahead(mocker):+ """Make every wall clock jump far past the CPU budget on each read,+ while the thread's own CPU time stands still.++ This is what a heavily loaded server looks like to one extraction:+ wall time passes (GIL waits, other requests' extractions) while this+ thread does no work of its own.+ """+ wall = {"now": 0.0}++ def racing():+ wall["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS * 10+ return wall["now"]++ for name in ("monotonic", "perf_counter"):+ mocker.patch(f"time.{name}", side_effect=racing)+ mocker.patch("time.thread_time", return_value=0.0)+++class TestServerLoadDoesNotCutHonestDocuments:+ """Wall-clock time spent waiting is not extraction work.++ A 60 s wall-clock deadline cut an honest 480-page document at page 416+ on an idle host, and at roughly page 65 with four extractions running+ at once; the partial text was saved as the complete document and+ never re-extracted. The budget must count only this thread's CPU.+ """++ def test_download_service_walks_every_page_under_load(+ self, service, mocker+ ):+ pages = _pages_of([f"page {i}" for i in range(20)])+ _pdf_with_pages(mocker, pages)+ _wall_clock_racing_ahead(mocker)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert all(page.extract_text.called for page in pages), (+ "extraction stopped early because WALL-clock time passed — "+ "server load, not this document's cost, cut it short"+ )+ assert text == "\n\n".join(f"page {i}" for i in range(20))++ def test_download_service_fallback_walks_every_page_under_load(+ self, service, mocker+ ):+ _pdf_with_pages(mocker, _pages_of([""]))+ fallback_pages = _pages_of([f"page {i}" for i in range(20)])+ reader = _reader_with_pages(mocker, fallback_pages)+ _wall_clock_racing_ahead(mocker)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert reader.called, (+ "the PyPDF fallback was skipped because WALL-clock time passed"+ )+ assert all(page.extract_text.called for page in fallback_pages)+ assert text == "\n\n".join(f"page {i}" for i in range(20))++ def test_base_downloader_walks_every_page_under_load(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ pages = _pages_of([f"page {i}" for i in range(20)])+ mock_reader = MagicMock()+ mock_reader.pages = pages+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)+ _wall_clock_racing_ahead(mocker)++ text = BaseDownloader.extract_text_from_pdf(b"%PDF-fake")++ assert all(page.extract_text.called for page in pages), (+ "the base downloader stopped early because WALL-clock time passed"+ )+ assert text == "\n".join(f"page {i}" for i in range(20))++ @pytest.mark.parametrize(+ ("contending_extractions", "dense_page_cpu_seconds", "margin"),+ [(4, 0.5, 2.0), (16, 0.66, 1.8)],+ )+ def test_budget_leaves_the_documented_margin_under_contention(+ self, contending_extractions, dense_page_cpu_seconds, margin+ ):+ """The budget must leave the margin the constant's comment claims+ over an honest document at the page ceiling.++ Measured with pdfplumber 0.11.10 on CPython 3.14: a synthetic+ page of ~12,700 characters of dense text costs ~0.29 s of the+ thread's CPU alone, ~0.5 s with four extractions contending for+ the GIL and ~0.66 s with sixteen. The comment claims 2x margin at+ four and ~1.8x at sixteen; a smaller budget would cut honest+ dense books short on a busy server.+ """+ assert MAX_PDF_EXTRACTION_CPU_SECONDS >= (+ margin * MAX_PDF_EXTRACTION_PAGES * dense_page_cpu_seconds+ ), (+ f"a {MAX_PDF_EXTRACTION_CPU_SECONDS}s budget does not fit a "+ f"dense {MAX_PDF_EXTRACTION_PAGES}-page document with "+ f"{margin}x margin at {contending_extractions} contending "+ "extractions"+ )++ def test_budget_is_still_a_bound(self):+ """The budget must stay small enough to stop pathological pages.++ A budget raised far past the honest-document sizing (say 1e9 s)+ would pass the lower bound above while bounding nothing; pin it+ to at most 15 minutes of CPU per extraction.+ """+ assert 0 < MAX_PDF_EXTRACTION_CPU_SECONDS <= 900, (+ f"a {MAX_PDF_EXTRACTION_CPU_SECONDS}s budget no longer bounds "+ "an extraction whose pages are pathologically expensive"+ )+++def _recording_pages(texts, events):+ """Mock pages that log ``("extract", i)``/``("close", i)`` to *events*."""+ pages = _pages_of(texts)+ for index, page in enumerate(pages):+ text = page.extract_text.return_value++ def extract(index=index, text=text):+ events.append(("extract", index))+ return text++ page.extract_text.side_effect = extract+ page.close.side_effect = lambda index=index: events.append(+ ("close", index)+ )+ return pages+++class TestPdfplumberPagesAreClosed:+ """Every pdfplumber page the walk reaches is closed before the next.++ pdfplumber caches each page's parsed layout objects and text map on+ the ``Page`` until ``close()``; ``PDF.close()`` on leaving the+ ``with`` block releases them only after the whole walk. Measured on a+ dense PDF, the walk retained ~25 MB per page without a per-page+ close (1 GB at 40 pages, ~12 GB at the page ceiling) and stayed flat+ with it. Closing frees only these caches; what the libraries decode+ and parse is released by the walk releasers (see+ ``TestDecodedStreamsAreReleased``). Closing all pages after the loop would not help, so the+ order is asserted, not just the count.+ """++ @staticmethod+ def _assert_each_extracted_page_closed_before_the_next(pages, events):+ extracted = [i for kind, i in events if kind == "extract"]+ assert extracted, "no page was extracted"+ for i in extracted:+ assert ("close", i) in events, (+ f"page {i} was extracted but never closed — its layout "+ "cache stays alive for the rest of the walk"+ )+ if ("extract", i + 1) in events:+ assert events.index(("close", i)) < events.index(+ ("extract", i + 1)+ ), f"page {i} was not closed before page {i + 1} was walked"++ def test_full_walk_closes_every_page(self, service, mocker):+ events = []+ pages = _recording_pages([f"page {i}" for i in range(5)], events)+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text == "\n\n".join(f"page {i}" for i in range(5))+ assert all(page.close.called for page in pages)+ self._assert_each_extracted_page_closed_before_the_next(pages, events)++ def test_page_ceiling_break_closes_every_walked_page(self, service, mocker):+ events = []+ pages = _recording_pages(+ ["page text"] * (MAX_PDF_EXTRACTION_PAGES + 5), events+ )+ _pdf_with_pages(mocker, pages)++ assert service._extract_text_from_pdf(b"%PDF-fake")++ assert all(+ page.close.called for page in pages[:MAX_PDF_EXTRACTION_PAGES]+ ), "a page walked before the page ceiling was left open"+ self._assert_each_extracted_page_closed_before_the_next(pages, events)++ def test_character_ceiling_break_closes_the_tripping_page(+ self, service, mocker+ ):+ events = []+ pages = _recording_pages(+ ["x" * 10, "y" * MAX_PDF_EXTRACTED_CHARS, "never reached"], events+ )+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert len(text) == MAX_PDF_EXTRACTED_CHARS+ assert pages[1].close.called, (+ "the page that tripped the character ceiling was left open"+ )+ assert not pages[2].extract_text.called+ self._assert_each_extracted_page_closed_before_the_next(pages, events)++ def test_cpu_budget_break_closes_every_walked_page(self, service, mocker):+ clock = {"now": 0.0}+ mocker.patch("time.thread_time", side_effect=lambda: clock["now"])+ events = []+ pages = _recording_pages(["first", "second", "third"], events)++ def slow_second_page():+ events.append(("extract", 1))+ clock["now"] += MAX_PDF_EXTRACTION_CPU_SECONDS + 1.0+ return "second"++ pages[1].extract_text.side_effect = slow_second_page+ _pdf_with_pages(mocker, pages)++ text = service._extract_text_from_pdf(b"%PDF-fake")++ assert text == "first\n\nsecond"+ assert not pages[2].extract_text.called+ assert pages[0].close.called and pages[1].close.called, (+ "a page walked before the CPU budget ran out was left open"+ )+ self._assert_each_extracted_page_closed_before_the_next(pages, events)++ def test_failing_page_is_closed(self, service, mocker):+ events = []+ pages = _recording_pages(["first", "second"], events)+ pages[1].extract_text.side_effect = ValueError("broken content stream")+ _pdf_with_pages(mocker, pages)++ assert service._extract_text_from_pdf(b"%PDF-fake") is None++ assert pages[0].close.called and pages[1].close.called, (+ "a page whose extraction raised was left open"+ )++ def test_failing_close_does_not_discard_the_text(self, service, mocker):+ pages = _pages_of(["first", "second"])+ pages[0].close.side_effect = RuntimeError("cache flush failed")+ _pdf_with_pages(mocker, pages)++ assert service._extract_text_from_pdf(b"%PDF-fake") == (+ "first\n\nsecond"+ ), "a close() failure aborted the extraction"+ assert pages[1].close.called+++# --- What the PDF libraries cache between pages ---------------------------+#+# The tests below use the real pdfplumber/pdfminer and pypdf on PDFs built+# here with the standard library, and inspect the libraries' caches+# directly: RSS is too noisy to assert on.++_SERVICE_MODULE = (+ "local_deep_research.research_library.services.download_service"+)+_RETAINED_LIMIT = (+ "local_deep_research.utilities.pdf_extraction_limits."+ "MAX_PDF_RETAINED_DECODED_BYTES"+)+_PAD = 2 * 1024 * 1024 # decoded bytes of whitespace per content stream+++def _crafted_pdf(pages, pad=0, *, late_refs=0, widths=0):+ """A PDF whose page ``i`` shows ``Page i``.++ Each page has its own FlateDecode content stream: the text operator+ plus *pad* spaces, which compress to almost nothing and yield no+ characters. ``late_refs`` appends that many extra pages that reuse+ the content streams of the first pages. ``widths`` gives every page+ its own non-standard font whose ``/Widths`` is a separate array+ object of that many entries.+ """+ import zlib++ bodies = []++ def add(body):+ bodies.append(body)+ return len(bodies)++ catalog = add(None)+ root = add(None)+ helvetica = add(b"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica >>")++ def page(font, stream):+ return add(+ b"<< /Type /Page /Parent %d 0 R /MediaBox [0 0 612 792] "+ b"/Resources << /Font << /F1 %d 0 R >> >> /Contents %d 0 R >>"+ % (root, font, stream)+ )++ kids, streams = [], []+ for i in range(pages):+ font = helvetica+ if widths:+ array = add(b"[" + b"500 " * widths + b"]")+ font = add(+ b"<< /Type /Font /Subtype /Type1 /BaseFont /Crafted%d "+ b"/FirstChar 0 /LastChar 255 /Widths %d 0 R >>" % (i, array)+ )+ data = zlib.compress(+ b"BT /F1 12 Tf 72 720 Td (Page %d) Tj ET\n" % i + b" " * pad+ )+ streams.append(+ add(+ b"<< /Length %d /Filter /FlateDecode >>\nstream\n" % len(data)+ + data+ + b"\nendstream"+ )+ )+ kids.append(page(font, streams[-1]))+ for i in range(late_refs):+ kids.append(page(helvetica, streams[i]))+ bodies[catalog - 1] = b"<< /Type /Catalog /Pages %d 0 R >>" % root+ bodies[root - 1] = b"<< /Type /Pages /Count %d /Kids [%s] >>" % (+ len(kids),+ b" ".join(b"%d 0 R" % kid for kid in kids),+ )+ out = bytearray(b"%PDF-1.4\n")+ offsets = []+ for number, body in enumerate(bodies, start=1):+ offsets.append(len(out))+ out += b"%d 0 obj\n" % number + body + b"\nendobj\n"+ xref = len(out)+ out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(bodies) + 1)+ out += b"".join(b"%010d 00000 n \n" % offset for offset in offsets)+ out += b"trailer\n<< /Size %d /Root %d 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (+ len(bodies) + 1,+ catalog,+ xref,+ )+ return bytes(out)+++def _page_texts(count, sep):+ return sep.join(f"Page {i}" for i in range(count))+++def _capture(mocker, target, real):+ """Patch *target* with a pass-through to *real* that records results."""+ made = []++ def passthrough(*args, **kwargs):+ made.append(real(*args, **kwargs))+ return made[-1]++ mocker.patch(target, side_effect=passthrough)+ return made+++def _pdfminer_decoded(pdf):+ """Decoded bytes pdfminer still holds: cached streams and the+ streams pages hold directly."""+ from pdfminer.pdftypes import PDFStream++ streams = [obj for obj, _ in pdf.doc._cached_objs.values()]+ for page in pdf.pages:+ streams.extend(page.page_obj.contents)+ return sum(+ len(obj.data)+ for obj in streams+ if isinstance(obj, PDFStream) and obj.data is not None+ )+++def _pypdf_decoded(reader):+ from pypdf.generic import EncodedStreamObject++ return sum(+ len(obj.decoded_self.get_data())+ for obj in reader.resolved_objects.values()+ if isinstance(obj, EncodedStreamObject) and obj.decoded_self is not None+ )+++class TestDecodedStreamsAreReleased:+ """A walk must not keep every page's decoded streams until it returns.++ ``page.close()`` frees only pdfplumber's per-page caches. pdfminer+ keeps each decoded stream on its ``PDFStream`` (``.data``) and pypdf+ on its ``EncodedStreamObject`` (``.decoded_self``), both reachable+ from the document's object cache. Measured before the release: a+ page of 20 MB of spaces (~20 KB compressed, no characters) added+ ~20 MB per page -- ~10 GB at the page ceiling from a ~10 MB file.+ """++ def test_pdfplumber_walk_releases_decoded_streams(self, service, mocker):+ import pdfplumber++ mocker.patch(_RETAINED_LIMIT, 1024 * 1024)+ opened = _capture(+ mocker, f"{_SERVICE_MODULE}.pdfplumber.open", pdfplumber.open+ )++ text = service._extract_text_from_pdf(_crafted_pdf(3, _PAD))++ assert text == _page_texts(3, "\n\n")+ assert _pdfminer_decoded(opened[0]) == 0, (+ "pdfminer still holds decoded content streams after the walk"+ )++ def test_decoded_streams_below_the_threshold_are_left_alone(+ self, service, mocker+ ):+ """Control: without a release the streams stay -- so the test+ above can see retention -- and honest documents, far below the+ threshold, are not touched."""+ import pdfplumber++ opened = _capture(+ mocker, f"{_SERVICE_MODULE}.pdfplumber.open", pdfplumber.open+ )++ text = service._extract_text_from_pdf(_crafted_pdf(3, _PAD))++ assert text == _page_texts(3, "\n\n")+ assert _pdfminer_decoded(opened[0]) >= 3 * _PAD++ def test_streams_a_later_page_reuses_still_extract(self, service, mocker):+ """Rewound streams decode again when a later page reads them:+ pages 3-5 reuse the content streams of pages 0-2, and a release+ runs after every page."""+ import pdfplumber++ mocker.patch(_RETAINED_LIMIT, 1)+ opened = _capture(+ mocker, f"{_SERVICE_MODULE}.pdfplumber.open", pdfplumber.open+ )++ text = service._extract_text_from_pdf(+ _crafted_pdf(3, _PAD, late_refs=3)+ )++ assert text == "\n\n".join([_page_texts(3, "\n\n")] * 2)+ assert _pdfminer_decoded(opened[0]) == 0++ def test_pypdf_fallback_releases_decoded_streams(self, service, mocker):+ from pypdf import PdfReader++ mocker.patch(_RETAINED_LIMIT, 1024 * 1024)+ _pdf_with_pages(mocker, _pages_of([""]))+ readers = _capture(mocker, f"{_SERVICE_MODULE}.PdfReader", PdfReader)++ text = service._extract_text_from_pdf(+ _crafted_pdf(3, _PAD, late_refs=3)+ )++ assert text == "\n\n".join([_page_texts(3, "\n\n")] * 2)+ assert _pypdf_decoded(readers[0]) == 0, (+ "pypdf still holds decoded content streams after the fallback"+ )++ def test_base_downloader_releases_decoded_streams(self, mocker):+ from pypdf import PdfReader++ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ mocker.patch(_RETAINED_LIMIT, 1024 * 1024)+ readers = _capture(mocker, "pypdf.PdfReader", PdfReader)++ text = BaseDownloader.extract_text_from_pdf(_crafted_pdf(3, _PAD))++ assert text == _page_texts(3, "\n")+ assert _pypdf_decoded(readers[0]) == 0++ def test_pypdf_below_the_threshold_is_left_alone(self, mocker):+ from pypdf import PdfReader++ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ readers = _capture(mocker, "pypdf.PdfReader", PdfReader)++ assert BaseDownloader.extract_text_from_pdf(+ _crafted_pdf(3, _PAD)+ ) == _page_texts(3, "\n")+ assert _pypdf_decoded(readers[0]) >= 3 * _PAD+++class TestParsedObjectsAreReleased:+ """Parsed objects a page resolves count against the same threshold.++ A page whose font points at a 500,000-entry ``/Widths`` array held+ ~60 MB more per page in either library (the array in the object+ cache; in pdfminer also the font object's widths), at ~2 s of CPU+ per page -- gigabytes within the CPU budget. Here each page's array+ has 20,000 entries (~2.6 MB at the releaser's per-element estimate)+ and the content streams are tiny, so only the parsed-object estimate+ can trip the 1 MiB threshold.+ """++ _ENTRIES = 20_000++ def test_pdfplumber_walk_drops_parsed_objects(self, service, mocker):+ import pdfplumber++ mocker.patch(_RETAINED_LIMIT, 1024 * 1024)+ opened = _capture(+ mocker, f"{_SERVICE_MODULE}.pdfplumber.open", pdfplumber.open+ )++ text = service._extract_text_from_pdf(+ _crafted_pdf(3, widths=self._ENTRIES)+ )++ pdf = opened[0]+ assert text is not None and "Page 2" in text+ assert not [+ obj+ for obj, _ in pdf.doc._cached_objs.values()+ if isinstance(obj, list) and len(obj) >= self._ENTRIES+ ], "a parsed /Widths array outlived the release"+ assert pdf.rsrcmgr._cached_fonts == {}, (+ "font objects (and their widths) outlived the release"+ )++ def test_pypdf_walk_drops_parsed_objects(self, mocker):+ from pypdf import PdfReader++ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ mocker.patch(_RETAINED_LIMIT, 1024 * 1024)+ readers = _capture(mocker, "pypdf.PdfReader", PdfReader)++ text = BaseDownloader.extract_text_from_pdf(+ _crafted_pdf(3, widths=self._ENTRIES)+ )++ assert text is not None and "Page 2" in text+ assert not [+ obj+ for obj in readers[0].resolved_objects.values()+ if isinstance(obj, list) and len(obj) >= self._ENTRIES+ ], "a parsed /Widths array outlived the release"++ def test_parsed_objects_below_the_threshold_are_left_alone(self, mocker):+ """Control for the two tests above."""+ from pypdf import PdfReader++ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ readers = _capture(mocker, "pypdf.PdfReader", PdfReader)++ BaseDownloader.extract_text_from_pdf(+ _crafted_pdf(3, widths=self._ENTRIES)+ )++ assert [+ obj+ for obj in readers[0].resolved_objects.values()+ if isinstance(obj, list) and len(obj) >= self._ENTRIES+ ]+++class TestWalkReleasers:+ """The releasers directly, on the real libraries."""++ def test_pdfminer_page_tree_is_kept_and_streams_keep_identity(self, mocker):+ import io++ import pdfplumber+ from pdfminer.pdftypes import PDFStream++ from local_deep_research.utilities.pdf_stream_release import (+ PdfplumberWalkReleaser,+ )++ mocker.patch(_RETAINED_LIMIT, 1)+ with pdfplumber.open(io.BytesIO(_crafted_pdf(2, _PAD))) as pdf:+ pages = pdf.pages+ pre_walk = dict(pdf.doc._cached_objs)+ releaser = PdfplumberWalkReleaser(pdf)+ for page in pages:+ releaser.before_page()+ assert page.extract_text() == f"Page {page.page_number - 1}"+ page.close()+ assert releaser.after_page() is True+ # after_page() without a before_page() has nothing to count.+ assert releaser.after_page() is False++ assert _pdfminer_decoded(pdf) == 0+ for objid, (obj, _) in pre_walk.items():+ assert pdf.doc._cached_objs[objid][0] is obj, (+ "an object cached before the walk was dropped or replaced"+ )+ for page in pages:+ for stream in page.page_obj.contents:+ assert isinstance(stream, PDFStream)+ assert pdf.doc._cached_objs[stream.objid][0] is stream, (+ "a page's content stream was replaced in the cache"+ )++ def test_pypdf_page_tree_is_kept_and_pre_walk_streams_are_released(+ self, mocker+ ):+ import io++ from pypdf import PdfReader++ from local_deep_research.utilities.pdf_stream_release import (+ PypdfWalkReleaser,+ )++ mocker.patch(_RETAINED_LIMIT, 1)+ reader = PdfReader(io.BytesIO(_crafted_pdf(2, _PAD)))+ pages = list(reader.pages)+ # Decode page 0's content stream before the walk begins, so the+ # stream is part of the pre-walk cache.+ pages[0].get_contents().get_data()+ pre_walk = dict(reader.resolved_objects)+ assert _pypdf_decoded(reader) >= _PAD++ releaser = PypdfWalkReleaser(reader)+ for page in pages:+ releaser.before_page()+ assert page.extract_text().strip().startswith("Page")+ assert releaser.after_page() is True++ assert _pypdf_decoded(reader) == 0+ for key, obj in pre_walk.items():+ assert reader.resolved_objects.get(key) is obj, (+ "an object cached before the walk was dropped or replaced"+ )++ def test_failed_rewind_leaves_the_stream_intact(self, mocker):+ """If the raw bytes cannot be read back, the decoded stream stays+ as it was -- same object, still cached, still decodable."""+ import io++ import pdfplumber+ from pdfminer.pdftypes import PDFStream++ from local_deep_research.utilities.pdf_stream_release import (+ PdfplumberWalkReleaser,+ )++ mocker.patch(_RETAINED_LIMIT, 1)+ with pdfplumber.open(io.BytesIO(_crafted_pdf(1, _PAD))) as pdf:+ page = pdf.pages[0]+ releaser = PdfplumberWalkReleaser(pdf)+ releaser.before_page()+ page.extract_text()+ page.close()+ cached = {+ objid: obj+ for objid, (obj, _) in pdf.doc._cached_objs.items()+ if isinstance(obj, PDFStream) and obj.data is not None+ }+ assert cached+ # Scoped: PDF.close() on leaving the with block walks the page+ # tree again through getobj().+ with patch.object(+ pdf.doc, "getobj", side_effect=RuntimeError("unreadable")+ ):+ assert releaser.after_page() is True++ for objid, obj in cached.items():+ assert pdf.doc._cached_objs[objid][0] is obj+ assert obj.get_data().startswith(b"BT")++ def test_releasers_never_raise_on_unexpected_objects(self):+ from local_deep_research.utilities.pdf_stream_release import (+ PdfplumberWalkReleaser,+ PypdfWalkReleaser,+ )++ class Exploding:+ def __getattr__(self, name):+ raise RuntimeError(name)++ for target in (MagicMock(), Exploding(), object()):+ for cls in (PdfplumberWalkReleaser, PypdfWalkReleaser):+ releaser = cls(target)+ releaser.before_page()+ assert releaser.after_page() is False++ def test_parsed_object_stream_cache_is_cleared(self, mocker):+ """pdfminer caches the objects parsed out of each object stream+ apart from the object cache; a release must drop those too."""+ from pdfminer.pdftypes import PDFStream++ from local_deep_research.utilities.pdf_stream_release import (+ PdfplumberWalkReleaser,+ )++ mocker.patch(_RETAINED_LIMIT, 1)+ stream = PDFStream({}, b"raw")+ stream.decode()+ pdf = MagicMock()+ pdf.doc._cached_objs = {}+ pdf.doc._parsed_objs = {7: ([[0] * 10], 1)}+ pdf.rsrcmgr._cached_fonts = {"F1": object()}+ releaser = PdfplumberWalkReleaser(pdf)+ releaser.before_page()+ pdf.doc._cached_objs[1] = (stream, 0)++ assert releaser.after_page() is True++ assert pdf.doc._parsed_objs == {}+ assert pdf.rsrcmgr._cached_fonts == {}++ def test_element_count_descends_containers_but_not_streams(self):+ from local_deep_research.utilities.pdf_stream_release import (+ _count_elements,+ )++ class Stream(dict):+ pass++ nested = {"a": [1, 2, [3, 4]], "b": Stream(x=list(range(100)))}+ # 2 dict entries + 3 list items + 2 nested list items; the+ # stream is skipped.+ assert _count_elements(nested, (Stream,)) == 7+++class TestLibraryInternalsTheReleaseReliesOn:+ """Canaries: fail when a pdfplumber/pdfminer or pypdf upgrade removes+ the internals ``utilities/pdf_stream_release`` relies on. The+ releasers swallow such failures by design, so without these an+ upgrade would silently bring the unbounded retention back.+ Written against pdfplumber 0.11.10, pdfminer.six 20260107 and pypdf+ 6.16.1."""++ def test_pdfminer_internals(self):+ import io++ import pdfplumber+ from pdfminer.pdftypes import PDFStream++ with pdfplumber.open(io.BytesIO(_crafted_pdf(1, 1024))) as pdf:+ page = pdf.pages[0]+ page.extract_text()+ assert isinstance(pdf.doc._cached_objs, dict)+ assert isinstance(pdf.doc._parsed_objs, dict)+ assert isinstance(pdf.rsrcmgr._cached_fonts, dict)+ (stream,) = page.page_obj.contents+ assert isinstance(stream, PDFStream)+ # decode() keeps the decoded bytes and drops the raw ones.+ assert stream.data is not None and stream.rawdata is None+ objid = stream.objid+ assert pdf.doc._cached_objs[objid][0] is stream+ # Parsing the object again yields its raw bytes.+ entry = pdf.doc._cached_objs.pop(objid)+ fresh = pdf.doc.getobj(objid)+ pdf.doc._cached_objs[objid] = entry+ assert isinstance(fresh, PDFStream) and fresh is not stream+ assert fresh.data is None and fresh.rawdata++ def test_pypdf_internals(self):+ import io++ from pypdf import PdfReader+ from pypdf.generic import EncodedStreamObject++ reader = PdfReader(io.BytesIO(_crafted_pdf(1, 1024)))+ page = reader.pages[0]+ page.extract_text()+ assert isinstance(reader.resolved_objects, dict)+ stream = page["/Contents"].get_object()+ assert isinstance(stream, EncodedStreamObject)+ assert stream.decoded_self is not None+ data = stream.get_data()+ # Dropping the cached copy is safe: it decodes again.+ stream.decoded_self = None+ assert stream.get_data() == data+++class TestReleaserWiring:+ """Every loop hands each page to its releaser in the right order."""++ @staticmethod+ def _recording_releaser(mocker, target, events):+ releaser = MagicMock()+ releaser.before_page.side_effect = lambda: events.append("before")+ releaser.after_page.side_effect = lambda: events.append("after")+ return mocker.patch(target, return_value=releaser)++ def test_pdfplumber_loop(self, service, mocker):+ events = []+ pages = _recording_pages(["first", "second"], events)+ pdf = _pdf_with_pages(mocker, pages)+ cls = self._recording_releaser(+ mocker,+ "local_deep_research.utilities.pdf_stream_release."+ "PdfplumberWalkReleaser",+ events,+ )++ assert service._extract_text_from_pdf(b"%PDF-fake") == (+ "first\n\nsecond"+ )++ cls.assert_called_once_with(pdf)+ assert events == [+ "before",+ ("extract", 0),+ ("close", 0),+ "after",+ "before",+ ("extract", 1),+ ("close", 1),+ "after",+ ]++ def test_pdfplumber_loop_releases_after_a_failing_page(+ self, service, mocker+ ):+ events = []+ pages = _recording_pages(["first", "second"], events)+ pages[1].extract_text.side_effect = ValueError("broken")+ _pdf_with_pages(mocker, pages)+ self._recording_releaser(+ mocker,+ "local_deep_research.utilities.pdf_stream_release."+ "PdfplumberWalkReleaser",+ events,+ )++ assert service._extract_text_from_pdf(b"%PDF-fake") is None++ assert events[-2:] == [("close", 1), "after"]++ def test_pypdf_fallback_loop(self, service, mocker):+ events = []+ _pdf_with_pages(mocker, _pages_of([""]))+ fallback = _recording_pages(["first", "second"], events)+ reader = _reader_with_pages(mocker, fallback)+ cls = self._recording_releaser(+ mocker,+ "local_deep_research.utilities.pdf_stream_release."+ "PypdfWalkReleaser",+ events,+ )++ assert service._extract_text_from_pdf(b"%PDF-fake") == (+ "first\n\nsecond"+ )++ cls.assert_called_once_with(reader.return_value)+ assert events == [+ "before",+ ("extract", 0),+ "after",+ "before",+ ("extract", 1),+ "after",+ ]++ def test_base_downloader_loop(self, mocker):+ from local_deep_research.research_library.downloaders.base import (+ BaseDownloader,+ )++ events = []+ pages = _recording_pages(["first", "second"], events)+ mock_reader = MagicMock()+ mock_reader.pages = pages+ mocker.patch("pypdf.PdfReader", return_value=mock_reader)+ cls = self._recording_releaser(+ mocker,+ "local_deep_research.research_library.downloaders.base."+ "PypdfWalkReleaser",+ events,+ )++ assert BaseDownloader.extract_text_from_pdf(b"%PDF-fake") == (+ "first\nsecond"+ )++ cls.assert_called_once_with(mock_reader)+ assert events == [+ "before",+ ("extract", 0),+ "after",+ "before",+ ("extract", 1),+ "after",+ ]