~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
fix(pdf): bound the size of resources fetched while rendering exports (#7248)
LearningCircuitb8849b2ce836
diff --git a/changelog.d/+pdf-export-resource-size-cap.security.md b/changelog.d/+pdf-export-resource-size-cap.security.mdnew file mode 100644index 0000000000..b833e49a48--- /dev/null+++ b/changelog.d/+pdf-export-resource-size-cap.security.md@@ -0,0 +1,3 @@+Limit what a PDF export fetches while rendering: the images, stylesheets, fonts and other resources a report references by URL. Each resource may cost at most 16 MiB, counted in body bytes read, in bytes after gzip or deflate decoding, and in decoding work (12 bytes per compressed byte, so a compressed body is refused past about 1.3 MiB of compressed data, and 4 KiB per gzip member). All of one export's fetches may cost 64 MiB together, refused and failed ones included. Once that is spent the export makes no further requests, so it reads at most 64 MiB of response bodies plus one byte. A body whose size is not declared is read to one byte past its limit before it is refused: measured in-process, an export that references 1000 such oversized images makes 4 requests and receives 64.2 MiB in all. A body whose `Content-Length` is past the limit is refused unread and charged the 64 KiB its first read would have asked for, so an export makes at most 1024 such requests. A resource over a limit is left out and the export still completes. Those byte caps bound reading and gzip/deflate decoding only, not the work the renderer does on a body afterwards, so bodies it would decode without a size bound are also refused: a web font it would otherwise decompress without limit (a WOFF or WOFF2 body, handed to zlib or brotli); a TTF, OTF or TTC font, any face of a collection, with colour-bitmap (`sbix`, `CBDT`/`CBLC`) or SVG (`SVG `) glyph tables, whose glyph images the renderer opens with Pillow each time it draws the glyph, outside the image limits below (a 704-byte font can declare a 13000x13000 glyph), and a font whose table directory cannot be read within bounds, or a Mac resource-fork (dfont) font; and a raster image that is not PNG, JPEG, GIF or WebP, or whose header declares more than 16,777,216 pixels (4096x4096, enough for a 12-megapixel photo or a 4K screenshot). A highly compressible image is tiny on the wire but costs the same to decode as a detailed one its size, so the renderer also limits the total pixels all of one export's images declare to 268,435,456 (about twenty-one 12-megapixel images, far more than an ordinary report); images past that are left out. This bounds the total declared raster pixels to decode; it is not an overall memory, CPU or elapsed-time quota for the renderer. SVG, which is drawn as vectors, and fonts whose glyphs are outlines only (TTF, OTF or TTC, COLR vector colour fonts included) are unaffected: the byte cap bounds such a font's file, not the work of shaping and drawing its glyphs. A refused font falls back to local fonts (so a colour-emoji web font is not used) and a refused image is left out, and the export still completes. Not counted: response heads, chunk framing and trailers, which Python's `http.client` reads itself, and what it buffers ahead; and responses whose body is never read, such as redirect and error statuses and empty bodies, are not limited in number. The cost of inflating a compressed body stays bounded by the budget: an export can inflate at most about 5.3 MiB of compressed data (64 MiB / 12), and the costliest such stream found in review — deflate blocks that decode to nothing — costs about 1 s of CPU per export on the test machine.++Image-format checks also cover Pillow formats without a signature detector. An allowed image with an unreadable header is skipped, so every accepted raster has a verified size before rendering. Reading an image's declared size no longer changes Python's process-wide warning filters, which was not safe while exports render in parallel.diff --git a/src/local_deep_research/web/services/pdf_service.py b/src/local_deep_research/web/services/pdf_service.pyindex 410b472d78..2beed88e23 100644--- a/src/local_deep_research/web/services/pdf_service.py+++ b/src/local_deep_research/web/services/pdf_service.py@@ -9,14 +9,23 @@ production Flask applications due to:- Good paged media features"""+import gzip+import http.clientimport ioimport platform+import re+import struct+import urllib.request+import urllib.response+import zlib+from contextvars import ContextVarfrom html import escape-from typing import Optional, Dict, Any+from typing import Optional, Dict, Any, NoReturnimport markdown # type: ignore[import-untyped]from loguru import loggerfrom ...security import redact_url_for_log, validate_url+from ...utilities.resource_utils import safe_close# WeasyPrint pulls in Pango/Cairo/fontTools — a heavy, multi-second import.@@ -62,7 +71,11 @@ def _ensure_weasyprint() -> None:HTML, CSS = _HTML, _CSSURLFetchingError = _URLFetchingErrorURLFetcherResponse = _Response- _URL_FETCHER = URLFetcher(allow_redirects=False)+ # Built in full before it is published, so no caller can pick up a+ # fetcher that does not have the size limits yet.+ fetcher = URLFetcher(allow_redirects=False)+ fetcher.add_handler(_BoundedBodyProcessor())+ _URL_FETCHER = fetcherWEASYPRINT_AVAILABLE = Trueexcept (OSError, ImportError):WEASYPRINT_AVAILABLE = False@@ -97,11 +110,42 @@ class UnsafePDFResourceURLError(ValueError):"""+class PDFResourceTooLargeError(ValueError):+ """Raised to refuse a resource over the PDF export's size limits.++ Raised inside the url_fetcher (by `_BoundedBodyProcessor`, or by+ `_refuse_if_budget_spent` before a fetch starts), so WeasyPrint+ handles it exactly as it handles `UnsafePDFResourceURLError`: the+ resource is skipped wherever a failed fetch is, and the retry in+ `_render_pdf` skips it where the first pass could not. The message+ never contains the URL.+ """+++class UnsupportedPDFResourceError(PDFResourceTooLargeError):+ """Refuse a resource WeasyPrint would decode without a size bound.++ The per-resource and export byte caps bound what the fetcher reads and+ what it inflates from a gzip or deflate ``Content-Encoding``. They do+ not bound the work WeasyPrint does *after* the fetch returns: it hands a+ ``wOFF``/``wOF2`` web font to zlib/brotli, every non-SVG image to+ Pillow, and every glyph image of a colour-bitmap font to Pillow, with no+ size limit of its own (`_refuse_unbounded_resource` has the exact call+ sites). A small body can expand enormously there, so those bodies are+ refused in the fetcher instead. Subclasses+ `PDFResourceTooLargeError`, so it is skipped and logged exactly like an+ over-cap resource, and the message never contains the URL.+ """++# Populated by _ensure_weasyprint() on first PDF use. The URLFetcher preserves# the allow_redirects=False posture that default_url_fetcher hard-coded.# Redirects disabled keeps the SSRF guard airtight — validate_url only inspects# the initial URL, so a 30x to a cloud metadata endpoint (see# ssrf_validator.ALWAYS_BLOCKED_METADATA_IPS) would otherwise slip past.+# It also carries a `_BoundedBodyProcessor`, which reads every body before the+# fetch returns, bounds what reading it may cost (`MAX_PDF_RESOURCE_BYTES`) and+# charges that cost to the export's budget (`MAX_PDF_EXPORT_RESOURCE_BYTES`)._URL_FETCHER = None# Populated alongside the fetcher; see `_render_pdf` for why it is needed.URLFetchingError = None@@ -120,9 +164,837 @@ _EMPTY_RESOURCE_MIME_TYPE = "application/octet-stream"# rest are counted. Every skipped URL also gets its own warning._SKIPPED_SUMMARY_LIMIT = 20+# Size limits for what WeasyPrint fetches while rendering an export: images,+# stylesheets, fonts and anything else a report references by URL. Applied by+# `_BoundedBodyProcessor` to what reading each body costs (`_FetchCost`): its+# body bytes, its decoded bytes if it is gzip or deflate, and the work of+# decoding it. A compressed body is therefore bounded by what it decodes to,+# and by the work of decoding it, not by its size on the wire.+#+# This byte cap bounds reading and `Content-Encoding` decoding only. It does+# not bound the work WeasyPrint does on the body after the fetch returns: a+# compressed web font goes to zlib/brotli, and a raster image, or a font's+# colour-bitmap glyph images, go to Pillow, each able to expand far past the+# body's size. `_refuse_unbounded_resource`+# refuses those bodies in the fetcher. `MAX_PDF_IMAGE_PIXELS` per image and+# `MAX_PDF_EXPORT_IMAGE_PIXELS` across the export separately bound declared+# raster pixels. These are resource budgets, not an overall memory, CPU or+# elapsed-time quota for the renderer.+#+# 16 MiB per resource is far above what report images, stylesheets and web+# fonts need. `safe_requests.MAX_RESPONSE_SIZE` (1 GiB) is sized for documents+# a user downloads on purpose, not for resources a report merely references.+MAX_PDF_RESOURCE_BYTES = 16 * 1024 * 1024+# The most pixels one raster image a report references may declare. WeasyPrint+# opens every non-SVG image with Pillow and decodes it to a raster buffer of 4+# bytes a pixel (RGB/RGBA) to re-encode it; Pillow's own decompression-bomb+# guard only errors past ~178 M pixels (~700 MiB), and `LOAD_TRUNCATED_IMAGES`+# lets a tiny truncated body decode to its full declared size. So+# `_refuse_unbounded_resource` reads the size an image declares in its header+# and refuses one past this cap. 16 M pixels (4096x4096) admits ordinary+# content -- a 12 MP phone photo (~12 M) and a 4K/8.3 MP screenshot -- while+# holding one image's transient decode to 64 MiB of RGBA. Images are decoded+# one at a time (WeasyPrint's `pdf.write_pdf` loop calls `get_x_object` per+# image; see the per-export note below). This bounds one image's dimensions,+# not the renderer's total peak memory, which includes other buffers.+MAX_PDF_IMAGE_PIXELS = 16 * 1024 * 1024+# The most pixels all of one export's accepted raster images may declare+# together. The byte caps do not bound this: a highly compressible image (a+# flat colour, a simple chart) is tiny on the wire yet costs the same to decode+# as a noisy one its size, so a report of thousands of distinct-URL images,+# each under the per-image cap and cheap on the wire, could impose unbounded+# total decode work. (It would not keep that many *resident*: WeasyPrint's+# `RasterImage` retains only the re-encoded, compressed bytes -- JPEG kept as+# is, everything else re-encoded to PNG -- in an in-memory cache keyed by URL,+# and decodes to raw pixels only transiently, one image at a time. Re-encoding+# can change the compressed size, so the fetched byte budget is not a bound+# on resident image memory. This separate cap bounds the total declared+# pixels to decode.)+# `_refuse_unbounded_resource` charges each accepted image its declared pixels+# against this budget and refuses the rest, so the export still completes. 256+# M pixels admits dozens of multi-megapixel images (about 21 at 12 MP, 64 at 4+# MP, 128 at 2 MP) -- far more than an ordinary report -- while a thousands-of-+# images reference set is cut off.+MAX_PDF_EXPORT_IMAGE_PIXELS = 256 * 1024 * 1024+# Raster image formats whose body `_refuse_unbounded_resource` accepts once it+# is within `MAX_PDF_IMAGE_PIXELS`; any other raster format Pillow would decode+# is refused. SVG is not here because it is vector XML, not a Pillow raster.+# Named as Pillow names them (``Image.OPEN``). A multi-picture JPEG (MPO) is+# opened through JPEG's own factory (``jpeg_factory``) and so classified as+# ``JPEG``; Pillow registers no separate ``MPO`` opener.+_ALLOWED_RASTER_FORMATS = frozenset({"PNG", "JPEG", "GIF", "WEBP"})+# These Pillow plugins have no signature sniffer, so their factories are run+# on the (already byte-capped) body to rule them out. Each was reviewed to+# decode no pixels and allocate nothing according to the declared image+# dimensions. Not all read a fixed-size header: PCD, SPIDER and TGA do, but+# IM and IMT read text header lines that may run to the end of the body+# (linear in the body, which the byte cap bounds). Only these factories may+# be called during classification. A new plugin without a sniffer fails closed until its+# header parser is reviewed.+_HEADER_ONLY_RASTER_FORMATS = frozenset({"IM", "IMT", "PCD", "SPIDER", "TGA"})+# Formats without a sniffer that are ruled out by a reviewed prefix instead of+# by running their factory. IPTC's ``_open`` walks every field to the end of+# the body before it can fail, but its first field must start with 0x1C (or+# the parse fails on a missing field), so a body starting with anything else+# cannot be IPTC, and one starting with 0x1C is refused as IPTC unparsed.+_PREFIX_RULED_RASTER_FORMATS = {"IPTC": b"\x1c"}+# Web-font magic numbers WeasyPrint decompresses after the fetch: ``wOFF`` via+# zlib (fontTools ``TTFont.save``) and ``wOF2`` via brotli (fontTools+# ``woff2.decompress``), both without a size limit. Refused outright.+_COMPRESSED_FONT_MAGIC = (b"wOFF", b"wOF2")+# The font containers HarfBuzz reads tables from (``OpenTypeFontFile`` in+# ``hb-open-file.hh``): a single sfnt face under one of these version tags, a+# collection of faces (``ttcf``), or a Mac resource-fork font (dfont). Pango+# builds each installed ``@font-face`` font's ``hb_face`` from the file, and+# WeasyPrint reads its colour-glyph tables through it.+_SFNT_VERSION_TAGS = frozenset({b"\x00\x01\x00\x00", b"OTTO", b"true", b"typ1"})+_FONT_COLLECTION_TAG = b"ttcf"+_RESOURCE_FORK_FONT_TAG = b"\x00\x00\x01\x00"+# Tables that make WeasyPrint draw glyphs as images rather than outlines.+# ``sbix`` and ``CBDT``/``CBLC`` hold a PNG per glyph: WeasyPrint+# (``pdf/fonts.py`` ``hb_ot_color_has_png``, ``draw/text.py``) opens each+# drawn glyph's PNG with an unrestricted ``Image.open`` and builds a+# ``RasterImage`` from it, for every occurrence, outside the fetcher and so+# outside `MAX_PDF_IMAGE_PIXELS` and the export's pixel budget: a 704-byte+# font can declare a 13000x13000 glyph. ``SVG `` holds an SVG document per+# glyph (``hb_ot_color_has_svg``), which can embed raster images of its own.+# A font with any of them is refused, so it falls back to local fonts.+_GLYPH_IMAGE_TABLES = frozenset({b"sbix", b"CBDT", b"CBLC", b"SVG "})+# Caps on the table directory `_font_refusal` reads: an ordinary font has a+# few dozen tables and a collection a few dozen faces. A directory past them+# is refused rather than walked.+_MAX_FONT_TABLES = 512+_MAX_FONT_COLLECTION_FACES = 256+# What all of one export's fetches may cost together: four resources at the+# per-resource limit. Every response is charged what reading it cost+# (`_FetchCost`), whether it was accepted, refused or failed part-way, so many+# oversized or undecodable references spend the budget just as many large+# resources do. No read asks for more than one byte past what is left, and+# once nothing is left every later fetch in the export is refused before it+# starts (`_refuse_if_budget_spent`). One export therefore reads at most this+# much body data plus one byte, and decodes at most this much plus one+# decoding step; `_FetchCost` lists what is not counted. Everything the+# export fetches draws on it, including the custom stylesheet's imports and+# the retry in `_render_pdf`.+MAX_PDF_EXPORT_RESOURCE_BYTES = 64 * 1024 * 1024+# The most one read of a body asks for, and the most one decoding step may+# produce.+_RESOURCE_READ_CHUNK_BYTES = 64 * 1024+# The most compressed data one decoding step takes in. Small, because a step+# that raises part-way is charged as if it had taken in all of it.+_INFLATE_INPUT_CHUNK_BYTES = 4 * 1024+# What each compressed byte that zlib takes in costs. Decoding work does not+# follow the output: a deflate stream whose blocks each build Huffman tables of+# their own can decode to nothing yet still cost CPU for every byte of it. At+# this cost a gzip or deflate body is refused once its compressed data passes+# 1/12 of its limit (about 1.3 MiB of the 16 MiB), and data that compresses+# less than 12 to 1 is charged more than its decoded size: text, at about 4 to+# 1, three times.+#+# 12 bounds, it does not measure. The budget lets at most ~5.3 MiB of+# compressed data be inflated across an export (64 MiB / 12), and the costliest+# such stream found in review -- dynamic-Huffman blocks that decode to nothing+# -- cost about 1 s of CPU per export when timed against this budget. The cost+# stays bounded by the budget whatever the stream: a larger constant would+# charge legitimate gzip- or deflate-encoded resources (which a CDN sends for+# CSS and SVG) more, dropping them sooner.+_INFLATED_BYTE_COST = 12+# What each member of a gzip body costs at least. A body may concatenate any+# number of members, and starting one costs about as much CPU as decoding a+# few KiB, however little it holds; an empty member is 20 bytes on the wire.+_GZIP_MEMBER_COST_BYTES = 4 * 1024++# Gzip header flags (RFC 1952, section 2.3.1), named as in ``gzip.py``.+_FHCRC, _FEXTRA, _FNAME, _FCOMMENT = 2, 4, 8, 16+# Zero padding, which gzip allows after a member.+_ZERO_BYTES = re.compile(rb"\0*")+++class _ResourceBudget:+ """What the current export's fetches may still cost.++ ``remaining`` is the byte budget (`MAX_PDF_EXPORT_RESOURCE_BYTES`) for+ reading and decoding bodies. ``image_pixels`` is a separate budget, in+ pixels, for the raster images accepted so far: the byte budget does not+ bound an image's decode, so each accepted image is charged its declared+ pixels here (`_refuse_unbounded_resource`), and once it is spent further+ images are refused.+ """++ __slots__ = (+ "remaining",+ "spent_reported",+ "image_pixels",+ "image_budget_reported",+ )++ def __init__(self, total: int, image_pixels: Optional[int] = None) -> None:+ self.remaining = total+ # Whether `_refuse_if_budget_spent` has logged its warning yet.+ self.spent_reported = False+ # Read the module global at call time (so a test may patch it); the+ # default is the whole export's image-decode budget.+ self.image_pixels = (+ MAX_PDF_EXPORT_IMAGE_PIXELS+ if image_pixels is None+ else image_pixels+ )+ # Whether the image-pixel-budget-spent warning has been logged yet.+ self.image_budget_reported = False++ def charge(self, cost: int) -> None:+ # A refused body can cost more than was left: one byte more on the+ # wire, or one decoding step more (see `_inflate`).+ self.remaining = max(0, self.remaining - cost)++ def charge_image_pixels(self, pixels: int) -> bool:+ """Deduct an accepted image's declared pixels; return whether they fit.++ A declaration past what is left spends the rest of the budget (so+ every later image is refused too) and returns ``False``.+ """+ fits = pixels <= self.image_pixels+ self.image_pixels = max(0, self.image_pixels - pixels)+ return fits+++# Set by `markdown_to_pdf` for the length of one export. A context variable,+# not state on a fetcher: `_safe_url_fetcher` and `_URL_FETCHER` are shared by+# every export in the process, concurrent ones included. Outside an export+# (None) only the per-resource limit applies.+_export_resource_budget: ContextVar[Optional[_ResourceBudget]] = ContextVar(+ "pdf_export_resource_budget", default=None+)+++def _over_limit(limit: int) -> PDFResourceTooLargeError:+ return PDFResourceTooLargeError(f"PDF resource larger than {limit} bytes")+++def _refuse_if_budget_spent(url) -> None:+ """Refuse a fetch before it starts if the export's budget is spent.++ Runs before `validate_url`, which resolves the host name, so an export+ whose budget is spent makes no further requests of any kind. Warns once+ per export, not once per resource: a report can reference any number.+ """+ budget = _export_resource_budget.get()+ if budget is None or budget.remaining > 0:+ return+ if not budget.spent_reported:+ budget.spent_reported = True+ logger.warning(+ "Refused a resource in PDF rendering without fetching it, as "+ "every later one in this export will be: the export's "+ f"{MAX_PDF_EXPORT_RESOURCE_BYTES}-byte resource budget is spent: "+ f"{redact_url_for_log(url)}"+ )+ raise PDFResourceTooLargeError(+ f"PDF export resource budget of {MAX_PDF_EXPORT_RESOURCE_BYTES} "+ "bytes is spent"+ )+++class _FetchCost:+ """What reading one response has cost so far.++ ``wire`` counts the body bytes read from the response: what+ ``http.client`` returns, after any chunked framing is removed.+ ``decoded`` counts every byte inflated from a gzip or deflate body,+ including what a failed attempt decoded. ``inflated`` counts the+ compressed bytes zlib took in to decode them, each charged+ `_INFLATED_BYTE_COST`, and ``members`` the gzip members, each charged+ `_GZIP_MEMBER_COST_BYTES`. Each read and each decoding step is counted+ before it runs, at the most it can cost, then corrected to what it did+ cost once it returns (`_read_body`, `_inflate`), so one that raises+ part-way stays counted.++ Not counted: what ``http.client`` reads around the body itself. That+ is the response head (a status line and up to 100 header lines of up+ to 64 KiB each, after any number of interim ``100 Continue``+ responses), the chunk framing (a size line of up to 64 KiB, extensions+ included, and a line end for every chunk, however small), the trailer+ (any number of lines of up to 64 KiB), and what its buffered reader+ takes from the connection ahead of what it returns (up to+ ``io.DEFAULT_BUFFER_SIZE`` per response: 128 KiB on Python 3.14). A+ server can therefore still keep one fetch reading without end, as it+ can by sending slowly. A body refused unread for a ``Content-Length``+ past the limit is charged as if its first read had been made+ (`_read_body`), so at most `MAX_PDF_EXPORT_RESOURCE_BYTES` /+ `_RESOURCE_READ_CHUNK_BYTES` (1024) of those are fetched per export.+ But a fetch whose body is never read costs nothing (a non-2xx status,+ which ``HTTPErrorProcessor`` raises on before `_BoundedBodyProcessor`+ runs, or an empty body), so the number of such fetches is not limited.+ """++ __slots__ = ("wire", "decoded", "inflated", "members")++ def __init__(self) -> None:+ self.wire = 0+ self.decoded = 0+ self.inflated = 0+ self.members = 0++ def total(self) -> int:+ work = (+ self.inflated * _INFLATED_BYTE_COST+ + self.members * _GZIP_MEMBER_COST_BYTES+ )+ return max(self.wire, self.decoded, work)++ def check(self, limit: int) -> None:+ """Refuse the resource once its cost has passed ``limit``."""+ if self.total() > limit:+ raise _over_limit(limit)+++def _read_body(response, limit: int, cost: _FetchCost) -> bytes:+ """Read ``response``'s body to its end, as it arrives on the wire.++ Refuses it (`_FetchCost.check`) as it is read, or before any of it is+ read if its ``Content-Length`` is already past ``limit`` (charged then+ as if its first read had been made: `_RESOURCE_READ_CHUNK_BYTES`, or+ one byte past ``limit`` if that is less). No read asks+ for more than one byte past ``limit``, and each is counted before it is+ made, at what it asks for: a read that raises part-way (a chunked body+ cut short, a timeout) stays counted. A body that ends before its+ ``Content-Length`` raises ``http.client.IncompleteRead``, as an unsized+ ``read()`` would: the sized reads used here return a short body as if+ it were complete. Without a ``Content-Length``, a chunked body that+ ends early raises ``IncompleteRead`` from ``http.client``; a body that+ ends when the connection closes cannot be told apart from a complete+ one.+ """+ declared = getattr(response, "length", None) # http.client's parsed value+ if isinstance(declared, int) and declared > limit:+ # Charged what its first read would have asked for, as if that read+ # had been made, so a declared length past the limit still spends+ # the budget and the number of such fetches is bounded too.+ cost.wire = min(_RESOURCE_READ_CHUNK_BYTES, limit + 1)+ raise _over_limit(limit)+ chunks = []+ while True:+ size = min(_RESOURCE_READ_CHUNK_BYTES, limit - cost.wire + 1)+ cost.wire += size+ chunk = response.read(size)+ cost.wire -= size - len(chunk)+ if not chunk:+ break+ cost.check(limit)+ chunks.append(chunk)+ data = b"".join(chunks)+ if isinstance(declared, int) and len(data) < declared:+ raise http.client.IncompleteRead(data, declared - len(data))+ return data+++def _inflate(+ inflater, data: bytes, start: int, limit: int, cost: _FetchCost, out: list+) -> Optional[int]:+ """Feed ``data[start:]`` to ``inflater`` until its stream ends.++ Each call takes in at most `_INFLATE_INPUT_CHUNK_BYTES` and puts out at+ most `_RESOURCE_READ_CHUNK_BYTES`. It is counted in ``cost`` before it+ runs, at those most, then corrected to what it took in and put out,+ refusing past ``limit`` before the next call. So a refused stream is+ decoded at most one call past its limit, and a call that raises+ part-way stays counted. Appends the output to ``out``. Returns the+ offset just past the end of the stream, or None if the data ends first.+ """+ view = memoryview(data)+ pos = start+ pending = b""+ while not inflater.eof:+ if pending:+ piece = pending+ else:+ piece = view[pos : pos + _INFLATE_INPUT_CHUNK_BYTES]+ pos += len(piece)+ cost.inflated += len(piece)+ cost.decoded += _RESOURCE_READ_CHUNK_BYTES+ try:+ chunk = inflater.decompress(piece, _RESOURCE_READ_CHUNK_BYTES)+ except zlib.error:+ # A step zlib rejects returns no output, so the output charged in+ # advance was never produced. The work stays charged.+ cost.decoded -= _RESOURCE_READ_CHUNK_BYTES+ raise+ # What zlib did not take in: left for the next call or, once the+ # stream has ended, past its end (CPython then sets both).+ cost.inflated -= len(+ inflater.unused_data if inflater.eof else inflater.unconsumed_tail+ )+ cost.decoded -= _RESOURCE_READ_CHUNK_BYTES - len(chunk)+ cost.check(limit)+ if not chunk and not piece:+ return None+ if chunk:+ out.append(chunk)+ pending = inflater.unconsumed_tail+ return pos - len(inflater.unused_data)+++def _inflate_capped(+ data: bytes, wbits: int, limit: int, cost: _FetchCost+) -> bytes:+ """``zlib.decompress(data, wbits)``, within ``limit`` (see `_inflate`)."""+ out: list = []+ if _inflate(zlib.decompressobj(wbits), data, 0, limit, cost, out) is None:+ # What zlib.decompress raises for a stream that ends early.+ raise zlib.error("incomplete or truncated stream")+ # Like zlib.decompress, ignores anything after the end of the stream.+ return b"".join(out)+++def _gzip_truncated() -> EOFError:+ # What GzipFile raises for a body that ends part-way through a member.+ return EOFError(+ "Compressed file ended before the end-of-stream marker was reached"+ )+++def _gzip_header_end(data: bytes, pos: int) -> int:+ """Where the deflate data of the gzip member at ``pos`` starts.++ Parses the header as ``gzip._read_gzip_header`` does. It checks the+ magic number and the compression method. It skips the optional extra+ field, name, comment and header CRC, and does not check that CRC. It+ ignores the other flag bits.+ """+ magic = data[pos : pos + 2]+ if magic != b"\037\213":+ raise gzip.BadGzipFile(f"Not a gzipped file ({magic!r})")+ if len(data) < pos + 10:+ raise _gzip_truncated()+ if data[pos + 2] != 8:+ raise gzip.BadGzipFile("Unknown compression method")+ flags = data[pos + 3]+ pos += 10+ if flags & _FEXTRA:+ if len(data) < pos + 2:+ raise _gzip_truncated()+ pos += 2 + int.from_bytes(data[pos : pos + 2], "little")+ if len(data) < pos:+ raise _gzip_truncated()+ for field in (_FNAME, _FCOMMENT):+ if flags & field: # ends at a zero byte, or at the end of the data+ end = data.find(b"\0", pos)+ pos = len(data) if end < 0 else end + 1+ if flags & _FHCRC:+ pos += 2+ if len(data) < pos:+ raise _gzip_truncated()+ return pos+++def _gunzip_capped(data: bytes, limit: int, cost: _FetchCost) -> bytes:+ """Decode a gzip body as ``GzipFile`` does, within ``limit``.++ Member by member, as ``gzip._GzipReader`` reads them: the header as+ `_gzip_header_end` parses it, the deflate data inflated raw, then the+ CRC-32 and length in the trailer checked, and any zero bytes after a+ member skipped. An empty body decodes to nothing. What it returns, and+ which bodies it rejects, match ``GzipFile``; two things differ, both in+ what decoding costs. Each member is charged to ``cost``+ (`_GZIP_MEMBER_COST_BYTES`), so a body of many tiny members is refused+ rather than decoded member after member. Zero padding is skipped in one+ step, not one byte at a time.+ """+ out: list = []+ pos = 0+ while pos < len(data):+ cost.members += 1+ cost.check(limit)+ first = len(out)+ end = _inflate(+ zlib.decompressobj(-zlib.MAX_WBITS),+ data,+ _gzip_header_end(data, pos),+ limit,+ cost,+ out,+ )+ if end is None or len(data) < end + 8:+ raise _gzip_truncated()+ crc, size = struct.unpack_from("<II", data, end)+ member_crc = 0+ member_size = 0+ for chunk in out[first:]:+ member_crc = zlib.crc32(chunk, member_crc)+ member_size += len(chunk)+ if crc != member_crc:+ raise gzip.BadGzipFile("CRC check failed")+ if size != member_size & 0xFFFFFFFF:+ raise gzip.BadGzipFile("Incorrect length of data produced")+ pos = _ZERO_BYTES.match(data, end + 8).end()+ return b"".join(out)+++def _decoded_body(response, limit: int, cost: _FetchCost) -> bytes:+ """Read and decode ``response``'s body as WeasyPrint's `URLFetcher` would.++ Refuses it with `PDFResourceTooLargeError` once what it costs+ (`_FetchCost`) passes ``limit``, counted as it is read and decoded. A+ declared ``Content-Length`` past the limit is refused before any of the+ body is read. The decoding follows ``URLFetcher.fetch``+ (``weasyprint/urls.py``): the first ``Content-Encoding`` header, compared+ exactly; ``gzip`` as ``GzipFile`` decodes it (`_gunzip_capped`);+ ``deflate`` as a zlib stream, retried as raw deflate; anything else+ passed through as it is.+ """+ data = _read_body(response, limit, cost)+ encoding = response.headers.get("Content-Encoding")+ if encoding == "gzip":+ return _gunzip_capped(data, limit, cost)+ if encoding == "deflate":+ try:+ return _inflate_capped(data, zlib.MAX_WBITS, limit, cost)+ except zlib.error:+ return _inflate_capped(data, -zlib.MAX_WBITS, limit, cost)+ return data+++def _pillow_raster_format(body: bytes) -> Optional[str]:+ """The Pillow format name for ``body``'s header, or None.++ Use registered ``accept`` sniffers where available. Pillow also opens+ formats without a sniffer: rule those out by a reviewed prefix+ (`_PREFIX_RULED_RASTER_FORMATS`) or by running their reviewed+ header-only factory (`_HEADER_ONLY_RASTER_FORMATS`) on the already+ byte-capped body. Never open an unreviewed format: some (ICO, ICNS, BLP)+ decode pixels during ``_open``. A header-only factory that raises+ ``ValueError`` or ``OSError`` (which ``Image.open`` would propagate, not+ skip) fails closed: its format is returned, so the body is refused with+ the guard's own warning. Non-raster bodies (SVG, ICC profiles,+ stylesheets, TTF/OTF fonts, text) return None.+ """+ from PIL import Image++ Image.init() # populate Image.OPEN with every plugin (idempotent)+ prefix = bytes(body[:32])+ for fmt, (factory, accept) in list(Image.OPEN.items()):+ if accept is None:+ ruled_by = _PREFIX_RULED_RASTER_FORMATS.get(fmt)+ if ruled_by is not None:+ if prefix.startswith(ruled_by):+ return fmt+ continue+ if fmt not in _HEADER_ONLY_RASTER_FORMATS:+ # Cannot rule out this format without running an unreviewed+ # decoder. Refuse the resource, leaving the export usable.+ return fmt+ try:+ with io.BytesIO(body) as stream, factory(stream):+ return fmt+ except (SyntaxError, IndexError, TypeError, struct.error):+ # Same non-matches Image.open skips while trying plugins.+ continue+ except (ValueError, OSError):+ # Image.open would raise these rather than try the next+ # plugin: fail closed under this format's name.+ return fmt+ try:+ matched = accept(prefix)+ except Exception:+ continue+ # A str result is a reason the format was *not* accepted (e.g. WebP+ # support missing), which Image.open treats as no match.+ if matched and not isinstance(matched, str):+ return fmt+ return None+++def _declared_image_pixels(body: bytes, fmt: str) -> Optional[int]:+ """The pixel count an allow-listed raster image declares in its header.++ Runs the identified format's own factory (``Image.OPEN[fmt][0]``), not+ ``Image.open``: so no other format's decoder is tried if this header+ fails, and Pillow's ``_decompression_bomb_check`` (which ``Image.open``+ runs after the factory, and which warns past ~89 M pixels) is never+ reached. Nothing here touches the process-wide ``warnings`` filters,+ which ``warnings.catch_warnings`` would, unsafely, while exports run in+ other threads; the cap below answers an oversized declaration instead.++ No factory decodes pixels, but not all of them read only a fixed-size+ header. PNG and GIF read their header chunks. JPEG walks its markers up+ to the first frame, so a body of markers alone is scanned to its end+ (linear in the body, which the byte cap bounds). WebP hands the body to+ libwebp's demuxer and animation decoder, which allocates a canvas buffer+ for the declared size (up to 16383x16383 RGBA, about 1 GiB of virtual+ memory, mostly never touched) before this cap is compared. Returns None+ if the header cannot be read (an allocation failure included), so the+ caller can refuse it.+ """+ from PIL import Image++ Image.init() # idempotent; Image.OPEN may not be populated yet+ opener = Image.OPEN.get(fmt)+ if opener is None:+ return None+ try:+ with opener[0](io.BytesIO(body), "") as image:+ width, height = image.size+ except Exception:+ return None+ return width * height+++class _MalformedFont(ValueError):+ """A font container `_font_refusal` cannot read within its caps."""+++def _sfnt_table_tags(body: bytes, offset: int) -> list:+ """The table tags of the sfnt table directory at ``offset``.++ Reads the fixed 12-byte header and ``numTables`` 16-byte records, each+ bounds-checked, and checks each table lies within the body. Raises+ `_MalformedFont` rather than reading past the body or past+ `_MAX_FONT_TABLES` records.+ """+ records = offset + 12+ if records > len(body):+ raise _MalformedFont("font table directory past the end of the body")+ (count,) = struct.unpack_from(">H", body, offset + 4)+ if count > _MAX_FONT_TABLES:+ raise _MalformedFont("font table directory too long")+ end = records + 16 * count+ if end > len(body):+ raise _MalformedFont("font table records past the end of the body")+ tags = []+ for record in range(records, end, 16):+ table_offset, length = struct.unpack_from(">II", body, record + 8)+ if table_offset + length > len(body):+ raise _MalformedFont("font table past the end of the body")+ tags.append(bytes(body[record : record + 4]))+ return tags+++def _font_face_offsets(body: bytes) -> list:+ """Where each face's table directory starts: one face, or a collection's."""+ if body[:4] != _FONT_COLLECTION_TAG:+ return [0]+ if len(body) < 12:+ raise _MalformedFont("font collection header past the end of the body")+ (count,) = struct.unpack_from(">I", body, 8)+ if count > _MAX_FONT_COLLECTION_FACES:+ raise _MalformedFont("font collection has too many faces")+ if 12 + 4 * count > len(body):+ raise _MalformedFont("font collection offsets past the end of the body")+ return list(struct.unpack_from(f">{count}I", body, 12))+++def _font_refusal(body: bytes) -> Optional[str]:+ """Why an sfnt or ``ttcf`` font body is refused, or None if it is not.++ Reads every face's table directory (pure ``struct``, bounded by+ `_MAX_FONT_TABLES` and `_MAX_FONT_COLLECTION_FACES`, every offset+ checked against the body) and refuses a font that has any of+ `_GLYPH_IMAGE_TABLES`. Every face of a collection is checked, since+ Fontconfig may install any of them. A directory that cannot be read+ within those bounds is refused too (fails closed): HarfBuzz may read+ more of a damaged font than this reader admits. The tags are compared+ in the records, not looked up by binary search as HarfBuzz does, so an+ unsorted directory cannot hide a table from this check.+ """+ try:+ for offset in _font_face_offsets(body):+ found = sorted(set(_sfnt_table_tags(body, offset)))+ images = [tag for tag in found if tag in _GLYPH_IMAGE_TABLES]+ if images:+ names = ", ".join(tag.decode("latin-1") for tag in images)+ return f"a font with image glyphs ({names.strip()})"+ except _MalformedFont as exc:+ return f"a font whose table directory is unreadable ({exc})"+ return None+++def _refuse_image_or_font(url, reason: str) -> NoReturn:+ """Log (redacted) and raise for a body WeasyPrint would decode unbounded."""+ logger.warning(+ f"Refused {reason} in PDF rendering, which WeasyPrint would decode "+ f"without a size bound: {redact_url_for_log(url)}"+ )+ raise UnsupportedPDFResourceError(f"PDF resource refused: {reason}")+++def _refuse_unbounded_resource(body: bytes, url) -> None:+ """Refuse a body WeasyPrint would decode without a size bound.++ Runs on the decoded body (after any ``Content-Encoding``), which is what+ WeasyPrint hands to fontTools, Fontconfig/HarfBuzz or Pillow. Refuses:++ - a compressed web font (``wOFF``/``wOF2``), decompressed with no limit;+ - an sfnt or ``ttcf`` font with colour-bitmap or SVG glyph tables+ (`_GLYPH_IMAGE_TABLES`), whose glyph images WeasyPrint opens with an+ unrestricted ``Image.open`` per drawn glyph, or whose table directory+ cannot be read within bounds (`_font_refusal`);+ - a Mac resource-fork (dfont) font, which HarfBuzz also reads tables+ from but this guard does not parse;+ - a raster image whose format is not allow-listed;+ - a raster image whose header declares more than `MAX_PDF_IMAGE_PIXELS`;+ - an allow-listed image once the export's `MAX_PDF_EXPORT_IMAGE_PIXELS`+ decode budget is spent. Each accepted image is charged its declared+ pixels against that budget, so many images that are cheap on the wire+ (a flat colour, a simple chart) but costly to decode cannot add up to+ unbounded total decode work. WeasyPrint keeps no raw image resident --+ its `RasterImage` retains only re-encoded, compressed bytes and decodes+ one image at a time -- so this bounds decode *work*, not resident RAM.++ Everything else -- SVG, an ICC colour profile, a stylesheet, a TTF/OTF+ font with outline glyphs only, plain text -- passes. The byte cap bounds+ such a font's file, not the work FreeType and HarfBuzz do shaping and+ drawing it (COLR vector glyphs included). Raising+ `UnsupportedPDFResourceError` fails the fetch exactly as an over-cap body+ does: skipped, and the export completes with the resource left out. The+ warning names no URL beyond scheme://host:port.+ """+ magic = body[:4]+ if magic in _COMPRESSED_FONT_MAGIC:+ # WeasyPrint sends wOFF to zlib and wOF2 to brotli with no size limit+ # (weasyprint/text/fonts.py); the export falls back to local fonts.+ _refuse_image_or_font(url, "a compressed web font")+ if magic in _SFNT_VERSION_TAGS or magic == _FONT_COLLECTION_TAG:+ # An uncompressed font: its outlines are bounded by the byte cap, its+ # glyph images are not (see _GLYPH_IMAGE_TABLES).+ reason = _font_refusal(body)+ if reason is not None:+ _refuse_image_or_font(url, reason)+ fmt = _pillow_raster_format(body)+ if fmt is None:+ if magic == _RESOURCE_FORK_FONT_TAG:+ # Pillow's ICO sniffer claims this prefix first; refused here too+ # in case that plugin is unavailable.+ _refuse_image_or_font(url, "a resource-fork (dfont) font")+ # Not a raster Pillow decodes: SVG, ICC profile, stylesheet, TTF/OTF,+ # text. Left for WeasyPrint to handle.+ return+ if fmt not in _ALLOWED_RASTER_FORMATS:+ # A raster Pillow would decode but we do not bound (its declared size+ # is not read here, and some formats decode on open).+ _refuse_image_or_font(+ url, f"a {fmt} image (not one of PNG, JPEG, GIF, WebP or SVG)"+ )+ pixels = _declared_image_pixels(body, fmt)+ if pixels is None:+ # Do not pass a body with an unverified size to the renderer, whose+ # unrestricted Image.open may try other formats after this fails.+ _refuse_image_or_font(url, f"a {fmt} image with an unreadable header")+ if pixels > MAX_PDF_IMAGE_PIXELS:+ _refuse_image_or_font(+ url,+ f"a {pixels}-pixel {fmt} image, over the "+ f"{MAX_PDF_IMAGE_PIXELS}-pixel per-image limit",+ )+ budget = _export_resource_budget.get()+ if budget is not None and not budget.charge_image_pixels(pixels):+ # Warn once per export, not once per image: a report can reference any+ # number. Every over-budget image is still refused (and dropped).+ if not budget.image_budget_reported:+ budget.image_budget_reported = True+ logger.warning(+ "Refused a raster image in PDF rendering: the export's "+ f"{MAX_PDF_EXPORT_IMAGE_PIXELS}-pixel image-decode budget is "+ f"spent, so further images are refused: "+ f"{redact_url_for_log(url)}"+ )+ raise UnsupportedPDFResourceError(+ "PDF export image-decode budget of "+ f"{MAX_PDF_EXPORT_IMAGE_PIXELS} pixels is spent"+ )+++class _BoundedBodyProcessor(urllib.request.BaseHandler):+ """Reads each http(s) body `_URL_FETCHER` receives, within the limits.++ WeasyPrint's ``URLFetcher.fetch`` sets no size limit: it decodes a+ deflate body in full inside the fetch, and returns a plain or gzip body+ as a stream that its consumers read to the end after the fetch has+ returned. This response processor runs inside the fetch instead: it+ reads and decodes the body (`_decoded_body`), drops the+ ``Content-Encoding`` header so ``fetch`` does not decode it a second+ time, and passes the decoded bytes on in memory.++ A body that costs more than `MAX_PDF_RESOURCE_BYTES`, or more than what+ is left of the export's `MAX_PDF_EXPORT_RESOURCE_BYTES`, raises+ `PDFResourceTooLargeError`. Whatever the outcome, accepted, refused or+ failed, the export's budget is charged what reading the body cost+ (`_FetchCost`). Once nothing is left, `_refuse_if_budget_spent` refuses+ each further fetch before it starts, so none reaches this processor.+ Because the body is read here, these also fail the fetch itself rather+ than a later read: a body cut short (``IncompleteRead``, see+ `_read_body`), a corrupt gzip or deflate body, and a read timeout+ part-way through the body.++ The byte cap bounds reading and `Content-Encoding` decoding only, not+ the work WeasyPrint does on the body afterwards. So the read, charged+ body is then passed to `_refuse_unbounded_resource`, which refuses a+ compressed web font, a font with colour-bitmap or SVG glyph tables (or+ an unreadable table directory), a raster image that is not an+ allow-listed format+ or declares more than `MAX_PDF_IMAGE_PIXELS`, and an allow-listed image+ once the export's `MAX_PDF_EXPORT_IMAGE_PIXELS` decode budget is spent+ (`UnsupportedPDFResourceError`) -- the bodies WeasyPrint would otherwise+ hand to zlib/brotli or Pillow unbounded. This processor is the one seam+ every http(s) body passes through; data:, file: and ftp: URLs never+ reach it, as ``validate_url`` refuses them before the fetch.++ Only the response is handled here; the request, ``validate_url`` and+ the refusal of redirects are left as they are.+ """++ # After HTTPErrorProcessor, which raises on any non-2xx status, so only+ # a body WeasyPrint would go on to use is read.+ handler_order = urllib.request.HTTPErrorProcessor.handler_order + 1++ def http_response(self, request, response):+ budget = _export_resource_budget.get()+ limit = MAX_PDF_RESOURCE_BYTES+ budget_bound = False+ if budget is not None and budget.remaining < limit:+ limit = budget.remaining+ budget_bound = True+ cost = _FetchCost()+ try:+ body = _decoded_body(response, limit, cost)+ except PDFResourceTooLargeError:+ which = (+ f"this export's remaining {limit} of "+ f"{MAX_PDF_EXPORT_RESOURCE_BYTES} bytes"+ if budget_bound+ else f"the {limit}-byte per-resource limit"+ )+ logger.warning(+ f"Refused a resource over {which} in PDF rendering: "+ f"{redact_url_for_log(request.full_url)}"+ )+ raise+ finally:+ safe_close(response, "PDF resource response")+ if budget is not None:+ budget.charge(cost.total())+ # The body is read and charged; now refuse one WeasyPrint would+ # decode without a size bound (a compressed web font, a font with+ # image glyphs, an oversized or non-allow-listed raster image). Raised after the charge, so+ # even a refused resource spends the budget for what reading it+ # cost, and outside the try above so its own warning stands rather+ # than the over-cap one.+ _refuse_unbounded_resource(body, request.full_url)+ del response.headers["Content-Encoding"]+ return urllib.response.addinfourl(+ io.BytesIO(body), response.headers, response.url, response.status+ )++ https_response = http_response+class _SafeUrlFetcher:- """WeasyPrint url_fetcher that blocks SSRF targets (GHSA-fj2m-qvh9-jq4q)."""+ """WeasyPrint url_fetcher that blocks SSRF targets (GHSA-fj2m-qvh9-jq4q).++ What it does fetch goes through `_URL_FETCHER`, whose+ `_BoundedBodyProcessor` bounds the size of every body. Once the+ export's budget is spent it fetches nothing at all+ (`_refuse_if_budget_spent`).+ """# WeasyPrint 70's fetch() context manager (weasyprint/urls.py) reads# ``_fail_on_errors`` off the *url_fetcher itself* after catching an@@ -151,6 +1023,7 @@ class _SafeUrlFetcher:_fail_on_errors = Falsedef __call__(self, url):+ _refuse_if_budget_spent(url)if not validate_url(url):# A rejected URL is adversarial-shaped and may carry the# operator's real credentials (RFC 3986 userinfo) or secret@@ -247,11 +1120,14 @@ class _SkippingUrlFetcher:bug. The exception *message* is never logged -- ``InvalidURL`` andfriends embed the raw URL, userinfo included.- Scope: only failures raised *inside* the fetch are skipped. WeasyPrint- 70 returns an identity or ``gzip`` body as a lazy stream that its- consumers read after the fetcher has returned, so an ``IncompleteRead``,- a corrupt gzip stream or a read timeout part-way through the body is- raised outside this object and can still fail the export.+ Scope: only failures raised *inside* the fetch are skipped, and the+ body is read inside it: `_BoundedBodyProcessor` reads and decodes every+ http(s) body before the fetch returns. So all of these fail the fetch+ itself: a body cut short (``IncompleteRead``: shorter than its+ ``Content-Length``, or a chunked body that ends early), a corrupt gzip+ or deflate stream, a read timeout part-way through the body, and a body+ over the size limits (`PDFResourceTooLargeError`, also raised before a+ fetch starts once the export's budget is spent)."""# WeasyPrint 70's fetch() context manager reads ``_fail_on_errors`` off@@ -462,6 +1338,11 @@ class PDFService:- Worker recycling after 100 requests"""_ensure_weasyprint()+ # One budget for everything this export fetches (see+ # MAX_PDF_EXPORT_RESOURCE_BYTES), reset when it ends.+ budget_token = _export_resource_budget.set(+ _ResourceBudget(MAX_PDF_EXPORT_RESOURCE_BYTES)+ )try:# Convert markdown to HTMLhtml_content = self._markdown_to_html(@@ -534,6 +1415,8 @@ class PDFService:except Exception:logger.exception("Error generating PDF")raise+ finally:+ _export_resource_budget.reset(budget_token)def _render_pdf(self, html_content, html_doc, css_list) -> bytes:"""Render, and retry once with refusals skipped rather than raised.diff --git a/tests/web/services/test_pdf_resource_size_limits.py b/tests/web/services/test_pdf_resource_size_limits.pynew file mode 100644index 0000000000..5278dcc16d--- /dev/null+++ b/tests/web/services/test_pdf_resource_size_limits.py@@ -0,0 +1,1915 @@+"""Size limits on the resources a PDF export fetches.++WeasyPrint fetches every image, stylesheet, font and other resource a+report references by URL. The module's ``URLFetcher`` carries a+``_BoundedBodyProcessor`` that reads each body inside the fetch and refuses+it once what it costs -- its body bytes, its decoded bytes if it is gzip or+deflate, and the work of decoding it -- passes ``MAX_PDF_RESOURCE_BYTES``+or what is left of the export's ``MAX_PDF_EXPORT_RESOURCE_BYTES``. Every+2xx response is charged to that budget, refused and failed ones included (a+non-2xx body is never read), and once it is spent no further request is+made. A refused resource fails like+a refused URL: WeasyPrint leaves it out and the export still completes. A+body shorter than its ``Content-Length`` fails the fetch too.++Every response comes from an in-process stub behind urllib's real HTTP+parsing (``http.client.HTTPResponse`` over an in-memory stream), patched in+as ``HTTPHandler.http_open``: no socket is opened and no name is resolved.+"""++from __future__ import annotations++import gzip+import http.client+import io+import struct+import sys+import threading+import urllib.error+import urllib.request+import warnings+import zlib+from collections.abc import Iterable+from unittest.mock import Mock, patch+from urllib.parse import urlsplit++import pytest+from loguru import logger++pytest.importorskip("weasyprint")++from local_deep_research.web.services import pdf_service # noqa: E402++ORIGIN = "http://pdf-resources.test"+MIB = 1024 * 1024+PIECE = 64 * 1024+# The decoded size of the oversized resources the renders below fetch; kept+# past MAX_PDF_RESOURCE_BYTES by test_oversized_is_past_the_limit.+OVERSIZED = 24 * MIB+++class _Stream(io.RawIOBase):+ """The server side of a connection: a response head, then body pieces.++ Pieces are produced only when read, and ``served`` counts every byte+ handed to the client, so a test can tell how much of a body was read.+ """++ def __init__(self, head: bytes, pieces: Iterable[bytes]):+ self._pending = head+ self._pieces = iter(pieces)+ self.served = 0++ def readable(self) -> bool:+ return True++ def readinto(self, target) -> int:+ while not self._pending:+ piece = next(self._pieces, None)+ if piece is None:+ return 0+ self._pending = piece+ size = min(len(target), len(self._pending))+ target[:size] = self._pending[:size]+ self._pending = self._pending[size:]+ self.served += size+ return size+++class _Socket:+ def __init__(self, stream: _Stream):+ self._stream = stream++ def makefile(self, *_args, **_kwargs):+ return io.BufferedReader(self._stream)+++def _chunked(pieces: Iterable[bytes]) -> Iterable[bytes]:+ for piece in pieces:+ yield b"%x\r\n" % len(piece) + piece + b"\r\n"+ yield b"0\r\n\r\n"+++def _zeros(total: int) -> Iterable[bytes]:+ for start in range(0, total, PIECE):+ yield b"\0" * min(PIECE, total - start)+++class _StubOrigin:+ """Canned responses for ``ORIGIN``, keyed by path and query."""++ def __init__(self):+ self._routes: dict = {}+ self.streams: dict[str, _Stream] = {}+ self.requested: list[str] = []++ def add(+ self,+ path,+ body=b"",+ *,+ headers=None,+ pieces=None,+ chunked=False,+ status="200 OK",+ ):+ """Serve ``body``, or ``pieces`` produced lazily (framed if ``chunked``).++ A ``Content-Length`` matching ``body`` is added only when ``pieces``+ is not given and ``headers`` do not name one. ``status`` is the+ status line's code and reason.+ """+ self._routes[path] = (+ body,+ dict(headers or {}),+ pieces,+ chunked,+ status,+ )++ def respond(self, request):+ # From the full URL, not request.selector, which a proxy rewrites.+ parts = urlsplit(request.full_url)+ key = parts.path + (f"?{parts.query}" if parts.query else "")+ self.requested.append(key)+ body, headers, pieces, chunked, status = self._routes[key]+ lines = [f"HTTP/1.1 {status}"]+ lines += [f"{name}: {value}" for name, value in headers.items()]+ if chunked:+ lines.append("Transfer-Encoding: chunked")+ pieces = _chunked(pieces)+ elif pieces is None:+ if "Content-Length" not in headers:+ lines.append(f"Content-Length: {len(body)}")+ pieces = [body]+ head = ("\r\n".join(lines) + "\r\n\r\n").encode("ascii")+ stream = _Stream(head, pieces)+ self.streams[key] = stream+ response = http.client.HTTPResponse(+ _Socket(stream), method=request.get_method(), url=request.full_url+ )+ response.begin()+ # What AbstractHTTPHandler.do_open sets on the response it returns.+ response.url = request.full_url+ response.msg = response.reason+ return response+++def origin():+ """The stub, reachable through the real wired fetcher and nothing else."""+ pdf_service._ensure_weasyprint() # build the real _URL_FETCHER first+ stub = _StubOrigin()++ def http_open(_handler, request):+ return stub.respond(request)++ def only_the_stub(url, *_args, **_kwargs):+ return url.startswith(ORIGIN + "/")++ def no_resolution(*_args, **_kwargs):+ raise OSError("name resolution is disabled in this test")++ with (+ patch.object(pdf_service, "validate_url", only_the_stub),+ patch.object(urllib.request.HTTPHandler, "http_open", http_open),+ patch("socket.getaddrinfo", side_effect=no_resolution),+ ):+ yield stub+++def pdf_logs():+ """This module's log lines (the package disables its loguru namespace)."""+ lines: list[str] = []++ def sink(message):+ if message.record["name"] == pdf_service.__name__:+ lines.append(str(message))++ handler_id = logger.add(sink, level="DEBUG")+ logger.enable("local_deep_research")+ try:+ yield lines+ finally:+ logger.disable("local_deep_research")+ logger.remove(handler_id)+++def _png(color: str, size: int = 0) -> bytes:+ """A 4x4 PNG, padded after its end marker to ``size`` bytes if given."""+ from PIL import Image++ out = io.BytesIO()+ Image.new("RGB", (4, 4), color).save(out, "PNG")+ image = out.getvalue()+ return image + b"\0" * max(0, size - len(image))+++def _tiny(fmt: str, mode: str = "RGB") -> bytearray:+ from PIL import Image++ out = io.BytesIO()+ Image.new(mode, (2, 2), (255, 0, 0)).save(out, fmt)+ return bytearray(out.getvalue())+++def _png_declaring(width: int, height: int) -> bytes:+ """A PNG header (signature, IHDR, IEND) declaring ``width`` x ``height``.++ No IDAT: no pixel data at all, the way Pillow's own decompression-bomb+ tests declare a size far past the body. Pillow reads the size from IHDR+ on open.+ """+ ihdr = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)+ ihdr = (+ struct.pack(">I", len(ihdr))+ + b"IHDR"+ + ihdr+ + struct.pack(">I", zlib.crc32(b"IHDR" + ihdr))+ )+ iend = (+ struct.pack(">I", 0) + b"IEND" + struct.pack(">I", zlib.crc32(b"IEND"))+ )+ return b"\x89PNG\r\n\x1a\n" + ihdr + iend+++def _gif_declaring(width: int, height: int) -> bytes:+ """A real 2x2 GIF with its logical-screen size patched to w x h."""+ gif = _tiny("GIF", "P")+ gif[6:10] = struct.pack("<HH", width, height)+ return bytes(gif)+++def _jpeg_declaring(width: int, height: int) -> bytes:+ """A real 2x2 JPEG with the size in its SOF frame header patched."""+ jpeg = _tiny("JPEG")+ at = 2+ while at < len(jpeg) - 1:+ if jpeg[at] == 0xFF and jpeg[at + 1] in (0xC0, 0xC1, 0xC2, 0xC3):+ jpeg[at + 5 : at + 7] = struct.pack(">H", height)+ jpeg[at + 7 : at + 9] = struct.pack(">H", width)+ return bytes(jpeg)+ if jpeg[at] == 0xFF and not (0xD0 <= jpeg[at + 1] <= 0xD9):+ at += 2 + struct.unpack(">H", jpeg[at + 2 : at + 4])[0]+ else:+ at += 1+ raise AssertionError("no SOF marker found") # pragma: no cover+++def _webp_declaring(width: int, height: int) -> bytes:+ """A real 2x2 lossy WebP with its frame-header size patched to w x h.++ Pillow writes a simple ``VP8 `` (lossy) body; its keyframe header carries+ the 14-bit width and height little-endian at these offsets, which is what+ libwebp reports as the image size.+ """+ webp = _tiny("WEBP")+ webp[26:28] = struct.pack("<H", width)+ webp[28:30] = struct.pack("<H", height)+ return bytes(webp)+++def _deflate_raw(data: bytes) -> bytes:+ compressor = zlib.compressobj(wbits=-zlib.MAX_WBITS)+ return compressor.compress(data) + compressor.flush()+++def _gzip_member(+ data: bytes,+ *,+ flags: int = 0,+ fields: bytes = b"",+ crc: int | None = None,+ deflated: bytes | None = None,+) -> bytes:+ """One gzip member; ``fields`` are the optional header fields ``flags``+ announce, ``crc`` overrides the CRC-32 in the trailer, ``deflated``+ the deflate data (which must decode to ``data``)."""+ head = b"\x1f\x8b\x08" + bytes([flags]) + b"\0\0\0\0\0\xff" + fields+ crc = zlib.crc32(data) if crc is None else crc+ deflated = _deflate_raw(data) if deflated is None else deflated+ return head + deflated + struct.pack("<II", crc, len(data))+++def _zlib_wrapped(deflated: bytes, data: bytes = b"") -> bytes:+ """Raw deflate data in a zlib stream; it must decode to ``data``."""+ return b"\x78\x01" + deflated + struct.pack(">I", zlib.adler32(data))+++# One unit of `_empty_dynamic_blocks`: eight blocks end on a byte boundary.+_EMPTY_BLOCKS_UNIT_BYTES = 93+++def _empty_dynamic_blocks(units: int, *, last: bytes = b"\x03\x00") -> bytes:+ """Raw deflate data of ``8 * units`` empty blocks that decodes to nothing.++ Each block carries Huffman codes of its own, as a real one does, that+ code nothing but its end: zlib builds tables from them, then reads the+ end of the block. 93 bits each. ``last`` follows them: by default the+ last block, empty, with fixed codes.+ """+ bits: list[int] = []++ def put(value: int, count: int) -> None: # least significant bit first+ bits.extend((value >> i) & 1 for i in range(count))++ order = (16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15)+ for _ in range(8):+ put(0b100, 3) # not the last block; dynamic Huffman codes+ put(0, 5) # 257 literal/length codes+ put(0, 5) # 1 distance code+ put(15, 4) # 19 code-length codes, in RFC 1951 order:+ for symbol in order:+ put(1 if symbol in (1, 18) else 0, 3) # 1 is '0', 18 is '1'+ put(1, 1)+ put(138 - 11, 7) # 18: 138 zero lengths+ put(1, 1)+ put(118 - 11, 7) # 18: 118 more, so no literal has a code+ put(0, 1) # 1: the end of the block has a 1-bit code+ put(0, 1) # 1: so has the one distance code+ put(0, 1) # the end of the block+ unit = bytes(+ sum(bit << i for i, bit in enumerate(bits[at : at + 8]))+ for at in range(0, len(bits), 8)+ )+ assert len(unit) == _EMPTY_BLOCKS_UNIT_BYTES+ return unit * units + last+++def _gunzip_as_weasyprint(wire: bytes) -> bytes:+ """What WeasyPrint's own gzip decoding (``GzipFile``) makes of ``wire``."""+ with gzip.GzipFile(fileobj=io.BytesIO(wire)) as stream:+ return stream.read()+++def _image_count(pdf: bytes) -> int:+ from pypdf import PdfReader++ return sum(len(page.images) for page in PdfReader(io.BytesIO(pdf)).pages)+++class TestOversizedBodiesAreRefused:+ def test_a_streamed_body_is_refused_once_it_passes_the_limit(self, origin):+ limit = pdf_service.MAX_PDF_RESOURCE_BYTES+ origin.add("/stream.bin", pieces=_zeros(2 * limit), chunked=True)++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/stream.bin")++ # Read up to the limit and stopped, not to the end of the stream.+ assert origin.streams["/stream.bin"].served < limit + MIB++ def test_a_declared_length_past_the_limit_is_refused_unread(self, origin):+ limit = pdf_service.MAX_PDF_RESOURCE_BYTES+ origin.add(+ "/declared.bin",+ headers={"Content-Length": str(limit + 1)},+ pieces=_zeros(limit + 1),+ )++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/declared.bin")++ assert origin.streams["/declared.bin"].served < PIECE++ def test_a_gzip_body_is_refused_on_its_decoded_size(self, origin):+ limit = pdf_service.MAX_PDF_RESOURCE_BYTES+ wire = gzip.compress(b"\0" * (limit + MIB))+ assert len(wire) < limit // 100 # small on the wire+ origin.add("/z.svg", wire, headers={"Content-Encoding": "gzip"})++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/z.svg")++ @pytest.mark.parametrize(+ "encode",+ [zlib.compress, _deflate_raw],+ ids=["zlib-wrapped", "raw"],+ )+ def test_a_deflate_body_is_refused_on_its_decoded_size(+ self, origin, encode+ ):+ limit = pdf_service.MAX_PDF_RESOURCE_BYTES+ wire = encode(b"\0" * (limit + MIB))+ assert len(wire) < limit // 100+ origin.add("/d.css", wire, headers={"Content-Encoding": "deflate"})++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/d.css")++ def test_the_refusal_is_logged_redacted_and_names_no_url(+ self, origin, pdf_logs+ ):+ path = "/figures/chart.png?token=T0K3N-9QX"+ origin.add(+ path,+ headers={+ "Content-Length": str(pdf_service.MAX_PDF_RESOURCE_BYTES + 1)+ },+ )++ with pytest.raises(pdf_service.PDFResourceTooLargeError) as refused:+ pdf_service._safe_url_fetcher(ORIGIN + path)++ assert "pdf-resources.test" not in str(refused.value)+ assert "T0K3N-9QX" not in str(refused.value)+ refusals = [line for line in pdf_logs if "Refused" in line]+ assert len(refusals) == 1, pdf_logs+ assert ORIGIN in refusals[0]+ assert "per-resource limit" in refusals[0]+ joined = "\n".join(pdf_logs)+ assert "T0K3N-9QX" not in joined+ assert "chart.png" not in joined+++class TestBodiesWithinTheLimitAreUnchanged:+ PAYLOAD = b"p { color: rebeccapurple; }\n" * 64++ @pytest.mark.parametrize(+ ("encoding", "encode"),+ [+ (None, bytes),+ ("gzip", gzip.compress),+ ("deflate", zlib.compress),+ ("deflate", _deflate_raw),+ ],+ ids=["identity", "gzip", "deflate-zlib-wrapped", "deflate-raw"],+ )+ def test_a_small_body_comes_back_whole_and_decoded(+ self, origin, encoding, encode+ ):+ headers = {"Content-Type": "text/css; charset=utf-8"}+ if encoding:+ headers["Content-Encoding"] = encoding+ origin.add("/style.css", encode(self.PAYLOAD), headers=headers)++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/style.css")+ try:+ assert resource.read() == self.PAYLOAD+ assert resource.content_type == "text/css"+ assert resource.charset == "utf-8"+ assert resource.url == ORIGIN + "/style.css"+ assert "Content-Encoding" not in resource.headers+ finally:+ resource.close()++ # Bodies WeasyPrint's GzipFile accepts, in the shapes a decoder can get+ # wrong: concatenated members with zero padding between and after them,+ # every optional header field (with a header CRC GzipFile does not+ # check, and reserved flag bits it ignores), and no body at all.+ @pytest.mark.parametrize(+ "wire",+ [+ gzip.compress(PAYLOAD)+ + b"\0" * 100+ + gzip.compress(b"/* second member */")+ + b"\0" * 7,+ _gzip_member(+ PAYLOAD,+ flags=0xE0 | 2 | 4 | 8 | 16, # reserved, FHCRC, FEXTRA,+ # FNAME, FCOMMENT; then those fields in that order+ fields=struct.pack("<H", 3)+ + b"xyz"+ + b"style.css\0"+ + b"a comment\0"+ + b"\xef\xbe",+ ),+ b"",+ ],+ ids=["members-and-padding", "optional-header-fields", "empty"],+ )+ def test_gzip_is_decoded_as_weasyprint_decodes_it(self, origin, wire):+ origin.add("/g.css", wire, headers={"Content-Encoding": "gzip"})++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/g.css")+ try:+ assert resource.read() == _gunzip_as_weasyprint(wire)+ finally:+ resource.close()++ @pytest.mark.parametrize(+ "wire",+ [+ _gzip_member(PAYLOAD, crc=0),+ gzip.compress(PAYLOAD) + b"junk",+ gzip.compress(PAYLOAD)[:-3],+ b"\0" + gzip.compress(PAYLOAD),+ ],+ ids=["bad-crc", "trailing-garbage", "cut-in-trailer", "leading-zero"],+ )+ def test_gzip_that_weasyprint_rejects_is_rejected(self, origin, wire):+ corrupt = (OSError, EOFError, zlib.error)+ with pytest.raises(corrupt):+ _gunzip_as_weasyprint(wire)+ origin.add("/g.css", wire, headers={"Content-Encoding": "gzip"})++ with pytest.raises(corrupt):+ pdf_service._safe_url_fetcher(ORIGIN + "/g.css")+++class TestTruncatedBodies:+ BODY = b"p { color: red; }\n" * 8++ def test_a_body_shorter_than_its_content_length_fails_the_fetch(+ self, origin+ ):+ # http.client's sized reads return these bytes as if complete.+ origin.add(+ "/cut.css",+ self.BODY,+ headers={"Content-Length": str(len(self.BODY) + 1)},+ )+ budget = pdf_service._ResourceBudget(MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(http.client.IncompleteRead):+ pdf_service._safe_url_fetcher(ORIGIN + "/cut.css")+ finally:+ pdf_service._export_resource_budget.reset(token)++ # Charged for what was read, though nothing was returned.+ assert budget.remaining == MIB - len(self.BODY)++ def test_a_chunked_body_cut_short_fails_the_fetch(self, origin):+ origin.add(+ "/cut.css",+ headers={"Transfer-Encoding": "chunked"},+ # One chunk, and no final zero-length chunk.+ pieces=[b"%x\r\n" % len(self.BODY) + self.BODY + b"\r\n"],+ )++ with pytest.raises(http.client.IncompleteRead):+ pdf_service._safe_url_fetcher(ORIGIN + "/cut.css")+++class TestTheLimitsAreWired:+ def test_the_processor_reads_after_the_status_check(self):+ pdf_service._ensure_weasyprint()++ for protocol in ("http", "https"):+ kinds = [+ type(processor)+ for processor in pdf_service._URL_FETCHER.process_response[+ protocol+ ]+ ]+ assert pdf_service._BoundedBodyProcessor in kinds, protocol+ assert kinds.index(urllib.request.HTTPErrorProcessor) < kinds.index(+ pdf_service._BoundedBodyProcessor+ ), protocol+++class TestExportsLeaveOutOnlyTheOversizedResource:+ def test_oversized_is_past_the_limit(self):+ assert OVERSIZED > pdf_service.MAX_PDF_RESOURCE_BYTES++ def test_a_small_image_renders_and_an_oversized_one_is_left_out(+ self, origin+ ):+ origin.add(+ "/small.png", _png("red"), headers={"Content-Type": "image/png"}+ )+ # A valid image, padded past the limit, gzip-encoded on the wire.+ origin.add(+ "/large.png",+ gzip.compress(_png("blue", size=OVERSIZED)),+ headers={"Content-Type": "image/png", "Content-Encoding": "gzip"},+ )+ markdown = (+ f"\n\n"+ f"\n\n"+ "Body text.\n"+ )++ pdf = pdf_service.PDFService().markdown_to_pdf(markdown, title="t")++ assert pdf.startswith(b"%PDF")+ assert sorted(origin.requested) == ["/large.png", "/small.png"]+ assert _image_count(pdf) == 1++ def test_an_oversized_color_profile_does_not_fail_the_export(+ self, origin, pdf_logs+ ):+ # The one at-rule whose failed fetch aborts a render: the export+ # must still complete, through the retry in _render_pdf.+ origin.add(+ "/profile.icc",+ gzip.compress(b"\0" * OVERSIZED),+ headers={"Content-Encoding": "gzip"},+ )+ markdown = (+ "<style>@color-profile --p { "+ f"src: url('{ORIGIN}/profile.icc'); components: 3; }}</style>\n\n"+ "# Report\n\nBody text.\n"+ )++ pdf = pdf_service.PDFService().markdown_to_pdf(markdown, title="t")++ assert pdf.startswith(b"%PDF")+ # Refused on the first pass, which it aborted, and again on the+ # retry, which skipped it.+ assert origin.requested == ["/profile.icc", "/profile.icc"]+ refusals = [line for line in pdf_logs if "Refused" in line]+ assert len(refusals) == 2, pdf_logs+ skips = [+ line+ for line in pdf_logs+ if "Skipping unavailable resource" in line+ and "(PDFResourceTooLargeError)" in line+ ]+ assert len(skips) == 1, pdf_logs+++class TestExportBudget:+ COLORS = {"a": "green", "b": "yellow", "c": "purple"}++ def test_a_body_refused_before_it_is_read_is_charged_its_first_read(+ self, origin+ ):+ for name, color in self.COLORS.items():+ origin.add(f"/{name}.png", _png(color, size=40 * 1024))+ budget = pdf_service._ResourceBudget(MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with patch.object(pdf_service, "MAX_PDF_RESOURCE_BYTES", 48 * 1024):+ for name in ("a", "b"):+ pdf_service._safe_url_fetcher(+ f"{ORIGIN}/{name}.png"+ ).close()+ assert budget.remaining == MIB - 80 * 1024+ with patch.object(+ pdf_service, "MAX_PDF_RESOURCE_BYTES", 32 * 1024+ ):+ # Its Content-Length is past the limit: refused unread.+ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(f"{ORIGIN}/c.png")+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert origin.streams["/c.png"].served < PIECE+ # Charged what its first read would have asked for: one byte past+ # the 32 KiB limit, as a body of undeclared size would have been.+ # Reverting the charge in _read_body leaves this at MIB - 80 KiB.+ assert budget.remaining == MIB - 80 * 1024 - (32 * 1024 + 1)++ def test_declared_oversized_bodies_spend_the_budget_so_fetches_stop(+ self, origin+ ):+ # Every one declares a Content-Length past the 16 MiB limit and is+ # refused unread at one read chunk (64 KiB) each, so a 1 MiB budget+ # admits 16 requests and no more, however many are referenced.+ urls = [f"/huge.bin?{i}" for i in range(40)]+ for url in urls:+ origin.add(+ url,+ headers={"Content-Length": str(64 * MIB)},+ pieces=_zeros(PIECE),+ )+ budget = pdf_service._ResourceBudget(MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ for url in urls:+ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + url)+ finally:+ pdf_service._export_resource_budget.reset(token)++ # Reverting the charge makes all 40 requests (they cost nothing).+ assert origin.requested == urls[: MIB // PIECE]+ assert budget.remaining == 0+ # The changelog's figure: at most this many such requests an export.+ assert (+ pdf_service.MAX_PDF_EXPORT_RESOURCE_BYTES+ // pdf_service._RESOURCE_READ_CHUNK_BYTES+ == 1024+ )++ @pytest.mark.parametrize(+ "status",+ ["302 Found", "404 Not Found", "500 Internal Server Error"],+ ids=["3xx", "4xx", "5xx"],+ )+ def test_a_non_2xx_body_is_never_read(self, origin, status):+ # Redirects are refused and error statuses raise in+ # HTTPErrorProcessor, before _BoundedBodyProcessor runs: the body,+ # however large, is not read, decoded or classified, and (as the+ # changelog says) costs the budget nothing.+ origin.add(+ "/status.png",+ headers={+ "Content-Length": str(4 * MIB),+ "Location": ORIGIN + "/elsewhere.png",+ },+ pieces=_zeros(4 * MIB),+ status=status,+ )+ budget = pdf_service._ResourceBudget(MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(urllib.error.HTTPError) as failed:+ pdf_service._safe_url_fetcher(ORIGIN + "/status.png")+ failed.value.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert failed.value.code == int(status.split()[0])+ assert origin.requested == ["/status.png"]+ assert origin.streams["/status.png"].served < PIECE+ assert budget.remaining == MIB++ def test_refused_bodies_spend_the_budget_so_later_ones_are_not_fetched(+ self, origin+ ):+ # Many references to oversized streams, differing only in the query+ # string: each is read up to its limit before it is refused.+ limit, total, count = MIB, 2 * MIB, 8+ urls = [f"/big.bin?{i}" for i in range(count)]+ for url in urls:+ origin.add(url, pieces=_zeros(4 * MIB), chunked=True)+ budget = pdf_service._ResourceBudget(total)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with patch.object(pdf_service, "MAX_PDF_RESOURCE_BYTES", limit):+ for url in urls:+ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + url)+ finally:+ pdf_service._export_resource_budget.reset(token)++ # The first two spend the budget, each read to one byte past what+ # it may cost; no request is made for the others.+ assert budget.remaining == 0+ assert origin.requested == urls[:2]+ served = [origin.streams[url].served for url in urls[:2]]+ assert served[0] > limit+ # The budget and one byte of body; besides, the heads, the chunk+ # framing and one piece each that the client buffered ahead.+ assert sum(served) < total + 2 * 2 * PIECE, served++ def test_no_request_is_made_once_the_budget_is_spent(+ self, origin, pdf_logs+ ):+ paths = [f"/{name}.png?token=T0K3N-9QX" for name in ("a", "b", "c")]+ for path in paths:+ origin.add(path, _png("red"))+ budget = pdf_service._ResourceBudget(0)+ checked = Mock(wraps=pdf_service.validate_url)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with patch.object(pdf_service, "validate_url", checked):+ for path in paths:+ with pytest.raises(+ pdf_service.PDFResourceTooLargeError+ ) as refused:+ pdf_service._safe_url_fetcher(ORIGIN + path)+ finally:+ pdf_service._export_resource_budget.reset(token)++ # Refused before the URL is checked (which resolves its host), so+ # not even a name is looked up, let alone a request made.+ assert origin.requested == []+ checked.assert_not_called()+ assert "pdf-resources.test" not in str(refused.value)+ assert "T0K3N-9QX" not in str(refused.value)+ # Warned once for the export, not once per resource; redacted.+ warnings = [line for line in pdf_logs if "without fetching it" in line]+ assert len(warnings) == 1, pdf_logs+ assert ORIGIN in warnings[0]+ assert "T0K3N-9QX" not in "\n".join(pdf_logs)++ def test_a_read_asks_for_at_most_one_byte_past_what_is_left(self, origin):+ # No Content-Length: a body read until the connection closes, which+ # the server sends in 1 KiB pieces.+ origin.add("/tail.bin", pieces=[b"\0" * 1024] * 256)+ budget = pdf_service._ResourceBudget(10)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/tail.bin")+ finally:+ pdf_service._export_resource_budget.reset(token)++ # One read of 11 bytes, so the client took in the head and the+ # first piece only, rather than a whole chunk of the body.+ assert origin.streams["/tail.bin"].served < 2 * 1024+ assert budget.remaining == 0++ def test_a_read_that_fails_part_way_is_charged(self, origin):+ # Its one chunk announces 64 KiB, then the body ends after 1 KiB:+ # http.client raises inside the read and returns none of it.+ origin.add(+ "/cut.bin",+ headers={"Transfer-Encoding": "chunked"},+ pieces=[b"%x\r\n" % PIECE + b"\0" * 1024],+ )+ budget = pdf_service._ResourceBudget(MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(http.client.IncompleteRead):+ pdf_service._safe_url_fetcher(ORIGIN + "/cut.bin")+ finally:+ pdf_service._export_resource_budget.reset(token)++ # Charged what the read asked for, the most it could have read.+ assert MIB - budget.remaining == PIECE++ def test_an_export_stops_fetching_once_its_budget_is_spent(+ self, origin, pdf_logs+ ):+ paths = [f"/big.png?{i}" for i in range(10)]+ for path in paths:+ origin.add(+ path,+ headers={"Content-Type": "image/png"},+ pieces=_zeros(MIB),+ chunked=True,+ )+ markdown = "".join(f"\n\n" for path in paths)++ with (+ patch.object(pdf_service, "MAX_PDF_RESOURCE_BYTES", 64 * 1024),+ patch.object(+ pdf_service, "MAX_PDF_EXPORT_RESOURCE_BYTES", 256 * 1024+ ),+ ):+ pdf = pdf_service.PDFService().markdown_to_pdf(markdown, title="t")++ assert pdf.startswith(b"%PDF")+ # Each fetch is refused one byte past what it may read: four spend+ # the budget, and the other six images are never requested.+ assert origin.requested == paths[:4]+ warnings = [line for line in pdf_logs if "without fetching it" in line]+ assert len(warnings) == 1, pdf_logs++ def test_an_undecodable_body_is_charged_what_was_decoded(self, origin):+ wire = bytearray(zlib.compress(b"\0" * (512 * 1024)))+ wire[-1] ^= 0xFF # a bad Adler-32: zlib rejects it at the very end+ origin.add(+ "/bad.css", bytes(wire), headers={"Content-Encoding": "deflate"}+ )+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(zlib.error):+ pdf_service._safe_url_fetcher(ORIGIN + "/bad.css")+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert len(wire) < 4 * 1024+ assert 16 * MIB - budget.remaining >= 512 * 1024 - PIECE++ def test_a_gzip_body_that_decodes_to_nothing_is_charged_its_wire_size(+ self, origin+ ):+ # One empty member, then zero padding: decodes to no bytes at all.+ wire = gzip.compress(b"") + b"\0" * (256 * 1024)+ origin.add("/empty.svg", wire, headers={"Content-Encoding": "gzip"})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/empty.svg")+ assert resource.read() == b""+ resource.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert 16 * MIB - budget.remaining == len(wire)++ def test_gzip_members_are_charged_however_small(self, origin):+ members = 100+ wire = gzip.compress(b"") * members # 20 bytes each+ origin.add("/empty.svg", wire, headers={"Content-Encoding": "gzip"})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/empty.svg")+ assert resource.read() == b""+ resource.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ charged = 16 * MIB - budget.remaining+ assert charged > len(wire)+ # Each member: its own charge, and the work of its deflate data.+ deflated = len(gzip.compress(b"")) - 18 # less header and trailer+ assert charged == members * (+ pdf_service._GZIP_MEMBER_COST_BYTES+ + deflated * pdf_service._INFLATED_BYTE_COST+ )++ def test_a_gzip_body_of_too_many_members_is_refused(self, origin):+ members = 5000+ wire = gzip.compress(b"") * members # 100,000 bytes, decodes to b""+ origin.add("/members.svg", wire, headers={"Content-Encoding": "gzip"})++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/members.svg")++ # Refused for its members alone: it is small on the wire.+ assert len(wire) < pdf_service.MAX_PDF_RESOURCE_BYTES // 100+ assert (+ members * pdf_service._GZIP_MEMBER_COST_BYTES+ > pdf_service.MAX_PDF_RESOURCE_BYTES+ )++ def test_each_export_has_its_own_budget(self, origin, pdf_logs):+ for name, color in self.COLORS.items():+ origin.add(+ f"/{name}.png",+ _png(color, size=40 * 1024),+ headers={"Content-Type": "image/png"},+ )+ markdown = "".join(+ f"\n\n" for name in self.COLORS+ )+ service = pdf_service.PDFService()++ with (+ patch.object(pdf_service, "MAX_PDF_RESOURCE_BYTES", 64 * 1024),+ patch.object(+ pdf_service, "MAX_PDF_EXPORT_RESOURCE_BYTES", 100 * 1024+ ),+ ):+ first = service.markdown_to_pdf(markdown, title="t")+ second = service.markdown_to_pdf(markdown, title="t")++ # Two of the three fit in each export's budget; the second export+ # starts from a full budget, not from what the first one left.+ assert _image_count(first) == 2+ assert _image_count(second) == 2+ assert pdf_service._export_resource_budget.get() is None+ budget_refusals = [+ line for line in pdf_logs if "this export's remaining" in line+ ]+ assert len(budget_refusals) == 2, pdf_logs+++class TestDecodingIsChargedForItsWork:+ """Decoding is charged for the compressed data zlib takes in, not only+ for what it puts out: a body of empty deflate blocks, each with Huffman+ codes of its own to build tables from, decodes to nothing and still+ costs CPU for every byte of it."""++ @pytest.mark.parametrize("encoding", ["deflate", "gzip"])+ def test_empty_blocks_are_charged_for_the_work(self, origin, encoding):+ blocks = _empty_dynamic_blocks(352) # 32,738 bytes+ if encoding == "gzip":+ wire = _gzip_member(b"", deflated=blocks)+ # zlib takes in the deflate data; the member is charged too.+ work = (+ len(blocks) * pdf_service._INFLATED_BYTE_COST+ + pdf_service._GZIP_MEMBER_COST_BYTES+ )+ else:+ wire = _zlib_wrapped(blocks)+ work = len(wire) * pdf_service._INFLATED_BYTE_COST+ origin.add("/b.css", wire, headers={"Content-Encoding": encoding})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/b.css")+ assert resource.read() == b""+ resource.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert 16 * MIB - budget.remaining == work+ assert work > 10 * len(wire)++ def test_a_body_of_empty_blocks_is_refused_on_the_work_alone(self, origin):+ limit = pdf_service.MAX_PDF_RESOURCE_BYTES+ units = (+ limit // pdf_service._INFLATED_BYTE_COST // _EMPTY_BLOCKS_UNIT_BYTES+ + 100+ )+ wire = _zlib_wrapped(_empty_dynamic_blocks(units))+ # A valid body that decodes to nothing and is small on the wire.+ assert zlib.decompress(wire) == b""+ assert len(wire) < limit // 10+ origin.add("/b.css", wire, headers={"Content-Encoding": "deflate"})++ with pytest.raises(pdf_service.PDFResourceTooLargeError):+ pdf_service._safe_url_fetcher(ORIGIN + "/b.css")++ def test_a_raising_step_is_charged_its_work_not_unproduced_output(+ self, origin+ ):+ # Empty blocks, then a block of a type that does not exist: zlib+ # builds every table, then raises in the same call, and the retry+ # as raw deflate raises at once. The whole body fits one input+ # chunk, so each attempt is one failed step. A failed step returns+ # no output, so the 64 KiB of output charged in advance is refunded+ # and only the work stays charged: every byte zlib was fed, twice.+ # Without the refund in _inflate this is 2 * PIECE (131072).+ wire = _zlib_wrapped(_empty_dynamic_blocks(30, last=b"\x07"))+ assert len(wire) < pdf_service._INFLATE_INPUT_CHUNK_BYTES+ origin.add("/b.css", wire, headers={"Content-Encoding": "deflate"})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(zlib.error):+ pdf_service._safe_url_fetcher(ORIGIN + "/b.css")+ finally:+ pdf_service._export_resource_budget.reset(token)++ charged = 16 * MIB - budget.remaining+ work = 2 * len(wire) * pdf_service._INFLATED_BYTE_COST+ assert charged == work+ assert charged < 2 * PIECE++ def test_a_raw_deflate_body_is_not_charged_the_failed_zlib_attempt(+ self, origin+ ):+ # Raw deflate under "Content-Encoding: deflate": the zlib-wrapped+ # attempt fails on its first step, then the raw retry decodes it.+ # Charged what it decoded, not that plus the 64 KiB the failed step+ # was charged in advance (the over-charge the refund removes).+ data = b"a small stylesheet, sent as raw deflate. " * 100+ wire = _deflate_raw(data)+ origin.add("/raw.css", wire, headers={"Content-Encoding": "deflate"})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/raw.css")+ assert resource.read() == data+ resource.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ charged = 16 * MIB - budget.remaining+ assert charged == max(+ len(data), 2 * len(wire) * pdf_service._INFLATED_BYTE_COST+ )+ assert charged < PIECE++ def test_the_work_charge_dominates_when_a_step_raises(self, origin):+ # About 5.5 KiB of empty blocks (> 64 KiB / 12), then a block of a+ # type that does not exist. As a single gzip member the body is+ # decoded once (no deflate zlib-then-raw retry), so the arithmetic+ # is exact: every byte fed to zlib is charged, the one failed step's+ # byte included, and the gzip header (10 bytes) is the only part not+ # fed. The work charge then exceeds the one 64 KiB the failed step+ # adds to the decoded-output charge, so it sets the total. Moving+ # `cost.inflated += len(piece)` after `inflater.decompress(...)`+ # drops the failed step's bytes and this assertion fails.+ blocks = _empty_dynamic_blocks(60, last=b"\x07")+ wire = _gzip_member(b"", deflated=blocks)+ origin.add("/b.css", wire, headers={"Content-Encoding": "gzip"})+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(zlib.error):+ pdf_service._safe_url_fetcher(ORIGIN + "/b.css")+ finally:+ pdf_service._export_resource_budget.reset(token)++ charged = 16 * MIB - budget.remaining+ inflated = len(wire) - 10 # the 10-byte gzip header is not fed+ work = (+ inflated * pdf_service._INFLATED_BYTE_COST+ + pdf_service._GZIP_MEMBER_COST_BYTES+ )+ assert charged == work+ assert work > PIECE # the work dominates the one step's 64 KiB+++# A raster side whose square is well past MAX_PDF_IMAGE_PIXELS (and under+# Pillow's own ~89 M-pixel warning, so opening the header stays quiet).+_OVERSIZED_SIDE = 8000+++class TestResourcesWeasyPrintWouldDecodeUnbounded:+ """Bodies the byte caps do not bound, because WeasyPrint decodes them+ after the fetch: a compressed web font (handed to zlib/brotli) and a+ raster image (handed to Pillow). These are refused inside the fetch, so+ the resource is left out and the export still completes.++ The fixtures are headers only: font magic plus padding, image headers+ that declare a size far past the body, and magic-only bodies for the+ refused formats. None carries compressed data that expands.+ """++ def test_the_refusal_is_the_same_family_as_an_over_cap_refusal(self):+ # So every handler that skips a PDFResourceTooLargeError skips this.+ assert issubclass(+ pdf_service.UnsupportedPDFResourceError,+ pdf_service.PDFResourceTooLargeError,+ )++ @pytest.mark.parametrize("magic", [b"wOFF", b"wOF2"], ids=["woff", "woff2"])+ def test_a_compressed_web_font_is_refused(self, origin, magic):+ origin.add("/f.woff", magic + b"\0" * 2048)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/f.woff")++ @pytest.mark.parametrize(+ "magic",+ [b"\x00\x01\x00\x00", b"OTTO", b"true", b"ttcf"],+ ids=["ttf", "otf", "true", "ttc"],+ )+ def test_an_uncompressed_font_is_left_alone(self, origin, magic):+ # TTF/OTF/TTC are not compressed: with no table that holds glyph+ # images (here, an empty table directory), they come back whole for+ # WeasyPrint to hand to Fontconfig.+ body = magic + b"\0" * 2048+ origin.add("/f.ttf", body)++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/f.ttf")+ try:+ assert resource.read() == body+ finally:+ resource.close()++ @pytest.mark.parametrize(+ ("name", "body"),+ [+ ("png", _png_declaring(_OVERSIZED_SIDE, _OVERSIZED_SIDE)),+ ("gif", _gif_declaring(_OVERSIZED_SIDE, _OVERSIZED_SIDE)),+ ("jpeg", _jpeg_declaring(_OVERSIZED_SIDE, _OVERSIZED_SIDE)),+ ("webp", _webp_declaring(_OVERSIZED_SIDE, _OVERSIZED_SIDE)),+ ],+ )+ def test_a_raster_declaring_too_many_pixels_is_refused(+ self, origin, name, body+ ):+ assert _OVERSIZED_SIDE**2 > pdf_service.MAX_PDF_IMAGE_PIXELS+ assert len(body) < pdf_service.MAX_PDF_RESOURCE_BYTES # small on wire+ origin.add(f"/big.{name}", body, headers={"Content-Type": "image/*"})++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + f"/big.{name}")++ def test_a_png_past_pillows_own_bomb_ceiling_is_refused(self, origin):+ # Declares 400 M pixels, past Pillow's own bomb ceiling. The size+ # probe runs the PNG factory directly, so Pillow's bomb check never+ # runs; the per-image cap refuses it.+ origin.add("/huge.png", _png_declaring(20000, 20000))++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/huge.png")++ @pytest.mark.parametrize(+ ("fmt", "mode"),+ [("PNG", "RGB"), ("GIF", "P"), ("JPEG", "RGB"), ("WEBP", "RGB")],+ )+ def test_a_small_image_of_each_allowed_format_comes_back_whole(+ self, origin, fmt, mode+ ):+ body = bytes(_tiny(fmt, mode))+ origin.add("/ok.img", body, headers={"Content-Type": "image/*"})++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/ok.img")+ try:+ assert resource.read() == body+ finally:+ resource.close()++ @pytest.mark.parametrize(+ ("name", "magic"),+ [+ ("ico", b"\x00\x00\x01\x00"),+ ("icns", b"icns"),+ ("blp", b"BLP2"),+ ("bmp", b"BM"),+ ("tiff", b"II*\x00"),+ ("dds", b"DDS "),+ ],+ )+ def test_a_non_allowlisted_raster_format_is_refused(+ self, origin, name, magic+ ):+ # Magic only: enough for Pillow to recognise the format, so the body+ # is refused before any Image.open that could decode on the open.+ origin.add(f"/x.{name}", magic + b"\0" * 64)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + f"/x.{name}")++ def test_an_svg_body_is_left_alone(self, origin):+ # SVG is vector XML, rendered without Pillow; it must pass through.+ body = b"<svg xmlns='http://www.w3.org/2000/svg'><rect/></svg>"+ origin.add(+ "/vector.svg", body, headers={"Content-Type": "image/svg+xml"}+ )++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/vector.svg")+ try:+ assert resource.read() == body+ finally:+ resource.close()++ def test_a_refused_font_still_spends_the_budget(self, origin):+ # The refusal is raised after the body is read and charged, so a+ # report of many refused fonts cannot fetch without bound.+ body = b"wOF2" + b"\0" * (120 * 1024)+ origin.add("/f.woff2", body)+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/f.woff2")+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert 16 * MIB - budget.remaining == len(body)++ def test_the_refusal_is_logged_without_the_url(self, origin, pdf_logs):+ path = "/fonts/secret.woff2?token=T0K3N-9QX"+ origin.add(path, b"wOF2" + b"\0" * 512)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError) as refused:+ pdf_service._safe_url_fetcher(ORIGIN + path)++ assert "pdf-resources.test" not in str(refused.value)+ assert "T0K3N-9QX" not in str(refused.value)+ refusals = [line for line in pdf_logs if "Refused a compressed" in line]+ assert len(refusals) == 1, pdf_logs+ assert ORIGIN in refusals[0]+ joined = "\n".join(pdf_logs)+ assert "T0K3N-9QX" not in joined+ assert "secret.woff2" not in joined++ def test_a_report_renders_with_an_oversized_image_left_out(self, origin):+ # A PNG that is tiny on the wire but declares a huge size: the kind+ # WeasyPrint would otherwise decode to that full size. It is refused,+ # so the export completes with only the small image.+ origin.add(+ "/small.png", _png("red"), headers={"Content-Type": "image/png"}+ )+ origin.add(+ "/big.png",+ _png_declaring(_OVERSIZED_SIDE, _OVERSIZED_SIDE),+ headers={"Content-Type": "image/png"},+ )+ markdown = (+ f"\n\n"+ f"\n\n"+ "Body text.\n"+ )++ pdf = pdf_service.PDFService().markdown_to_pdf(markdown, title="t")++ assert pdf.startswith(b"%PDF")+ assert sorted(origin.requested) == ["/big.png", "/small.png"]+ assert _image_count(pdf) == 1+++class TestExportImageDecodeBudget:+ """Bound the TOTAL pixels an export's accepted images declare, not just+ each image. A highly compressible image (a flat colour, a simple chart)+ is tiny on the wire yet costs the same to decode as a detailed one its+ size, so without this a report of thousands of distinct-URL images, each+ under the per-image cap, could impose unbounded total decode work.++ Header-only fixtures only: images that DECLARE a size, with no pixel data.+ """++ def test_an_accepted_image_charges_its_declared_pixels(self, origin):+ origin.add(+ "/img.png",+ _png_declaring(1234, 567),+ headers={"Content-Type": "image/png"},+ )+ budget = pdf_service._ResourceBudget(16 * MIB)+ start = budget.image_pixels+ token = pdf_service._export_resource_budget.set(budget)+ try:+ pdf_service._safe_url_fetcher(ORIGIN + "/img.png").close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert start == pdf_service.MAX_PDF_EXPORT_IMAGE_PIXELS+ assert start - budget.image_pixels == 1234 * 567++ def test_images_past_the_export_pixel_budget_are_refused(self, origin):+ for i in range(3):+ origin.add(+ f"/i{i}.png",+ _png_declaring(2000, 2000), # 4 M pixels each+ headers={"Content-Type": "image/png"},+ )+ # Room for two of the three 4 M-pixel images, not the third.+ budget = pdf_service._ResourceBudget(+ 16 * MIB, image_pixels=10 * 1024 * 1024+ )+ token = pdf_service._export_resource_budget.set(budget)+ try:+ pdf_service._safe_url_fetcher(ORIGIN + "/i0.png").close()+ pdf_service._safe_url_fetcher(ORIGIN + "/i1.png").close()+ # Two 4 M-pixel images fit; the budget is now below one more.+ after_two = budget.image_pixels+ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/i2.png")+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert after_two == 10 * 1024 * 1024 - 2 * (2000 * 2000)+ # The over-budget third image spends the rest, so later ones are+ # refused too.+ assert budget.image_pixels == 0+ # All three are fetched: the budget check reads the header first.+ assert origin.requested == ["/i0.png", "/i1.png", "/i2.png"]++ def test_the_image_budget_is_reported_once(self, origin, pdf_logs):+ for i in range(3):+ origin.add(+ f"/x{i}.png?token=T0K3N-9QX",+ _png_declaring(2000, 2000),+ headers={"Content-Type": "image/png"},+ )+ budget = pdf_service._ResourceBudget(16 * MIB, image_pixels=1024)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ for i in range(3):+ with pytest.raises(+ pdf_service.UnsupportedPDFResourceError+ ) as refused:+ pdf_service._safe_url_fetcher(+ f"{ORIGIN}/x{i}.png?token=T0K3N-9QX"+ )+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert "T0K3N-9QX" not in str(refused.value)+ spent = [+ line for line in pdf_logs if "image-decode budget is spent" in line+ ]+ assert len(spent) == 1, pdf_logs+ assert ORIGIN in spent[0]+ assert "T0K3N-9QX" not in "\n".join(pdf_logs)++ def test_an_ordinary_megapixel_image_is_accepted(self, origin):+ # 3000x3000 = 9 M pixels: an ordinary photo/screenshot. Over the old+ # 4 M-pixel per-image cap, within the current 16 M one.+ assert 3000 * 3000 > 4 * 1024 * 1024+ assert 3000 * 3000 <= pdf_service.MAX_PDF_IMAGE_PIXELS+ origin.add(+ "/photo.png",+ _png_declaring(3000, 3000),+ headers={"Content-Type": "image/png"},+ )+ budget = pdf_service._ResourceBudget(16 * MIB)+ token = pdf_service._export_resource_budget.set(budget)+ try:+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/photo.png")+ resource.close()+ finally:+ pdf_service._export_resource_budget.reset(token)++ assert (+ pdf_service.MAX_PDF_EXPORT_IMAGE_PIXELS - budget.image_pixels+ == 3000 * 3000+ )++ def test_a_report_completes_when_images_pass_the_export_budget(+ self, origin+ ):+ colors = {"a": "red", "b": "green", "c": "blue"}+ for name, color in colors.items():+ origin.add(+ f"/{name}.png",+ _png(color),+ headers={"Content-Type": "image/png"},+ )+ markdown = "".join(f"\n\n" for n in colors)++ # Each 4x4 image declares 16 pixels; a 32-pixel budget admits two.+ with patch.object(pdf_service, "MAX_PDF_EXPORT_IMAGE_PIXELS", 32):+ pdf = pdf_service.PDFService().markdown_to_pdf(markdown, title="t")++ assert pdf.startswith(b"%PDF")+ # All three are fetched (the budget check reads each header first).+ assert sorted(origin.requested) == ["/a.png", "/b.png", "/c.png"]+ # Only the first two are embedded; the third is over budget.+ assert _image_count(pdf) == 2+++_FORMATS_WITHOUT_SNIFFERS = ("IM", "IMT", "IPTC", "PCD", "SPIDER", "TGA")+++def _raster_without_sniffer(fmt: str) -> bytes:+ """Small valid images, or headers when Pillow has no encoder."""+ from PIL import Image++ if fmt in {"IM", "SPIDER", "TGA"}:+ out = io.BytesIO()+ mode = "F" if fmt == "SPIDER" else "RGB"+ options = {"compression": "tga_rle"} if fmt == "TGA" else {}+ Image.new(mode, (2, 2)).save(out, fmt, **options)+ return out.getvalue()+ if fmt == "IMT":+ return b"width 2\nheight 2\npixel n8\n\x0c" + bytes(4)+ if fmt == "IPTC":++ def field(record, tag, data):+ return (+ bytes([28, record, tag]) + len(data).to_bytes(2, "big") + data+ )++ return (+ field(3, 60, b"\1\0") # one grayscale component+ + field(3, 20, b"\2") # width+ + field(3, 30, b"\2") # height+ + field(3, 120, b"\1") # raw compression+ + field(8, 10, bytes(4))+ )+ assert fmt == "PCD"+ # PCD's header fixes the size at 768x512; no pixel data is included.+ return bytes(2048) + b"PCD_" + bytes(1535)+++class TestPillowFormatBoundary:+ def test_every_format_without_a_sniffer_has_a_fixture(self):+ from PIL import Image++ Image.init()+ formats = {+ name for name, (_, accept) in Image.OPEN.items() if accept is None+ }+ # A Pillow upgrade adding another such plugin needs a header-only+ # review and a real fixture, not an unchecked call to its factory.+ assert formats == set(_FORMATS_WITHOUT_SNIFFERS)++ @pytest.mark.parametrize("fmt", _FORMATS_WITHOUT_SNIFFERS)+ def test_a_format_without_a_sniffer_is_refused_without_decoding(+ self, origin, fmt+ ):+ from PIL import Image, ImageFile++ body = _raster_without_sniffer(fmt)+ with Image.open(io.BytesIO(body)) as image:+ assert image.format == fmt+ assert min(image.size) > 0+ assert Image.OPEN[fmt][1] is None+ origin.add("/image.bin", body)++ # Even classification must not load any pixel data.+ with (+ patch.object(Image.Image, "load") as image_load,+ patch.object(ImageFile.ImageFile, "load") as file_load,+ pytest.raises(pdf_service.UnsupportedPDFResourceError),+ ):+ pdf_service._safe_url_fetcher(ORIGIN + "/image.bin")+ image_load.assert_not_called()+ file_load.assert_not_called()++ @pytest.mark.parametrize("fmt", ["TGA", "IM"])+ def test_render_omits_an_unapproved_image_but_keeps_the_png(+ self, origin, fmt+ ):+ from PIL import Image++ body = _raster_without_sniffer(fmt)+ # Complete, decodable 2x2 images: a successful PDF alone would not+ # prove they were refused. Inspect its embedded image count too.+ with Image.open(io.BytesIO(body)) as image:+ image.load()+ origin.add("/unapproved", body)+ origin.add("/approved", _png("red"))++ pdf = pdf_service.PDFService().markdown_to_pdf(+ f"\n\n"+ f"\n\nBody text.",+ title="t",+ )++ assert pdf.startswith(b"%PDF")+ assert sorted(origin.requested) == ["/approved", "/unapproved"]+ assert _image_count(pdf) == 1++ @pytest.mark.parametrize(+ "body",+ [+ b"\x89PNG\r\n\x1a\n",+ b"\xff\xd8\xff",+ b"GIF89a",+ b"RIFF\0\0\0\0WEBPVP8 ",+ ],+ ids=["png", "jpeg", "gif", "webp"],+ )+ def test_an_unreadable_allowed_image_is_refused(self, origin, body):+ origin.add("/broken", body)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/broken")++ def test_a_failed_size_probe_does_not_try_other_formats(self, origin):+ from PIL import Image++ Image.init()+ factory, accept = Image.OPEN["TGA"]+ other_factory = Mock(wraps=factory)+ origin.add("/broken", b"\x89PNG\r\n\x1a\n")++ with (+ patch.dict(Image.OPEN, {"TGA": (other_factory, accept)}),+ pytest.raises(pdf_service.UnsupportedPDFResourceError),+ ):+ pdf_service._safe_url_fetcher(ORIGIN + "/broken")+ other_factory.assert_not_called()++ def test_an_unreviewed_plugin_is_not_opened(self, origin):+ from PIL import Image++ Image.init()+ factory = Mock(side_effect=AssertionError("unreviewed image decoder"))+ origin.add("/body", b"not an image")++ with (+ patch.dict(Image.OPEN, {"NEW_FORMAT": (factory, None)}),+ pytest.raises(pdf_service.UnsupportedPDFResourceError),+ ):+ pdf_service._safe_url_fetcher(ORIGIN + "/body")+ factory.assert_not_called()++ @pytest.mark.parametrize(+ "body",+ [+ b"body { color: red; }\n",+ b"<svg xmlns='http://www.w3.org/2000/svg'><rect/></svg>",+ bytes(36) + b"acsp" + bytes(128), # ICC profile signature+ b"Plain text.\nAnother line.",+ ],+ ids=["css", "svg", "icc", "text"],+ )+ def test_non_raster_resources_still_pass(self, origin, body):+ origin.add("/body", body)++ resource = pdf_service._safe_url_fetcher(ORIGIN + "/body")+ try:+ assert resource.read() == body+ finally:+ resource.close()+++class TestHeaderOnlyFactories:+ def test_the_reviewed_format_sets_are_not_widened(self):+ # Widening any of these runs a factory, or accepts a format, that+ # no one reviewed for decoding on open; it needs that review first.+ assert pdf_service._HEADER_ONLY_RASTER_FORMATS == {+ "IM",+ "IMT",+ "PCD",+ "SPIDER",+ "TGA",+ }+ assert pdf_service._PREFIX_RULED_RASTER_FORMATS == {"IPTC": b"\x1c"}+ assert pdf_service._ALLOWED_RASTER_FORMATS == {+ "PNG",+ "JPEG",+ "GIF",+ "WEBP",+ }+ # Every format without a sniffer is ruled out one way or the other.+ assert set(_FORMATS_WITHOUT_SNIFFERS) == set(+ pdf_service._HEADER_ONLY_RASTER_FORMATS+ ) | set(pdf_service._PREFIX_RULED_RASTER_FORMATS)++ def test_iptc_is_ruled_out_without_running_its_parser(self, origin):+ # IPTC's _open walks every field to the end of the body; it is+ # decided by its first byte instead. Putting IPTC back among the+ # header-only formats calls the factory and fails this test.+ from PIL import Image++ Image.init()+ factory = Mock(side_effect=AssertionError("IPTC parser was run"))+ origin.add("/style.css", b"body { color: red; }\n")+ origin.add("/image.bin", _raster_without_sniffer("IPTC"))++ with patch.dict(Image.OPEN, {"IPTC": (factory, None)}):+ resource = pdf_service._safe_url_fetcher(ORIGIN + "/style.css")+ resource.close()+ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/image.bin")+ factory.assert_not_called()++ def test_a_header_parser_that_raises_value_error_fails_closed(+ self, origin, pdf_logs+ ):+ # IMT's header parser reads "width 100px" and int() raises+ # ValueError, which Image.open propagates rather than skips. The+ # guard refuses the body under its own warning instead of letting+ # the ValueError escape the fetch unlogged.+ body = b"width 100px\nheight 3\n"+ origin.add("/text.bin", body)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ pdf_service._safe_url_fetcher(ORIGIN + "/text.bin")+ refusals = [line for line in pdf_logs if "Refused a IMT image" in line]+ assert len(refusals) == 1, pdf_logs+++class TestSizeProbeLeavesWarningsAlone:+ """The size probe runs while other exports render in other threads, so+ it must not touch the process-wide ``warnings`` filters, which+ ``warnings.catch_warnings`` replaces and restores non-atomically."""++ def test_the_probe_uses_neither_catch_warnings_nor_image_open(self):+ body = _png_declaring(10000, 10000) # past Pillow's warning, 89 M+ refuse = AssertionError("must not be called")+ with (+ patch.object(warnings, "catch_warnings", side_effect=refuse),+ patch.object(warnings, "simplefilter", side_effect=refuse),+ patch("PIL.Image.open", side_effect=refuse),+ ):+ pixels = pdf_service._declared_image_pixels(body, "PNG")+ assert pixels == 10000 * 10000++ def test_an_image_past_pillows_warning_raises_no_warning(self):+ from PIL import Image++ body = _png_declaring(10000, 10000)+ assert 10000 * 10000 > Image.MAX_IMAGE_PIXELS+ with warnings.catch_warnings(record=True) as caught:+ warnings.simplefilter("always")+ assert pdf_service._declared_image_pixels(body, "PNG") == 10**8+ assert not [+ w+ for w in caught+ if issubclass(w.category, Image.DecompressionBombWarning)+ ]++ def test_concurrent_probes_leave_the_warning_filters_unchanged(self):+ body = _png("red")+ before = list(warnings.filters)+ failures: list = []++ def probe() -> None:+ try:+ for _ in range(2000):+ assert pdf_service._declared_image_pixels(body, "PNG") == 16+ except BaseException as exc: # pragma: no cover - reported below+ failures.append(exc)++ threads = [threading.Thread(target=probe) for _ in range(4)]+ interval = sys.getswitchinterval()+ sys.setswitchinterval(1e-6) # switch threads as often as possible+ try:+ for thread in threads:+ thread.start()+ for thread in threads:+ thread.join(timeout=60)+ finally:+ sys.setswitchinterval(interval)++ assert not any(thread.is_alive() for thread in threads)+ assert not failures, failures+ assert list(warnings.filters) == before+++# Tables an ordinary outline font has, none of which holds glyph images.+_OUTLINE_TABLES = (+ b"OS/2",+ b"cmap",+ b"glyf",+ b"head",+ b"hhea",+ b"hmtx",+ b"loca",+ b"maxp",+ b"name",+ b"post",+)+++def _sfnt(tags, *, version=b"\x00\x01\x00\x00", base=0) -> bytes:+ """A minimal sfnt: a table directory naming ``tags``, each an 8-byte+ table. ``base`` is where the face sits in a collection, since table+ offsets count from the start of the file."""+ directory = 12 + 16 * len(tags)+ records = b"".join(+ tag + struct.pack(">III", 0, base + directory + 8 * i, 8)+ for i, tag in enumerate(tags)+ )+ head = version + struct.pack(">HHHH", len(tags), 0, 0, 0)+ return head + records + bytes(8 * len(tags))+++def _ttc(*faces) -> bytes:+ """A font collection of faces, each given as its table tags."""+ at = 12 + 4 * len(faces)+ offsets, bodies = [], []+ for tags in faces:+ body = _sfnt(tags, base=at)+ offsets.append(at)+ bodies.append(body)+ at += len(body)+ head = b"ttcf\0\1\0\0" + struct.pack(+ f">I{len(faces)}I", len(faces), *offsets+ )+ return head + b"".join(bodies)+++def _real_font(*, cff: bool = False, sbix_side: int = 0):+ """A complete font built with fontTools, as a browser would load it;+ with ``sbix_side``, one glyph is a PNG header declaring that square."""+ pytest.importorskip("fontTools")+ from fontTools.fontBuilder import FontBuilder+ from fontTools.pens.t2CharStringPen import T2CharStringPen+ from fontTools.pens.ttGlyphPen import TTGlyphPen+ from fontTools.ttLib import newTable+ from fontTools.ttLib.tables import sbixGlyph, sbixStrike++ names = [".notdef", "A"]+ builder = FontBuilder(1000, isTTF=not cff)+ builder.setupGlyphOrder(names)+ builder.setupCharacterMap({0x41: "A"})+ if cff:+ pen = T2CharStringPen(600, None)+ pen.moveTo((0, 0))+ pen.lineTo((0, 500))+ pen.closePath()+ charstring = pen.getCharString()+ builder.setupCFF("T-Regular", {}, {n: charstring for n in names}, {})+ else:+ pen = TTGlyphPen(None)+ pen.moveTo((0, 0))+ pen.lineTo((0, 500))+ pen.lineTo((500, 0))+ pen.closePath()+ builder.setupGlyf({n: pen.glyph() for n in names})+ builder.setupHorizontalMetrics({n: (600, 0) for n in names})+ builder.setupHorizontalHeader(ascent=800, descent=-200)+ builder.setupNameTable({"familyName": "T", "styleName": "Regular"})+ builder.setupOS2()+ builder.setupPost()+ font = builder.font+ if sbix_side:+ strike = sbixStrike.Strike(ppem=20, resolution=72)+ strike.glyphs["A"] = sbixGlyph.Glyph(+ glyphName="A",+ graphicType="png ",+ imageData=_png_declaring(sbix_side, sbix_side),+ originOffsetX=0,+ originOffsetY=0,+ )+ font["sbix"] = newTable("sbix")+ font["sbix"].strikes[20] = strike+ return font+++def _saved(font) -> bytes:+ out = io.BytesIO()+ font.save(out)+ return out.getvalue()+++class TestFontsWithImageGlyphs:+ """A colour-bitmap font (``sbix``, ``CBDT``/``CBLC``) holds a PNG per+ glyph, which WeasyPrint opens with an unrestricted ``Image.open`` each+ time it draws the glyph, outside the fetcher's image limits; an+ ``SVG `` table holds SVG glyphs that can embed raster images. Such a+ font is refused in the fetcher. Removing that check accepts every font+ these tests refuse."""++ def _fetch(self, origin, body):+ origin.add("/f.ttf", body, headers={"Content-Type": "font/ttf"})+ return pdf_service._safe_url_fetcher(ORIGIN + "/f.ttf")++ @pytest.mark.parametrize("tag", [b"sbix", b"CBDT", b"CBLC", b"SVG "])+ @pytest.mark.parametrize(+ "version",+ [b"\x00\x01\x00\x00", b"OTTO", b"true", b"typ1"],+ ids=["ttf", "otf", "true", "typ1"],+ )+ def test_a_font_with_an_image_glyph_table_is_refused(+ self, origin, pdf_logs, tag, version+ ):+ body = _sfnt([*_OUTLINE_TABLES, tag], version=version)++ with pytest.raises(pdf_service.UnsupportedPDFResourceError) as refused:+ self._fetch(origin, body)+ assert tag.decode().strip() in str(refused.value)+ refusals = [line for line in pdf_logs if "image glyphs" in line]+ assert len(refusals) == 1, pdf_logs++ def test_an_unsorted_directory_cannot_hide_the_table(self, origin):+ body = _sfnt([b"zzzz", b"sbix", b"aaaa"])++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ self._fetch(origin, body)++ def test_a_collection_with_an_image_glyph_face_is_refused(self, origin):+ # Only the second face has sbix: every face is checked, since+ # Fontconfig may install any of them.+ body = _ttc(_OUTLINE_TABLES, (*_OUTLINE_TABLES, b"sbix"))++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ self._fetch(origin, body)++ @pytest.mark.parametrize(+ "body",+ [+ _sfnt(_OUTLINE_TABLES),+ _sfnt(_OUTLINE_TABLES, version=b"OTTO"),+ _sfnt((*_OUTLINE_TABLES, b"COLR", b"CPAL")), # vector colour+ _sfnt((*_OUTLINE_TABLES, b"EBDT", b"EBLC")), # mono bitmaps+ _ttc(_OUTLINE_TABLES, _OUTLINE_TABLES),+ ],+ ids=["ttf", "otf", "colr", "ebdt", "ttc"],+ )+ def test_a_font_without_image_glyphs_comes_back_whole(self, origin, body):+ resource = self._fetch(origin, body)+ try:+ assert resource.read() == body+ finally:+ resource.close()++ @pytest.mark.parametrize("cff", [False, True], ids=["ttf", "otf"])+ def test_a_real_outline_font_comes_back_whole(self, origin, cff):+ body = _saved(_real_font(cff=cff))+ assert body[:4] == (b"OTTO" if cff else b"\x00\x01\x00\x00")++ resource = self._fetch(origin, body)+ try:+ assert resource.read() == body+ finally:+ resource.close()++ def test_a_real_sbix_font_declaring_a_huge_glyph_is_refused(self, origin):+ # The gate's case: a few hundred bytes declaring a 13000x13000+ # glyph, which WeasyPrint would build a RasterImage from per use.+ body = _saved(_real_font(sbix_side=13000))+ assert len(body) < 4096+ assert 13000 * 13000 > pdf_service.MAX_PDF_IMAGE_PIXELS++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ self._fetch(origin, body)++ def test_a_real_collection_with_an_sbix_face_is_refused(self, origin):+ from fontTools.ttLib import TTCollection++ collection = TTCollection()+ collection.fonts = [_real_font(), _real_font(sbix_side=64)]+ body = _saved(collection)+ assert body[:4] == b"ttcf"++ with pytest.raises(pdf_service.UnsupportedPDFResourceError):+ self._fetch(origin, body)++ @pytest.mark.parametrize(+ "body",+ [+ b"\x00\x01\x00\x00\x00\x01", # header cut short+ _sfnt(_OUTLINE_TABLES)[:40], # records cut short+ _sfnt(_OUTLINE_TABLES)[:-1], # last table past the end+ b"OTTO" + struct.pack(">H", 0xFFFF) + bytes(16 * 0xFFFF),+ b"ttcf\0\1", # collection header cut short+ b"ttcf\0\1\0\0" + struct.pack(">I", 1 << 20) + bytes(64),+ b"ttcf\0\1\0\0" + struct.pack(">II", 1, 1 << 30),+ _ttc(_OUTLINE_TABLES)[:30], # face directory cut short+ ],+ ids=[+ "short-header",+ "short-records",+ "table-past-end",+ "too-many-tables",+ "short-ttc-header",+ "too-many-faces",+ "face-past-end",+ "short-face",+ ],+ )+ def test_an_unreadable_table_directory_fails_closed(self, origin, body):+ with pytest.raises(pdf_service.UnsupportedPDFResourceError) as refused:+ self._fetch(origin, body)+ assert "table directory is unreadable" in str(refused.value)++ def test_a_resource_fork_font_is_refused(self, origin):+ # HarfBuzz reads dfont too; its prefix is ICO's, which Pillow's+ # sniffer refuses first. Without that plugin it is still refused.+ from PIL import Image++ Image.init()+ body = b"\x00\x00\x01\x00" + bytes(252)+ with patch.dict(Image.OPEN):+ del Image.OPEN["ICO"]+ with pytest.raises(pdf_service.UnsupportedPDFResourceError) as e:+ self._fetch(origin, body)+ assert "dfont" in str(e.value)+++class TestDocumentedFigures:+ def test_the_figures_the_changelog_and_comments_state(self):+ # The changelog and the comments in pdf_service quote these; a+ # change here must change them too.+ assert pdf_service.MAX_PDF_RESOURCE_BYTES == 16 * MIB+ assert pdf_service.MAX_PDF_EXPORT_RESOURCE_BYTES == 64 * MIB+ assert pdf_service._INFLATED_BYTE_COST == 12+ assert pdf_service._GZIP_MEMBER_COST_BYTES == 4 * 1024+ assert pdf_service._RESOURCE_READ_CHUNK_BYTES == 64 * 1024+ assert pdf_service.MAX_PDF_IMAGE_PIXELS == 16_777_216 == 4096 * 4096+ assert pdf_service.MAX_PDF_EXPORT_IMAGE_PIXELS == 268_435_456+ # "refused past about 1.3 MiB of compressed data"+ per_resource = pdf_service.MAX_PDF_RESOURCE_BYTES+ assert round(per_resource / 12 / MIB, 1) == 1.3+ # "at most about 5.3 MiB of compressed data (64 MiB / 12)"+ per_export = pdf_service.MAX_PDF_EXPORT_RESOURCE_BYTES+ assert round(per_export / 12 / MIB, 1) == 5.3+ # "about twenty-one 12-megapixel images"+ pixels = pdf_service.MAX_PDF_EXPORT_IMAGE_PIXELS+ assert pixels // (12 * MIB) == 21+ # "at most 1024 such requests": 64 KiB charged each+ assert per_export // pdf_service._RESOURCE_READ_CHUNK_BYTES == 1024