Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Update the docs

Adrián Chaves1087bb7b2eab
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index adf3d0f4a..2d1227cf8 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -702,6 +702,7 @@ Those are:
* :reqmeta:`download_fail_on_dataloss`
* :reqmeta:`download_latency`
* :reqmeta:`download_maxsize`
+* :reqmeta:`download_warnsize`
* :reqmeta:`download_timeout`
* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)
* ``ftp_user`` (See :setting:`FTP_USER` for more info)
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 7cdfb8768..eb24b834a 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -873,40 +873,42 @@ The amount of time (in secs) that the downloader will wait before timing out.
Request.meta key.
.. setting:: DOWNLOAD_MAXSIZE
+.. reqmeta:: download_maxsize
DOWNLOAD_MAXSIZE
----------------
-Default: ``1073741824`` (1024MB)
-
-The maximum response size (in bytes) that downloader will download.
+Default: ``1073741824`` (1 GiB)
-If you want to disable it set to 0.
+The maximum response body size (in bytes) allowed. Bigger responses are
+aborted and ignored.
-.. reqmeta:: download_maxsize
+This applies both before and after compression. If decompressing a response
+body would exceed this limit, decompression is aborted and the response is
+ignored.
-.. note::
+Use ``0`` to disable this limit.
- This size can be set per spider using :attr:`download_maxsize`
- spider attribute and per-request using :reqmeta:`download_maxsize`
- Request.meta key.
+This limit can be set per spider using the :attr:`download_maxsize` spider
+attribute and per request using the :reqmeta:`download_maxsize` Request.meta
+key.
.. setting:: DOWNLOAD_WARNSIZE
+.. reqmeta:: download_warnsize
DOWNLOAD_WARNSIZE
-----------------
-Default: ``33554432`` (32MB)
-
-The response size (in bytes) that downloader will start to warn.
+Default: ``33554432`` (32 MiB)
-If you want to disable it set to 0.
+If the size of a response exceeds this value, before or after compression, a
+warning will be logged about it.
-.. note::
+Use ``0`` to disable this limit.
- This size can be set per spider using :attr:`download_warnsize`
- spider attribute and per-request using :reqmeta:`download_warnsize`
- Request.meta key.
+This limit can be set per spider using the :attr:`download_warnsize` spider
+attribute and per request using the :reqmeta:`download_warnsize` Request.meta
+key.
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS