Scrapy, a fast high-level web crawling & scraping framework for Python.
Update the docs
Adrián Chaves1087bb7b2eab
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex adf3d0f4a..2d1227cf8 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -702,6 +702,7 @@ Those are:* :reqmeta:`download_fail_on_dataloss`* :reqmeta:`download_latency`* :reqmeta:`download_maxsize`+* :reqmeta:`download_warnsize`* :reqmeta:`download_timeout`* ``ftp_password`` (See :setting:`FTP_PASSWORD` for more info)* ``ftp_user`` (See :setting:`FTP_USER` for more info)diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 7cdfb8768..eb24b834a 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -873,40 +873,42 @@ The amount of time (in secs) that the downloader will wait before timing out.Request.meta key... setting:: DOWNLOAD_MAXSIZE+.. reqmeta:: download_maxsizeDOWNLOAD_MAXSIZE-----------------Default: ``1073741824`` (1024MB)--The maximum response size (in bytes) that downloader will download.+Default: ``1073741824`` (1 GiB)-If you want to disable it set to 0.+The maximum response body size (in bytes) allowed. Bigger responses are+aborted and ignored.-.. reqmeta:: download_maxsize+This applies both before and after compression. If decompressing a response+body would exceed this limit, decompression is aborted and the response is+ignored.-.. note::+Use ``0`` to disable this limit.- This size can be set per spider using :attr:`download_maxsize`- spider attribute and per-request using :reqmeta:`download_maxsize`- Request.meta key.+This limit can be set per spider using the :attr:`download_maxsize` spider+attribute and per request using the :reqmeta:`download_maxsize` Request.meta+key... setting:: DOWNLOAD_WARNSIZE+.. reqmeta:: download_warnsizeDOWNLOAD_WARNSIZE------------------Default: ``33554432`` (32MB)--The response size (in bytes) that downloader will start to warn.+Default: ``33554432`` (32 MiB)-If you want to disable it set to 0.+If the size of a response exceeds this value, before or after compression, a+warning will be logged about it.-.. note::+Use ``0`` to disable this limit.- This size can be set per spider using :attr:`download_warnsize`- spider attribute and per-request using :reqmeta:`download_warnsize`- Request.meta key.+This limit can be set per spider using the :attr:`download_warnsize` spider+attribute and per request using the :reqmeta:`download_warnsize` Request.meta+key... setting:: DOWNLOAD_FAIL_ON_DATALOSS