Scrapy, a fast high-level web crawling & scraping framework for Python.
Document HTTPCACHE_IGNORE_HTTP_CODES/RETRY_HTTP_CODES interaction (#8202)
Adrian8fab802f626c
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 8550ece4c..67b606d84 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -642,6 +642,11 @@ Default: ``[]``Don't cache response with these HTTP codes.+If you also retry requests (see :setting:`RETRY_HTTP_CODES`), add those+same status codes here. Otherwise, a response that gets retried may also+get cached, and further retries of that request could be served that+cached response instead of reaching the server again.+.. setting:: HTTPCACHE_IGNORE_MISSINGHTTPCACHE_IGNORE_MISSING@@ -1090,6 +1095,10 @@ In some cases you may want to add 400 to :setting:`RETRY_HTTP_CODES` becauseit is a common code used to indicate server overload. It is not included bydefault because HTTP specs say so.+If you also cache responses (see :setting:`HTTPCACHE_ENABLED`), see+:setting:`HTTPCACHE_IGNORE_HTTP_CODES` to keep retried responses out of the+cache.+.. setting:: RETRY_EXCEPTIONSRETRY_EXCEPTIONS