Scrapy, a fast high-level web crawling & scraping framework for Python.
Document the trust boundary of JOBDIR and HTTPCACHE_DIR (#8252)
Adrian55c3e31cf939
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex f045dd9fe..d952a2232 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -644,6 +644,10 @@ The directory to use for storing the (low-level) HTTP cache. If empty, the HTTPcache will be disabled. If a relative path is given, is taken relative to theproject data dir. For more info see: :ref:`topics-project-structure`.+.. warning:: Treat the cache directory with the same security care as your+ Scrapy project source code. Do not point :setting:`HTTPCACHE_DIR` to a+ path that untrusted parties can write to.+.. setting:: HTTPCACHE_IGNORE_HTTP_CODESHTTPCACHE_IGNORE_HTTP_CODESdiff --git a/docs/topics/security.rst b/docs/topics/security.rstindex 583e446e4..ffceb7e4c 100644--- a/docs/topics/security.rst+++ b/docs/topics/security.rst@@ -338,3 +338,12 @@ request is written within its own transaction:* **Con:** scheduling requests becomes slower, which can be noticeable incrawls with a high request throughput.++.. _security-local-data:++Local data directories+======================++Scrapy loads the contents of the :setting:`JOBDIR` and :setting:`HTTPCACHE_DIR`+directories with :mod:`pickle`, which can run arbitrary code, so protect them+from write access by untrusted parties as you would your project source code.