Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Document the trust boundary of JOBDIR and HTTPCACHE_DIR (#8252)

Adrian55c3e31cf939
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index f045dd9fe..d952a2232 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -644,6 +644,10 @@ The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP
cache will be disabled. If a relative path is given, is taken relative to the
project data dir. For more info see: :ref:`topics-project-structure`.
+.. warning:: Treat the cache directory with the same security care as your
+ Scrapy project source code. Do not point :setting:`HTTPCACHE_DIR` to a
+ path that untrusted parties can write to.
+
.. setting:: HTTPCACHE_IGNORE_HTTP_CODES
HTTPCACHE_IGNORE_HTTP_CODES
diff --git a/docs/topics/security.rst b/docs/topics/security.rst
index 583e446e4..ffceb7e4c 100644
--- a/docs/topics/security.rst
+++ b/docs/topics/security.rst
@@ -338,3 +338,12 @@ request is written within its own transaction:
* **Con:** scheduling requests becomes slower, which can be noticeable in
crawls with a high request throughput.
+
+.. _security-local-data:
+
+Local data directories
+======================
+
+Scrapy loads the contents of the :setting:`JOBDIR` and :setting:`HTTPCACHE_DIR`
+directories with :mod:`pickle`, which can run arbitrary code, so protect them
+from write access by untrusted parties as you would your project source code.