Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

doc for creating custom cache storage backend.

Anubhav Patel044318920a46
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 0d976077b..f913b059d 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -496,6 +496,59 @@ In order to use this storage backend:
.. _LevelDB: https://github.com/google/leveldb
.. _leveldb python bindings: https://pypi.python.org/pypi/leveldb
+.. _httpcache-storage-custom:
+
+Implementing custom cache storage backend
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can implement custom cache storage backend by creating a Python class that
+defines the methods described below.
+
+.. module:: scrapy.extensions.httpcache
+
+.. class:: CacheStorage
+
+ .. method:: open_spider(spider)
+
+ This method gets called after a spider has been opened for crawling.
+
+ :param spider: the spider which has been opened
+ :type spider: :class:`~scrapy.spiders.Spider` object
+
+ .. method:: close_spider(spider)
+
+ This method gets called after a spider has been closed.
+
+ :param spider: the spider which has been closed
+ :type spider: :class:`~scrapy.spiders.Spider` object
+
+ .. method:: retrieve_response(spider, request)
+
+ Returns response if present in cache, or ``None`` otherwise.
+
+ :param spider: the spider which generated the request
+ :type spider: :class:`~scrapy.spiders.Spider` object
+
+ :param request: the request to find cached reponse for
+ :type request: :class:`~scrapy.http.Request` object
+
+ .. method:: store_response(spider, request, response)
+
+ Stores the given response in the cache.
+
+ :param spider: the spider for which the response is intended
+ :type spider: :class:`~scrapy.spiders.Spider` object
+
+ :param request: corresponding request the spider generated
+ :type request: :class:`~scrapy.http.Request` object
+
+ :param response: the response to store in the cache
+ :type response: :class:`~scrapy.http.Response` object
+
+In order to use your storage backend, set:
+
+* :setting:`HTTPCACHE_STORAGE` to path of your custom storage class.
+
HTTPCache middleware settings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~