Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Mark the LevelDB storage backend as deprecated

Adrián Chaves3d0df419c4ee
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 2892b9b79..8048e1c86 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -348,7 +348,6 @@ HttpCacheMiddleware
* :ref:`httpcache-storage-fs`
* :ref:`httpcache-storage-dbm`
- * :ref:`httpcache-storage-leveldb`
You can change the HTTP cache storage backend with the :setting:`HTTPCACHE_STORAGE`
setting. Or you can also :ref:`implement your own storage backend. <httpcache-storage-custom>`
@@ -478,27 +477,6 @@ DBM storage backend
By default, it uses the anydbm_ module, but you can change it with the
:setting:`HTTPCACHE_DBM_MODULE` setting.
-.. _httpcache-storage-leveldb:
-
-LevelDB storage backend
-~~~~~~~~~~~~~~~~~~~~~~~
-
-.. class:: LeveldbCacheStorage
-
- .. versionadded:: 0.23
-
- A LevelDB_ storage backend is also available for the HTTP cache middleware.
-
- This backend is not recommended for development because only one process
- can access LevelDB databases at the same time, so you can't run a crawl and
- open the scrapy shell in parallel for the same spider.
-
- In order to use this storage backend, install the `LevelDB python
- bindings`_ (e.g. ``pip install leveldb``).
-
- .. _LevelDB: https://github.com/google/leveldb
- .. _leveldb python bindings: https://pypi.python.org/pypi/leveldb
-
.. _httpcache-storage-custom:
Writing your own storage backend
diff --git a/scrapy/extensions/httpcache.py b/scrapy/extensions/httpcache.py
index c6094643d..7c650a91e 100644
--- a/scrapy/extensions/httpcache.py
+++ b/scrapy/extensions/httpcache.py
@@ -1,19 +1,24 @@
from __future__ import print_function
-import os
+
import gzip
import logging
-from six.moves import cPickle as pickle
+import os
+from email.utils import mktime_tz, parsedate_tz
from importlib import import_module
from time import time
+from warnings import warn
from weakref import WeakKeyDictionary
-from email.utils import mktime_tz, parsedate_tz
+
+from six.moves import cPickle as pickle
from w3lib.http import headers_raw_to_dict, headers_dict_to_raw
+
+from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.http import Headers, Response
from scrapy.responsetypes import responsetypes
-from scrapy.utils.request import request_fingerprint
-from scrapy.utils.project import data_path
from scrapy.utils.httpobj import urlparse_cached
+from scrapy.utils.project import data_path
from scrapy.utils.python import to_bytes, to_unicode, garbage_collect
+from scrapy.utils.request import request_fingerprint
logger = logging.getLogger(__name__)
@@ -345,6 +350,8 @@ class FilesystemCacheStorage(object):
class LeveldbCacheStorage(object):
def __init__(self, settings):
+ warn("The LevelDB storage backend is deprecated.",
+ ScrapyDeprecationWarning, stacklevel=2)
import leveldb
self._leveldb = leveldb
self.cachedir = data_path(settings['HTTPCACHE_DIR'], createdir=True)