Scrapy, a fast high-level web crawling & scraping framework for Python.
docs: document subclassing RedirectMiddleware to allow/deny redirects by target URL (#8061)
Adrian8d390db33ab4
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 5eda7d92c..0417ce9e1 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -855,6 +855,41 @@ allow on a per-request basis. You can also set the meta key``handle_httpstatus_all`` to ``True`` if you want to allow any response codefor a request.+To decide whether to follow a redirect based on its target URL, subclass+:class:`RedirectMiddleware` and override+:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`:++.. code-block:: python++ import re++ from scrapy import Request+ from scrapy.downloadermiddlewares.redirect import RedirectMiddleware+ from scrapy.exceptions import IgnoreRequest+++ class AllowedRedirectMiddleware(RedirectMiddleware):+ def process_response(self, request, response, spider):+ result = super().process_response(request, response, spider)+ if isinstance(result, Request) and not re.search(r"/product/", result.url):+ raise IgnoreRequest(f"Redirect target not allowed: {result.url}")+ return result++Enable it in place of :class:`RedirectMiddleware`, using the same priority:++.. code-block:: python++ from scrapy import Spider+++ class MySpider(Spider):+ custom_settings = {+ "DOWNLOADER_MIDDLEWARES": {+ "scrapy.downloadermiddlewares.redirect.RedirectMiddleware": None,+ "myproject.middlewares.AllowedRedirectMiddleware": 600,+ }+ }+RedirectMiddleware settings~~~~~~~~~~~~~~~~~~~~~~~~~~~