Scrapy, a fast high-level web crawling & scraping framework for Python.
Add content-based image filtering example (#4954)
smellslikemlfb3455304d9e
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex b542c6c05..c67d2d627 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -774,4 +774,28 @@ To enable your custom media pipeline component you must add its class import patITEM_PIPELINES = {"myproject.pipelines.MyImagesPipeline": 300}+Content-based image filtering pipeline+--------------------------------------++This example overrides ``get_images()`` to filter images using a classifier,+such as a TensorFlow_ model. Override ``is_valid_image()`` with your+classification logic:++.. code-block:: python++ from scrapy.pipelines.images import ImagesPipeline, ImageException+++ class ImageClassifierPipeline(ImagesPipeline):+ def is_valid_image(self, image):+ raise NotImplementedError++ def get_images(self, response, request, info, *, item=None):+ for path, image, buf in super().get_images(response, request, info, item=item):+ if not self.is_valid_image(image):+ raise ImageException("Image does not match criteria")+ yield path, image, buf++.. _MD5 hash: https://en.wikipedia.org/wiki/MD5+.. _TensorFlow: https://tensorflow.org