Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Update docs.

djunzu6988e9cd4ba4
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index a34f4c053..3da243d29 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -77,30 +77,6 @@ PIL.
.. _Python Imaging Library: http://www.pythonware.com/products/pil/
-Usage example
-=============
-
-In order to use a media pipeline first, :ref:`enable it
-<topics-media-pipeline-enabling>`.
-
-Then, if a spider returns a dict with the URLs key ('file_urls' or
-'image_urls', for the Files or Images Pipeline respectively), the pipeline will
-put the results under respective key ('files' or images').
-
-If you prefer to use :class:`~.Item`, then define a custom item with the
-necessary fields, like in this example for Images Pipeline::
-
- import scrapy
-
- class MyItem(scrapy.Item):
-
- # ... other item fields ...
- image_urls = scrapy.Field()
- images = scrapy.Field()
-
-If you need something more complex and want to override the custom pipeline
-behaviour, see :ref:`topics-media-pipeline-override`.
-
.. _topics-media-pipeline-enabling:
Enabling your Media Pipeline
@@ -171,6 +147,51 @@ Where:
used). For more info see :ref:`topics-images-thumbnails`.
+Usage example
+=============
+
+.. setting:: FILES_URLS_FIELD
+.. setting:: FILES_RESULT_FIELD
+.. setting:: IMAGES_URLS_FIELD
+.. setting:: IMAGES_RESULT_FIELD
+
+In order to use a media pipeline first, :ref:`enable it
+<topics-media-pipeline-enabling>`.
+
+Then, if a spider returns a dict with the URLs key (``file_urls`` or
+``image_urls``, for the Files or Images Pipeline respectively), the pipeline will
+put the results under respective key (``files`` or ``images``).
+
+If you prefer to use :class:`~.Item`, then define a custom item with the
+necessary fields, like in this example for Images Pipeline::
+
+ import scrapy
+
+ class MyItem(scrapy.Item):
+
+ # ... other item fields ...
+ image_urls = scrapy.Field()
+ images = scrapy.Field()
+
+If you want to use another field name for the URLs key or for the results key,
+it is also possible to override it.
+
+For the Files Pipeline, set :setting:`FILES_URLS_FIELD` and/or
+:setting:`FILES_RESULT_FIELD` settings::
+
+ FILES_URLS_FIELD = 'field_name_for_your_files_urls'
+ FILES_RESULT_FIELD = 'field_name_for_your_processed_files'
+
+For the Images Pipeline, set :setting:`IMAGES_URLS_FIELD` and/or
+:setting:`IMAGES_RESULT_FIELD` settings::
+
+ IMAGES_URLS_FIELD = 'field_name_for_your_images_urls'
+ IMAGES_RESULT_FIELD = 'field_name_for_your_processed_images'
+
+If you need something more complex and want to override the custom pipeline
+behaviour, see :ref:`topics-media-pipeline-override`.
+
+
Additional features
===================
@@ -185,12 +206,14 @@ adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or
:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which
specifies the delay in number of days::
- # 90 days of delay for files expiration
- FILES_EXPIRES = 90
+ # 120 days of delay for files expiration
+ FILES_EXPIRES = 120
# 30 days of delay for images expiration
IMAGES_EXPIRES = 30
+The default value for both settings is 90 days.
+
.. _topics-images-thumbnails:
Thumbnail generation for images
@@ -249,7 +272,13 @@ For example::
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
-Note: these size constraints don't affect thumbnail generation at all.
+.. note::
+ The size constraints don't affect thumbnail generation at all.
+
+It is possible to set just one size constraint or both. When setting both of
+them, only images that satisfy both minimum sizes will be saved. For the
+above example, images of sizes (105 x 105) or (105 x 200) or (200 x 105) will
+all be dropped because at least one dimension is shorter than the constraint.
By default, there are no size constraints, so all images are processed.