Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries

BroodingKangaroo111a58fe3d8c
diff --cc docs/topics/feed-exports.rst
index 0f15044b3,9e5968a29..42c4e2267
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@@ -430,40 -429,3 +430,40 @@@ format in :setting:`FEED_EXPORTERS`. E.
.. _Amazon S3: https://aws.amazon.com/s3/
.. _botocore: https://github.com/boto/botocore
.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+
+.. setting:: FEED_STORAGE_BATCH_ITEM_COUNT
+
+FEED_STORAGE_BATCH_ITEM_COUNT
- ----------------------
++-----------------------------
+Default: ``None``
+
+An integer number that represents the number of scraped items stored in each output
+file. Whenever the number of items exceeds this setting, a new file is
+created and the output is redirected to it. The name of the new file will be selected
+based on the timestamp when the feed is being created and/or on the batch sequence number.
+Therefore you must specify %(batch_time)s or %(batch_id)s or both in :setting:`FEED_URI`.
+
+* ``%(batch_time)s`` - gets replaced by a timestamp when the feed is being created
+(e.g. `2020-03-28T14-45-08.237134`)
+
+* ``%(batch_id)s`` - gets replaced by the batch sequence number of batch
+(e.g. `2` for the second file)
+
+For instance, if your settings include::
+
+ FEED_STORAGE_BATCH_ITEM_COUNT=100
+
+And your :command:`crawl` command line is::
+
+ scrapy crawl spidername -o dirname/%(batch_id)s-filename%(batch_time)s.json
+
+The resulting directory tree of above can be like::
+
+->projectname
+-->dirname
+--->1-filename2020-03-28T14-45-08.237134.json
+--->2-filename2020-03-28T14-45-09.148903.json
+--->3-filename2020-03-28T14-45-10.046092.json
+
+Where the first and second files contain exactly 100 items. The last one contains
+<= 100 items.