Scrapy, a fast high-level web crawling & scraping framework for Python.
Merge remote-tracking branch 'upstream/master' into ISSUE-4250-add_batch_deliveries
BroodingKangaroo111a58fe3d8c
diff --cc docs/topics/feed-exports.rstindex 0f15044b3,9e5968a29..42c4e2267--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@@ -430,40 -429,3 +430,40 @@@ format in :setting:`FEED_EXPORTERS`. E... _Amazon S3: https://aws.amazon.com/s3/.. _botocore: https://github.com/boto/botocore.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl++.. setting:: FEED_STORAGE_BATCH_ITEM_COUNT++FEED_STORAGE_BATCH_ITEM_COUNT- ----------------------++-----------------------------+Default: ``None``++An integer number that represents the number of scraped items stored in each output+file. Whenever the number of items exceeds this setting, a new file is+created and the output is redirected to it. The name of the new file will be selected+based on the timestamp when the feed is being created and/or on the batch sequence number.+Therefore you must specify %(batch_time)s or %(batch_id)s or both in :setting:`FEED_URI`.++* ``%(batch_time)s`` - gets replaced by a timestamp when the feed is being created+(e.g. `2020-03-28T14-45-08.237134`)++* ``%(batch_id)s`` - gets replaced by the batch sequence number of batch+(e.g. `2` for the second file)++For instance, if your settings include::++ FEED_STORAGE_BATCH_ITEM_COUNT=100++And your :command:`crawl` command line is::++ scrapy crawl spidername -o dirname/%(batch_id)s-filename%(batch_time)s.json++The resulting directory tree of above can be like::++->projectname+-->dirname+--->1-filename2020-03-28T14-45-08.237134.json+--->2-filename2020-03-28T14-45-09.148903.json+--->3-filename2020-03-28T14-45-10.046092.json++Where the first and second files contain exactly 100 items. The last one contains+<= 100 items.