Scrapy, a fast high-level web crawling & scraping framework for Python.
Support and prefer the .jsonl file extension (#4848)
Michel Ace6e878490e823
diff --git a/docs/intro/overview.rst b/docs/intro/overview.rstindex f3d652621..cfa6bfa83 100644--- a/docs/intro/overview.rst+++ b/docs/intro/overview.rst@@ -45,9 +45,9 @@ https://quotes.toscrape.com, following the pagination::Put this in a text file, name it to something like ``quotes_spider.py``and run the spider using the :command:`runspider` command::- scrapy runspider quotes_spider.py -o quotes.jl+ scrapy runspider quotes_spider.py -o quotes.jsonl-When this finishes you will have in the ``quotes.jl`` file a list of the+When this finishes you will have in the ``quotes.jsonl`` file a list of thequotes in JSON Lines format, containing text and author, looking like this::{"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"}diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rstindex cde1b1ef4..75928077e 100644--- a/docs/intro/tutorial.rst+++ b/docs/intro/tutorial.rst@@ -482,7 +482,7 @@ to append new content to any existing file. However, appending to a JSON filemakes the file contents invalid JSON. When appending to a file, considerusing a different serialization format, such as `JSON Lines`_::- scrapy crawl quotes -o quotes.jl+ scrapy crawl quotes -o quotes.jsonlThe `JSON Lines`_ format is useful because it's stream-like, you can easilyappend new records to it. It doesn't have the same problem of JSON when you rundiff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex 9a13eb82f..398f80633 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -638,6 +638,7 @@ Default::{'json': 'scrapy.exporters.JsonItemExporter','jsonlines': 'scrapy.exporters.JsonLinesItemExporter',+ 'jsonl': 'scrapy.exporters.JsonLinesItemExporter','jl': 'scrapy.exporters.JsonLinesItemExporter','csv': 'scrapy.exporters.CsvItemExporter','xml': 'scrapy.exporters.XmlItemExporter',@@ -763,7 +764,7 @@ source spider in the feed URI:#. Use ``%(spider_name)s`` in your feed URI::- scrapy crawl <spider_name> -o "%(spider_name)s.jl"+ scrapy crawl <spider_name> -o "%(spider_name)s.jsonl".. _URIs: https://en.wikipedia.org/wiki/Uniform_Resource_Identifierdiff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rstindex 882ff5661..af294f52c 100644--- a/docs/topics/item-pipeline.rst+++ b/docs/topics/item-pipeline.rst@@ -99,11 +99,11 @@ contain a price::raise DropItem(f"Missing price in {item}")-Write items to a JSON file---------------------------+Write items to a JSON lines file+--------------------------------The following pipeline stores all scraped items (from all spiders) into a-single ``items.jl`` file, containing one item per line serialized in JSON+single ``items.jsonl`` file, containing one item per line serialized in JSONformat::import json@@ -113,7 +113,7 @@ format::class JsonWriterPipeline:def open_spider(self, spider):- self.file = open('items.jl', 'w')+ self.file = open('items.jsonl', 'w')def close_spider(self, spider):self.file.close()diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.pyindex f5a3efe69..ff86af125 100644--- a/scrapy/settings/default_settings.py+++ b/scrapy/settings/default_settings.py@@ -154,6 +154,7 @@ FEED_EXPORTERS = {}FEED_EXPORTERS_BASE = {'json': 'scrapy.exporters.JsonItemExporter','jsonlines': 'scrapy.exporters.JsonLinesItemExporter',+ 'jsonl': 'scrapy.exporters.JsonLinesItemExporter','jl': 'scrapy.exporters.JsonLinesItemExporter','csv': 'scrapy.exporters.CsvItemExporter','xml': 'scrapy.exporters.XmlItemExporter',