Scrapy, a fast high-level web crawling & scraping framework for Python.
[wip] draft 1.6 release notes
Mikhail Korobov706910790b6e
diff --git a/docs/news.rst b/docs/news.rstindex adf679ded..99a339cea 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -6,11 +6,156 @@ Release notesScrapy 1.6.0 (unreleased)-------------------------+Highlights for this release:++* better Windows compatibility;+* Python 3.7 compatibility;+* big documentation improvements, including a switch+ from ``.extract() / .extract_first()`` API to ``.get() / .getall()`` API;+* Feed exports, FilePipeline and MediaPipeline improvements;+* ``scrapy.contracts`` fixes and new features;+* large clean-up of deprecated code+* TODO++parsel 1.5+~~~~~~~~~~++TODO+While this is not a change in Scrapy itself, a new version of ``parsel``+is released; Scrapy now depends on ``parsel >= 1.5``.++Feed export improvements+~~~~~~~~~~~~~~~~~~~~~~~~++* ``from_crawler`` support is added to feed exporters and feed storages. This,+ among other things, allow to access Scrapy settings from custom storages+ and exporters (:issue:`1605`, :issue:`3348`).+* fixed issue with extra blank lines in .csv exports under Windows+ (:issue:`3039`);+* better error message when an exporter is disabled (:issue:`3358`);++FilePipeline and MediaPipeline improvements+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++* Expose more options for S3FilesStore: :setting:`AWS_ENDPOINT_URL`,+ :setting:`AWS_USE_SSL`, :setting:`AWS_VERIFY`, :setting:`AWS_REGION_NAME`.+ For example, this allows to use alternative or self-hosted+ AWS-compatible providers (:issue:`2609`).+* ACL support for Google Cloud Storage: :setting:`FILES_STORE_GCS_ACL` and+ :setting:`IMAGES_STORE_GCS_ACL` (:issue:`3199`).++``scrapy.contracts`` improvements+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++* Exceptions in contracts code are handled better (:issue:`3377`);+* ``dont_filter=True`` is used for contract requests, which allows to test+ different callbacks with the same URL (:issue:`3381`);+* ``request_cls`` attribute in Contract subclasses allow to use different+ Request classes in contracts, for example FormRequest (:issue:`3383`).+* Fixed errback handling in contracts, e.g. for cases where a contract+ is executed for URL which returns non-200 response (:issue:`3371`).++Documentation improvements+~~~~~~~~~~~~~~~~~~~~~~~~~~++* Docs are re-written to suggest .get/.getall API instead of+ .extract/.extract_first. Also, :ref:`topics-selectors` docs are updated+ and re-structured to match latest parsel docs; they now contain more topics,+ such as :ref:`selecting-attributes` or :ref:`topics-selectors-css-extensions`+ (:issue:`3390`).+* :ref:`topics-developer-tools` is a new tutorial which replaces+ old Firefox and Firebug tutorials (:issue:`3400`).+* SCRAPY_PROJECT environment variable is documented (:issue:`3518`);+* troubleshooting section is added to install instructions (:issue:`3517`);+* improved links to beginner resources in the tutorial+ (:issue:`3367`, :issue:`3468`);+* fixed :setting:`RETRY_HTTP_CODES` default values in docs (:issue:`3335`);+* remove unused `DEPTH_STATS` option from docs (:issue:`3245`);+* other cleanups (:issue:`3347`, :issue:`3350`, :issue:`3445`).++Better Windows support+~~~~~~~~~~~~~~~~~~~~~~++* All Scrapy tests now pass on Windows; Scrapy testing suite is executed+ in a Windows environment on CI (:issue:`3315`).+* Scrapy used to produce unnecessary blank lines in .csv exports on Windows,+ this is fixed (:issue:`3039`).++Testing fixes+~~~~~~~~~~~~~++* Python 3.7 support (:issue:`3326`, :issue:`3150`, :issue:`3547`)+* Testing and CI fixes (:issue:`3526`, :issue:`3538`, :issue:`3308`,+ :issue:`3311`, :issue:`3309`, :issue:`3305`, :issue:`3210`, :issue:`3299`)++Deprecation removals+~~~~~~~~~~~~~~~~~~~~++Compatibility shims for pre-1.0 Scrapy module names are removed+(:issue:`3318`):++* ``scrapy.command``+* ``scrapy.contrib`` (with all submodules)+* ``scrapy.contrib_exp`` (with all submodules)+* ``scrapy.dupefilter``+* ``scrapy.linkextractor``+* ``scrapy.project``+* ``scrapy.spider``+* ``scrapy.spidermanager``+* ``scrapy.squeue``+* ``scrapy.stats``+* ``scrapy.statscol``+* ``scrapy.utils.decorator``++See :ref:`module_relocations` for more information, or use suggestions+from Scrapy 1.5.x deprecation warnings to update your code.++Other deprecation removals:++* Deprecated scrapy.interfaces.ISpiderManager is removed; please use+ scrapy.interfaces.ISpiderLoader.+* Deprecated ``CrawlerSettings`` class is removed (:issue:`3327`).+* Deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes+ are removed (:issue:`3327`, :issue:`3359`).++Internal improvements+~~~~~~~~~~~~~~~~~~~~~++* ``from_crawler`` support is added to dupefilters (:issue:`2956`); this allows+ to access e.g. settings or a spider from a dupefilter.+* :signal:`item_error` is fired when an error happens in a pipeline+ (:issue:`3256`);+* :signal:`request_reached_downloader` is fired when Downloader gets+ a new Request; this signal can be useful e.g. for custom Schedulers+ (:issue:`3393`).+* ``scrapy.http.cookies.CookieJar.clear`` accepts "domain", "path" and "name"+ optional arguments (:issue:`3231`).++Usability improvements+~~~~~~~~~~~~~~~~~~~~~~++* more stats for RobotsTxtMiddleware (:issue:`3100`)+* INFO log level is used to show telnet host/port (:issue:`3115`)+* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`)+* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`)+* non-zero exit code is returned from Scrapy commands when error happens+ on spider inititalization (:issue:`3226`)+* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`);+ "flv" is added to common video extensions (:issue:`3165`)++Bug fixes+~~~~~~~~~+* proper handling of pickling errors in Python 3 when serializing objects+ for disk queues (:issue:`3082`)+* flags are now preserved when copying Requests (:issue:`3342`);+* FormRequest.from_response clickdata shouldn't ignore elements with+ ``input[type=image]`` (:issue:`3153`).+* FormRequest.from_response should preserve duplicate keys (:issue:`3247`)+Cleanups~~~~~~~~--* Remove deprecated ``CrawlerSettings`` class.-* Remove deprecated ``Settings.overrides`` and ``Settings.defaults`` attributes.+* additional files are included to sdist (:issue:`3495`);+* code style fixes (:issue:`3405`, :issue:`3304`)Scrapy 1.5.2 (2019-01-22)-------------------------@@ -1080,6 +1225,8 @@ until it reaches a stable status.See more examples for scripts running Scrapy: :ref:`topics-practices`+.. _module_relocations:+Module Relocations~~~~~~~~~~~~~~~~~~