Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Merge remote-tracking branch 'origin/master' into release-notes-1.1

Paul Tremberth728e505e531e
diff --cc docs/news.rst
index 25f459a49,3e60b2b4f..3f118f218
--- a/docs/news.rst
+++ b/docs/news.rst
@@@ -3,158 -3,14 +3,167 @@@
Release notes
=============
+1.1.0
+-----
+
+This 1.1 release brings a lot of interesting features and bug fixes:
+
- - Scrapy 1.1 has basic Python 3 support (requires Twisted >= 15.5). See
++- Scrapy 1.1 has basic Python 3 support (requires Twisted >= 15.5). See
+ :ref:`news_basicpy3` for more details and some limitations.
+- Hot new features:
+
+ - ItemLoaders now support nested loaders (:issue:`1467`).
+ - ``FormRequest.from_response`` improvements (:issue:`1382`, :issue:`1137`).
+ - Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improved
+ AutoThrottle docs (:issue:`1324`).
+ - Added ``response.text`` to get body as unicode (:issue:`1730`).
+ - Anonymous S3 connections (:issue:`1358`).
- - Deferreds in downloader middlewares (:issue:`1473`). This enables better
++ - Deferreds in downloader middlewares (:issue:`1473`). This enables better
+ robots.txt handling (:issue:`1471`).
+ - HTTP cache improvements (:issue:`1151`).
+
+- These bug fixes may require your attention:
+
+ - Don't retry bad requests (HTTP 400) (:issue:`1289`).
+ - Fix shell files argument handling (:issue:`1710`, :issue:`1550`).
+ - Fixes on robots.txt handling (:issue:`1783`).
+ - Exporters work on unicode (:issue:`1080`).
+ - Fix xmliter to accept nodenames with dots (:issue:`1533`).
+
+Keep reading for more details on other improvements and bug fixes.
+
+.. _news_basicpy3:
+
+Basic Python 3 Support
+~~~~~~~~~~~~~~~~~~~~~~
+
+We have been hard at work to make Scrapy run on Python 3. As a result, now you
+can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some
+features are still missing (and some may never be ported).
+
+Almost all addons/middlewares are expected to work. However, we are aware of
+some limitations:
+
+- S3 downloads are not supported (see :issue:`1718`)
+- Sending emails is not supported
+- FTP download handler is not supported (non-Python 3 ported Twisted
+ dependency)
+- Telnet is not supported (non-Python 3 ported Twisted dependency)
+- There are problems with non-ASCII URLs in Python 3
+- Reported problems with HTTP caches created by Scrapy in Python 2.x which
+ can't be reused in Scrapy in Python 3.x (to be checked)
+- There is also a nasty issue with `cryptography` library: recent versions
+ don't work well on OS X + Python 3.5
+ (see https://github.com/pyca/cryptography/issues/2690). As a workaround, you
+ can downgrade the library to an older version.
+
+Additional New Features and Enhancements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Scrapy now has a `Code of Conduct`_ (:issue:`1681`).
+- Command line tool now has completion for zsh (:issue:`934`).
+- Improvements to ``scrapy shell``:
+
+ - Support for bpython and configure preferred Python shell via
+ ``SCRAPY_PYTHON_SHELL`` (:issue:`1100`, :issue:`1444`).
+ - Support URLs without scheme (:issue:`1498`)
+ **Warning: backwards incompatible!**
+ - Bring back support for relative file path (:issue:`1710`, :issue:`1550`).
+
- - Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check
++- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check
+ interval (:issue:`1282`).
+- HTTP caching now follows RFC2616 more closely, added settings
+ :setting:`HTTPCACHE_ALWAYS_STORE` and
+ :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:issue:`1151`).
+- Download handlers are now lazy-loaded on first request using their
+ scheme (:issue:`1390`, :issue:`1421`).
+- ``RedirectMiddleware`` now skips the status codes from
+ ``handle_httpstatus_list`` on spider attribute
+ or in ``Request``'s ``meta`` key (:issue:`1334`, :issue:`1364`,
+ :issue:`1447`).
+- Form submission:
+
+ - now works with ``<button>`` elements too (:issue:`1469`).
+ - an empty string is now used for submit buttons without a value
+ (:issue:`1472`)
+
+- Item Loaders now support nested loaders (:issue:`1467`).
+- Dict-like settings now have per-key priorities
+ (:issue:`1135`, :issue:`1149` and :issue:`1586`).
+- robots.txt compliance now enabled by default for new projects (:issue:`1724`).
+- ``CloseSpider`` and ``SpiderState`` extensions now get disabled if no relevant
+ setting is set (:issue:`1723`, :issue:`1725`).
- - Added method ``ExecutionEngine.close`` (:issue:`1423`).
++- Added method ``ExecutionEngine.close`` (:issue:`1423`).
+- Added method ``CrawlerRunner.create_crawler`` (:issue:`1528`).
+- Tons of documentation updates and related fixes (:issue:`1291`, :issue:`1302`,
+ :issue:`1335`, :issue:`1683`, :issue:`1660`, :issue:`1642`, :issue:`1721`,
+ :issue:`1727`).
+- Other refactoring, optimizations and cleanup (:issue:`1476`, :issue:`1481`,
+ :issue:`1477` and :issue:`1315`).
+- Added ``to_bytes`` and ``to_unicode``, deprecated ``str_to_unicode`` and
+ ``unicode_to_str`` functions (:issue:`778`).
+- Extracted ``CrawlerRunner._crawl`` method which always expects ``Crawler``
+ instance (:issue:`1290`).
+.. _`Code of Conduct`: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
+
+
+Deprecations and Removals
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- The ``optional_features`` set has been removed (:issue:`1359`).
+- The ``--lsprof`` command line option has been removed (:issue:`1689`).
+ **Warning: backward incompatible**, but doesn't break user code.
+- The following datatypes were deprecated (:issue:`1720`):
+
+ + ``scrapy.utils.datatypes.MultiValueDictKeyError``
+ + ``scrapy.utils.datatypes.MultiValueDict``
+ + ``scrapy.utils.datatypes.SiteNode``
+
+
+Relocations
+~~~~~~~~~~~
+
+- Selectors were ported to use parsel_ (:issue:`1409`).
+- ``telnetconsole`` was relocated to ``extensions/`` (:issue:`1524`).
+
+ + Note: telnet is not enabled on Python 3
+ (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)
+
+.. _parsel: https://github.com/scrapy/parsel
+
+
+Bugfixes
+~~~~~~~~
+
+- Scrapy does not retry requests that got a ``HTTP 400 Bad Request``
+ response anymore (:issue:`1289`). **Warning: backwards incompatible!**
+- Support empty password for http_proxy config (:issue:`1274`).
+- Interpret ``application/x-json`` as ``TextResponse`` (:issue:`1333`).
+- Support link rel attribute with multiple values (:issue:`1201`).
+- Fixed ``scrapy.http.FormRequest.from_response`` when there is a ``<base>``
+ tag (:issue:`1564`).
+- Fixed :setting:`TEMPLATES_DIR` handling (:issue:`1575`).
+- Various ``FormRequest`` fixes (:issue:`1595`, :issue:`1596`, :issue:`1597`).
+- Makes ``_monkeypatches`` more robust (:issue:`1634`).
- - Fixed bug on ``XMLItemExporter`` with non-string fields in
++- Fixed bug on ``XMLItemExporter`` with non-string fields in
+ items (:issue:`1738`).
+- Fixed startproject command in OS X (:issue:`1635`).
+- Fixed PythonItemExporter and CSVExporter for non-string item
+ types (:issue:`1737`).
+- Various logging related fixes (:issue:`1294`, :issue:`1419`, :issue:`1263`,
+ :issue:`1624`, :issue:`1654`, :issue:`1722`, :issue:`1726` and :issue:`1303`).
+- Fixed bug in ``utils.template.render_templatefile()`` (:issue:`1212`).
+
+
+ 1.0.5 (2016-02-04)
+ ------------------
+
+ - FIX: [Backport] Ignore bogus links in LinkExtractors (fixes :issue:`907`, :commit:`108195e`)
+ - TST: Changed buildbot makefile to use 'pytest' (:commit:`1f3d90a`)
+ - DOC: Fixed typos in tutorial and media-pipeline (:commit:`808a9ea` and :commit:`803bd87`)
+ - DOC: Add AjaxCrawlMiddleware to DOWNLOADER_MIDDLEWARES_BASE in settings docs (:commit:`aa94121`)
+
++
1.0.4 (2015-12-30)
------------------