Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Updates to release notes

Valdir Stumm Jrdd7b0eb359dc
diff --git a/docs/news.rst b/docs/news.rst
index 900c6d1c5..25f459a49 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -3,107 +3,109 @@
Release notes
=============
-1.1.0 (unreleased)
-------------------
+1.1.0
+-----
+
+This 1.1 release brings a lot of interesting features and bug fixes:
+
+- Scrapy 1.1 has basic Python 3 support (requires Twisted >= 15.5). See
+ :ref:`news_basicpy3` for more details and some limitations.
+- Hot new features:
+
+ - ItemLoaders now support nested loaders (:issue:`1467`).
+ - ``FormRequest.from_response`` improvements (:issue:`1382`, :issue:`1137`).
+ - Added setting :setting:`AUTOTHROTTLE_TARGET_CONCURRENCY` and improved
+ AutoThrottle docs (:issue:`1324`).
+ - Added ``response.text`` to get body as unicode (:issue:`1730`).
+ - Anonymous S3 connections (:issue:`1358`).
+ - Deferreds in downloader middlewares (:issue:`1473`). This enables better
+ robots.txt handling (:issue:`1471`).
+ - HTTP cache improvements (:issue:`1151`).
+
+- These bug fixes may require your attention:
+
+ - Don't retry bad requests (HTTP 400) (:issue:`1289`).
+ - Fix shell files argument handling (:issue:`1710`, :issue:`1550`).
+ - Fixes on robots.txt handling (:issue:`1783`).
+ - Exporters work on unicode (:issue:`1080`).
+ - Fix xmliter to accept nodenames with dots (:issue:`1533`).
-Python 3 Support (basic)
-~~~~~~~~~~~~~~~~~~~~~~~~
+Keep reading for more details on other improvements and bug fixes.
+
+.. _news_basicpy3:
+
+Basic Python 3 Support
+~~~~~~~~~~~~~~~~~~~~~~
We have been hard at work to make Scrapy run on Python 3. As a result, now you
-can run spiders on Python 3.3, 3.4 and 3.5, although some features are still
-missing (some of them may never be ported to Python 3).
+can run spiders on Python 3.3, 3.4 and 3.5 (Twisted >= 15.5 required). Some
+features are still missing (and some may never be ported).
Almost all addons/middlewares are expected to work. However, we are aware of
some limitations:
-- s3 downloads are not supported (see :issue:`1718`)
-- sending emails is not supported
+- S3 downloads are not supported (see :issue:`1718`)
+- Sending emails is not supported
- FTP download handler is not supported (non-Python 3 ported Twisted
dependency)
-- telnet is not supported (non-Python 3 ported Twisted dependency)
-- there are problems with non-ASCII URLs in Python 3
-- reported problems with HTTP caches created by Scrapy in Python 2.x which
+- Telnet is not supported (non-Python 3 ported Twisted dependency)
+- There are problems with non-ASCII URLs in Python 3
+- Reported problems with HTTP caches created by Scrapy in Python 2.x which
can't be reused in Scrapy in Python 3.x (to be checked)
-- there is also a nasty issue with `cryptography` library: recent versions
+- There is also a nasty issue with `cryptography` library: recent versions
don't work well on OS X + Python 3.5
(see https://github.com/pyca/cryptography/issues/2690). As a workaround, you
can downgrade the library to an older version.
-New Features and Enhancements
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Additional New Features and Enhancements
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Scrapy now has a `Code of Conduct`_ (:issue:`1681`).
- Command line tool now has completion for zsh (:issue:`934`).
-- ``scrapy shell`` got a few changes of its own:
-
- - it now checks a new ``SCRAPY_PYTHON_SHELL`` environment
- variable to launch the interactive shell of your choice
- (one of ``ipython``, ``bpython`` or ``python`` that is);
- - it will try ``bpython`` if ``ipython`` is not available
- (:issue:`1444`).
- - it uses ``http://`` as the default scheme for URLs (:issue:`1498`)
- (try ``scrapy shell scrapy.org``)
+- Improvements to ``scrapy shell``:
+
+ - Support for bpython and configure preferred Python shell via
+ ``SCRAPY_PYTHON_SHELL`` (:issue:`1100`, :issue:`1444`).
+ - Support URLs without scheme (:issue:`1498`)
**Warning: backwards incompatible!**
- - unless argument looks like a relative file path, which works again;
- this was a regression identified in 1.0+ releases
- (:issue:`1710`, :issue:`1550`).
-
-- Autothrottle code has been cleaned up and its docs have been improved;
- there's also a new ``AUTOTHROTTLE_TARGET_CONCURRENCY`` setting which
- allows to send more than 1 concurrent request on average (:issue:`1324`).
-- Memory usage extension got a new ``MEMUSAGE_CHECK_INTERVAL_SECONDS``
- setting to change default check interval (:issue:`1282`).
-- HTTP caching now follows RFC2616 more closely (TODO: link to docs);
- 2 new settings can be used to control level of compliancy:
- ``HTTPCACHE_ALWAYS_STORE`` and ``HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS``
- (:issue:`1151`).
+ - Bring back support for relative file path (:issue:`1710`, :issue:`1550`).
+
+- Added :setting:`MEMUSAGE_CHECK_INTERVAL_SECONDS` setting to change default check
+ interval (:issue:`1282`).
+- HTTP caching now follows RFC2616 more closely, added settings
+ :setting:`HTTPCACHE_ALWAYS_STORE` and
+ :setting:`HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS` (:issue:`1151`).
- Download handlers are now lazy-loaded on first request using their
scheme (:issue:`1390`, :issue:`1421`).
-- RedirectMiddleware now skips the status codes from
- ``handle_httpstatus_list``. You can set it either as a spider attribute
- or in ``Request``'s ``meta`` key (:issue:`1334`, :issue:`1364`, :issue:`1447`).
+- ``RedirectMiddleware`` now skips the status codes from
+ ``handle_httpstatus_list`` on spider attribute
+ or in ``Request``'s ``meta`` key (:issue:`1334`, :issue:`1364`,
+ :issue:`1447`).
- Form submission:
- now works with ``<button>`` elements too (:issue:`1469`).
- - an empty string is now used for submit buttons without a ``value``
-
-- Scrapy does not retry requests that got a ``HTTP 400 Bad Request``
- response anymore (:issue:`1289`).
- **Warning: backwards incompatible!**
-- Middlewares now can return deferreds (:issue:`1473`);
+ - an empty string is now used for submit buttons without a value
+ (:issue:`1472`)
- - as a consequence, ``RobotsTxtMiddleware`` now fully respects `robots.txt`
- (:issue:`1471`).
-
-- ItemLoaders now support nested loaders (:issue:`1467`).
-- dict-like settings now have per-key priorities
+- Item Loaders now support nested loaders (:issue:`1467`).
+- Dict-like settings now have per-key priorities
(:issue:`1135`, :issue:`1149` and :issue:`1586`).
-- Anonymous S3 connections are now supported (:issue:`1358`).
-- ``/robots.txt`` compliance is now enabled by default for new projects (:issue:`1724`).
- **Warning: backwards incompatible**
+- robots.txt compliance now enabled by default for new projects (:issue:`1724`).
- ``CloseSpider`` and ``SpiderState`` extensions now get disabled if no relevant
setting is set (:issue:`1723`, :issue:`1725`).
-
+- Added method ``ExecutionEngine.close`` (:issue:`1423`).
+- Added method ``CrawlerRunner.create_crawler`` (:issue:`1528`).
+- Tons of documentation updates and related fixes (:issue:`1291`, :issue:`1302`,
+ :issue:`1335`, :issue:`1683`, :issue:`1660`, :issue:`1642`, :issue:`1721`,
+ :issue:`1727`).
+- Other refactoring, optimizations and cleanup (:issue:`1476`, :issue:`1481`,
+ :issue:`1477` and :issue:`1315`).
+- Added ``to_bytes`` and ``to_unicode``, deprecated ``str_to_unicode`` and
+ ``unicode_to_str`` functions (:issue:`778`).
+- Extracted ``CrawlerRunner._crawl`` method which always expects ``Crawler``
+ instance (:issue:`1290`).
.. _`Code of Conduct`: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
-API changes
-~~~~~~~~~~~
-
-- ``FormRequest.from_response`` now allows:
-
- - to set which Form you want to get through a new ``formid`` parameter (:issue:`1137`).
-
- - to define through CSS selectors which form from the response should be used.
- It previously supported only XPath (:issue:`1382`).
-
-- New methods:
-
- - ``ExecutionEngine.close`` (:issue:`1423`).
- - ``CrawlerRunner.create_crawler`` (:issue:`1528`).
-
-- ``.text`` is a new attribute on ``TextResponse``. It's a shortcut to the
- ``.body_as_unicode()`` method (:issue:`1730`).
-
Deprecations and Removals
~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -111,8 +113,7 @@ Deprecations and Removals
- The ``optional_features`` set has been removed (:issue:`1359`).
- The ``--lsprof`` command line option has been removed (:issue:`1689`).
**Warning: backward incompatible**, but doesn't break user code.
-- The following datatypes were deprecated, because they were not used anywhere
- (:issue:`1720`):
+- The following datatypes were deprecated (:issue:`1720`):
+ ``scrapy.utils.datatypes.MultiValueDictKeyError``
+ ``scrapy.utils.datatypes.MultiValueDict``
@@ -123,192 +124,35 @@ Relocations
~~~~~~~~~~~
- Selectors were ported to use parsel_ (:issue:`1409`).
-- ``telnetconsole`` was relocated to ``extensions/`` (:issue:`1524`).\
+- ``telnetconsole`` was relocated to ``extensions/`` (:issue:`1524`).
+ Note: telnet is not enabled on Python 3
(https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)
.. _parsel: https://github.com/scrapy/parsel
-Documentation
-~~~~~~~~~~~~~
-
-- DOC SignalManager docstrings. See GH-713. #1291 (https://github.com/scrapy/scrapy/commit/5bd0395be4dc6d8315ad2726f1dbbd9c0b57b143)
-- Improvements for docs on how to access settings #1302 (https://github.com/scrapy/scrapy/commit/8b3ca4f250b4d831403c7fcfa72efe7ecdfa5247)
- (closes: https://github.com/scrapy/scrapy/issues/1300)
-- Make Sphinx autodoc use local, not system-wide Scrapy PR #1335 (https://github.com/scrapy/scrapy/commit/b6eb3404a287508949ddb215e3f553a10fe43b8c)
-- DOCS: Update deprecated examples #1660 (https://github.com/scrapy/scrapy/commit/95e8ff8ba1dff3ec045dce931b6ea4314e887399)
-- DOCS: Update Stats Collection documentation for @master #1683 (https://github.com/scrapy/scrapy/commit/3f1f15bc4d3ee81612bce00fa0106ed16a7f72e5)
-- DOCS: DOC: Update MetaRefreshMiddlware's setting variables #1642 (https://github.com/scrapy/scrapy/commit/b1e44436bc4629773388d25ad9ab7b8ecf43d15e)
-
- REDIRECT_MAX_METAREFRESH_DELAY has been deprecated and was renamed to METAREFRESH_MAXDELAY.
- Merge duplicate documents about METAREFRESH_MAXDELAY appeared both in the settings page and the downloader-middlewares page.
-
- Leftover from https://github.com/scrapy/scrapy/commit/defc4f89b542b756276f0920921dc00fe3ec4675
-- DOCS;TESTS: tests+doc for subdomains in offsite middleware #1721
-- DOCS: Clarify priority adjust settings docs #1727
-
Bugfixes
~~~~~~~~
-- Support empty password for http_proxy config #1313 (https://github.com/scrapy/scrapy/commit/07f4f12e8b5417fe3e9f70560f7b60bc488570e8)
- Fixes #1274 HTTP_PROXY variable with username and empty password not supported
-- interpreting application/x-json as TextResponse #1333 (https://github.com/scrapy/scrapy/commit/2a7dc31f4cab7b13aacb632bdc78c50af754e76f)
-- Support link rel attribute with multiple values #1214 (https://github.com/scrapy/scrapy/commit/aa31811cfdc85eda07ddab25178d5003155523ec)
- Fixes: nofollow doesnt work correcly when there multiple values in rel attribute #1201
-- BUG FIX: for Incorrectly picked URL in `scrapy.http.FormRequest.from_response` when there is a `<base>` tag #1562
- PR #1563 (https://github.com/scrapy/scrapy/commit/9548691fdd47077a53f85daace091ef4af599cb9)
-- Startproject templates override #1575 (https://github.com/scrapy/scrapy/commit/3881eaff456d0d2704aa126f7c389080580d8f6c)
- Closes: Override of TEMPLATES_DIR does not work for "startproject" command (https://github.com/scrapy/scrapy/issues/671)
-- BUG FIX: Various FormRequest tests+fixes #1597 (https://github.com/scrapy/scrapy/commit/dc6502639556efbd06d45319efa8320e84e88fde)
- Fixes: FormRequest should consider input type values case-insensitive #1595
- Fixes: FormRequest doesn't handle input elements without type attribute #1596
-- BUG FIX: for Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. #1564
- PR #1565 (https://github.com/scrapy/scrapy/commit/17aba44f169fc3a86b6a1f46f30cf5fe29500db1)
-- BUG FIX: BF: robustify _monkeypatches check for twisted - str() name first (Closes #1634) #1644 (https://github.com/scrapy/scrapy/commit/57f99fc34ebc7cb8a2a84371b89552e6623c9e9d)
- Fixes: https://github.com/scrapy/scrapy/issues/1634
-- Fix bug on XMLItemExporter with non-string fields in items #1747
- Fixes: AttributeError when exporting non-string types through XMLFeedExporter #1738
-- change os.mknod() for open() #1657
- Fixes: Test for startproject command fails in OS X #1635
-- BUG FIX: Fix PythonItemExporter and CSVExporter for non-string item types #1737
-
-
-Python 3 porting effort
-~~~~~~~~~~~~~~~~~~~~~~~
-
-- Python 3: PY3 port scrapy.utils.python PR #1379
-- Python 3: In-progress Python 3 port PR #1384
- TODO: worth describing?
-- Python 3: fix form requests tests on py3 (https://github.com/scrapy/scrapy/commit/de6e013b9a8080cf759096e793272f6814e3617d)
-- Python 3: Port scrapy/responsetypes.py https://github.com/scrapy/scrapy/commit/d05cf6e0af8c26863cbb1edc7a8199165eaeeb5d
-- Python 3: remove scrapy.utils.testsite from PY3 ignores #1397
-- Python 3: PY3 port scrapy.utils.response #1396
-- Python 3: PY3 port http cookies handling #1398 (https://github.com/scrapy/scrapy/commit/95e6bd2f8da9c0ed79c3667ae0619d35541de346)
-
-- Python 3: PY3 port scrapy.utils.reqser #1408 (https://github.com/scrapy/scrapy/commit/311293ffdc63892bd5ab8494310529a6da0f5b62)
-
-- Python 3: nyov's PY3 changes #1415
- Various files:
- requirements-py3.txt
- scrapy/cmdline.py
- scrapy/core/downloader/handlers/s3.py
- scrapy/core/downloader/middleware.py
- scrapy/core/spidermw.py
- scrapy/linkextractors/htmlparser.py
- scrapy/pipelines/files.py
- scrapy/pipelines/images.py
- scrapy/utils/testproc.py
- tests/py3-ignores.txt
- tests/requirements-py3.txt
- tests/test_cmdline/__init__.py
- tests/test_command_version.py
- tests/test_crawl.py
- tests/test_loader.py
- tests/test_pipeline_files.py
- tests/test_pipeline_images.py
- tests/test_selector_csstranslator.py
- tests/test_selector_lxmldocument.py
- tests/test_utils_iterators.py
- tests/test_utils_reqser.py
- tox.ini
-- Python 3: py3: port dictionary itervalues call (666ebfa1d97264bc4e6adb78fe4ce1a9ea15cc1f)
-- Python 3: PY3: port scrapy.utils.trackref #1420 (https://github.com/scrapy/scrapy/commit/fa3d84b0504e25f7478f7fac723a45)
-- Python 3: Small Python 3 fixes #1456 (https://github.com/scrapy/scrapy/commit/026a1caffb9f0bafbefba4f56af61a7347750f20)
-- Python 3: enable console tests in PY3 (8ecc4544b3747eb9be33153483b62c6441bd7c56)
-- Python 3: assorted Python 3 porting #1461 (https://github.com/scrapy/scrapy/commit/0018caf0b61e4f10857e61cddb347c3854bacc4b)
- Port LxmlLinkExtractor and leave other link extractors Python 2.x - only.
-
- refactor test_linkextractors
- move tests for deprecated link extractors to another file and ignore it in Python 3
- port LxmlLinkExtractor to Python 3
- + scrapy.spiders and a couple more things
-
-- port some downloader middlewares to Python 3 #1470 (https://github.com/scrapy/scrapy/commit/3919ad64c5873d360aa1a412bee5270aad121760)
- scrapy/downloadermiddlewares/httpauth.py
- scrapy/downloadermiddlewares/useragent.py
-- Python 3: PY3 redirect downloader mware #1488 (https://github.com/scrapy/scrapy/commit/4d1c5c3d32591c37e37f879f0e77e50db7124603)
-- PY3 port bench, startproject, genspider, list and runspider commands #1535 (https://github.com/scrapy/scrapy/commit/411174cf38ebda00422529637b427a591c114eff)
- Fixes: PY3 enable test_commands.ParseCommandTest #1536
-- Python 3:
- - py3: fix webclient #1676 (https://github.com/scrapy/scrapy/commit/49fe631d8946f87e783c59e44a498f3d43083e2e)
- - Py3: port http downloaders #1678 (https://github.com/scrapy/scrapy/commit/b4fb9d35342bc41a0149b74ecca38c056beaa220)
- - Raise minimal twisted version for py3 #1694 (https://github.com/scrapy/scrapy/commit/d59d3f1e296795116704baa01780ff11870257f1)
- - Cleanup http11 tunneling connection after #1678 #1701
- - Py3: port downloader cache and compression middlewares #1680
- - Add Python 3.5 tox env + Python 3.5 tests in Travis #1674 (https://github.com/scrapy/scrapy/commit/8fb9a6f8191dc0bf2dfb39ef01b1eb63e49bc23b)
- - Py3: port test_engine #1691
- - Py3: port commands fetch and shell #1693
- - py3 fix HttpProxy and Retry Middlewares #1637
- - PY3 fixed scrapy bench command #1708
- - Py3: port test crawl #1692
- - PY3 enable tests for scrapy parse command #1711
- - py3: fix test_mail #1715
- - py3: reviewed passing test_spidermiddleware_httperror.py #1717
- - py3: test_pipeline_files and test_pipeline_images #1716
- - PY3 exporters #1499
- - PY3 fix downloader slots GC #1741
-- Python 3: PY3: port utils/iterators #1661 (https://github.com/scrapy/scrapy/commit/f01fd076420f0e58a1a165be31ec505eeb561ef4)
-
-
-Tests, CI and Deploys
-~~~~~~~~~~~~~~~~~~~~~
-
-- BF: fail if docs failed to build #1319
-- Run on new travis-ci infra (https://github.com/scrapy/scrapy/commit/805a491647fabfed58acb9d2)
- no more travis workarounds (removed .travis-workarounds.sh)
-- Unset environment proxies for tests #1353 (https://github.com/scrapy/scrapy/commit/cbfb24dbeb82c791e82f1d9249685aa4d75fed3e)
-- Coverage and reports at codecov.io and coveralls.io #1433 (https://github.com/scrapy/scrapy/commit/9adb5c31c06bc22d1b5243a04633a)
-- drop coveralls support #1537 (https://github.com/scrapy/scrapy/commit/65f4ba349cb341736b67c0307074cef2cf0bd12e)
-- Add some missing tests for scrapy.settings #1570 (https://github.com/scrapy/scrapy/commit/9424ca0fdbdd492f3049fe08be8848f92e84fde3)
-- DOCS;TESTS: tests+doc for subdomains in offsite middleware #1721
-- TESTS: Include tests for non-string items to Exporters #1742
-
-
-Logging
-~~~~~~~
-
-- Ignore ScrapyDeprecationWarning warnings properly (:issue:`1294`)
-- Do not fail representing non-HTTP requests (:issue:`1419`)
-- Make list of enabled middlewares more readable (:issue:`1263`)
-- Be more verbose when download is cancelled because of size limit (:issue:`1624`)
-- Show warning on download size only once (:issue:`1654`)
-- Fix logging of enabled middlewares in startup logs (:issue:`1722` and :issue:`1726`)
-
-
-Code refactoring
-~~~~~~~~~~~~~~~~
-
-- Optimization on lists (:issue:`1476` and :issue:`1481`)
-- equal_attributes function optimization (:issue:`1477`)
-- Downloader slots cleanup (:issue:`1315`)
-
-
-Other changes
-~~~~~~~~~~~~~
-
-
-- Extend regex for tags that deploy to PyPI to support new release cycle (:commit:`26f50d3`)
-- rename str_to_unicode and unicode_to_str functions (ISSUE #778) (https://github.com/scrapy/scrapy/commit/61cd27e5c7b777a54)
-
-- fix utils.template.render_templatefile() bug +test #1212 (https://github.com/scrapy/scrapy/commit/71bd79e70fb10ed4899b15ca3ffa9aaa16567727)
-
-- style fixes for settings.py created by `scrapy startproject` #1496 (https://github.com/scrapy/scrapy/commit/5279da9916c00c7a6679cfc555f9a2b1863b4821)
- Adds AUTOTHROTTLE_TARGET_CONCURRENCY to settings.py
-
-- (MINOR) Simplify if statement #1686 (https://github.com/scrapy/scrapy/commit/9ef25d7b68fe90c5e6b94bd3e81755089e743080)
- Note: in conftest.py
-
-- (MINOR) fix indentation #1687 (https://github.com/scrapy/scrapy/commit/66f41aba3cbfa642b37354e8419e3d1437b88348)
- Note: in scrapy/downloadermiddlewares/retry.py
-- (MINOR) fixed typo You -> you #1698 (https://github.com/scrapy/scrapy/commit/e8b26e2ab25ac7ec15c03d3c0b766c7aa8f48cce)
- Fixes DOWNLOAD_WARNSIZE is too verbose #1303
-
-- extract CrawlerRunner._crawl method which always expects Crawler #1290 (https://github.com/scrapy/scrapy/commit/5bcda9b7d13b9c3b486c2b247fd6d87a7b59df1a)
- Provides an extension point where crawler instance is available;
- makes it easier to write alternative CrawlerRunner.crawl implementations.
- User can override CrawlerRunner._crawl method and connect signals there.
+- Scrapy does not retry requests that got a ``HTTP 400 Bad Request``
+ response anymore (:issue:`1289`). **Warning: backwards incompatible!**
+- Support empty password for http_proxy config (:issue:`1274`).
+- Interpret ``application/x-json`` as ``TextResponse`` (:issue:`1333`).
+- Support link rel attribute with multiple values (:issue:`1201`).
+- Fixed ``scrapy.http.FormRequest.from_response`` when there is a ``<base>``
+ tag (:issue:`1564`).
+- Fixed :setting:`TEMPLATES_DIR` handling (:issue:`1575`).
+- Various ``FormRequest`` fixes (:issue:`1595`, :issue:`1596`, :issue:`1597`).
+- Makes ``_monkeypatches`` more robust (:issue:`1634`).
+- Fixed bug on ``XMLItemExporter`` with non-string fields in
+ items (:issue:`1738`).
+- Fixed startproject command in OS X (:issue:`1635`).
+- Fixed PythonItemExporter and CSVExporter for non-string item
+ types (:issue:`1737`).
+- Various logging related fixes (:issue:`1294`, :issue:`1419`, :issue:`1263`,
+ :issue:`1624`, :issue:`1654`, :issue:`1722`, :issue:`1726` and :issue:`1303`).
+- Fixed bug in ``utils.template.render_templatefile()`` (:issue:`1212`).
1.0.4 (2015-12-30)