Scrapy, a fast high-level web crawling & scraping framework for Python.
Add 1.1 release notes (draft)
Paul Tremberthdb0697bc0654
diff --git a/docs/news.rst b/docs/news.rstindex 4d7dc4d41..6a83a6163 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -3,6 +3,336 @@Release notes=============+1.1.0 (unreleased)+------------------++Python 3 Support (basic)+~~~~~~~~~~~~~~~~~~~~~~~~++We have been hard at work to make Scrapy work on Python 3. Some features+are still missing (and may never be ported to Python 3, see below),+but you can now run spiders on Python 3.3, 3.4 and 3.5.++Almost all of addons/middleware should work, but here are the current+limitations we know of:++- s3 downloads are not supported (see :issue:`1718`)+- sending emails is not supported+- FTP download handler is not supported (non-Python 3 ported Twisted dependency)+- telnet is not supported (non-Python 3 ported Twisted dependency)+- there are problems with non-ASCII URLs in Python 3+- reported problems with HTTP cache created in Python 2.x which can't be used in 3.x (to be checked)+- there is also a nasty issue with cryptography library:+ recent versions don't work well on OS X + Python 3.5 (see https://github.com/pyca/cryptography/issues/2690),+ downgrading to an older version helps+++New Features and Enhancements+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~++- Command line tool has completion for zsh (:issue:`934`)+- ``scrapy shell`` works with local files again ; this was a regression+ identified in 1.0+ releases (:issue:`1710`, :issue:`1550`)+- ``scrapy shell`` now also checks a new ``SCRAPY_PYTHON_SHELL`` environment+ variable to launch the interactive shell of your choice ;+ ``bpython`` is a newly supported option too (:issue:`1444`)+- Autothrottle has gotten a code cleanup and better docs ;+ there's also a new ``AUTOTHROTTLE_TARGET_CONCURRENCY`` setting which+ allows to send more than 1 concurrent request on average (:issue:`1324`)+- Memory usage extension has a new ``MEMUSAGE_CHECK_INTERVAL_SECONDS``+ setting to change default check interval (:issue:`1282`)+- HTTP caching follows RFC2616 more closely (TODO: link to docs);+ 2 new settings can be used to control level of compliancy:+ ``HTTPCACHE_ALWAYS_STORE`` and ``HTTPCACHE_IGNORE_RESPONSE_CACHE_CONTROLS``+ (:issue:`1151`)+- Scheme Download handlers are now lazy-loaded on first request using+ that scheme (``http(s)://``, ``ftp://``, ``file://``, ``s3://``)+ (:issue:`1390`, :issue:`1421`)+- RedirectMiddleware now skips status codes in ``handle_httpstatus_list``,+ set either as spider attribute or ``Request``'s ``meta`` key+ (:issue:`1334`, :issue:`1364`, :issue:`1447`)+++- Form submit button plain #1469 (https://github.com/scrapy/scrapy/commit/b876755f1cee619d8c421357777d223037d5289c)+ Fixes: Form submit button (https://github.com/scrapy/scrapy/issues/1354)+- Implement FormRequest.from_response CSS support #1382 (https://github.com/scrapy/scrapy/commit/a6e5c848feb672c117f3380976077b6d0f42e3a6)+ + Fix version number to appear new feature #1706++- Incomplete submit button #1472 (https://github.com/scrapy/scrapy/commit/bc499cb552dad362494b86082e47d1f732095874)++- dont retry 400 #1289 (https://github.com/scrapy/scrapy/milestones/Scrapy%201.1)+ + DOC fix docs after GH-1289. #1530 (https://github.com/scrapy/scrapy/commit/451318ef7a4e8ee7837b83e73b158da98f579980)+ WARNING: BACKWARDS INCOMPATIBLE!+- DOC fix docs after GH-1289. #1530 (https://github.com/scrapy/scrapy/commit/451318ef7a4e8ee7837b83e73b158da98f579980)++- Support for returning deferreds in middlewares #1473 (https://github.com/scrapy/scrapy/commit/dd473145f2e1ae2d3c9462c489f3289a96e447f4)+ Adds support for returning deferreds in middlewares, and makes use of this to fix a limitation in RobotsTxtMiddleware.+ Fixes #1471+- add support for a nested loaders #1467 (https://github.com/scrapy/scrapy/commit/3c596dcf4606315e4eb88608e3ecde430fe18c29)+ Closes: https://github.com/scrapy/scrapy/pull/818+ Adds a nested_xpath()/nested_css() methods to ItemLoader. (TODO: add links to docs)++- add_scheme_if_missing for `scrapy shell` command #1498 (https://github.com/scrapy/scrapy/commit/fe15f93e533be36e81e0385691fe5571c88b0b31)+ Fixes: #1487+ Warning: backward incompatible++ + see: https://github.com/scrapy/scrapy/issues/1550, https://github.com/scrapy/scrapy/pull/1710+- Per-key priorities for dict-like settings by promoting dicts to Settings instances #1149 (https://github.com/scrapy/scrapy/commit/dd9f777ba725d7a7dbb192302cc52a120005ad64)+ + Backwards compatible per key priorities #1586 (https://github.com/scrapy/scrapy/commit/54216d7afe9d545031c57b5821f2c821faa2ccc3)++ Fixes: Per-key priorities for dictionary-like settings #1135+ Obsoletes: Settings.updatedict() method to update dictionary-like settings #1110+- Support anonymous S3DownloadHandler (boto) connections #1358 (https://github.com/scrapy/scrapy/commit/5ec4319885e4be87b0248cb80b5213f68829129e)+ + optional_features has been removed #1699++- Enable robots.txt handling by default for new projects. #1724 (https://github.com/scrapy/scrapy/commit/0d368c5d6fd468aed301ed5967f8bfe9d5e86101)+ WARNING: backwards incompatible++- Disable CloseSpider extension if no CLOSPIDER_* setting set #1723 (https://github.com/scrapy/scrapy/commit/2246280bb6f71d7d52e24aca5b4ce955b3aa1363)+- Disable SpiderState extension if no JOBDIR set #1725++- Add Code of Conduct Version 1.3.0 from http://contributor-covenant.org/ #1681+++API changes+~~~~~~~~~~~++- Update form.py to improve existing capability PR #1137 (https://github.com/scrapy/scrapy/commit/786f62664b41f264bf4213a1ee3805774d82ed69)+ Adds "formid" parameter for Form from_response()++- Add ExecutionEngine.close() method #1423 (https://github.com/scrapy/scrapy/commit/caf2080b8095acd11de6018911025076ead23585)+ Adds a new method as a single entry point for shutting down the engine+ and integrates it into Crawler.crawl() for graceful error handling during the crawling process.++ TODO: explain what this does+- public Crawler.create_crawler method #1528 (https://github.com/scrapy/scrapy/commit/57f87b95d4d705f8afdd8fb9f7551033a7d88ee2)+ Note: this is a Core API change+ Note: this is CrawlerRunner.create_crawler(), not Crawler.create_crawler+ http://doc.scrapy.org/en/master/topics/api.html?#scrapy.crawler.CrawlerRunner.create_crawler++ Return a Crawler object.++ If crawler_or_spidercls is a Crawler, it is returned as-is.+ If crawler_or_spidercls is a Spider subclass, a new Crawler is constructed for it.+ If crawler_or_spidercls is a string, this function finds a spider with this name in a Scrapy project (using spider loader), then creates a Crawler instance for it.++- API CHANGE: response.text #1730 + micro-optimize response.text #1740+ New `.text` attribute on TextResponses+ Response body, as unicode.+++Deprecations and Removals+~~~~~~~~~~~~~~~~~~~~~~~~~++- drop deprecated "optional_features" set #1359 (https://github.com/scrapy/scrapy/commit/7d187735ffecb0f49cffce1a9058961146212f59)+- Remove --lsprof command-line option. #1689 (https://github.com/scrapy/scrapy/commit/56b69d2ea85ccdebfa5ec7945f1ed1df54b4b87f)+ WARNING: backward incompatible, but doesnt break user code++- deprecated unused and untested code in scrapy.utils.datatypes #1720+ DEPRECATION: these will be removed in next releases+ scrapy.utils.datatypes.MultiValueDictKeyError+ scrapy.utils.datatypes.MultiValueDict+ scrapy.utils.datatypes.SiteNode+++Relocations+~~~~~~~~~~~++- Migrating selectors to use parsel #1409 (https://github.com/scrapy/scrapy/commit/15c1300d35e4764ea343d98c133bc83f7c90c2d6)+ + Replace usage of deprecated class by its parsel\'s counterpart #1431 (https://github.com/scrapy/scrapy/commit/12bebb61725272cdd977ce914d18a4b18ec0cb77)+ closes Scrapy.selector Enhancement Proposal (https://github.com/scrapy/scrapy/issues/906)+- Relocate telnetconsole to extensions/ #1524 (https://github.com/scrapy/scrapy/commit/72eeead6db7a5fdbce49a59102bb6a7125d56bc1)+ Fixes: Move scrapy.telnet to scrapy.extensions.telnet #1520++ See discussion on disabling telnet by default: (still open) https://github.com/scrapy/scrapy/issues/1572+ Note that telnet is not enabled on Python 3 (https://github.com/scrapy/scrapy/pull/1524#issuecomment-146985595)+++Documentation+~~~~~~~~~~~~~++- DOC SignalManager docstrings. See GH-713. #1291 (https://github.com/scrapy/scrapy/commit/5bd0395be4dc6d8315ad2726f1dbbd9c0b57b143)+- Improvements for docs on how to access settings #1302 (https://github.com/scrapy/scrapy/commit/8b3ca4f250b4d831403c7fcfa72efe7ecdfa5247)+ (closes: https://github.com/scrapy/scrapy/issues/1300)+- Make Sphinx autodoc use local, not system-wide Scrapy PR #1335 (https://github.com/scrapy/scrapy/commit/b6eb3404a287508949ddb215e3f553a10fe43b8c)+- DOCS: Update deprecated examples #1660 (https://github.com/scrapy/scrapy/commit/95e8ff8ba1dff3ec045dce931b6ea4314e887399)+- DOCS: Update Stats Collection documentation for @master #1683 (https://github.com/scrapy/scrapy/commit/3f1f15bc4d3ee81612bce00fa0106ed16a7f72e5)+- DOCS: DOC: Update MetaRefreshMiddlware's setting variables #1642 (https://github.com/scrapy/scrapy/commit/b1e44436bc4629773388d25ad9ab7b8ecf43d15e)++ REDIRECT_MAX_METAREFRESH_DELAY has been deprecated and was renamed to METAREFRESH_MAXDELAY.+ Merge duplicate documents about METAREFRESH_MAXDELAY appeared both in the settings page and the downloader-middlewares page.++ Leftover from https://github.com/scrapy/scrapy/commit/defc4f89b542b756276f0920921dc00fe3ec4675+- DOCS;TESTS: tests+doc for subdomains in offsite middleware #1721+- DOCS: Clarify priority adjust settings docs #1727+++Bugfixes+~~~~~~~~++- Support empty password for http_proxy config #1313 (https://github.com/scrapy/scrapy/commit/07f4f12e8b5417fe3e9f70560f7b60bc488570e8)+ Fixes #1274 HTTP_PROXY variable with username and empty password not supported+- interpreting application/x-json as TextResponse #1333 (https://github.com/scrapy/scrapy/commit/2a7dc31f4cab7b13aacb632bdc78c50af754e76f)+- Support link rel attribute with multiple values #1214 (https://github.com/scrapy/scrapy/commit/aa31811cfdc85eda07ddab25178d5003155523ec)+ Fixes: nofollow doesnt work correcly when there multiple values in rel attribute #1201+- BUG FIX: for Incorrectly picked URL in `scrapy.http.FormRequest.from_response` when there is a `<base>` tag #1562+ PR #1563 (https://github.com/scrapy/scrapy/commit/9548691fdd47077a53f85daace091ef4af599cb9)+- Startproject templates override #1575 (https://github.com/scrapy/scrapy/commit/3881eaff456d0d2704aa126f7c389080580d8f6c)+ Closes: Override of TEMPLATES_DIR does not work for "startproject" command (https://github.com/scrapy/scrapy/issues/671)+- BUG FIX: Various FormRequest tests+fixes #1597 (https://github.com/scrapy/scrapy/commit/dc6502639556efbd06d45319efa8320e84e88fde)+ Fixes: FormRequest should consider input type values case-insensitive #1595+ Fixes: FormRequest doesn't handle input elements without type attribute #1596+- BUG FIX: for Incorrectly picked URL in `scrapy.linkextractors.regex.RegexLinkExtractor` when there is a `<base>` tag. #1564+ PR #1565 (https://github.com/scrapy/scrapy/commit/17aba44f169fc3a86b6a1f46f30cf5fe29500db1)+- BUG FIX: BF: robustify _monkeypatches check for twisted - str() name first (Closes #1634) #1644 (https://github.com/scrapy/scrapy/commit/57f99fc34ebc7cb8a2a84371b89552e6623c9e9d)+ Fixes: https://github.com/scrapy/scrapy/issues/1634+- Fix bug on XMLItemExporter with non-string fields in items #1747+ Fixes: AttributeError when exporting non-string types through XMLFeedExporter #1738+- change os.mknod() for open() #1657+ Fixes: Test for startproject command fails in OS X #1635+- BUG FIX: Fix PythonItemExporter and CSVExporter for non-string item types #1737+++Python 3 porting effort+~~~~~~~~~~~~~~~~~~~~~~~++- Python 3: PY3 port scrapy.utils.python PR #1379+- Python 3: In-progress Python 3 port PR #1384+ TODO: worth describing?+- Python 3: fix form requests tests on py3 (https://github.com/scrapy/scrapy/commit/de6e013b9a8080cf759096e793272f6814e3617d)+- Python 3: Port scrapy/responsetypes.py https://github.com/scrapy/scrapy/commit/d05cf6e0af8c26863cbb1edc7a8199165eaeeb5d+- Python 3: remove scrapy.utils.testsite from PY3 ignores #1397+- Python 3: PY3 port scrapy.utils.response #1396+- Python 3: PY3 port http cookies handling #1398 (https://github.com/scrapy/scrapy/commit/95e6bd2f8da9c0ed79c3667ae0619d35541de346)++- Python 3: PY3 port scrapy.utils.reqser #1408 (https://github.com/scrapy/scrapy/commit/311293ffdc63892bd5ab8494310529a6da0f5b62)++- Python 3: nyov's PY3 changes #1415+ Various files:+ requirements-py3.txt+ scrapy/cmdline.py+ scrapy/core/downloader/handlers/s3.py+ scrapy/core/downloader/middleware.py+ scrapy/core/spidermw.py+ scrapy/linkextractors/htmlparser.py+ scrapy/pipelines/files.py+ scrapy/pipelines/images.py+ scrapy/utils/testproc.py+ tests/py3-ignores.txt+ tests/requirements-py3.txt+ tests/test_cmdline/__init__.py+ tests/test_command_version.py+ tests/test_crawl.py+ tests/test_loader.py+ tests/test_pipeline_files.py+ tests/test_pipeline_images.py+ tests/test_selector_csstranslator.py+ tests/test_selector_lxmldocument.py+ tests/test_utils_iterators.py+ tests/test_utils_reqser.py+ tox.ini+- Python 3: py3: port dictionary itervalues call (666ebfa1d97264bc4e6adb78fe4ce1a9ea15cc1f)+- Python 3: PY3: port scrapy.utils.trackref #1420 (https://github.com/scrapy/scrapy/commit/fa3d84b0504e25f7478f7fac723a45)+- Python 3: Small Python 3 fixes #1456 (https://github.com/scrapy/scrapy/commit/026a1caffb9f0bafbefba4f56af61a7347750f20)+- Python 3: enable console tests in PY3 (8ecc4544b3747eb9be33153483b62c6441bd7c56)+- Python 3: assorted Python 3 porting #1461 (https://github.com/scrapy/scrapy/commit/0018caf0b61e4f10857e61cddb347c3854bacc4b)+ Port LxmlLinkExtractor and leave other link extractors Python 2.x - only.++ refactor test_linkextractors+ move tests for deprecated link extractors to another file and ignore it in Python 3+ port LxmlLinkExtractor to Python 3+ + scrapy.spiders and a couple more things++- port some downloader middlewares to Python 3 #1470 (https://github.com/scrapy/scrapy/commit/3919ad64c5873d360aa1a412bee5270aad121760)+ scrapy/downloadermiddlewares/httpauth.py+ scrapy/downloadermiddlewares/useragent.py+- Python 3: PY3 redirect downloader mware #1488 (https://github.com/scrapy/scrapy/commit/4d1c5c3d32591c37e37f879f0e77e50db7124603)+- PY3 port bench, startproject, genspider, list and runspider commands #1535 (https://github.com/scrapy/scrapy/commit/411174cf38ebda00422529637b427a591c114eff)+ Fixes: PY3 enable test_commands.ParseCommandTest #1536+- Python 3:+ - py3: fix webclient #1676 (https://github.com/scrapy/scrapy/commit/49fe631d8946f87e783c59e44a498f3d43083e2e)+ - Py3: port http downloaders #1678 (https://github.com/scrapy/scrapy/commit/b4fb9d35342bc41a0149b74ecca38c056beaa220)+ - Raise minimal twisted version for py3 #1694 (https://github.com/scrapy/scrapy/commit/d59d3f1e296795116704baa01780ff11870257f1)+ - Cleanup http11 tunneling connection after #1678 #1701+ - Py3: port downloader cache and compression middlewares #1680+ - Add Python 3.5 tox env + Python 3.5 tests in Travis #1674 (https://github.com/scrapy/scrapy/commit/8fb9a6f8191dc0bf2dfb39ef01b1eb63e49bc23b)+ - Py3: port test_engine #1691+ - Py3: port commands fetch and shell #1693+ - py3 fix HttpProxy and Retry Middlewares #1637+ - PY3 fixed scrapy bench command #1708+ - Py3: port test crawl #1692+ - PY3 enable tests for scrapy parse command #1711+ - py3: fix test_mail #1715+ - py3: reviewed passing test_spidermiddleware_httperror.py #1717+ - py3: test_pipeline_files and test_pipeline_images #1716+ - PY3 exporters #1499+ - PY3 fix downloader slots GC #1741+- Python 3: PY3: port utils/iterators #1661 (https://github.com/scrapy/scrapy/commit/f01fd076420f0e58a1a165be31ec505eeb561ef4)+++Tests, CI and Deploys+~~~~~~~~~~~~~~~~~~~~~++- BF: fail if docs failed to build #1319+- Run on new travis-ci infra (https://github.com/scrapy/scrapy/commit/805a491647fabfed58acb9d2)+ no more travis workarounds (removed .travis-workarounds.sh)+- Unset environment proxies for tests #1353 (https://github.com/scrapy/scrapy/commit/cbfb24dbeb82c791e82f1d9249685aa4d75fed3e)+- Coverage and reports at codecov.io and coveralls.io #1433 (https://github.com/scrapy/scrapy/commit/9adb5c31c06bc22d1b5243a04633a)+- drop coveralls support #1537 (https://github.com/scrapy/scrapy/commit/65f4ba349cb341736b67c0307074cef2cf0bd12e)+- Add some missing tests for scrapy.settings #1570 (https://github.com/scrapy/scrapy/commit/9424ca0fdbdd492f3049fe08be8848f92e84fde3)+- DOCS;TESTS: tests+doc for subdomains in offsite middleware #1721+- TESTS: Include tests for non-string items to Exporters #1742+++Logging+~~~~~~~++- Ignore ScrapyDeprecationWarning warnings properly. #1294 (https://github.com/scrapy/scrapy/commit/64466526350820bdb424dc70968b4e015fd13641)+- Do not fail representing non-http requests #1419 (https://github.com/scrapy/scrapy/commit/bdcc78b4ddf47b6161b962b9d9fc8851b11f0117)+- Make list of enabled middlewares more readable #1263 (https://github.com/scrapy/scrapy/commit/a7787628ff53322e295be315e5595c555eb8e057)+- added more verbosity for log and for exception when download is cancelled because of a size limit #1624 (https://github.com/scrapy/scrapy/commit/fdc3c9d561ad87e417447fcee9adcc8cd6dbc594)+- LOGGING: show download warnsize once #1654 (https://github.com/scrapy/scrapy/commit/6827eab2c59e93d8ec46ef308bc751c6c00f32fd)+- LOGGING: Fix logging of enabled middlewares #1722 + Use long classes names for enabled middlewares in startup logs #1726+++Code refactoring+~~~~~~~~~~~~~~~~++- Avoid creation of temporary list object in iflatten #1476 (https://github.com/scrapy/scrapy/commit/6ae8963256f52bcc26ea8b4edc938743b07b6b2c)+- equal_attributes function optimization #1477 (https://github.com/scrapy/scrapy/commit/6490cb534e8e9a9068a8e298a8c6edb6be9725c5)+- Optimization - avoid temporary list objects, unnecessary function call #1481 (https://github.com/scrapy/scrapy/commit/3e13740a5765152e1b8241ad4db91efac5c746d7)+- Small downloader slots cleanup #1315 (https://github.com/scrapy/scrapy/commit/8a140b6ba1cf89e4a3bb74f8afb6e81c283e298b)+ downloader.Slot becomes unaware of Scrapy settings;+ it got __str__ and __repr__ methods useful in manhole;+ unused import is dropped;+ absolute_imports future import is added (I like adding it everywhere).+- extract CrawlerRunner._crawl method which always expects Crawler #1290 (https://github.com/scrapy/scrapy/commit/5bcda9b7d13b9c3b486c2b247fd6d87a7b59df1a)+ Provides an extension point where crawler instance is available;+ makes it easier to write alternative CrawlerRunner.crawl implementations.+ User can override CrawlerRunner._crawl method and connect signals there.+++Other changes+~~~~~~~~~~~~~+++- Extend regex for tags that deploy to PyPI to support new release cycle (:commit:`26f50d3`)+- rename str_to_unicode and unicode_to_str functions (ISSUE #778) (https://github.com/scrapy/scrapy/commit/61cd27e5c7b777a54)++- fix utils.template.render_templatefile() bug +test #1212 (https://github.com/scrapy/scrapy/commit/71bd79e70fb10ed4899b15ca3ffa9aaa16567727)++- style fixes for settings.py created by `scrapy startproject` #1496 (https://github.com/scrapy/scrapy/commit/5279da9916c00c7a6679cfc555f9a2b1863b4821)+ Adds AUTOTHROTTLE_TARGET_CONCURRENCY to settings.py++- (MINOR) Simplify if statement #1686 (https://github.com/scrapy/scrapy/commit/9ef25d7b68fe90c5e6b94bd3e81755089e743080)+ Note: in conftest.py++- (MINOR) fix indentation #1687 (https://github.com/scrapy/scrapy/commit/66f41aba3cbfa642b37354e8419e3d1437b88348)+ Note: in scrapy/downloadermiddlewares/retry.py+- (MINOR) fixed typo You -> you #1698 (https://github.com/scrapy/scrapy/commit/e8b26e2ab25ac7ec15c03d3c0b766c7aa8f48cce)+ Fixes DOWNLOAD_WARNSIZE is too verbose #1303++1.0.4 (2015-12-30)------------------@@ -590,7 +920,7 @@ Enhancements- Document `request_scheduled` signal (:issue:`746`)- Add a note about reporting security issues (:issue:`697`)- Add LevelDB http cache storage backend (:issue:`626`, :issue:`500`)-- Sort spider list output of `scrapy list` command (:issue:`742`)+- Sort spider list output of `scrapy list` command (:issue:`742`)- Multiple documentation enhancemens and fixes(:issue:`575`, :issue:`587`, :issue:`590`, :issue:`596`, :issue:`610`,:issue:`617`, :issue:`618`, :issue:`627`, :issue:`613`, :issue:`643`,