Scrapy, a fast high-level web crawling & scraping framework for Python.
fix typos
Marc Hernandez Cabotc0d84f0962a4
diff --git a/docs/contributing.rst b/docs/contributing.rstindex 3aebb3d50..b56295027 100644--- a/docs/contributing.rst+++ b/docs/contributing.rst@@ -217,7 +217,7 @@ the tests with Python 3.6 use::tox -e py36-You can also specify a comma-separated list of environmets, and use :ref:`tox’s+You can also specify a comma-separated list of environments, and use :ref:`tox’sparallel mode <tox:parallel_mode>` to run the tests on multiple environments inparallel::diff --git a/docs/faq.rst b/docs/faq.rstindex 080d81981..aae2411e0 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -338,7 +338,7 @@ How to split an item into multiple items in an item pipeline?input item. :ref:`Create a spider middleware <custom-spider-middleware>`instead, and use its:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`-method for this puspose. For example::+method for this purpose. For example::from copy import deepcopydiff --git a/docs/news.rst b/docs/news.rstindex 9dfd28508..28db40e3c 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -678,7 +678,7 @@ Usability improvements* a message is added to IgnoreRequest in RobotsTxtMiddleware (:issue:`3113`)* better validation of ``url`` argument in ``Response.follow`` (:issue:`3131`)* non-zero exit code is returned from Scrapy commands when error happens- on spider inititalization (:issue:`3226`)+ on spider initialization (:issue:`3226`)* Link extraction improvements: "ftp" is added to scheme list (:issue:`3152`);"flv" is added to common video extensions (:issue:`3165`)* better error message when an exporter is disabled (:issue:`3358`);@@ -1156,7 +1156,7 @@ Bug fixes- Fix :command:`view` command ; it was a regression in v1.3.0 (:issue:`2503`).- Fix tests regarding ``*_EXPIRES settings`` with Files/Images pipelines (:issue:`2460`).- Fix name of generated pipeline class when using basic project template (:issue:`2466`).-- Fix compatiblity with Twisted 17+ (:issue:`2496`, :issue:`2528`).+- Fix compatibility with Twisted 17+ (:issue:`2496`, :issue:`2528`).- Fix ``scrapy.Item`` inheritance on Python 3.6 (:issue:`2511`).- Enforce numeric values for components order in ``SPIDER_MIDDLEWARES``,``DOWNLOADER_MIDDLEWARES``, ``EXTENIONS`` and ``SPIDER_CONTRACTS`` (:issue:`2420`).@@ -1164,7 +1164,7 @@ Bug fixesDocumentation~~~~~~~~~~~~~-- Reword Code of Coduct section and upgrade to Contributor Covenant v1.4+- Reword Code of Conduct section and upgrade to Contributor Covenant v1.4(:issue:`2469`).- Clarify that passing spider arguments converts them to spider attributes(:issue:`2483`).@@ -1178,7 +1178,7 @@ DocumentationCleanups~~~~~~~~-- Remove reduntant check in ``MetaRefreshMiddleware`` (:issue:`2542`).+- Remove redundant check in ``MetaRefreshMiddleware`` (:issue:`2542`).- Faster checks in ``LinkExtractor`` for allow/deny patterns (:issue:`2538`).- Remove dead code supporting old Twisted versions (:issue:`2544`).@@ -1204,7 +1204,7 @@ New Features- ``MailSender`` now accepts single strings as values for ``to`` and ``cc``arguments (:issue:`2272`)- ``scrapy fetch url``, ``scrapy shell url`` and ``fetch(url)`` inside- scrapy shell now follow HTTP redirections by default (:issue:`2290`);+ Scrapy shell now follow HTTP redirections by default (:issue:`2290`);See :command:`fetch` and :command:`shell` for details.- ``HttpErrorMiddleware`` now logs errors with ``INFO`` level instead of ``DEBUG``;this is technically **backward incompatible** so please check your log parsers.@@ -1705,7 +1705,7 @@ Scrapy 1.0.4 (2015-12-30)- fix ValueError: Invalid XPath: //div/[id="not-exists"]/text() on selectors.rst (:commit:`ca8d60f`)- Typos corrections (:commit:`7067117`)- fix typos in downloader-middleware.rst and exceptions.rst, middlware -> middleware (:commit:`32f115c`)-- Add note to ubuntu install section about debian compatibility (:commit:`23fda69`)+- Add note to Ubuntu install section about Debian compatibility (:commit:`23fda69`)- Replace alternative OSX install workaround with virtualenv (:commit:`98b63ee`)- Reference Homebrew's homepage for installation instructions (:commit:`1925db1`)- Add oldest supported tox version to contributing docs (:commit:`5d10d6d`)@@ -1758,7 +1758,7 @@ Scrapy 1.0.1 (2015-07-01)- include tests/ to source distribution in MANIFEST.in (:commit:`eca227e`)- DOC Fix SelectJmes documentation (:commit:`b8567bc`)- DOC Bring Ubuntu and Archlinux outside of Windows subsection (:commit:`392233f`)-- DOC remove version suffix from ubuntu package (:commit:`5303c66`)+- DOC remove version suffix from Ubuntu package (:commit:`5303c66`)- DOC Update release date for 1.0 (:commit:`c89fa29`).. _release-1.0.0:@@ -2211,7 +2211,7 @@ Scrapy 0.24.2 (2014-07-08)- Use a mutable mapping to proxy deprecated settings.overrides and settings.defaults attribute (:commit:`e5e8133`)- there is not support for python3 yet (:commit:`3cd6146`)-- Update python compatible version set to debian packages (:commit:`fa5d76b`)+- Update python compatible version set to Debian packages (:commit:`fa5d76b`)- DOC fix formatting in release notes (:commit:`c6a9e20`)Scrapy 0.24.1 (2014-06-27)@@ -2229,12 +2229,12 @@ Enhancements- Improve Scrapy top-level namespace (:issue:`494`, :issue:`684`)- Add selector shortcuts to responses (:issue:`554`, :issue:`690`)-- Add new lxml based LinkExtractor to replace unmantained SgmlLinkExtractor+- Add new lxml based LinkExtractor to replace unmaintained SgmlLinkExtractor(:issue:`559`, :issue:`761`, :issue:`763`)- Cleanup settings API - part of per-spider settings **GSoC project** (:issue:`737`)- Add UTF8 encoding header to templates (:issue:`688`, :issue:`762`)- Telnet console now binds to 127.0.0.1 by default (:issue:`699`)-- Update debian/ubuntu install instructions (:issue:`509`, :issue:`549`)+- Update Debian/Ubuntu install instructions (:issue:`509`, :issue:`549`)- Disable smart strings in lxml XPath evaluations (:issue:`535`)- Restore filesystem based cache as default for httpcache middleware (:issue:`541`, :issue:`500`, :issue:`571`)@@ -2267,7 +2267,7 @@ Enhancements- Tests and docs for ``request_fingerprint`` function (:issue:`597`)- Update SEP-19 for GSoC project ``per-spider settings`` (:issue:`705`)- Set exit code to non-zero when contracts fails (:issue:`727`)-- Add a setting to control what class is instanciated as Downloader component+- Add a setting to control what class is instantiated as Downloader component(:issue:`738`)- Pass response in ``item_dropped`` signal (:issue:`724`)- Improve ``scrapy check`` contracts command (:issue:`733`, :issue:`752`)@@ -2276,7 +2276,7 @@ Enhancements- Add a note about reporting security issues (:issue:`697`)- Add LevelDB http cache storage backend (:issue:`626`, :issue:`500`)- Sort spider list output of ``scrapy list`` command (:issue:`742`)-- Multiple documentation enhancemens and fixes+- Multiple documentation enhancements and fixes(:issue:`575`, :issue:`587`, :issue:`590`, :issue:`596`, :issue:`610`,:issue:`617`, :issue:`618`, :issue:`627`, :issue:`613`, :issue:`643`,:issue:`654`, :issue:`675`, :issue:`663`, :issue:`711`, :issue:`714`)@@ -2321,19 +2321,19 @@ Scrapy 0.22.1 (released 2014-02-08)- BaseSgmlLinkExtractor: Added unit test of a link with an inner tag (:commit:`c1cb418`)- BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag (:commit:`7e4d627`)- Fix tests for Travis-CI build (:commit:`76c7e20`)-- replace unencodeable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)+- replace unencodable codepoints with html entities. fixes #562 and #285 (:commit:`5f87b17`)- RegexLinkExtractor: encode URL unicode value when creating Links (:commit:`d0ee545`)- Updated the tutorial crawl output with latest output. (:commit:`8da65de`)- Updated shell docs with the crawler reference and fixed the actual shell output. (:commit:`875b9ab`)- PEP8 minor edits. (:commit:`f89efaf`)-- Expose current crawler in the scrapy shell. (:commit:`5349cec`)+- Expose current crawler in the Scrapy shell. (:commit:`5349cec`)- Unused re import and PEP8 minor edits. (:commit:`387f414`)- Ignore None's values when using the ItemLoader. (:commit:`0632546`)- DOC Fixed HTTPCACHE_STORAGE typo in the default value which is now Filesystem instead Dbm. (:commit:`cde9a8c`)-- show ubuntu setup instructions as literal code (:commit:`fb5c9c5`)+- show Ubuntu setup instructions as literal code (:commit:`fb5c9c5`)- Update Ubuntu installation instructions (:commit:`70fb105`)- Merge pull request #550 from stray-leone/patch-1 (:commit:`6f70b6a`)-- modify the version of scrapy ubuntu package (:commit:`725900d`)+- modify the version of Scrapy Ubuntu package (:commit:`725900d`)- fix 0.22.0 release date (:commit:`af0219a`)- fix typos in news.rst and remove (not released yet) header (:commit:`b7f58f4`)@@ -2354,7 +2354,7 @@ Enhancements- Improve test coverage and forthcoming Python 3 support (:issue:`525`)- Promote startup info on settings and middleware to INFO level (:issue:`520`)- Support partials in ``get_func_args`` util (:issue:`506`, issue:`504`)-- Allow running indiviual tests via tox (:issue:`503`)+- Allow running individual tests via tox (:issue:`503`)- Update extensions ignored by link extractors (:issue:`498`)- Add middleware methods to get files/images/thumbs paths (:issue:`490`)- Improve offsite middleware tests (:issue:`478`)@@ -2411,7 +2411,7 @@ Enhancements- scrapy.mail.MailSender now can connect over TLS or upgrade using STARTTLS (:issue:`327`)- New FilesPipeline with functionality factored out from ImagesPipeline (:issue:`370`, :issue:`409`)- Recommend Pillow instead of PIL for image handling (:issue:`317`)-- Added debian packages for Ubuntu quantal and raring (:commit:`86230c0`)+- Added Debian packages for Ubuntu Quantal and raring (:commit:`86230c0`)- Mock server (used for tests) can listen for HTTPS requests (:issue:`410`)- Remove multi spider support from multiple core components(:issue:`422`, :issue:`421`, :issue:`420`, :issue:`419`, :issue:`423`, :issue:`418`)@@ -2430,7 +2430,7 @@ Bugfixes- Fix tests under Django 1.6 (:commit:`b6bed44c`)- Lot of bugfixes to retry middleware under disconnections using HTTP 1.1 download handler- Fix inconsistencies among Twisted releases (:issue:`406`)-- Fix scrapy shell bugs (:issue:`418`, :issue:`407`)+- Fix Scrapy shell bugs (:issue:`418`, :issue:`407`)- Fix invalid variable name in setup.py (:issue:`429`)- Fix tutorial references (:issue:`387`)- Improve request-response docs (:issue:`391`)@@ -2512,15 +2512,15 @@ Scrapy 0.18.1 (released 2013-08-27)- test PotentiaDataLoss errors on unbound responses (:commit:`b15470d`)- Treat responses without content-length or Transfer-Encoding as good responses (:commit:`c4bf324`)- do no include ResponseFailed if http11 handler is not enabled (:commit:`6cbe684`)-- New HTTP client wraps connection losts in ResponseFailed exception. fix #373 (:commit:`1a20bba`)+- New HTTP client wraps connection lost in ResponseFailed exception. fix #373 (:commit:`1a20bba`)- limit travis-ci build matrix (:commit:`3b01bb8`)- Merge pull request #375 from peterarenot/patch-1 (:commit:`fa766d7`)- Fixed so it refers to the correct folder (:commit:`3283809`)-- added quantal & raring to support ubuntu releases (:commit:`1411923`)+- added Quantal & raring to support Ubuntu releases (:commit:`1411923`)- fix retry middleware which didn't retry certain connection errors after the upgrade to http1 client, closes GH-373 (:commit:`bb35ed0`)- fix XmlItemExporter in Python 2.7.4 and 2.7.5 (:commit:`de3e451`)- minor updates to 0.18 release notes (:commit:`c45e5f1`)-- fix contributters list format (:commit:`0b60031`)+- fix contributors list format (:commit:`0b60031`)Scrapy 0.18.0 (released 2013-08-09)-----------------------------------@@ -2617,7 +2617,7 @@ contributors sorted by number of commits::Scrapy 0.16.5 (released 2013-05-30)------------------------------------- obey request method when scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)+- obey request method when Scrapy deploy is redirected to a new endpoint (:commit:`8c4fcee`)- fix inaccurate downloader middleware documentation. refs #280 (:commit:`40667cb`)- doc: remove links to diveintopython.org, which is no longer available. closes #246 (:commit:`bd58bfa`)- Find form nodes in invalid html5 documents (:commit:`e3d6945`)@@ -2631,8 +2631,8 @@ Scrapy 0.16.4 (released 2013-01-23)- Fixed error message formatting. log.err() doesn't support cool formatting and when error occurred, the message was: "ERROR: Error processing %(item)s" (:commit:`c16150c`)- lint and improve images pipeline error logging (:commit:`56b45fc`)- fixed doc typos (:commit:`243be84`)-- add documentation topics: Broad Crawls & Common Practies (:commit:`1fbb715`)-- fix bug in scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)+- add documentation topics: Broad Crawls & Common Practices (:commit:`1fbb715`)+- fix bug in Scrapy parse command when spider is not specified explicitly. closes #209 (:commit:`c72e682`)- Update docs/topics/commands.rst (:commit:`28eac7a`)Scrapy 0.16.3 (released 2012-12-07)@@ -2651,11 +2651,11 @@ Scrapy 0.16.3 (released 2012-12-07)Scrapy 0.16.2 (released 2012-11-09)------------------------------------- scrapy contracts: python2.6 compat (:commit:`a4a9199`)-- scrapy contracts verbose option (:commit:`ec41673`)-- proper unittest-like output for scrapy contracts (:commit:`86635e4`)+- Scrapy contracts: python2.6 compat (:commit:`a4a9199`)+- Scrapy contracts verbose option (:commit:`ec41673`)+- proper unittest-like output for Scrapy contracts (:commit:`86635e4`)- added open_in_browser to debugging doc (:commit:`c9b690d`)-- removed reference to global scrapy stats from settings doc (:commit:`dd55067`)+- removed reference to global Scrapy stats from settings doc (:commit:`dd55067`)- Fix SpiderState bug in Windows platforms (:commit:`58998f4`)@@ -2665,7 +2665,7 @@ Scrapy 0.16.1 (released 2012-10-26)- fixed LogStats extension, which got broken after a wrong merge before the 0.16 release (:commit:`8c780fd`)- better backward compatibility for scrapy.conf.settings (:commit:`3403089`)- extended documentation on how to access crawler stats from extensions (:commit:`c4da0b5`)-- removed .hgtags (no longer needed now that scrapy uses git) (:commit:`d52c188`)+- removed .hgtags (no longer needed now that Scrapy uses git) (:commit:`d52c188`)- fix dashes under rst headers (:commit:`fa4f7f9`)- set release date for 0.16.0 in news (:commit:`e292246`)@@ -2680,8 +2680,7 @@ Scrapy changes:- documented :doc:`topics/autothrottle` and added to extensions installed by default. You still need to enable it with :setting:`AUTOTHROTTLE_ENABLED`- major Stats Collection refactoring: removed separation of global/per-spider stats, removed stats-related signals (``stats_spider_opened``, etc). Stats are much simpler now, backward compatibility is kept on the Stats Collector API and signals.- added :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests` method to spider middlewares-- dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info.-- dropped Signals singleton. Signals should now be accesed through the Crawler.signals attribute. See the signals documentation for more info.+- dropped Signals singleton. Signals should now be accessed through the Crawler.signals attribute. See the signals documentation for more info.- dropped Stats Collector singleton. Stats can now be accessed through the Crawler.stats attribute. See the stats collection documentation for more info.- documented :ref:`topics-api`- ``lxml`` is now the default selectors backend instead of ``libxml2``@@ -2715,7 +2714,7 @@ Scrapy changes:Scrapy 0.14.4--------------- added precise to supported ubuntu distros (:commit:`b7e46df`)+- added precise to supported Ubuntu distros (:commit:`b7e46df`)- fixed bug in json-rpc webservice reported in https://groups.google.com/forum/#!topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py (:commit:`340fbdb`)- meta tag attributes for content-type http equiv can be in any order. #123 (:commit:`0cb68af`)- replace "import Image" by more standard "from PIL import Image". closes #88 (:commit:`4d17048`)@@ -2728,11 +2727,11 @@ Scrapy 0.14.3- include egg files used by testsuite in source distribution. #118 (:commit:`c897793`)- update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 (:commit:`2548dcc`)- added note to docs/topics/firebug.rst about google directory being shut down (:commit:`668e352`)-- dont discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)+- don't discard slot when empty, just save in another dict in order to recycle if needed again. (:commit:`8e9f607`)- do not fail handling unicode xpaths in libxml2 backed selectors (:commit:`b830e95`)- fixed minor mistake in Request objects documentation (:commit:`bf3c9ee`)- fixed minor defect in link extractors documentation (:commit:`ba14f38`)-- removed some obsolete remaining code related to sqlite support in scrapy (:commit:`0665175`)+- removed some obsolete remaining code related to sqlite support in Scrapy (:commit:`0665175`)Scrapy 0.14.2-------------diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rstindex f5542495b..8816a028c 100644--- a/docs/topics/jobs.rst+++ b/docs/topics/jobs.rst@@ -74,7 +74,7 @@ Request serializationFor persistence to work, :class:`~scrapy.http.Request` objects must beserializable with :mod:`pickle`, except for the ``callback`` and ``errback``values passed to their ``__init__`` method, which must be methods of the-runnning :class:`~scrapy.spiders.Spider` class.+running :class:`~scrapy.spiders.Spider` class.If you wish to log the requests that couldn't be serialized, you can set the:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rstindex 793636f59..83b4d8154 100644--- a/docs/topics/leaks.rst+++ b/docs/topics/leaks.rst@@ -110,7 +110,7 @@ ties the response lifetime to the requests' one, and that would definitelycause memory leaks.Let's see how we can discover the cause (without knowing it-a-priori, of course) by using the ``trackref`` tool.+a priori, of course) by using the ``trackref`` tool.After the crawler is running for a few minutes and we notice its memory usagehas grown a lot, we can enter its telnet console and check the livediff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex 206e7cfa5..a40682e5b 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -558,7 +558,7 @@ See here the methods that you can override in your custom Images Pipeline:Custom Images pipeline example==============================-Here is a full example of the Images Pipeline whose methods are examplified+Here is a full example of the Images Pipeline whose methods are exemplifiedabove::import scrapydiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex a1d15a760..f138b7064 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -884,7 +884,7 @@ LOG_FORMATDefault: ``'%(asctime)s [%(name)s] %(levelname)s: %(message)s'``-String for formatting log messsages. Refer to the `Python logging documentation`_ for the whole list of available+String for formatting log messages. Refer to the `Python logging documentation`_ for the whole list of availableplaceholders... _Python logging documentation: https://docs.python.org/2/library/logging.html#logrecord-attributesdiff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rstindex 7db7e4f6b..ecdc5423c 100644--- a/docs/topics/telnetconsole.rst+++ b/docs/topics/telnetconsole.rst@@ -48,7 +48,7 @@ autogenerated Password can be seen on scrapy logs like the example below::2018-10-16 14:35:21 [scrapy.extensions.telnet] INFO: Telnet Password: 16f92501e8a59326-Default Username and Password can be overriden by the settings+Default Username and Password can be overridden by the settings:setting:`TELNETCONSOLE_USERNAME` and :setting:`TELNETCONSOLE_PASSWORD`... warning::diff --git a/sep/sep-001.rst b/sep/sep-001.rstindex 2a66f9802..00226283f 100644--- a/sep/sep-001.rst+++ b/sep/sep-001.rst@@ -254,8 +254,8 @@ ItemForm#!pythonclass MySiteForm(ItemForm):- witdth = adaptor(ItemForm.witdh, default_unit='cm')- volume = adaptor(ItemForm.witdh, default_unit='lt')+ width = adaptor(ItemForm.width, default_unit='cm')+ volume = adaptor(ItemForm.width, default_unit='lt')ia['width'] = x.x('//p[@class="width"]')ia['volume'] = x.x('//p[@class="volume"]')diff --git a/sep/sep-019.rst b/sep/sep-019.rstindex 9fbf6a223..84f3a96c3 100644--- a/sep/sep-019.rst+++ b/sep/sep-019.rst@@ -185,7 +185,7 @@ These ideas translate to the following changes on the ``SpiderManager`` class:will return a spider class, not an instance. It's basically a ``__get__``to ``self._spiders``.-- All remaining functions should be deprecated or remove accordantly, since a+- All remaining functions should be deprecated or remove accordingly, since acrawler reference is no longer needed.- New helper ``get_spider_manager_class_from_scrapycfg`` in