Scrapy, a fast high-level web crawling & scraping framework for Python.
Fix typos
Kian-Meng, Ang3a263280bad5
diff --git a/docs/news.rst b/docs/news.rstindex 5e590f027..509366c17 100644--- a/docs/news.rst+++ b/docs/news.rst@@ -1830,7 +1830,7 @@ New features* A new scheduler priority queue,``scrapy.pqueues.DownloaderAwarePriorityQueue``, may be:ref:`enabled <broad-crawls-scheduler-priority-queue>` for a significant- scheduling improvement on crawls targetting multiple web domains, at the+ scheduling improvement on crawls targeting multiple web domains, at thecost of no :setting:`CONCURRENT_REQUESTS_PER_IP` support (:issue:`3520`)* A new :attr:`Request.cb_kwargs <scrapy.http.Request.cb_kwargs>` attribute@@ -2868,7 +2868,7 @@ Bug fixes- Fix for selected callbacks when using ``CrawlSpider`` with :command:`scrapy parse <parse>`(:issue:`2225`).- Fix for invalid JSON and XML files when spider yields no items (:issue:`872`).-- Implement ``flush()`` fpr ``StreamLogger`` avoiding a warning in logs (:issue:`2125`).+- Implement ``flush()`` for ``StreamLogger`` avoiding a warning in logs (:issue:`2125`).Refactoring~~~~~~~~~~~@@ -3731,7 +3731,7 @@ Scrapy 0.24.3 (2014-08-09)- adding some xpath tips to selectors docs (:commit:`2d103e0`)- fix tests to account for https://github.com/scrapy/w3lib/pull/23 (:commit:`f8d366a`)- get_func_args maximum recursion fix #728 (:commit:`81344ea`)-- Updated input/ouput processor example according to #560. (:commit:`f7c4ea8`)+- Updated input/output processor example according to #560. (:commit:`f7c4ea8`)- Fixed Python syntax in tutorial. (:commit:`db59ed9`)- Add test case for tunneling proxy (:commit:`f090260`)- Bugfix for leaking Proxy-Authorization header to remote host when using tunneling (:commit:`d8793af`)@@ -4393,7 +4393,7 @@ Scrapyd changes~~~~~~~~~~~~~~~- Scrapyd now uses one process per spider-- It stores one log file per spider run, and rotate them keeping the lastest 5 logs per spider (by default)+- It stores one log file per spider run, and rotate them keeping the latest 5 logs per spider (by default)- A minimal web ui was added, available at http://localhost:6800 by default- There is now a ``scrapy server`` command to start a Scrapyd server of the current project@@ -4429,7 +4429,7 @@ New features and improvements- Added two new methods to item pipeline open_spider(), close_spider() with deferred support (#195)- Support for overriding default request headers per spider (#181)- Replaced default Spider Manager with one with similar functionality but not depending on Twisted Plugins (#186)-- Splitted Debian package into two packages - the library and the service (#187)+- Split Debian package into two packages - the library and the service (#187)- Scrapy log refactoring (#188)- New extension for keeping persistent spider contexts among different runs (#203)- Added ``dont_redirect`` request.meta key for avoiding redirects (#233)diff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 2ab2020fa..19a549a02 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -1566,7 +1566,7 @@ If a reactor is already installed,:meth:`CrawlerRunner.__init__ <scrapy.crawler.CrawlerRunner.__init__>` raises:exc:`Exception` if the installed reactor does not match the-:setting:`TWISTED_REACTOR` setting; therfore, having top-level+:setting:`TWISTED_REACTOR` setting; therefore, having top-level:mod:`~twisted.internet.reactor` imports in project files and importedthird-party libraries will make Scrapy raise :exc:`Exception` whenit checks which reactor is installed.@@ -1658,7 +1658,7 @@ Default: ``"Scrapy/VERSION (+https://scrapy.org)"``The default User-Agent to use when crawling, unless overridden. This user agent isalso used by :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware`if :setting:`ROBOTSTXT_USER_AGENT` setting is ``None`` and-there is no overridding User-Agent header specified for the request.+there is no overriding User-Agent header specified for the request.Settings documented elsewhere:diff --git a/docs/topics/shell.rst b/docs/topics/shell.rstindex 8c90a506c..007e9fc2f 100644--- a/docs/topics/shell.rst+++ b/docs/topics/shell.rst@@ -99,7 +99,7 @@ Available Shortcutsshortcuts- ``fetch(url[, redirect=True])`` - fetch a new response from the given URL- and update all related objects accordingly. You can optionaly ask for HTTP+ and update all related objects accordingly. You can optionally ask for HTTP3xx redirections to not be followed by passing ``redirect=False``- ``fetch(request)`` - fetch a new response from the given request and updatediff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex 4d3d32941..99e74233a 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -372,7 +372,7 @@ CrawlSpiderdescribed below. If multiple rules match the same link, the first onewill be used, according to the order they're defined in this attribute.- This spider also exposes an overrideable method:+ This spider also exposes an overridable method:.. method:: parse_start_url(response, **kwargs)@@ -534,7 +534,7 @@ XMLFeedSpideritertag = 'n:url'# ...- Apart from these new attributes, this spider has the following overrideable+ Apart from these new attributes, this spider has the following overridablemethods too:.. method:: adapt_response(response)diff --git a/docs/versioning.rst b/docs/versioning.rstindex 57643ea9a..9d02757b0 100644--- a/docs/versioning.rst+++ b/docs/versioning.rst@@ -13,7 +13,7 @@ There are 3 numbers in a Scrapy version: *A.B.C*large changes.* *B* is the release number. This will include many changes including featuresand things that possibly break backward compatibility, although we strive to- keep theses cases at a minimum.+ keep these cases at a minimum.* *C* is the bugfix release number.Backward-incompatibilities are explicitly mentioned in the :ref:`release notes <news>`,diff --git a/extras/qpsclient.py b/extras/qpsclient.pyindex f9fb70342..28703650d 100644--- a/extras/qpsclient.py+++ b/extras/qpsclient.py@@ -1,5 +1,5 @@"""-A spider that generate light requests to meassure QPS troughput+A spider that generate light requests to meassure QPS throughputusage:diff --git a/scrapy/downloadermiddlewares/retry.py b/scrapy/downloadermiddlewares/retry.pyindex f1fdc3858..c6cc7c56d 100644--- a/scrapy/downloadermiddlewares/retry.py+++ b/scrapy/downloadermiddlewares/retry.py@@ -2,7 +2,7 @@An extension to retry failed requests that are potentially caused by temporaryproblems such as a connection timeout or HTTP 500 error.-You can change the behaviour of this middleware by modifing the scraping settings:+You can change the behaviour of this middleware by modifying the scraping settings:RETRY_TIMES - how many times to retry a failed pageRETRY_HTTP_CODES - which HTTP response codes to retrydiff --git a/scrapy/exporters.py b/scrapy/exporters.pyindex fb4b565cf..36cca2d05 100644--- a/scrapy/exporters.py+++ b/scrapy/exporters.py@@ -30,7 +30,7 @@ class BaseItemExporter:self._configure(kwargs, dont_fail=dont_fail)def _configure(self, options, dont_fail=False):- """Configure the exporter by poping options from the ``options`` dict.+ """Configure the exporter by popping options from the ``options`` dict.If dont_fail is set, it won't raise an exception on unexpected options(useful for using with keyword arguments in subclasses ``__init__`` methods)"""diff --git a/scrapy/linkextractors/lxmlhtml.py b/scrapy/linkextractors/lxmlhtml.pyindex e941c4321..b5d2585a8 100644--- a/scrapy/linkextractors/lxmlhtml.py+++ b/scrapy/linkextractors/lxmlhtml.py@@ -88,7 +88,7 @@ class LxmlParserLinkExtractor:def _process_links(self, links):""" Normalize and filter extracted links- The subclass should override it if neccessary+ The subclass should override it if necessary"""return self._deduplicate_if_needed(links)diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.pyindex 8766ef66f..5c52c6c28 100644--- a/scrapy/pipelines/files.py+++ b/scrapy/pipelines/files.py@@ -85,7 +85,7 @@ class S3FilesStore:AWS_USE_SSL = NoneAWS_VERIFY = None- POLICY = 'private' # Overriden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settings+ POLICY = 'private' # Overridden from settings.FILES_STORE_S3_ACL in FilesPipeline.from_settingsHEADERS = {'Cache-Control': 'max-age=172800',}@@ -142,7 +142,7 @@ class S3FilesStore:**extra)def _headers_to_botocore_kwargs(self, headers):- """ Convert headers to botocore keyword agruments.+ """ Convert headers to botocore keyword arguments."""# This is required while we need to support both boto and botocore.mapping = CaselessDict({@@ -190,7 +190,7 @@ class GCSFilesStore:CACHE_CONTROL = 'max-age=172800'# The bucket's default object ACL will be applied to the object.- # Overriden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings.+ # Overridden from settings.FILES_STORE_GCS_ACL in FilesPipeline.from_settings.POLICY = Nonedef __init__(self, uri):@@ -291,7 +291,7 @@ class FilesPipeline(MediaPipeline):"""Abstract pipeline that implement the file downloadingThis pipeline tries to minimize network transfers and file processing,- doing stat of the files and determining if file is new, uptodate or+ doing stat of the files and determining if file is new, up-to-date orexpired.``new`` files are those that pipeline never processed and needs to bediff --git a/scrapy/spiders/feed.py b/scrapy/spiders/feed.pyindex 6ed17e4dd..bef2d6b24 100644--- a/scrapy/spiders/feed.py+++ b/scrapy/spiders/feed.py@@ -43,7 +43,7 @@ class XMLFeedSpider(Spider):return responsedef parse_node(self, response, selector):- """This method must be overriden with your custom spider functionality"""+ """This method must be overridden with your custom spider functionality"""if hasattr(self, 'parse_item'): # backward compatibilityreturn self.parse_item(response, selector)raise NotImplementedError@@ -113,7 +113,7 @@ class CSVFeedSpider(Spider):return responsedef parse_row(self, response, row):- """This method must be overriden with your custom spider functionality"""+ """This method must be overridden with your custom spider functionality"""raise NotImplementedErrordef parse_rows(self, response):diff --git a/scrapy/utils/console.py b/scrapy/utils/console.pyindex 133261fd7..1bc0bd45f 100644--- a/scrapy/utils/console.py+++ b/scrapy/utils/console.py@@ -14,7 +14,7 @@ def _embed_ipython_shell(namespace={}, banner=''):@wraps(_embed_ipython_shell)def wrapper(namespace=namespace, banner=''):config = load_default_config()- # Always use .instace() to ensure _instance propagation to all parents+ # Always use .instance() to ensure _instance propagation to all parents# this is needed for <TAB> completion works well for new imports# and clear the instance to always have the fresh env# on repeated breaks like with inspect_response()diff --git a/scrapy/utils/datatypes.py b/scrapy/utils/datatypes.pyindex e31284a7f..47df8a717 100644--- a/scrapy/utils/datatypes.py+++ b/scrapy/utils/datatypes.py@@ -41,7 +41,7 @@ class CaselessDict(dict):return key.lower()def normvalue(self, value):- """Method to normalize values prior to be setted"""+ """Method to normalize values prior to be set"""return valuedef get(self, key, def_val=None):diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.pyindex b317c12a3..b02bfdccb 100644--- a/scrapy/utils/defer.py+++ b/scrapy/utils/defer.py@@ -34,7 +34,7 @@ def defer_succeed(result) -> Deferred:"""Same as twisted.internet.defer.succeed but delay calling callback untilnext reactor loop- It delays by 100ms so reactor has a chance to go trough readers and writers+ It delays by 100ms so reactor has a chance to go through readers and writersbefore attending pending delayed calls, so do not set delay to zero."""from twisted.internet import reactordiff --git a/scrapy/utils/request.py b/scrapy/utils/request.pyindex 57dcc5f2c..70ef3ba2b 100644--- a/scrapy/utils/request.py+++ b/scrapy/utils/request.py@@ -48,7 +48,7 @@ def request_fingerprint(the fingerprint.For this reason, request headers are ignored by default when calculating- the fingeprint. If you want to include specific headers use the+ the fingerprint. If you want to include specific headers use theinclude_headers argument, which is a list of Request headers to include.Also, servers usually ignore fragments in urls when handling requests,@@ -78,7 +78,7 @@ def request_fingerprint(def request_authenticate(request: Request, username: str, password: str) -> None:- """Autenticate the given request (in place) using the HTTP basic access+ """Authenticate the given request (in place) using the HTTP basic accessauthentication mechanism (RFC 2617) and the given username and password"""request.headers['Authorization'] = basic_auth_header(username, password)diff --git a/sep/sep-001.rst b/sep/sep-001.rstindex 00226283f..f704e113f 100644--- a/sep/sep-001.rst+++ b/sep/sep-001.rst@@ -260,7 +260,7 @@ ItemFormia['width'] = x.x('//p[@class="width"]')ia['volume'] = x.x('//p[@class="volume"]')- # another example passing parametes on instance+ # another example passing parameters on instanceia = NewsForm(response, encoding='utf-8')ia['name'] = x.x('//p[@class="name"]')diff --git a/sep/sep-005.rst b/sep/sep-005.rstindex e795838e4..08ed367b3 100644--- a/sep/sep-005.rst+++ b/sep/sep-005.rst@@ -107,7 +107,7 @@ gUsing default_builderThis will use default_builder as the builder for every field in the item class.-As a reducer is not set reducers will be set based on Item Field classess.+As a reducer is not set reducers will be set based on Item Field classes.gReset default_builder for a field==================================diff --git a/sep/sep-014.rst b/sep/sep-014.rstindex 8ca81824d..0859e3f7c 100644--- a/sep/sep-014.rst+++ b/sep/sep-014.rst@@ -64,7 +64,7 @@ Request Processors takes requests objects and can perform any action to them,like filtering or modifying on the fly.The current ``LinkExtractor`` had integrated link processing, like-canonicalize. Request Processors can be reutilized and applied in serie.+canonicalize. Request Processors can be reutilized and applied in series.Request Generator-----------------diff --git a/sep/sep-021.rst b/sep/sep-021.rstindex 372429791..c1ec16f7f 100644--- a/sep/sep-021.rst+++ b/sep/sep-021.rst@@ -22,7 +22,7 @@ Instead, the hooks are spread over:* Downloader handlers (DOWNLOADER_HANDLERS)* Item pipelines (ITEM_PIPELINES)* Feed exporters and storages (FEED_EXPORTERS, FEED_STORAGES)-* Overrideable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc)+* Overridable components (DUPEFILTER_CLASS, STATS_CLASS, SCHEDULER, SPIDER_MANAGER_CLASS, ITEM_PROCESSOR, etc)* Generic extensions (EXTENSIONS)* CLI commands (COMMANDS_MODULE)diff --git a/tests/test_http_response.py b/tests/test_http_response.pyindex c376a46cd..0ec5257e1 100644--- a/tests/test_http_response.py+++ b/tests/test_http_response.py@@ -19,7 +19,7 @@ class BaseResponseTest(unittest.TestCase):response_class = Responsedef test_init(self):- # Response requires url in the consturctor+ # Response requires url in the constructorself.assertRaises(Exception, self.response_class)self.assertTrue(isinstance(self.response_class('http://example.com/'), self.response_class))self.assertRaises(TypeError, self.response_class, b"http://example.com")@@ -392,7 +392,7 @@ class TextResponseTest(BaseResponseTest):def test_declared_encoding_invalid(self):"""Check that unknown declared encodings are ignored"""r = self.response_class("http://www.example.com",- headers={"Content-type": ["text/html; charset=UKNOWN"]},+ headers={"Content-type": ["text/html; charset=UNKNOWN"]},body=b"\xc2\xa3")self.assertEqual(r._declared_encoding(), None)self._assert_response_values(r, 'utf-8', "\xa3")diff --git a/tests/test_request_attribute_binding.py b/tests/test_request_attribute_binding.pyindex 00c532c41..25d9657d5 100644--- a/tests/test_request_attribute_binding.py+++ b/tests/test_request_attribute_binding.py@@ -106,9 +106,9 @@ class CrawlTestCase(TestCase):"""Downloader middleware which returns a response with an specific 'request' attribute.- * The spider callback should receive the overriden response.request- * Handlers listening to the response_received signal should receive the overriden response.request- * The "crawled" log message should show the overriden response.request+ * The spider callback should receive the overridden response.request+ * Handlers listening to the response_received signal should receive the overridden response.request+ * The "crawled" log message should show the overridden response.request"""signal_params = {}@@ -144,7 +144,7 @@ class CrawlTestCase(TestCase):An exception is raised but caught by the next middleware, whichreturns a Response with a specific 'request' attribute.- The spider callback should receive the overriden response.request+ The spider callback should receive the overridden response.request"""url = self.mockserver.url("/status?n=200")runner = CrawlerRunner(settings={diff --git a/tests/test_utils_defer.py b/tests/test_utils_defer.pyindex 7a5f458c7..032dbc8c5 100644--- a/tests/test_utils_defer.py+++ b/tests/test_utils_defer.py@@ -23,7 +23,7 @@ class MustbeDeferredTest(unittest.TestCase):dfd = mustbe_deferred(_append, 1)dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred- steps.append(2) # add another value, that should be catched by assertEqual+ steps.append(2) # add another value, that should be caught by assertEqualreturn dfddef test_unfired_deferred(self):@@ -37,7 +37,7 @@ class MustbeDeferredTest(unittest.TestCase):dfd = mustbe_deferred(_append, 1)dfd.addCallback(self.assertEqual, [1, 2]) # it is [1] with maybeDeferred- steps.append(2) # add another value, that should be catched by assertEqual+ steps.append(2) # add another value, that should be caught by assertEqualreturn dfddiff --git a/tests/test_utils_template.py b/tests/test_utils_template.pyindex 5ff2e41ef..1d5e63363 100644--- a/tests/test_utils_template.py+++ b/tests/test_utils_template.py@@ -36,7 +36,7 @@ class UtilsRenderTemplateFileTestCase(unittest.TestCase):self.assertEqual(result.read().decode('utf8'), rendered)os.remove(render_path)- assert not os.path.exists(render_path) # Failure of test iself+ assert not os.path.exists(render_path) # Failure of test itselfif '__main__' == __name__: