Scrapy, a fast high-level web crawling & scraping framework for Python.
Initial Python 2 removal (#4091)
Andrey Rahmatullin229e722a03ac
diff --git a/.travis.yml b/.travis.ymlindex 4c2498053..2352ef124 100644--- a/.travis.yml+++ b/.travis.yml@@ -7,14 +7,6 @@ branches:- /^\d\.\d+\.\d+(rc\d+|\.dev\d+)?$/matrix:include:- - env: TOXENV=py27- python: 2.7- - env: TOXENV=py27-pinned- python: 2.7- - env: TOXENV=py27-extra-deps- python: 2.7- - env: TOXENV=pypy- python: 2.7- env: TOXENV=pypy3python: 3.5- env: TOXENV=py35@@ -70,4 +62,4 @@ deploy:on:tags: truerepo: scrapy/scrapy- condition: "$TOXENV == py27 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"+ condition: "$TOXENV == py37 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"diff --git a/README.rst b/README.rstindex bd82bff06..fb4ca8e4f 100644--- a/README.rst+++ b/README.rst@@ -40,7 +40,7 @@ https://scrapy.orgRequirements============-* Python 2.7 or Python 3.5++* Python 3.5+* Works on Linux, Windows, Mac OSX, BSDInstalldiff --git a/docs/faq.rst b/docs/faq.rstindex 9733471bf..080d81981 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -69,11 +69,11 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parsWhat Python versions does Scrapy support?------------------------------------------Scrapy is supported under Python 2.7 and Python 3.5++Scrapy is supported under Python 3.5+under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).-Python 2.6 support was dropped starting at Scrapy 0.20.Python 3 support was added in Scrapy 1.1.PyPy support was added in Scrapy 1.4, PyPy3 support was added in Scrapy 1.5.+Python 2 support was dropped in Scrapy 2.0... note::For Python 3 support on Windows, it is recommended to usediff --git a/docs/intro/install.rst b/docs/intro/install.rstindex 51b41b4d7..e924b5303 100644--- a/docs/intro/install.rst+++ b/docs/intro/install.rst@@ -7,7 +7,7 @@ Installation guideInstalling Scrapy=================-Scrapy runs on Python 2.7 and Python 3.5 or above+Scrapy runs on Python 3.5 or aboveunder CPython (default Python implementation) and PyPy (starting with PyPy 5.9).If you're using `Anaconda`_ or `Miniconda`_, you can install the package from@@ -102,10 +102,8 @@ just like any other Python package.(See :ref:`platform-specific guides <intro-install-platform-notes>`below for non-Python dependencies that you may need to install beforehand).-Python virtualenvs can be created to use Python 2 by default, or Python 3 by default.--* If you want to install scrapy with Python 3, install scrapy within a Python 3 virtualenv.-* And if you want to install scrapy with Python 2, install scrapy within a Python 2 virtualenv.+Python virtualenvs can be created to use Python 2 by default, or Python 3 by default. As Scrapy+only supports Python 3, make sure you created a Python 3 virtualenv... _virtualenv: https://virtualenv.pypa.io.. _virtualenv installation instructions: https://virtualenv.pypa.io/en/stable/installation/@@ -149,16 +147,12 @@ typically too old and slow to catch up with latest Scrapy.To install scrapy on Ubuntu (or Ubuntu-based) systems, you need to installthese dependencies::- sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev+ sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev-- ``python-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``+- ``python3-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``are required for ``lxml``- ``libssl-dev`` and ``libffi-dev`` are required for ``cryptography``-If you want to install scrapy on Python 3, you’ll also need Python 3 development headers::-- sudo apt-get install python3 python3-dev-Inside a :ref:`virtualenv <intro-using-virtualenv>`,you can install Scrapy with ``pip`` after that::diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rstindex af541db78..7481b1a99 100644--- a/docs/topics/feed-exports.rst+++ b/docs/topics/feed-exports.rst@@ -99,12 +99,12 @@ The storages backends supported out of the box are:* :ref:`topics-feed-storage-fs`* :ref:`topics-feed-storage-ftp`- * :ref:`topics-feed-storage-s3` (requires botocore_ or boto_)+ * :ref:`topics-feed-storage-s3` (requires botocore_)* :ref:`topics-feed-storage-stdout`Some storage backends may be unavailable if the required external libraries arenot available. For example, the S3 backend is only available if the botocore_-or boto_ library is installed (Scrapy supports boto_ only on Python 2).+library is installed... _topics-feed-uri-params:@@ -182,7 +182,7 @@ The feeds are stored on `Amazon S3`_.* ``s3://mybucket/path/to/export.csv``* ``s3://aws_key:aws_secret@mybucket/path/to/export.csv``- * Required external libraries: `botocore`_ (Python 2 and Python 3) or `boto`_ (Python 2 only)+ * Required external libraries: `botocore`_The AWS credentials can be passed as user/password in the URI, or they can bepassed through the following settings:@@ -399,6 +399,5 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter.. _URI: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier.. _Amazon S3: https://aws.amazon.com/s3/-.. _boto: https://github.com/boto/boto.. _botocore: https://github.com/boto/botocore.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acldiff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rstindex 8278e9849..793636f59 100644--- a/docs/topics/leaks.rst+++ b/docs/topics/leaks.rst@@ -260,7 +260,7 @@ knowledge about Python internals. For more info about Guppy, refer to theDebugging memory leaks with muppy=================================-If you're using Python 3, you can use muppy from `Pympler`_.+You can use muppy from `Pympler`_... _Pympler: https://pypi.org/project/Pympler/diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rstindex 0ce431ff5..431cc6027 100644--- a/docs/topics/media-pipeline.rst+++ b/docs/topics/media-pipeline.rst@@ -171,7 +171,7 @@ policy::For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.-Because Scrapy uses ``boto`` / ``botocore`` internally you can also use other S3-like storages. Storages like+Because Scrapy uses ``botocore`` internally you can also use other S3-like storages. Storages likeself-hosted `Minio`_ or `s3.scality`_. All you need to do is set endpoint option in you Scrapy settings::AWS_ENDPOINT_URL = 'http://minio.example.com:9000'diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rstindex 727c67482..5a76e189e 100644--- a/docs/topics/request-response.rst+++ b/docs/topics/request-response.rst@@ -596,8 +596,8 @@ Response objects(for single valued headers) or lists (for multi-valued headers).:type headers: dict- :param body: the response body. To access the decoded text as str (unicode- in Python 2) you can use ``response.text`` from an encoding-aware+ :param body: the response body. To access the decoded text as str you can use+ ``response.text`` from an encoding-aware:ref:`Response subclass <topics-request-response-ref-response-subclasses>`,such as :class:`TextResponse`.:type body: bytesdiff --git a/docs/topics/settings.rst b/docs/topics/settings.rstindex 56375664f..a1d15a760 100644--- a/docs/topics/settings.rst+++ b/docs/topics/settings.rst@@ -188,7 +188,6 @@ AWS_ENDPOINT_URLDefault: ``None``Endpoint URL used for S3-like storage, for example Minio or s3.scality.-Only supported with ``botocore`` library... setting:: AWS_USE_SSL@@ -199,7 +198,6 @@ Default: ``None``Use this option if you want to disable SSL connection for communication withS3 or S3-like storage. By default SSL will be used.-Only supported with ``botocore`` library... setting:: AWS_VERIFY@@ -209,7 +207,7 @@ AWS_VERIFYDefault: ``None``Verify SSL connection between Scrapy and S3 or S3-like storage. By default-SSL verification will occur. Only supported with ``botocore`` library.+SSL verification will occur... setting:: AWS_REGION_NAME@@ -219,7 +217,6 @@ AWS_REGION_NAMEDefault: ``None``The name of the region associated with the AWS client.-Only supported with ``botocore`` library... setting:: BOT_NAMEdiff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rstindex d60c93be6..d65a43afd 100644--- a/docs/topics/spiders.rst+++ b/docs/topics/spiders.rst@@ -72,8 +72,6 @@ scrapy.Spiderspider that crawls ``mywebsite.com`` would often be called``mywebsite``.- .. note:: In Python 2 this must be ASCII only.-.. attribute:: allowed_domainsAn optional list of strings containing domains that this spider isdiff --git a/requirements-py2.txt b/requirements-py2.txtdeleted file mode 100644index dde8d1c9c..000000000--- a/requirements-py2.txt+++ /dev/null@@ -1,18 +0,0 @@-parsel>=1.5.0-PyDispatcher>=2.0.5-w3lib>=1.17.0-protego>=0.1.15--pyOpenSSL>=16.2.0 # Earlier versions fail with "AttributeError: module 'lib' has no attribute 'SSL_ST_INIT'"-queuelib>=1.4.2 # Earlier versions fail with "AttributeError: '...QueueTest' object has no attribute 'qpath'"-cryptography>=2.0 # Earlier versions would fail to install--# Reference versions taken from-# https://packages.ubuntu.com/xenial/python/-# https://packages.ubuntu.com/xenial/zope/-cssselect>=0.9.1-lxml>=3.5.0-service_identity>=16.0.0-six>=1.10.0-Twisted>=16.0.0-zope.interface>=4.1.3diff --git a/scrapy/__init__.py b/scrapy/__init__.pyindex 03ec6c667..230e5cee3 100644--- a/scrapy/__init__.py+++ b/scrapy/__init__.py@@ -14,8 +14,8 @@ del pkgutil# Check minimum required Python versionimport sys-if sys.version_info < (2, 7):- print("Scrapy %s requires Python 2.7" % __version__)+if sys.version_info < (3, 5):+ print("Scrapy %s requires Python 3.5" % __version__)sys.exit(1)# Ignore noisy twisted deprecation warningsdiff --git a/setup.py b/setup.pyindex 2f5fca4c9..8f5f14f0d 100644--- a/setup.py+++ b/setup.py@@ -50,8 +50,6 @@ setup('License :: OSI Approved :: BSD License','Operating System :: OS Independent','Programming Language :: Python',- 'Programming Language :: Python :: 2',- 'Programming Language :: Python :: 2.7','Programming Language :: Python :: 3','Programming Language :: Python :: 3.5','Programming Language :: Python :: 3.6',@@ -63,10 +61,9 @@ setup('Topic :: Software Development :: Libraries :: Application Frameworks','Topic :: Software Development :: Libraries :: Python Modules',],- python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',+ python_requires='>=3.5',install_requires=[- 'Twisted>=16.0.0;python_version=="2.7"',- 'Twisted>=17.9.0;python_version>="3.5"',+ 'Twisted>=17.9.0','cryptography>=2.0','cssselect>=0.9.1','lxml>=3.5.0',diff --git a/tests/requirements-py2.txt b/tests/requirements-py2.txtdeleted file mode 100644index f621eb4eb..000000000--- a/tests/requirements-py2.txt+++ /dev/null@@ -1,15 +0,0 @@-# Tests requirements-brotlipy-jmespath-mitmproxy==0.10.1-mock-netlib==0.10.1-pytest-pytest-cov-pytest-twisted-pytest-xdist-testfixtures--# optional for shell wrapper tests-bpython-ipython<6.0diff --git a/tox.ini b/tox.iniindex fe925951b..821144381 100644--- a/tox.ini+++ b/tox.ini@@ -4,18 +4,16 @@# and then run "tox" from this directory.[tox]-envlist = py27+envlist = py35[testenv]deps =-ctests/constraints.txt- -rrequirements-py2.txt+ -rrequirements-py3.txt+ -rtests/requirements-py3.txt# Extrasbotocore>=1.3.23- google-cloud-storage- leveldbPillow>=3.4.2- -rtests/requirements-py2.txtpassenv =S3_TEST_FILE_URIAWS_ACCESS_KEY_ID@@ -25,42 +23,8 @@ passenv =commands =py.test --cov=scrapy --cov-report= {posargs:scrapy tests}-[testenv:py27-pinned]-basepython = python2.7-deps =- -ctests/constraints.txt- cryptography==2.0- cssselect==0.9.1- lxml==3.5.0- parsel==1.5.0- Protego==0.1.15- PyDispatcher==2.0.5- pyOpenSSL==16.2.0- queuelib==1.4.2- service_identity==16.0.0- six==1.10.0- Twisted==16.0.0- w3lib==1.17.0- zope.interface==4.1.3- -rtests/requirements-py2.txt- # Extras- botocore==1.3.23- Pillow==3.4.2--[testenv:pypy]-basepython = pypy-commands =- py.test {posargs:scrapy tests}-[testenv:py35]basepython = python3.5-deps =- -ctests/constraints.txt- -rrequirements-py3.txt- -rtests/requirements-py3.txt- # Extras- botocore>=1.3.23- Pillow>=3.4.2[testenv:py35-pinned]basepython = python3.5@@ -86,19 +50,15 @@ deps =[testenv:py36]basepython = python3.6-deps = {[testenv:py35]deps}[testenv:py37]basepython = python3.7-deps = {[testenv:py35]deps}[testenv:py38]basepython = python3.8-deps = {[testenv:py35]deps}[testenv:pypy3]basepython = pypy3-deps = {[testenv:py35]deps}commands =py.test {posargs:scrapy tests}@@ -127,13 +87,6 @@ commands =[testenv:py38-extra-deps]basepython = python3.8-deps =- {[testenv:py35]deps}- reppy- robotexclusionrulesparser--[testenv:py27-extra-deps]-basepython = python2.7deps ={[testenv]deps}reppy