Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Initial Python 2 removal (#4091)

Andrey Rahmatullin229e722a03ac
diff --git a/.travis.yml b/.travis.yml
index 4c2498053..2352ef124 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,14 +7,6 @@ branches:
- /^\d\.\d+\.\d+(rc\d+|\.dev\d+)?$/
matrix:
include:
- - env: TOXENV=py27
- python: 2.7
- - env: TOXENV=py27-pinned
- python: 2.7
- - env: TOXENV=py27-extra-deps
- python: 2.7
- - env: TOXENV=pypy
- python: 2.7
- env: TOXENV=pypy3
python: 3.5
- env: TOXENV=py35
@@ -70,4 +62,4 @@ deploy:
on:
tags: true
repo: scrapy/scrapy
- condition: "$TOXENV == py27 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"
+ condition: "$TOXENV == py37 && $TRAVIS_TAG =~ ^[0-9]+[.][0-9]+[.][0-9]+(rc[0-9]+|[.]dev[0-9]+)?$"
diff --git a/README.rst b/README.rst
index bd82bff06..fb4ca8e4f 100644
--- a/README.rst
+++ b/README.rst
@@ -40,7 +40,7 @@ https://scrapy.org
Requirements
============
-* Python 2.7 or Python 3.5+
+* Python 3.5+
* Works on Linux, Windows, Mac OSX, BSD
Install
diff --git a/docs/faq.rst b/docs/faq.rst
index 9733471bf..080d81981 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -69,11 +69,11 @@ Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML pars
What Python versions does Scrapy support?
-----------------------------------------
-Scrapy is supported under Python 2.7 and Python 3.5+
+Scrapy is supported under Python 3.5+
under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
-Python 2.6 support was dropped starting at Scrapy 0.20.
Python 3 support was added in Scrapy 1.1.
PyPy support was added in Scrapy 1.4, PyPy3 support was added in Scrapy 1.5.
+Python 2 support was dropped in Scrapy 2.0.
.. note::
For Python 3 support on Windows, it is recommended to use
diff --git a/docs/intro/install.rst b/docs/intro/install.rst
index 51b41b4d7..e924b5303 100644
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@@ -7,7 +7,7 @@ Installation guide
Installing Scrapy
=================
-Scrapy runs on Python 2.7 and Python 3.5 or above
+Scrapy runs on Python 3.5 or above
under CPython (default Python implementation) and PyPy (starting with PyPy 5.9).
If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
@@ -102,10 +102,8 @@ just like any other Python package.
(See :ref:`platform-specific guides <intro-install-platform-notes>`
below for non-Python dependencies that you may need to install beforehand).
-Python virtualenvs can be created to use Python 2 by default, or Python 3 by default.
-
-* If you want to install scrapy with Python 3, install scrapy within a Python 3 virtualenv.
-* And if you want to install scrapy with Python 2, install scrapy within a Python 2 virtualenv.
+Python virtualenvs can be created to use Python 2 by default, or Python 3 by default. As Scrapy
+only supports Python 3, make sure you created a Python 3 virtualenv.
.. _virtualenv: https://virtualenv.pypa.io
.. _virtualenv installation instructions: https://virtualenv.pypa.io/en/stable/installation/
@@ -149,16 +147,12 @@ typically too old and slow to catch up with latest Scrapy.
To install scrapy on Ubuntu (or Ubuntu-based) systems, you need to install
these dependencies::
- sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
+ sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
-- ``python-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``
+- ``python3-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``
are required for ``lxml``
- ``libssl-dev`` and ``libffi-dev`` are required for ``cryptography``
-If you want to install scrapy on Python 3, you’ll also need Python 3 development headers::
-
- sudo apt-get install python3 python3-dev
-
Inside a :ref:`virtualenv <intro-using-virtualenv>`,
you can install Scrapy with ``pip`` after that::
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index af541db78..7481b1a99 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -99,12 +99,12 @@ The storages backends supported out of the box are:
* :ref:`topics-feed-storage-fs`
* :ref:`topics-feed-storage-ftp`
- * :ref:`topics-feed-storage-s3` (requires botocore_ or boto_)
+ * :ref:`topics-feed-storage-s3` (requires botocore_)
* :ref:`topics-feed-storage-stdout`
Some storage backends may be unavailable if the required external libraries are
not available. For example, the S3 backend is only available if the botocore_
-or boto_ library is installed (Scrapy supports boto_ only on Python 2).
+library is installed.
.. _topics-feed-uri-params:
@@ -182,7 +182,7 @@ The feeds are stored on `Amazon S3`_.
* ``s3://mybucket/path/to/export.csv``
* ``s3://aws_key:aws_secret@mybucket/path/to/export.csv``
- * Required external libraries: `botocore`_ (Python 2 and Python 3) or `boto`_ (Python 2 only)
+ * Required external libraries: `botocore`_
The AWS credentials can be passed as user/password in the URI, or they can be
passed through the following settings:
@@ -399,6 +399,5 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
.. _URI: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier
.. _Amazon S3: https://aws.amazon.com/s3/
-.. _boto: https://github.com/boto/boto
.. _botocore: https://github.com/boto/botocore
.. _Canned ACL: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index 8278e9849..793636f59 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -260,7 +260,7 @@ knowledge about Python internals. For more info about Guppy, refer to the
Debugging memory leaks with muppy
=================================
-If you're using Python 3, you can use muppy from `Pympler`_.
+You can use muppy from `Pympler`_.
.. _Pympler: https://pypi.org/project/Pympler/
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index 0ce431ff5..431cc6027 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -171,7 +171,7 @@ policy::
For more information, see `canned ACLs`_ in the Amazon S3 Developer Guide.
-Because Scrapy uses ``boto`` / ``botocore`` internally you can also use other S3-like storages. Storages like
+Because Scrapy uses ``botocore`` internally you can also use other S3-like storages. Storages like
self-hosted `Minio`_ or `s3.scality`_. All you need to do is set endpoint option in you Scrapy settings::
AWS_ENDPOINT_URL = 'http://minio.example.com:9000'
diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst
index 727c67482..5a76e189e 100644
--- a/docs/topics/request-response.rst
+++ b/docs/topics/request-response.rst
@@ -596,8 +596,8 @@ Response objects
(for single valued headers) or lists (for multi-valued headers).
:type headers: dict
- :param body: the response body. To access the decoded text as str (unicode
- in Python 2) you can use ``response.text`` from an encoding-aware
+ :param body: the response body. To access the decoded text as str you can use
+ ``response.text`` from an encoding-aware
:ref:`Response subclass <topics-request-response-ref-response-subclasses>`,
such as :class:`TextResponse`.
:type body: bytes
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 56375664f..a1d15a760 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -188,7 +188,6 @@ AWS_ENDPOINT_URL
Default: ``None``
Endpoint URL used for S3-like storage, for example Minio or s3.scality.
-Only supported with ``botocore`` library.
.. setting:: AWS_USE_SSL
@@ -199,7 +198,6 @@ Default: ``None``
Use this option if you want to disable SSL connection for communication with
S3 or S3-like storage. By default SSL will be used.
-Only supported with ``botocore`` library.
.. setting:: AWS_VERIFY
@@ -209,7 +207,7 @@ AWS_VERIFY
Default: ``None``
Verify SSL connection between Scrapy and S3 or S3-like storage. By default
-SSL verification will occur. Only supported with ``botocore`` library.
+SSL verification will occur.
.. setting:: AWS_REGION_NAME
@@ -219,7 +217,6 @@ AWS_REGION_NAME
Default: ``None``
The name of the region associated with the AWS client.
-Only supported with ``botocore`` library.
.. setting:: BOT_NAME
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index d60c93be6..d65a43afd 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -72,8 +72,6 @@ scrapy.Spider
spider that crawls ``mywebsite.com`` would often be called
``mywebsite``.
- .. note:: In Python 2 this must be ASCII only.
-
.. attribute:: allowed_domains
An optional list of strings containing domains that this spider is
diff --git a/requirements-py2.txt b/requirements-py2.txt
deleted file mode 100644
index dde8d1c9c..000000000
--- a/requirements-py2.txt
+++ /dev/null
@@ -1,18 +0,0 @@
-parsel>=1.5.0
-PyDispatcher>=2.0.5
-w3lib>=1.17.0
-protego>=0.1.15
-
-pyOpenSSL>=16.2.0 # Earlier versions fail with "AttributeError: module 'lib' has no attribute 'SSL_ST_INIT'"
-queuelib>=1.4.2 # Earlier versions fail with "AttributeError: '...QueueTest' object has no attribute 'qpath'"
-cryptography>=2.0 # Earlier versions would fail to install
-
-# Reference versions taken from
-# https://packages.ubuntu.com/xenial/python/
-# https://packages.ubuntu.com/xenial/zope/
-cssselect>=0.9.1
-lxml>=3.5.0
-service_identity>=16.0.0
-six>=1.10.0
-Twisted>=16.0.0
-zope.interface>=4.1.3
diff --git a/scrapy/__init__.py b/scrapy/__init__.py
index 03ec6c667..230e5cee3 100644
--- a/scrapy/__init__.py
+++ b/scrapy/__init__.py
@@ -14,8 +14,8 @@ del pkgutil
# Check minimum required Python version
import sys
-if sys.version_info < (2, 7):
- print("Scrapy %s requires Python 2.7" % __version__)
+if sys.version_info < (3, 5):
+ print("Scrapy %s requires Python 3.5" % __version__)
sys.exit(1)
# Ignore noisy twisted deprecation warnings
diff --git a/setup.py b/setup.py
index 2f5fca4c9..8f5f14f0d 100644
--- a/setup.py
+++ b/setup.py
@@ -50,8 +50,6 @@ setup(
'License :: OSI Approved :: BSD License',
'Operating System :: OS Independent',
'Programming Language :: Python',
- 'Programming Language :: Python :: 2',
- 'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
@@ -63,10 +61,9 @@ setup(
'Topic :: Software Development :: Libraries :: Application Frameworks',
'Topic :: Software Development :: Libraries :: Python Modules',
],
- python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
+ python_requires='>=3.5',
install_requires=[
- 'Twisted>=16.0.0;python_version=="2.7"',
- 'Twisted>=17.9.0;python_version>="3.5"',
+ 'Twisted>=17.9.0',
'cryptography>=2.0',
'cssselect>=0.9.1',
'lxml>=3.5.0',
diff --git a/tests/requirements-py2.txt b/tests/requirements-py2.txt
deleted file mode 100644
index f621eb4eb..000000000
--- a/tests/requirements-py2.txt
+++ /dev/null
@@ -1,15 +0,0 @@
-# Tests requirements
-brotlipy
-jmespath
-mitmproxy==0.10.1
-mock
-netlib==0.10.1
-pytest
-pytest-cov
-pytest-twisted
-pytest-xdist
-testfixtures
-
-# optional for shell wrapper tests
-bpython
-ipython<6.0
diff --git a/tox.ini b/tox.ini
index fe925951b..821144381 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,18 +4,16 @@
# and then run "tox" from this directory.
[tox]
-envlist = py27
+envlist = py35
[testenv]
deps =
-ctests/constraints.txt
- -rrequirements-py2.txt
+ -rrequirements-py3.txt
+ -rtests/requirements-py3.txt
# Extras
botocore>=1.3.23
- google-cloud-storage
- leveldb
Pillow>=3.4.2
- -rtests/requirements-py2.txt
passenv =
S3_TEST_FILE_URI
AWS_ACCESS_KEY_ID
@@ -25,42 +23,8 @@ passenv =
commands =
py.test --cov=scrapy --cov-report= {posargs:scrapy tests}
-[testenv:py27-pinned]
-basepython = python2.7
-deps =
- -ctests/constraints.txt
- cryptography==2.0
- cssselect==0.9.1
- lxml==3.5.0
- parsel==1.5.0
- Protego==0.1.15
- PyDispatcher==2.0.5
- pyOpenSSL==16.2.0
- queuelib==1.4.2
- service_identity==16.0.0
- six==1.10.0
- Twisted==16.0.0
- w3lib==1.17.0
- zope.interface==4.1.3
- -rtests/requirements-py2.txt
- # Extras
- botocore==1.3.23
- Pillow==3.4.2
-
-[testenv:pypy]
-basepython = pypy
-commands =
- py.test {posargs:scrapy tests}
-
[testenv:py35]
basepython = python3.5
-deps =
- -ctests/constraints.txt
- -rrequirements-py3.txt
- -rtests/requirements-py3.txt
- # Extras
- botocore>=1.3.23
- Pillow>=3.4.2
[testenv:py35-pinned]
basepython = python3.5
@@ -86,19 +50,15 @@ deps =
[testenv:py36]
basepython = python3.6
-deps = {[testenv:py35]deps}
[testenv:py37]
basepython = python3.7
-deps = {[testenv:py35]deps}
[testenv:py38]
basepython = python3.8
-deps = {[testenv:py35]deps}
[testenv:pypy3]
basepython = pypy3
-deps = {[testenv:py35]deps}
commands =
py.test {posargs:scrapy tests}
@@ -127,13 +87,6 @@ commands =
[testenv:py38-extra-deps]
basepython = python3.8
-deps =
- {[testenv:py35]deps}
- reppy
- robotexclusionrulesparser
-
-[testenv:py27-extra-deps]
-basepython = python2.7
deps =
{[testenv]deps}
reppy