Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Use pytest-xdist

Adrián Chaves5a6fb3daa6e6
diff --git a/docs/contributing.rst b/docs/contributing.rst
index aac0f4496..b462ae331 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -205,6 +205,29 @@ To run a specific test (say ``tests/test_loader.py``) use:
``tox -- tests/test_loader.py``
+To run the tests on a specific tox_ environment, use ``-e <name>`` with an
+environment name from ``tox.ini``. For example, to run the tests with Python
+3.6 use::
+
+ tox -e py36
+
+You can also specify a comma-separated list of environmets, and use `tox’s
+parallel mode`_ to run the tests on multiple environments in parallel::
+
+ tox -e py27,py36 -p auto
+
+To pass command-line options to pytest_, add them after ``--`` in your call to
+tox_. Using ``--`` overrides the default positional arguments defined in
+``tox.ini``, so you must include those default positional arguments
+(``scrapy tests``) after ``--`` as well::
+
+ tox -- scrapy tests -x # stop after first failure
+
+You can also use the `pytest-xdist`_ plugin. For example, to run all tests on
+the Python 3.6 tox_ environment using all your CPU cores::
+
+ tox -e py36 -- scrapy tests -n auto
+
To see coverage report install `coverage`_ (``pip install coverage``) and run:
``coverage report``
@@ -238,4 +261,7 @@ And their unit-tests are in::
.. _tests/: https://github.com/scrapy/scrapy/tree/master/tests
.. _open issues: https://github.com/scrapy/scrapy/issues
.. _pull request: https://help.github.com/send-pull-requests/
+.. _pytest: https://docs.pytest.org/en/latest/usage.html
+.. _pytest-xdist: https://docs.pytest.org/en/3.0.0/xdist.html
.. _tox: https://pypi.python.org/pypi/tox
+.. _tox’s parallel mode: https://tox.readthedocs.io/en/latest/example/basic.html#parallel-mode
diff --git a/tests/requirements-py2.txt b/tests/requirements-py2.txt
index 790f29d34..be809b151 100644
--- a/tests/requirements-py2.txt
+++ b/tests/requirements-py2.txt
@@ -2,9 +2,10 @@
mock
mitmproxy==0.10.1
netlib==0.10.1
-pytest==2.9.2
+pytest
+pytest-cov
pytest-twisted
-pytest-cov==2.2.1
+pytest-xdist
jmespath
brotlipy
testfixtures
diff --git a/tests/requirements-py3.txt b/tests/requirements-py3.txt
index 7c1aacd81..ed7bf0be0 100644
--- a/tests/requirements-py3.txt
+++ b/tests/requirements-py3.txt
@@ -1,6 +1,7 @@
-pytest==3.6.3
+pytest
+pytest-cov
pytest-twisted
-pytest-cov==2.5.1
+pytest-xdist
testfixtures
jmespath
leveldb; sys_platform != "win32"