Scrapy, a fast high-level web crawling & scraping framework for Python.
Update the installation docs (#7878)
Adriand88bb9ab58d1
diff --git a/docs/intro/install.rst b/docs/intro/install.rstindex dfc21cb84..9c632cf8e 100644--- a/docs/intro/install.rst+++ b/docs/intro/install.rst@@ -16,28 +16,27 @@ Scrapy requires Python 3.11+.Installing Scrapy=================-If you're using `Anaconda`_ or `Miniconda`_, you can install the package from-the `conda-forge`_ channel, which has up-to-date packages for Linux, Windows-and macOS.+Install Scrapy and its dependencies from PyPI with::-To install Scrapy using ``conda``, run::+ pip install scrapy++We strongly recommend that you install Scrapy in :ref:`a dedicated virtual+environment <intro-using-virtualenv>`, to avoid conflicting with your system+packages.- conda install -c conda-forge scrapy+Alternatively:-Alternatively, if you’re already familiar with installing Python packages, you-can install Scrapy and its dependencies from PyPI with::+- From a `uv`_ project, e.g. one created with ``uv init``. ``uv`` manages the+ virtual environment for you::- pip install Scrapy+ uv add scrapy-We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv <intro-using-virtualenv>`,-to avoid conflicting with your system packages.+- From the `conda-forge`_ channel, with ``conda``::-Note that this may sometimes require solving compilation issues for some Scrapy-dependencies depending on your operating system, so be sure to check the-:ref:`intro-install-platform-notes`.+ conda install -c conda-forge scrapy-For more detailed and platform-specific instructions, as well as-troubleshooting information, read on.+If the installation fails while building one of the dependencies, see+:ref:`install-no-wheel`.Things that are good to know@@ -51,17 +50,9 @@ Scrapy is written in pure Python and depends on a few key Python packages (among* `twisted`_, an asynchronous networking framework* `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs-Some of these packages themselves depend on non-Python packages-that might require additional installation steps depending on your platform.-Please check :ref:`platform-specific guides below <intro-install-platform-notes>`.--In case of any trouble related to these dependencies,-please refer to their respective installation instructions:--* `lxml installation`_-* :doc:`cryptography installation <cryptography:installation>`--.. _lxml installation: https://lxml.de/installation.html+Some of these packages include compiled code. They provide binary wheels for+common platforms, so that installing them requires no compiler; see+:ref:`install-no-wheel` if no wheel matches your platform... _intro-using-virtualenv:@@ -85,8 +76,6 @@ See :ref:`tut-venv` on how to create your virtual environment.Once you have created a virtual environment, you can install Scrapy inside it with ``pip``,just like any other Python package.-(See :ref:`platform-specific guides <intro-install-platform-notes>`-below for non-Python dependencies that you may need to install beforehand)... _extras:@@ -144,132 +133,91 @@ The following extras are available:- `uvloop <https://github.com/MagicStack/uvloop>`_ event loop+.. _install-notes:.. _intro-install-platform-notes:Platform-specific installation notes====================================+.. _install-windows:.. _intro-install-windows:Windows--------Though it's possible to install Scrapy on Windows using pip, we recommend you-install `Anaconda`_ or `Miniconda`_ and use the package from the-`conda-forge`_ channel, which will avoid most installation issues.--Once you've installed `Anaconda`_ or `Miniconda`_, install Scrapy with::-- conda install -c conda-forge scrapy--To install Scrapy on Windows using ``pip``:--.. warning::- This installation method requires “Microsoft Visual C++” for installing some- Scrapy dependencies, which demands significantly more disk space than Anaconda.+Install Scrapy :ref:`as described above <intro-install-scrapy>`. Dependencies+that include compiled code provide wheels for 64-bit x86 Windows, so no+compiler is needed.-#. Download and execute `Microsoft C++ Build Tools`_ to install the Visual Studio Installer.--#. Run the Visual Studio Installer.--#. Under the Workloads section, select **C++ build tools**.--#. Check the installation details and make sure the following packages are- selected as optional components:-- * **MSVC** (e.g. MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) )-- * **Windows SDK** (e.g. Windows 10 SDK (10.0.18362.0))--#. Install the Visual Studio Build Tools.--Now, you should be able to :ref:`install Scrapy <intro-install-scrapy>` using ``pip``.+On 32-bit or ARM64 Windows some of those wheels are missing, see+:ref:`install-no-wheel`. An additional option there is `WSL`_, which lets you+follow the :ref:`Linux instructions <install-linux>` instead. Note that your+code then runs on Linux, so Windows paths do not work in settings like+:setting:`FEEDS`.+.. _install-linux:.. _intro-install-ubuntu:-Ubuntu 14.04 or above------------------------Scrapy is currently tested with recent-enough versions of lxml,-twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions.-But it should support older versions of Ubuntu too, like Ubuntu 14.04,-albeit with potential issues with TLS connections.--**Don't** use the ``python-scrapy`` package provided by Ubuntu; it is typically-too old and slow to catch up with the latest Scrapy release.---To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to install-these dependencies::-- sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev--- ``python3-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``- are required for ``lxml``-- ``libssl-dev`` and ``libffi-dev`` are required for ``cryptography``--Inside a :ref:`virtualenv <intro-using-virtualenv>`,-you can install Scrapy with ``pip`` after that::-- pip install scrapy+Linux+------.. note::- The same non-Python dependencies can be used to install Scrapy in Debian- Jessie (8.0) and above.+Install Scrapy :ref:`as described above <intro-install-scrapy>`. Dependencies+that include compiled code provide ``manylinux`` and ``musllinux`` wheels, so+no compiler is needed.+**Don't** use the Scrapy package provided by your distribution, e.g.+``python-scrapy`` on Debian and Ubuntu, they are typically too old and slow to+catch up with the latest Scrapy release.+.. _install-macos:.. _intro-install-macos:macOS------Building Scrapy's dependencies requires the presence of a C compiler and-development headers. On macOS this is typically provided by Apple’s Xcode-development tools. To install the Xcode command-line tools, open a terminal-window and run::-- xcode-select --install+Install Scrapy :ref:`as described above <intro-install-scrapy>`. Dependencies+that include compiled code provide wheels for both Intel and Apple silicon, so+no compiler is needed.-There's a `known issue <https://github.com/pypa/pip/issues/2468>`_ that-prevents ``pip`` from updating system packages. This has to be addressed to-successfully install Scrapy and its dependencies. Here are some proposed-solutions:+We recommend against using the Python interpreter that comes with macOS.+Install a separate one instead, e.g. with `homebrew`_::-* *(Recommended)* **Don't** use system Python. Install a new, updated version- that doesn't conflict with the rest of your system. Here's how to do it using- the `homebrew`_ package manager:+ brew install python- * Install `homebrew`_ following the instructions in https://brew.sh/- * Update your ``PATH`` variable to state that homebrew packages should be- used before system packages (change ``.bashrc`` to ``.zshrc`` accordingly- if you're using `zsh`_ as the default shell)::-- echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc+.. _install-troubleshooting:+.. _intro-install-troubleshooting:- * Reload ``.bashrc`` to ensure the changes have taken place::+Troubleshooting+===============- source ~/.bashrc+.. _install-no-wheel:- * Install Python::+Installation fails while building a dependency+----------------------------------------------- brew install python+When ``pip`` finds no wheel matching your Python version and platform, it+builds the dependency from its source distribution instead, which requires a+build toolchain and the libraries that the dependency wraps. To check whether+that is what fails for you, ask ``pip`` for wheels only::-* *(Optional)* :ref:`Install Scrapy inside a Python virtual environment- <intro-using-virtualenv>`.+ pip install --only-binary :all: scrapy- This method is a workaround for the above macOS issue, but it's an overall- good practice for managing dependencies and can complement the first method.+With ``uv``, use ``uv add --no-build scrapy`` instead.-After any of these workarounds, you should be able to install Scrapy::+If that fails, no wheel is available, and you have the following options,+easiest first:- pip install Scrapy+- Use an older Python version. Wheels for a new Python version may take+ weeks to become available after its release.+- Use ``conda``, e.g. through `Miniforge`_, and install Scrapy from the+ `conda-forge`_ channel.-.. _intro-install-troubleshooting:--Troubleshooting-===============+- Build the dependency from source, following its own installation+ instructions, e.g. `lxml installation`_ or :doc:`cryptography installation+ <cryptography:installation>`. Beyond a C compiler, this may require a Rust+ compiler, development files of libraries like libxml2 or OpenSSL, or both.AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'----------------------------------------------------------------@@ -294,17 +242,15 @@ reinstall Twisted with the :code:`tls` extra option::For details, see `Issue #2473 <https://github.com/scrapy/scrapy/issues/2473>`_.-.. _Python: https://www.python.org/.. _lxml: https://lxml.de/index.html+.. _lxml installation: https://lxml.de/installation.html.. _parsel: https://pypi.org/project/parsel/.. _w3lib: https://pypi.org/project/w3lib/.. _twisted: https://twisted.org/.. _cryptography: https://cryptography.io/en/latest/.. _pyOpenSSL: https://pypi.org/project/pyOpenSSL/-.. _setuptools: https://pypi.org/pypi/setuptools.. _homebrew: https://brew.sh/-.. _zsh: https://www.zsh.org/-.. _Anaconda: https://www.anaconda.com/docs/main-.. _Miniconda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html-.. _Microsoft C++ Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/+.. _uv: https://docs.astral.sh/uv/+.. _Miniforge: https://conda-forge.org/download/.. _conda-forge: https://conda-forge.org/+.. _WSL: https://learn.microsoft.com/en-us/windows/wsl/install