Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

8.2 KB
Raw Blame
.. _topics-debug:

=================
Debugging Spiders
=================

This document explains the most common techniques for debugging spiders.
Consider the following Scrapy spider:

.. skip: next
.. code-block:: python

    import scrapy
    from myproject.items import MyItem


    class MySpider(scrapy.Spider):
        name = "myspider"
        start_urls = (
            "http://example.com/page1",
            "http://example.com/page2",
        )

        def parse(self, response):
            # <processing code not shown>
            # collect `item_urls`
            for item_url in item_urls:
                yield scrapy.Request(item_url, self.parse_item)

        def parse_item(self, response):
            # <processing code not shown>
            item = MyItem()
            # populate `item` fields
            # and extract item_details_url
            yield scrapy.Request(
                item_details_url, self.parse_details, cb_kwargs={"item": item}
            )

        def parse_details(self, response, item):
            # populate more `item` fields
            return item

Basically, this is a simple spider that parses two pages of items (the start
URLs). Items also have a details page with additional information, so we use
the ``cb_kwargs`` functionality of :class:`~scrapy.Request` to pass a partially
populated item.


Parse Command
=============

The most basic way of checking the output of your spider is to use the
:command:`parse` command. It allows you to check the behaviour of different
parts of the spider at the method level. It has the advantage of being flexible
and simple to use, but it does not allow debugging code inside a method.

.. highlight:: none

.. skip: start

In order to see the item scraped from a specific url::

    $ scrapy parse --spider=myspider -c parse_item -d 2 <item_url>
    [ ... scrapy log lines crawling example.com spider ... ]

    >>> STATUS DEPTH LEVEL 2 <<<
    # Scraped Items  ------------------------------------------------------------
    [{'url': <item_url>}]

    # Requests  -----------------------------------------------------------------
    []

Using the ``--verbose`` or ``-v`` option we can see the status at each depth level::

    $ scrapy parse --spider=myspider -c parse_item -d 2 -v <item_url>
    [ ... scrapy log lines crawling example.com spider ... ]

    >>> DEPTH LEVEL: 1 <<<
    # Scraped Items  ------------------------------------------------------------
    []

    # Requests  -----------------------------------------------------------------
    [<GET item_details_url>]


    >>> DEPTH LEVEL: 2 <<<
    # Scraped Items  ------------------------------------------------------------
    [{'url': <item_url>}]

    # Requests  -----------------------------------------------------------------
    []

Checking items scraped from a single start URL can also be easily achieved
using::

    $ scrapy parse --spider=myspider -d 3 'http://example.com/page1'

.. skip: end


Scrapy Shell
============

While the :command:`parse` command is very useful for checking the behaviour of
a spider, it is of little help when checking what happens inside a callback
besides showing the response received and the output. How do you debug the
situation when ``parse_details`` sometimes receives no item?

.. highlight:: python

Fortunately, the :command:`shell` is your bread and butter in this case (see
:ref:`topics-shell-inspect-response`):

.. code-block:: python

    from scrapy.shell import inspect_response


    def parse_details(self, response, item=None):
        if item:
            # populate more `item` fields
            return item
        else:
            inspect_response(response, self)

See also: :ref:`topics-shell-inspect-response`.


Scrapy MCP server
=================

You can use the :ref:`Scrapy MCP server <using-mcp-server>` to connect a coding
agent to a running crawl, so that it can check the progress of the crawl and
investigate or even modify its runtime state.


Open in browser
===============

Sometimes you just want to see how a certain response looks in a browser; you
can use the :func:`~scrapy.utils.response.open_in_browser` function for that:

.. autofunction:: scrapy.utils.response.open_in_browser


Logging
=======

Logging is another useful option for getting information about your spider run.
Although not as convenient, it comes with the advantage that the logs will be
available in all future runs should you need them again:

.. code-block:: python

    def parse_details(self, response, item=None):
        if item:
            # populate more `item` fields
            return item
        else:
            self.logger.warning("No item received for %s", response.url)

For more information, check the :ref:`topics-logging` section.

.. _debug-live-traffic:

Inspecting live traffic
=======================

Sometimes it's important to see what exactly was sent to the server or received
from it, such as header values, formatting and order (Scrapy cannot log this,
as underlying HTTP libraries produce the final values for request headers and
canonicalize response ones) or TLS handshake details. There are two ways to see
and log the real traffic of a running spider:

-   Capture the traffic with a tool such as Wireshark_. As your requests likely
    use TLS, you will need to decrypt the traffic (see the `Wireshark TLS
    documentation`_ for detailed instructions). You will need the encryption
    key which you can save as described in :ref:`debug-tls`. As this way of
    capturing traffic is passive, it cannot interfere with the spider.

-   Use mitmproxy_ between the spider and the server, as described below. This
    is easier to set up and in addition to inspecting the traffic allows
    modifying it, but it's not passive: there is now a connection between
    Scrapy and mitmproxy and another one between mitmproxy and the server
    instead of a direct connection between Scrapy and the server. Due to this,
    low-level connection behavior is different from normal crawls, which may
    change the server behavior, and you cannot easily use mitmproxy and regular
    proxies in the same crawl.

Using mitmdump
--------------

You should refer to the mitmproxy documentation for more details, additional
interception modes and advanced features but here is one simple way to use it.
First, run a ``mitmdump`` instance (it will use the port 8080 by default),
asking it to log the traffic details on the terminal (``--flow-detail 2`` will
log headers but not bodies):

.. code-block:: shell

    mitmdump --flow-detail 2

Then configure ``http://127.0.0.1:8080`` as a proxy in your spider using
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`:

.. code-block:: shell

    https_proxy=http://127.0.0.1:8080 scrapy crawl myspider

To inspect only some requests, set their :reqmeta:`proxy` meta key instead.

.. _mitmproxy: https://mitmproxy.org/
.. _Wireshark: https://www.wireshark.org/
.. _Wireshark TLS documentation: https://wiki.wireshark.org/TLS

.. _debug-tls:

Decrypting TLS traffic
======================

Scrapy writes the session keys of its HTTPS connections to the file that the
``SSLKEYLOGFILE`` environment variable points to, using the `NSS key log
format`_ that traffic analysis tools such as Wireshark understand.

.. versionadded:: 2.18.0

.. code-block:: shell

    SSLKEYLOGFILE=/tmp/sslkeylog scrapy crawl myspider

.. _NSS key log format: https://firefox-source-docs.mozilla.org/security/nss/legacy/key_log_format/index.html

.. warning:: Anyone who can read the key log file can decrypt the traffic of
    the connections recorded in it, including any credentials that they carry.

.. _debug-vscode:

Visual Studio Code
==================

.. highlight:: json

To debug spiders with Visual Studio Code you can use the following ``launch.json``::

    {
        "version": "0.1.0",
        "configurations": [
            {
                "name": "Python: Launch Scrapy Spider",
                "type": "python",
                "request": "launch",
                "module": "scrapy",
                "args": [
                    "runspider",
                    "${file}"
                ],
                "console": "integratedTerminal"
            }
        ]
    }


Also, make sure you enable "User Uncaught Exceptions", to catch exceptions in
your Scrapy spider.