Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add a beginner's guide to Scrapy's main concepts (#8025)

Adrian4f0b5fa5c3a2
diff --git a/docs/index.rst b/docs/index.rst
index de06e3488..34eb003e6 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -69,6 +69,7 @@ Basic concepts
:caption: Basic concepts
:hidden:
+ topics/concepts
topics/commands
topics/spiders
topics/selectors
@@ -83,6 +84,9 @@ Basic concepts
topics/settings
topics/exceptions
+:doc:`topics/concepts`
+ Get a quick map of Scrapy's main concepts and when to use each.
+
:doc:`topics/commands`
Learn about the command-line tool used to manage your Scrapy project.
diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst
index c60c43f3c..9394fc2a7 100644
--- a/docs/topics/architecture.rst
+++ b/docs/topics/architecture.rst
@@ -125,18 +125,9 @@ Downloader middlewares
Downloader middlewares are specific hooks that sit between the Engine and the
Downloader and process requests when they pass from the Engine to the
-Downloader, and responses that pass from Downloader to the Engine.
-
-Use a Downloader middleware if you need to do one of the following:
-
-* process a request just before it is sent to the Downloader
- (i.e. right before Scrapy sends the request to the website);
-* change received response before passing it to a spider;
-* send a new Request instead of passing received response to a spider;
-* pass response to a spider without fetching a web page;
-* silently drop some requests.
-
-For more information see :ref:`topics-downloader-middleware`.
+Downloader, and responses that pass from Downloader to the Engine. See
+:ref:`concepts` for when to use one, and :ref:`topics-downloader-middleware`
+for more information.
.. _component-spider-middleware:
@@ -145,17 +136,17 @@ Spider middlewares
Spider middlewares are specific hooks that sit between the Engine and the
Spiders and are able to process spider input (responses) and output (items and
-requests).
+requests). See :ref:`concepts` for when to use one, and
+:ref:`topics-spider-middleware` for more information.
-Use a Spider middleware if you need to
+.. _component-extensions:
-* post-process output of spider callbacks - change/add/remove requests or items;
-* post-process start requests or items;
-* handle spider exceptions;
-* call errback instead of callback for some of the requests based on response
- content.
+Extensions
+----------
-For more information see :ref:`topics-spider-middleware`.
+Extensions do not have a specific role in the data flow like the other
+components above. See :ref:`concepts` for when to use one, and
+:ref:`topics-extensions` for more information.
Event-driven networking
=======================
diff --git a/docs/topics/concepts.rst b/docs/topics/concepts.rst
new file mode 100644
index 000000000..290ed3b1f
--- /dev/null
+++ b/docs/topics/concepts.rst
@@ -0,0 +1,102 @@
+.. _concepts:
+
+========================
+Scrapy's building blocks
+========================
+
+Scrapy splits the work of crawling and scraping across a handful of pieces,
+each responsible for one part of the problem. Placing a given piece of custom
+logic where it belongs keeps that logic reusable across spiders and the
+project easy to follow — this separation is what lets Scrapy scale from a
+single spider to large, maintainable projects, rather than one script per
+site.
+
+This page is a quick map of those pieces, what problem each one solves, and
+when to reach for it.
+
+Spiders
+=======
+
+A :ref:`spider <topics-spiders>` defines which requests to send and how to
+parse their responses. It is where a scraping project usually starts: one
+spider per site (or group of similar sites), yielding :ref:`items <topics-items>`
+and further requests as it parses each response.
+
+Items
+=====
+
+An :ref:`item <topics-items>` is the data you extract, as a plain key-value
+object. Spiders yield them, and everything downstream (item pipelines, feed
+exports) works with them regardless of which item type a project uses.
+
+Settings
+========
+
+:ref:`Settings <topics-settings>` configure everything else on this page:
+which middlewares and extensions are enabled and in what order, how item
+pipelines behave, feed export destinations, and Scrapy's own defaults.
+Settings are usually project-wide, but a spider can override them for itself
+alone through :attr:`~scrapy.Spider.custom_settings`, which is often the
+simplest way to turn something project-wide, such as a downloader middleware,
+into behavior that only applies to the spiders that need it.
+
+Downloader middleware
+=====================
+
+A :ref:`downloader middleware <topics-downloader-middleware>` sees every
+request Scrapy sends and every response it receives, regardless of which
+spider is running — though a spider can narrow that to itself alone through
+:attr:`~scrapy.Spider.custom_settings`. Reach for one when the logic is about
+the HTTP layer itself, such as adding proxies or authentication headers,
+retrying or redirecting based on the response, or short-circuiting a request
+without hitting the network.
+
+Spider middleware
+=================
+
+A :ref:`spider middleware <topics-spider-middleware>` sits between the engine
+and a spider's callbacks, processing the responses going in and the items and
+requests coming out. Reach for one to post-process what a spider produces
+(filtering items, adding requests, handling exceptions) without changing the
+spider's own parsing code, especially when the same post-processing should
+apply across several spiders.
+
+Item pipelines
+==============
+
+An :ref:`item pipeline <topics-item-pipeline>` processes items after a spider
+yields them: cleaning up fields, validating data, dropping duplicates, or
+storing items in a database. This is the usual place for anything that acts on
+a *scraped item* rather than on a request or response.
+
+Feed exports
+============
+
+:ref:`Feed exports <topics-feed-exports>` write scraped items straight to a
+file or storage (JSON, CSV, XML, and more) without any custom code. Reach for
+them before writing an item pipeline if all you need is to dump the scraped
+items somewhere.
+
+Extensions
+==========
+
+An :ref:`extension <topics-extensions>` is the wildcard: use it for
+functionality that does not fit any of the roles above, such as collecting
+stats, enforcing a memory limit, or logging crawl progress. Extensions
+typically hook into :ref:`signals <topics-signals>` rather than into the
+request/response/item flow.
+
+Signals
+=======
+
+:ref:`Signals <topics-signals>` notify your code when something happens during
+a crawl, such as a spider opening or closing, or a response being downloaded.
+Extensions are the usual place to connect to them, but any component can.
+
+Other components
+================
+
+The pieces above are what most projects touch directly. Scrapy has more of
+them under the hood, such as download handlers, the scheduler, and feed
+storages, that most projects never need to customize; see
+:ref:`topics-components` for the full list.
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 29e20ca1c..5eda7d92c 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -6,7 +6,8 @@ Downloader Middleware
The downloader middleware is a framework of hooks into Scrapy's
request/response processing. It's a light, low-level system for globally
-altering Scrapy's requests and responses.
+altering Scrapy's requests and responses at the HTTP level. See
+:ref:`concepts` for a rundown of other alternatives.
.. _topics-downloader-middleware-setting:
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 6d61cc342..a29c2aed3 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -9,7 +9,8 @@ own custom functionality into Scrapy.
Unlike other components, extensions do not have a specific role in Scrapy. They
are “wildcard” components that can be used for anything that does not fit the
-role of any other type of component.
+role of any other type of component, such as collecting stats or enforcing a
+memory limit; see :ref:`concepts` for a rundown of other alternatives.
Loading and activating extensions
=================================
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index 1c9ee0c77..219eff206 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -7,7 +7,8 @@ Spider Middleware
The spider middleware is a framework of hooks into Scrapy's spider processing
mechanism where you can plug custom functionality to process the responses that
are sent to :ref:`topics-spiders` for processing and to process the requests
-and items that are generated from spiders.
+and items that come out of a spider's callback. See :ref:`concepts` for a
+rundown of other alternatives.
.. _topics-spider-middleware-setting:
@@ -125,6 +126,10 @@ one or more of these methods:
*result* is lazy: a generator callback runs as *result* is iterated, so
code that runs before that iteration runs before the callback body.
+ The number of objects yielded need not match the number received: drop
+ some, pass others through unchanged, or yield more than were received,
+ e.g. turning one item into several.
+
.. seealso:: :ref:`universal-spider-middleware`.
:param response: the response which generated this output from the