Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

DOC move Data Flow below the picture; add links to components

Mikhail Korobov495d3226912b
diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst
index ba0e2c61c..39e54ee99 100644
--- a/docs/topics/architecture.rst
+++ b/docs/topics/architecture.rst
@@ -16,20 +16,71 @@ components and an outline of the data flow that takes place inside the system
below with links for more detailed information about them. The data flow is
also described below.
+.. _data-flow:
+
+Data flow
+=========
+
.. image:: _images/scrapy_architecture_02.png
:width: 700
:height: 470
:alt: Scrapy architecture
+The data flow in Scrapy is controlled by the execution engine, and goes like
+this:
+
+1. The :ref:`Engine <component-engine>` gets the first URLs to crawl from the
+ :ref:`Spider <component-spiders>`.
+
+2. The :ref:`Engine <component-engine>` schedules the URLs in the
+ :ref:`Scheduler <component-scheduler>` as Requests and asks for the
+ next URLs to crawl.
+
+3. The :ref:`Scheduler <component-scheduler>` returns the next URLs to crawl
+ to the :ref:`Engine <component-engine>`.
+
+4. The :ref:`Engine <component-engine>` sends the URLs to the
+ :ref:`Downloader <component-downloader>`, passing through the
+ :ref:`Downloader Middleware <component-downloader-middleware>`
+ (request direction).
+
+5. Once the page finishes downloading the
+ :ref:`Downloader <component-downloader>` generates a Response (with
+ that page) and sends it to the Engine, passing through the
+ :ref:`Downloader Middleware <component-downloader-middleware>`
+ (response direction).
+
+6. The :ref:`Engine <component-engine>` receives the Response from the
+ :ref:`Downloader <component-downloader>` and sends it to the
+ :ref:`Spider <component-spiders>` for processing, passing
+ through the :ref:`Spider Middleware <component-spider-middleware>`
+ (input direction).
+
+7. The :ref:`Spider <component-spiders>` processes the Response and returns
+ scraped items and new Requests (to follow) to the
+ :ref:`Engine <component-engine>`, passing through the
+ :ref:`Spider Middleware <component-spider-middleware>` (output direction).
+
+8. The :ref:`Engine <component-engine>` sends processed items to
+ :ref:`Item Pipelines <component-pipelines>` and processed Requests to
+ the :ref:`Scheduler <component-scheduler>`.
+
+9. The process repeats (from step 1) until there are no more requests from the
+ :ref:`Scheduler <component-scheduler>`.
+
Components
==========
+.. _component-engine:
+
Scrapy Engine
-------------
The engine is responsible for controlling the data flow between all components
-of the system, and triggering events when certain actions occur. See the Data
-Flow section below for more details.
+of the system, and triggering events when certain actions occur. See the
+:ref:`Data Flow <data-flow>` section above for more details.
+
+.. _component-scheduler:
Scheduler
---------
@@ -37,12 +88,16 @@ Scheduler
The Scheduler receives requests from the engine and enqueues them for feeding
them later (also to the engine) when the engine requests them.
+.. _component-downloader:
+
Downloader
----------
The Downloader is responsible for fetching web pages and feeding them to the
engine which, in turn, feeds them to the spiders.
+.. _component-spiders:
+
Spiders
-------
@@ -50,6 +105,8 @@ Spiders are custom classes written by Scrapy users to parse responses and
extract items (aka scraped items) from them or additional URLs (requests) to
follow. For more information see :ref:`topics-spiders`.
+.. _component-pipelines:
+
Item Pipeline
-------------
@@ -58,6 +115,8 @@ extracted (or scraped) by the spiders. Typical tasks include cleansing,
validation and persistence (like storing the item in a database). For more
information see :ref:`topics-item-pipeline`.
+.. _component-downloader-middleware:
+
Downloader middlewares
----------------------
@@ -76,6 +135,8 @@ Use a Downloader middleware if you need to do one of the following:
For more information see :ref:`topics-downloader-middleware`.
+.. _component-spider-middleware:
+
Spider middlewares
------------------
@@ -93,39 +154,6 @@ Use a Spider middleware if you need to
For more information see :ref:`topics-spider-middleware`.
-Data flow
-=========
-
-The data flow in Scrapy is controlled by the execution engine, and goes like
-this:
-
-1. The Engine gets the first URLs to crawl from the Spider.
-
-2. The Engine schedules the URLs in the Scheduler as Requests and asks for the
- next URLs to crawl.
-
-3. The Scheduler returns the next URLs to crawl to the Engine.
-
-4. The Engine sends the URLs to the Downloader, passing through the
- Downloader Middleware (request direction).
-
-5. Once the page finishes downloading the Downloader generates a Response (with
- that page) and sends it to the Engine, passing through the Downloader
- Middleware (response direction).
-
-6. The Engine receives the Response from the Downloader and sends it to the
- Spider for processing, passing through the Spider Middleware (input direction).
-
-7. The Spider processes the Response and returns scraped items and new Requests
- (to follow) to the Engine, passing through the Spider Middleware
- (output direction).
-
-8. The Engine sends processed items to Item Pipelines and processed Requests to
- the Scheduler.
-
-9. The process repeats (from step 1) until there are no more requests from the
- Scheduler.
-
Event-driven networking
=======================