Scrapy, a fast high-level web crawling & scraping framework for Python.
DOC move Data Flow below the picture; add links to components
Mikhail Korobov495d3226912b
diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rstindex ba0e2c61c..39e54ee99 100644--- a/docs/topics/architecture.rst+++ b/docs/topics/architecture.rst@@ -16,20 +16,71 @@ components and an outline of the data flow that takes place inside the systembelow with links for more detailed information about them. The data flow isalso described below.+.. _data-flow:++Data flow+=========+.. image:: _images/scrapy_architecture_02.png:width: 700:height: 470:alt: Scrapy architecture+The data flow in Scrapy is controlled by the execution engine, and goes like+this:++1. The :ref:`Engine <component-engine>` gets the first URLs to crawl from the+ :ref:`Spider <component-spiders>`.++2. The :ref:`Engine <component-engine>` schedules the URLs in the+ :ref:`Scheduler <component-scheduler>` as Requests and asks for the+ next URLs to crawl.++3. The :ref:`Scheduler <component-scheduler>` returns the next URLs to crawl+ to the :ref:`Engine <component-engine>`.++4. The :ref:`Engine <component-engine>` sends the URLs to the+ :ref:`Downloader <component-downloader>`, passing through the+ :ref:`Downloader Middleware <component-downloader-middleware>`+ (request direction).++5. Once the page finishes downloading the+ :ref:`Downloader <component-downloader>` generates a Response (with+ that page) and sends it to the Engine, passing through the+ :ref:`Downloader Middleware <component-downloader-middleware>`+ (response direction).++6. The :ref:`Engine <component-engine>` receives the Response from the+ :ref:`Downloader <component-downloader>` and sends it to the+ :ref:`Spider <component-spiders>` for processing, passing+ through the :ref:`Spider Middleware <component-spider-middleware>`+ (input direction).++7. The :ref:`Spider <component-spiders>` processes the Response and returns+ scraped items and new Requests (to follow) to the+ :ref:`Engine <component-engine>`, passing through the+ :ref:`Spider Middleware <component-spider-middleware>` (output direction).++8. The :ref:`Engine <component-engine>` sends processed items to+ :ref:`Item Pipelines <component-pipelines>` and processed Requests to+ the :ref:`Scheduler <component-scheduler>`.++9. The process repeats (from step 1) until there are no more requests from the+ :ref:`Scheduler <component-scheduler>`.+Components==========+.. _component-engine:+Scrapy Engine-------------The engine is responsible for controlling the data flow between all components-of the system, and triggering events when certain actions occur. See the Data-Flow section below for more details.+of the system, and triggering events when certain actions occur. See the+:ref:`Data Flow <data-flow>` section above for more details.++.. _component-scheduler:Scheduler---------@@ -37,12 +88,16 @@ SchedulerThe Scheduler receives requests from the engine and enqueues them for feedingthem later (also to the engine) when the engine requests them.+.. _component-downloader:+Downloader----------The Downloader is responsible for fetching web pages and feeding them to theengine which, in turn, feeds them to the spiders.+.. _component-spiders:+Spiders-------@@ -50,6 +105,8 @@ Spiders are custom classes written by Scrapy users to parse responses andextract items (aka scraped items) from them or additional URLs (requests) tofollow. For more information see :ref:`topics-spiders`.+.. _component-pipelines:+Item Pipeline-------------@@ -58,6 +115,8 @@ extracted (or scraped) by the spiders. Typical tasks include cleansing,validation and persistence (like storing the item in a database). For moreinformation see :ref:`topics-item-pipeline`.+.. _component-downloader-middleware:+Downloader middlewares----------------------@@ -76,6 +135,8 @@ Use a Downloader middleware if you need to do one of the following:For more information see :ref:`topics-downloader-middleware`.+.. _component-spider-middleware:+Spider middlewares------------------@@ -93,39 +154,6 @@ Use a Spider middleware if you need toFor more information see :ref:`topics-spider-middleware`.-Data flow-=========--The data flow in Scrapy is controlled by the execution engine, and goes like-this:--1. The Engine gets the first URLs to crawl from the Spider.--2. The Engine schedules the URLs in the Scheduler as Requests and asks for the- next URLs to crawl.--3. The Scheduler returns the next URLs to crawl to the Engine.--4. The Engine sends the URLs to the Downloader, passing through the- Downloader Middleware (request direction).--5. Once the page finishes downloading the Downloader generates a Response (with- that page) and sends it to the Engine, passing through the Downloader- Middleware (response direction).--6. The Engine receives the Response from the Downloader and sends it to the- Spider for processing, passing through the Spider Middleware (input direction).--7. The Spider processes the Response and returns scraped items and new Requests- (to follow) to the Engine, passing through the Spider Middleware- (output direction).--8. The Engine sends processed items to Item Pipelines and processed Requests to- the Scheduler.--9. The process repeats (from step 1) until there are no more requests from the- Scheduler.-Event-driven networking=======================