Scrapy, a fast high-level web crawling & scraping framework for Python.
DOC improved Architecture overview
Mikhail Korobov3ca977a8cb3b
diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rstindex aef033dc3..dc03d47e8 100644--- a/docs/topics/architecture.rst+++ b/docs/topics/architecture.rst@@ -48,8 +48,7 @@ SpidersSpiders are custom classes written by Scrapy users to parse responses andextract items (aka scraped items) from them or additional URLs (requests) to-follow. Each spider is able to handle a specific domain (or group of domains).-For more information see :ref:`topics-spiders`.+follow. For more information see :ref:`topics-spiders`.Item Pipeline-------------@@ -64,18 +63,35 @@ Downloader middlewaresDownloader middlewares are specific hooks that sit between the Engine and theDownloader and process requests when they pass from the Engine to the-Downloader, and responses that pass from Downloader to the Engine. They provide-a convenient mechanism for extending Scrapy functionality by plugging custom-code. For more information see :ref:`topics-downloader-middleware`.+Downloader, and responses that pass from Downloader to the Engine.++Use a Downloader middleware if you need to do one of the following:++* process a request just before it is sent to the Downloader+ (i.e. right before Scrapy sends the request to the website);+* change received response before passing it to a spider;+* send a new Request instead of passing received response to a spider;+* pass response to a spider without fetching a web page;+* silently drop some requests.++For more information see :ref:`topics-downloader-middleware`.Spider middlewares------------------Spider middlewares are specific hooks that sit between the Engine and theSpiders and are able to process spider input (responses) and output (items and-requests). They provide a convenient mechanism for extending Scrapy-functionality by plugging custom code. For more information see-:ref:`topics-spider-middleware`.+requests).++Use a Spider middleware if you need to++* post-process output of spider callbacks - change/add/remove requests or items;+* post-process start_requests;+* handle spider exceptions;+* call errback instead of callback for some of the requests based on response+ content.++For more information see :ref:`topics-spider-middleware`.Data flow=========@@ -83,32 +99,30 @@ Data flowThe data flow in Scrapy is controlled by the execution engine, and goes likethis:-1. The Engine opens a domain, locates the Spider that handles that domain, and- asks the spider for the first URLs to crawl.--2. The Engine gets the first URLs to crawl from the Spider and schedules them+1. The Engine gets the first URLs to crawl from the Spider and schedules themin the Scheduler, as Requests.-3. The Engine asks the Scheduler for the next URLs to crawl.+2. The Engine asks the Scheduler for the next URLs to crawl.-4. The Scheduler returns the next URLs to crawl to the Engine and the Engine+3. The Scheduler returns the next URLs to crawl to the Engine and the Enginesends them to the Downloader, passing through the Downloader Middleware(request direction).-5. Once the page finishes downloading the Downloader generates a Response (with+4. Once the page finishes downloading the Downloader generates a Response (withthat page) and sends it to the Engine, passing through the DownloaderMiddleware (response direction).-6. The Engine receives the Response from the Downloader and sends it to the+5. The Engine receives the Response from the Downloader and sends it to theSpider for processing, passing through the Spider Middleware (input direction).-7. The Spider processes the Response and returns scraped items and new Requests+6. The Spider processes the Response and returns scraped items and new Requests(to follow) to the Engine.-8. The Engine sends scraped items (returned by the Spider) to the Item Pipeline- and Requests (returned by spider) to the Scheduler+7. The Engine passes scraped items and new Requests returned by a spider+ through Spider Middleware (output direction), and then sends processed+ items to Item Pipelines and processed Requests to the Scheduler.-9. The process repeats (from step 2) until there are no more requests from the+8. The process repeats (from step 2) until there are no more requests from theScheduler, and the Engine closes the domain.Event-driven networking