Scrapy, a fast high-level web crawling & scraping framework for Python.
Add a beginner's guide to Scrapy's main concepts (#8025)
Adrian4f0b5fa5c3a2
diff --git a/docs/index.rst b/docs/index.rstindex de06e3488..34eb003e6 100644--- a/docs/index.rst+++ b/docs/index.rst@@ -69,6 +69,7 @@ Basic concepts:caption: Basic concepts:hidden:+ topics/conceptstopics/commandstopics/spiderstopics/selectors@@ -83,6 +84,9 @@ Basic conceptstopics/settingstopics/exceptions+:doc:`topics/concepts`+ Get a quick map of Scrapy's main concepts and when to use each.+:doc:`topics/commands`Learn about the command-line tool used to manage your Scrapy project.diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rstindex c60c43f3c..9394fc2a7 100644--- a/docs/topics/architecture.rst+++ b/docs/topics/architecture.rst@@ -125,18 +125,9 @@ Downloader middlewaresDownloader middlewares are specific hooks that sit between the Engine and theDownloader and process requests when they pass from the Engine to the-Downloader, and responses that pass from Downloader to the Engine.--Use a Downloader middleware if you need to do one of the following:--* process a request just before it is sent to the Downloader- (i.e. right before Scrapy sends the request to the website);-* change received response before passing it to a spider;-* send a new Request instead of passing received response to a spider;-* pass response to a spider without fetching a web page;-* silently drop some requests.--For more information see :ref:`topics-downloader-middleware`.+Downloader, and responses that pass from Downloader to the Engine. See+:ref:`concepts` for when to use one, and :ref:`topics-downloader-middleware`+for more information... _component-spider-middleware:@@ -145,17 +136,17 @@ Spider middlewaresSpider middlewares are specific hooks that sit between the Engine and theSpiders and are able to process spider input (responses) and output (items and-requests).+requests). See :ref:`concepts` for when to use one, and+:ref:`topics-spider-middleware` for more information.-Use a Spider middleware if you need to+.. _component-extensions:-* post-process output of spider callbacks - change/add/remove requests or items;-* post-process start requests or items;-* handle spider exceptions;-* call errback instead of callback for some of the requests based on response- content.+Extensions+-----------For more information see :ref:`topics-spider-middleware`.+Extensions do not have a specific role in the data flow like the other+components above. See :ref:`concepts` for when to use one, and+:ref:`topics-extensions` for more information.Event-driven networking=======================diff --git a/docs/topics/concepts.rst b/docs/topics/concepts.rstnew file mode 100644index 000000000..290ed3b1f--- /dev/null+++ b/docs/topics/concepts.rst@@ -0,0 +1,102 @@+.. _concepts:++========================+Scrapy's building blocks+========================++Scrapy splits the work of crawling and scraping across a handful of pieces,+each responsible for one part of the problem. Placing a given piece of custom+logic where it belongs keeps that logic reusable across spiders and the+project easy to follow — this separation is what lets Scrapy scale from a+single spider to large, maintainable projects, rather than one script per+site.++This page is a quick map of those pieces, what problem each one solves, and+when to reach for it.++Spiders+=======++A :ref:`spider <topics-spiders>` defines which requests to send and how to+parse their responses. It is where a scraping project usually starts: one+spider per site (or group of similar sites), yielding :ref:`items <topics-items>`+and further requests as it parses each response.++Items+=====++An :ref:`item <topics-items>` is the data you extract, as a plain key-value+object. Spiders yield them, and everything downstream (item pipelines, feed+exports) works with them regardless of which item type a project uses.++Settings+========++:ref:`Settings <topics-settings>` configure everything else on this page:+which middlewares and extensions are enabled and in what order, how item+pipelines behave, feed export destinations, and Scrapy's own defaults.+Settings are usually project-wide, but a spider can override them for itself+alone through :attr:`~scrapy.Spider.custom_settings`, which is often the+simplest way to turn something project-wide, such as a downloader middleware,+into behavior that only applies to the spiders that need it.++Downloader middleware+=====================++A :ref:`downloader middleware <topics-downloader-middleware>` sees every+request Scrapy sends and every response it receives, regardless of which+spider is running — though a spider can narrow that to itself alone through+:attr:`~scrapy.Spider.custom_settings`. Reach for one when the logic is about+the HTTP layer itself, such as adding proxies or authentication headers,+retrying or redirecting based on the response, or short-circuiting a request+without hitting the network.++Spider middleware+=================++A :ref:`spider middleware <topics-spider-middleware>` sits between the engine+and a spider's callbacks, processing the responses going in and the items and+requests coming out. Reach for one to post-process what a spider produces+(filtering items, adding requests, handling exceptions) without changing the+spider's own parsing code, especially when the same post-processing should+apply across several spiders.++Item pipelines+==============++An :ref:`item pipeline <topics-item-pipeline>` processes items after a spider+yields them: cleaning up fields, validating data, dropping duplicates, or+storing items in a database. This is the usual place for anything that acts on+a *scraped item* rather than on a request or response.++Feed exports+============++:ref:`Feed exports <topics-feed-exports>` write scraped items straight to a+file or storage (JSON, CSV, XML, and more) without any custom code. Reach for+them before writing an item pipeline if all you need is to dump the scraped+items somewhere.++Extensions+==========++An :ref:`extension <topics-extensions>` is the wildcard: use it for+functionality that does not fit any of the roles above, such as collecting+stats, enforcing a memory limit, or logging crawl progress. Extensions+typically hook into :ref:`signals <topics-signals>` rather than into the+request/response/item flow.++Signals+=======++:ref:`Signals <topics-signals>` notify your code when something happens during+a crawl, such as a spider opening or closing, or a response being downloaded.+Extensions are the usual place to connect to them, but any component can.++Other components+================++The pieces above are what most projects touch directly. Scrapy has more of+them under the hood, such as download handlers, the scheduler, and feed+storages, that most projects never need to customize; see+:ref:`topics-components` for the full list.diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 29e20ca1c..5eda7d92c 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -6,7 +6,8 @@ Downloader MiddlewareThe downloader middleware is a framework of hooks into Scrapy'srequest/response processing. It's a light, low-level system for globally-altering Scrapy's requests and responses.+altering Scrapy's requests and responses at the HTTP level. See+:ref:`concepts` for a rundown of other alternatives... _topics-downloader-middleware-setting:diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rstindex 6d61cc342..a29c2aed3 100644--- a/docs/topics/extensions.rst+++ b/docs/topics/extensions.rst@@ -9,7 +9,8 @@ own custom functionality into Scrapy.Unlike other components, extensions do not have a specific role in Scrapy. Theyare “wildcard” components that can be used for anything that does not fit the-role of any other type of component.+role of any other type of component, such as collecting stats or enforcing a+memory limit; see :ref:`concepts` for a rundown of other alternatives.Loading and activating extensions=================================diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rstindex 1c9ee0c77..219eff206 100644--- a/docs/topics/spider-middleware.rst+++ b/docs/topics/spider-middleware.rst@@ -7,7 +7,8 @@ Spider MiddlewareThe spider middleware is a framework of hooks into Scrapy's spider processingmechanism where you can plug custom functionality to process the responses thatare sent to :ref:`topics-spiders` for processing and to process the requests-and items that are generated from spiders.+and items that come out of a spider's callback. See :ref:`concepts` for a+rundown of other alternatives... _topics-spider-middleware-setting:@@ -125,6 +126,10 @@ one or more of these methods:*result* is lazy: a generator callback runs as *result* is iterated, socode that runs before that iteration runs before the callback body.+ The number of objects yielded need not match the number received: drop+ some, pass others through unchanged, or yield more than were received,+ e.g. turning one item into several.+.. seealso:: :ref:`universal-spider-middleware`.:param response: the response which generated this output from the