Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

1.0 KB
Raw Blame
.. _topics-scheduler:

=========
Scheduler
=========

.. module:: scrapy.core.scheduler

The scheduler component receives requests from the :ref:`engine
<component-engine>` and stores them into persistent and/or non-persistent data
structures. It also gets those requests and feeds them back to the engine when
it asks for the next request to be downloaded.


Overriding the default scheduler
================================

You can use your own custom scheduler class by supplying its full
Python path in the :setting:`SCHEDULER` setting.


Minimal scheduler interface
===========================

.. autoclass:: BaseScheduler
   :members:


Default scheduler
=================

.. autoclass:: Scheduler()
   :members:
   :special-members: __init__, __len__


Priority queues
===============

.. autoclass:: scrapy.pqueues.PriorityQueueProtocol
    :members: from_crawler, push, pop, close, changed, next_priority, state
    :special-members: __len__

.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue

.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue