Scrapy, a fast high-level web crawling & scraping framework for Python.
Add a two-legged OAuth request-signing recipe (#8208)
Adrian62e90b77fd11
diff --git a/docs/faq.rst b/docs/faq.rstindex 456fc9ae9..3fac4df41 100644--- a/docs/faq.rst+++ b/docs/faq.rst@@ -165,6 +165,11 @@ Can I use Basic HTTP Authentication in my spiders?Yes, see :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`.+Can I sign my requests with OAuth?+----------------------------------++Yes, see :ref:`mw-oauth`.+Why does Scrapy download pages in English instead of my native language?------------------------------------------------------------------------diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rstindex 8fcad6239..8550ece4c 100644--- a/docs/topics/downloader-middleware.rst+++ b/docs/topics/downloader-middleware.rst@@ -214,6 +214,41 @@ requests as well, and the middleware above downloads a token for each of them.Cache the task that downloads the token, and not only its result, to downloadthe token only once.+.. _mw-oauth:++Signing requests with OAuth+===========================++To sign requests, e.g. for two-legged OAuth, compute the signed headers in+:meth:`process_request` with a third-party OAuth 1 client, such as the one from+`oauthlib`_:++.. skip: next++.. code-block:: python++ from oauthlib.oauth1 import Client+++ class OAuthMiddleware:+ def __init__(self, client):+ self.client = client++ @classmethod+ def from_crawler(cls, crawler):+ settings = crawler.settings+ client = Client(+ settings["OAUTH_CONSUMER_KEY"],+ client_secret=settings["OAUTH_CONSUMER_SECRET"],+ )+ return cls(client)++ def process_request(self, request):+ _, headers, _ = self.client.sign(request.url, http_method=request.method)+ request.headers["Authorization"] = headers["Authorization"]++.. _oauthlib: https://oauthlib.readthedocs.io/+.. _topics-downloader-middleware-ref:Built-in downloader middleware reference