Skip to content

Scrapy, a fast high-level web crawling & scraping framework for Python.

Add a two-legged OAuth request-signing recipe (#8208)

Adrian62e90b77fd11
diff --git a/docs/faq.rst b/docs/faq.rst
index 456fc9ae9..3fac4df41 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -165,6 +165,11 @@ Can I use Basic HTTP Authentication in my spiders?
Yes, see :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`.
+Can I sign my requests with OAuth?
+----------------------------------
+
+Yes, see :ref:`mw-oauth`.
+
Why does Scrapy download pages in English instead of my native language?
------------------------------------------------------------------------
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 8fcad6239..8550ece4c 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -214,6 +214,41 @@ requests as well, and the middleware above downloads a token for each of them.
Cache the task that downloads the token, and not only its result, to download
the token only once.
+.. _mw-oauth:
+
+Signing requests with OAuth
+===========================
+
+To sign requests, e.g. for two-legged OAuth, compute the signed headers in
+:meth:`process_request` with a third-party OAuth 1 client, such as the one from
+`oauthlib`_:
+
+.. skip: next
+
+.. code-block:: python
+
+ from oauthlib.oauth1 import Client
+
+
+ class OAuthMiddleware:
+ def __init__(self, client):
+ self.client = client
+
+ @classmethod
+ def from_crawler(cls, crawler):
+ settings = crawler.settings
+ client = Client(
+ settings["OAUTH_CONSUMER_KEY"],
+ client_secret=settings["OAUTH_CONSUMER_SECRET"],
+ )
+ return cls(client)
+
+ def process_request(self, request):
+ _, headers, _ = self.client.sign(request.url, http_method=request.method)
+ request.headers["Authorization"] = headers["Authorization"]
+
+.. _oauthlib: https://oauthlib.readthedocs.io/
+
.. _topics-downloader-middleware-ref:
Built-in downloader middleware reference