Scrapy, a fast high-level web crawling & scraping framework for Python.
[Docs] mention curl2scrapy in Request.from_curl
Aditi Dutta010edfe85caa
diff --git a/docs/topics/developer-tools.rst b/docs/topics/developer-tools.rstindex f1b0964c6..4e87a00f2 100644--- a/docs/topics/developer-tools.rst+++ b/docs/topics/developer-tools.rst@@ -292,6 +292,9 @@ Alternatively, if you want to know the arguments needed to recreate thatrequest you can use the :func:`scrapy.utils.curl.curl_to_request_kwargs`function to get a dictionary with the equivalent arguments.+Note that to translate a cURL command into a Scrapy request,+you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.+As you can see, with a few inspections in the `Network`-tool wewere able to easily replicate the dynamic requests of the scrollingfunctionality of the page. Crawling dynamic pages can be quitediff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex b98133676..aa326868b 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -104,6 +104,9 @@ If you get the expected response `sometimes`, but not always, the issue isprobably not your request, but the target server. The target server might bebuggy, overloaded, or :ref:`banning <bans>` some of your requests.+Note that to translate a cURL command into a Scrapy request,+you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.+.. _topics-handling-response-formats:Handling different response formatsdiff --git a/scrapy/http/request/__init__.py b/scrapy/http/request/__init__.pyindex 6c536cb71..0a6637af8 100644--- a/scrapy/http/request/__init__.py+++ b/scrapy/http/request/__init__.py@@ -129,6 +129,9 @@ class Request(object_ref)::class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`,may modify the :class:`~scrapy.http.Request` object.+ To translate a cURL command into a Scrapy request,+ you may use `curl2scrapy <https://michael-shub.github.io/curl2scrapy/>`_.+"""request_kwargs = curl_to_request_kwargs(curl_command, ignore_unknown_options)request_kwargs.update(kwargs)