Scrapy, a fast high-level web crawling & scraping framework for Python.
Update headless browser docs
Eugenio Lacuestabdabc500aaa3
diff --git a/docs/topics/dynamic-content.rst b/docs/topics/dynamic-content.rstindex 495111b56..7450de4a2 100644--- a/docs/topics/dynamic-content.rst+++ b/docs/topics/dynamic-content.rst@@ -248,8 +248,34 @@ Using a headless browserA `headless browser`_ is a special web browser that provides an API forautomation.-The easiest way to use a headless browser with Scrapy is to use Selenium_,-along with `scrapy-selenium`_ for seamless integration.+Since version 2.0, it is possible to integrate libraries that use the+``async/await`` syntax. One such library is `pyppeteer`_ (an unnoficial+Python port of `puppeteer`_), which uses headless Chrome to download and+render pages.+The following is a simple snippet to illustrate its usage within Scrapy::++ import pyppeteer+ import scrapy++ class PyppeteerSpider(scrapy.Spider):+ name = "pyppeteer"+ start_urls = ["data:,"] # avoid making an actual upstream request++ async def parse(self, response):+ browser = await pyppeteer.launch()+ page = await browser.newPage()+ await page.goto("https:/example.org")+ title = await page.title()+ yield {"title": title}++Keep in mind that this is just a proof of concept, since it circumvents+most of the Scrapy components (middlewares, dupefilter, etc).++There are some 3rd party projects which provider better integration:++* https://github.com/elacuesta/scrapy-pyppeteer+* https://github.com/lopuhin/scrapy-pyppeteer+* https://github.com/clemfromspace/scrapy-puppeteer.. _AJAX: https://en.wikipedia.org/wiki/Ajax_%28programming%29@@ -259,11 +285,11 @@ along with `scrapy-selenium`_ for seamless integration... _headless browser: https://en.wikipedia.org/wiki/Headless_browser.. _JavaScript: https://en.wikipedia.org/wiki/JavaScript.. _js2xml: https://github.com/scrapinghub/js2xml+.. _puppeteer: https://pptr.dev/+.. _pyppeteer: https://pyppeteer.github.io/pyppeteer/.. _pytesseract: https://github.com/madmaze/pytesseract-.. _scrapy-selenium: https://github.com/clemfromspace/scrapy-selenium.. _scrapy-splash: https://github.com/scrapy-plugins/scrapy-splash-.. _Selenium: https://www.selenium.dev/.. _Splash: https://github.com/scrapinghub/splash.. _tabula-py: https://github.com/chezou/tabula-py.. _wget: https://www.gnu.org/software/wget/-.. _wgrep: https://github.com/stav/wgrep\ No newline at end of file+.. _wgrep: https://github.com/stav/wgrep