Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urban50c64bd5f450
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 1e5439f7e..e0564a5d8 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -135,9 +135,9 @@ await crawler.run();<p> This is useful if you need to extend the page load timeout or select different criteriato determine that the navigation succeeded.</p><p> Note that a single page object is only used to process a single request and it is closed afterwards.</p>-<p> See source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L9" target="_blank">GitHub</a>- for default behavior.</p>+<p> By default, the function invokes Puppeteer's <code>Page.goto()</code> with a timeout of 60 seconds.+ For details, see source code on+ <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L9" target="_blank">GitHub</a>.</p></td></tr><tr><td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr>diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.mdindex b6aec499c..1ceec6261 100644--- a/docs/api/PuppeteerPool.md+++ b/docs/api/PuppeteerPool.md@@ -112,7 +112,7 @@ await puppeteerPool.destroy();<td colspan="3"><p>Options used by <code>Apify.launchPuppeteer()</code> to start new Puppeteer instances.See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p></td></tr><tr>-<td><code>[options.recycleDiskCache]</code></td><td><code>Boolean</code></td><td></td>+<td><code>[options.recycleDiskCache]</code></td><td><code>Boolean</code></td><td><code>false</code></td></tr><tr><td colspan="3"><p>Enables recycling of disk cache directories by Chrome instances.diff --git a/docs/api/social.md b/docs/api/social.mdindex 12382d43c..800e907f2 100644--- a/docs/api/social.md+++ b/docs/api/social.md@@ -370,7 +370,7 @@ const Apify = require('apify');const browser = await Apify.launchPuppeteer();const page = await browser.newPage();await page.goto('http://www.example.com');-const html = await puppeteer.content();+const html = await page.content();const result = Apify.utils.social.parseHandlesFromHtml(html);console.log('Social handles:');