Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update docs
Ondra Urban0c5dfd74545d
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex e42c06a07..ff8feac5a 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -8,7 +8,7 @@ Provides a framework for the parallel crawling of web pages using plain HTTP req<a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a> HTML parser.`CheerioCrawler` downloads each URL using a plain HTTP request,-parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>+parses the HTML content using <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>and then invokes the user-provided [`handlePageFunction()`](#new_CheerioCrawler_new) to extract page datausing a <a href="https://jquery.com/" target="_blank">jQuery</a>-like interface to the parsed HTML DOM.@@ -24,7 +24,8 @@ The crawler finishes when there are no more [`Request`](request) objects to crawBy default, `CheerioCrawler` downloads HTML using the<a href="https://www.npmjs.com/package/request-promise" target="_blank">request-promise</a> NPM package.-You can override this behavior by setting the `requestFunction` option.+You can override this behavior by setting the `requestFunction` option. If you want to keep `request-promise`,+but use different than default options, use the `requestOptions` parameter.New requests are only dispatched when there is enough free CPU and memory available,using the functionality provided by the [`AutoscaledPool`](autoscaledpool) class.@@ -70,13 +71,13 @@ await crawler.run();* [CheerioCrawler](cheeriocrawler)- * [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)+ * [`new CheerioCrawler(options, [useApifyProxy], [apifyProxyGroups], [apifyProxySession])`](#new_CheerioCrawler_new)* [`.run()`](#CheerioCrawler+run) ⇒ <code>Promise</code>* [`.abort()`](#CheerioCrawler+abort) ⇒ <code>Promise</code><a name="new_CheerioCrawler_new"></a>-## `new CheerioCrawler(options)`+## `new CheerioCrawler(options, [useApifyProxy], [apifyProxyGroups], [apifyProxySession])`<table><thead><tr>@@ -120,9 +121,17 @@ await crawler.run();<td><code>[options.requestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>-<td colspan="3"><p>Overrides the function that performs the HTTP request to get the raw HTML needed for Cheerio.+<td colspan="3"><p>Overrides the default function that performs the HTTP request to get the raw HTML needed for Cheerio.See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L264">GitHub</a> for default behavior.</p></td></tr><tr>+<td><code>[options.requestOptions]</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Represents the options passed to the <code>requestFunction</code>, which are essentially the options passed to+ <a href="https://www.npmjs.com/package/request-promise" target="_blank">request-promise</a> to make the HTTP call.+ Provided <code>requestOptions</code> are merged with defaults so if you only need to add a parameter, there's no need to duplicate+ the whole object.</p>+</td></tr><tr><td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>300</code></td></tr><tr>@@ -139,6 +148,37 @@ await crawler.run();<td colspan="3"><p>If set to true, SSL certificate errors will be ignored. This is dependent on using the defaultrequest function. If using a custom <code>options.requestFunction</code>, user needs to implement this functionality.</p></td></tr><tr>+<td><code>[useApifyProxy]</code></td><td><code>Boolean</code></td><td><code>false</code></td>+</tr>+<tr>+<td colspan="3"><p>If set to <code>true</code>, <code>CheerioCrawler</code> will be configured to use+ <a href="https://my.apify.com/proxy" target="_blank">Apify Proxy</a> for all connections.+ For more information, see the <a href="https://www.apify.com/docs/proxy" target="_blank">documentation</a></p>+</td></tr><tr>+<td><code>[apifyProxyGroups]</code></td><td><code>Array<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of proxy groups to be used+ by the <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>.+ Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>+</td></tr><tr>+<td><code>[apifyProxySession]</code></td><td><code>String</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Apify Proxy session identifier to be used with requests made by <code>CheerioCrawler</code>.+ All HTTP requests going through the proxy with the same session identifier+ will use the same target proxy server (i.e. the same IP address).+ The identifier can only contain the following characters: <code>0-9</code>, <code>a-z</code>, <code>A-Z</code>, <code>"."</code>, <code>"_"</code> and <code>"~"</code>.+ Only applied if the <code>useApifyProxy</code> option is <code>true</code>.</p>+</td></tr><tr>+<td><code>[options.proxyUrls]</code></td><td><code>Array<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>CheerioCrawler</code> instance.+ The provided custom proxies' order will be randomized and the resulting list rotated.+ Custom proxies are not compatible with Apify Proxy and an attempt to use both+ configuration options will cause an error to be thrown on startup.</p>+</td></tr><tr><td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 05530adb7..7e5bab942 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -177,6 +177,14 @@ await crawler.run();after which the browser is closed even if there are pending tabs. See<code>killInstanceAfterMillis</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.</p></td></tr><tr>+<td><code>[options.proxyUrls]</code></td><td><code>Array<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of custom proxy URLs to be used by the <a href="puppeteerpool"><code>PuppeteerPool</code></a> instance.+ The provided custom proxies' order will be randomized and the resulting list rotated.+ Custom proxies are not compatible with Apify Proxy and an attempt to use both+ configuration options will cause an error to be thrown on startup.</p>+</td></tr><tr><td><code>[options.launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td></tr><tr>diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.mdindex 31033dfd2..50673c860 100644--- a/docs/api/PuppeteerPool.md+++ b/docs/api/PuppeteerPool.md@@ -114,6 +114,14 @@ await puppeteerPool.destroy();where <code>X</code> is the approximate maximum number of bytes for disk cache.</p><p> <em>IMPORTANT:</em> Currently this feature only works in <strong>headful</strong> mode, because of a bug in Chromium.</p><p> The <code>options.recycleDiskCache</code> setting should not be used together with <code>--disk-cache-dir</code> argument in <code>options.launchPuppeteerOptions.args</code>.</p>+</td></tr><tr>+<td><code>[options.proxyUrls]</code></td><td><code>Array<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of custom proxy URLs to be used by the <code>PuppeteerPool</code> instance.+ The provided custom proxies' order will be randomized and the resulting list rotated.+ Custom proxies are not compatible with Apify Proxy and an attempt to use both+ configuration options will cause an error to be thrown on startup.</p></td></tr></tbody></table><a name="PuppeteerPool+newPage"></a>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 19030386c..9843ff673 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -27,7 +27,7 @@ await puppeteer.injectJQuery(page);* [`.injectFile(page, filePath)`](#puppeteer.injectFile) ⇒ <code>Promise</code>* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ <code>Promise</code>* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ <code>Promise</code>- * [`.enqueueLinks(page, selector, pseudoUrls, requestQueue)`](#puppeteer.enqueueLinks) ⇒ <code>Promise<Array<RequestOperationInfo>></code>+ * [`.enqueueLinks(page, selector, requestQueue, [pseudoUrls])`](#puppeteer.enqueueLinks) ⇒ <code>Promise<Array<RequestOperationInfo>></code>* [`.blockResources(page, resourceTypes)`](#puppeteer.blockResources) ⇒ <code>Promise</code>* [`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ <code>Promise</code>* [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ <code>function</code>@@ -125,7 +125,7 @@ libraries included by the page that use the same variable.</table><a name="puppeteer.enqueueLinks"></a>-## `puppeteer.enqueueLinks(page, selector, pseudoUrls, requestQueue)` ⇒ <code>Promise<Array<RequestOperationInfo>></code>+## `puppeteer.enqueueLinks(page, selector, requestQueue, [pseudoUrls])` ⇒ <code>Promise<Array<RequestOperationInfo>></code>Finds HTML elements matching a CSS selector, clicks the elements and if a redirect is triggered and destination URL matchesone of the provided [`PseudoUrl`](pseudourl)s, then the function enqueues that URL to a given request queue.To create a Request object function uses `requestTemplate` from a matching [`PseudoUrl`](pseudourl).@@ -152,15 +152,18 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse<tr><td colspan="3"><p>CSS selector matching elements to be clicked.</p></td></tr><tr>-<td><code>pseudoUrls</code></td><td><code><a href="pseudourl">Array<PseudoUrl></a></code></td>+<td><code>requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td></tr><tr>-<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued.</p>+<td colspan="3"><p><a href="requestqueue"><code>RequestQueue</code></a> instance where URLs will be enqueued.</p></td></tr><tr>-<td><code>requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td>+<td><code>[pseudoUrls]</code></td><td><code><a href="pseudourl">Array<PseudoUrl></a></code> | <code>Array<Object></code> | <code>Array<String></code></td></tr><tr>-<td colspan="3"><p>Request queue object where URLs will be enqueued.</p>+<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,+ or an array of Strings or Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s should be constructed+ The Objects must include at least a <code>purl</code> property, which holds a pseudoUrl string.+ All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.</p></td></tr></tbody></table><a name="puppeteer.blockResources"></a>