Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Build docs
Ondra Urban7199b5595f08
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 1569faa51..e73766614 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -472,6 +472,12 @@ The `launchPuppeteer()` function alters the following Puppeteer options:Also it adds <code>--enable-resource-load-scheduler=false</code>to make crawling of pages in all tabs run equally fast.</li>+ <li>+ Sets <code>defaultViewport</code> Puppeteer option (if not already set)+ to a more reasonable default for screenshots and debugging.+ You can set `options.defaultViewport` to `null` if you prefer to let Puppeteer+ choose the default viewport size.+ </li></ul>To use this function, you need to have the <a href="https://www.npmjs.com/package/puppeteer" target="_blank">puppeteer</a>diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 5aa47e422..77ee85757 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -84,10 +84,10 @@ await crawler.run();<tr><td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each URL to crawl.</p><p> The function receives the following object as an argument:</p>-<pre><code> {- request: Request- }-</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.+<pre><code>{+ request: Request+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance represents the URL to crawl.The function must return a promise.</p></td></tr><tr><td><code>options.requestList</code></td><td><code><a href="requestlist">RequestList</a></code></td><td></td>@@ -105,10 +105,17 @@ await crawler.run();<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>-<td colspan="3"><p>Function that handles requests that failed more then <code>options.maxRequestRetries</code> times.- See source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">GitHub</a>- for default behavior.</p>+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>+<p> The function receives the following object as an argument:</p>+<pre><code>{+ request: Request,+ error: Error,+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+ represents the last error thrown during processing of the request.</p>+<p> See+ <a href="https://github.com/apifytech/apify-js/blob/master/src/basic_crawler.js#L11" target="_blank">source code</a>+ for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td></tr>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex c79d6b67d..5e52e1978 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -97,12 +97,12 @@ await crawler.run();<td colspan="3"><p>User-provided function that performs the logic of the crawler. It is called for each pageloaded and parsed by the crawler.</p><p> The function receives the following object as an argument:</p>-<pre><code> {- $: Cheerio, // the Cheerio object with parsed HTML- html: String // the raw HTML of the page- request: Request,- response: Object // a http.IncomingMessage object with properties such as the `statusCode`- }+<pre><code>{+ $: Cheerio, // the Cheerio object with parsed HTML+ html: String // the raw HTML of the page+ request: Request,+ response: Object // a http.IncomingMessage object with properties such as the `statusCode`+}</code></pre><p> With the <a href="request"><code>Request</code></a> object representing the URL to crawl.If the function returns a promise, it is awaited.</p></td></tr><tr>@@ -181,8 +181,16 @@ await crawler.run();<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>-<td colspan="3"><p>Function that handles requests that failed more then <code>option.maxRequestRetries</code> times.- See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">GitHub</a> for default behavior.</p>+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>+<p> The function receives the following object as an argument:</p>+<pre><code>{+ request: Request,+ error: Error,+}+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+ represents the last error thrown during processing of the request.</p>+<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>+ for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td></tr>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 42bd74f8e..ad3e7cb96 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -87,12 +87,12 @@ await crawler.run();<tr><td colspan="3"><p>Function that is called to process each request.It is passed an object with the following fields:</p>-<pre><code> {- request: Request,- response: Response,- page: Page,- puppeteerPool: PuppeteerPool- }+<pre><code>{+ request: Request,+ response: Response,+ page: Page,+ puppeteerPool: PuppeteerPool+}</code></pre><p> <code>request</code> is an instance of the <a href="request"><code>Request</code></a> object with details about the URL to open, HTTP method etc.<code>response</code> is an instance of the <code>Puppeteer</code><a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a>@@ -134,10 +134,17 @@ await crawler.run();<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td></tr><tr>-<td colspan="3"><p>Function to handle requests that failed more than <code>option.maxRequestRetries</code> times.- See source code on- <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">GitHub</a>- for default behavior.</p>+<td colspan="3"><p>A function to handle requests that failed more than <code>option.maxRequestRetries</code> times.</p>+<p> The function receives the following object as an argument:</p>+<pre><code>{+ request: Request,+ error: Error,+}+</code></pre><p> Where the <a href="request"><code>Request</code></a> instance corresponds to the failed request, and the <code>Error</code> instance+ represents the last error thrown during processing of the request.</p>+<p> See+ <a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_crawler.js#L11" target="_blank">source code</a>+ for the default implementation of this function.</p></td></tr><tr><td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td></tr>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex f1abe4c24..2dd1421fb 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -28,13 +28,13 @@ await Apify.utils.sleep(1500);<a name="utils.URL_NO_COMMAS_REGEX"></a>-## `utils.URL\_NO\_COMMAS\_REGEX`+## `utils.URL_NO_COMMAS_REGEX`Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL charactersand does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).<a name="utils.URL_WITH_COMMAS_REGEX"></a>-## `utils.URL\_WITH\_COMMAS\_REGEX`+## `utils.URL_WITH_COMMAS_REGEX`Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.