Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update changelog, build docs
Ondra Urbanfd064915fd03
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex c89c7a821..6716e2841 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,7 +1,9 @@-xxx+NEXT====================-- Bugfix in BasicCrawler: async calls in `isFinishedFunction` were not awaited-- Better logging of memory overload errors+- Improve logging of memory overload errors.+- Fix multiple log lines appearing when a crawler was about to finish.+- Add `Apify.utils.puppeteer.enqueueLinksByClickingElements()` function which enables you+ to add requests to the queue from pure JavaScript navigations, form submissions etc.0.14.15 / 2019-05-31====================diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 86c5ee368..f2bd8b697 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -148,8 +148,9 @@ await crawler.run();</tr><tr><td colspan="3"><p>Custom options passed to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> constructor.- Note that the <code>runTaskFunction</code>, <code>isTaskReadyFunction</code> and <code>isFinishedFunction</code> options- are provided by <code>BasicCrawler</code> and cannot be overridden.</p>+ Note that the <code>runTaskFunction</code> and <code>isTaskReadyFunction</code> options+ are provided by <code>BasicCrawler</code> and cannot be overridden.+ However, you can provide a custom implementation of <code>isFinishedFunction</code>.</p></td></tr><tr><td><code>[options.minConcurrency]</code></td><td><code>Object</code></td><td><code>1</code></td></tr>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 0d4160221..af160be1a 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -136,14 +136,16 @@ await crawler.run();<td colspan="3"><p>Represents the options passed to<a href="https://www.npmjs.com/package/request" target="_blank">request</a> to make the HTTP call.Provided <code>requestOptions</code> are added to internal defaults that cannot be overridden to ensure- the operation of <code>CheerioCrawler</code> and all its options. If you need more granular control over- your requests, use <a href="basiccrawler"><code>BasicCrawler</code></a>.</p>-<p> The mandatory internal defaults:</p>+ the operation of <code>CheerioCrawler</code> and all its options. Headers will not be merged,+ use <a href="requestlist"><code>RequestList</code></a> and/or <a href="requestqueue"><code>RequestQueue</code></a> to initialize your <a href="request"><code>Request</code></a> with the+ correct headers or use <code>options.prepareRequestFunction</code> to modify your <a href="request"><code>Request</code></a> dynamically.+ If you need more granular control over your requests, use <a href="basiccrawler"><code>BasicCrawler</code></a>.</p>+<p> The mandatory internal defaults that <strong>CANNOT BE OVERRIDDEN</strong> by <code>requestOptions</code>:</p><pre><code> {- url,- method,- headers,- payload, // Are provided by RequestList and/or RequestQueue+ url, // Provided by RequestList and/or RequestQueue+ method, // Provided by RequestList and/or RequestQueue+ headers, // Provided by RequestList and/or RequestQueue+ payload, // Provided by RequestList and/or RequestQueuestrictSSL, // Use options.ignoreSslErrorsproxy, // Use options.useApifyProxy or options.proxyUrls}</code></pre></td></tr><tr>diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.mdindex 0d4dba8de..34a947f03 100644--- a/docs/api/PseudoUrl.md+++ b/docs/api/PseudoUrl.md@@ -57,7 +57,7 @@ if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('M* [PseudoUrl](pseudourl)* [`new PseudoUrl(purl, requestTemplate)`](#new_PseudoUrl_new)* [`.matches(url)`](#PseudoUrl+matches) ⇒ `Boolean`- * [`.createRequest(url)`](#PseudoUrl+createRequest) ⇒ [`Request`](request)+ * [`.createRequest(urlOrProps)`](#PseudoUrl+createRequest) ⇒ [`Request`](request)<a name="new_PseudoUrl_new"></a>@@ -106,8 +106,12 @@ Determines whether a URL matches this pseudo-URL pattern.</table><a name="PseudoUrl+createRequest"></a>-## `pseudoUrl.createRequest(url)` ⇒ [`Request`](request)-Creates a Request object from a provided `requestTemplate` and given URL.+## `pseudoUrl.createRequest(urlOrProps)` ⇒ [`Request`](request)+Creates a Request object from a provided `requestTemplate` and a given URL+or an object that specifies $[`Request`](request) properties. In case of a collision+the properties will override the template, except for `userData`, which will+be merged together, with the `userData` property having preference over the template.+This enables dynamic overriding of the template.<table><thead>@@ -117,7 +121,7 @@ Creates a Request object from a provided `requestTemplate` and given URL.</thead><tbody><tr>-<td><code>url</code></td><td><code>String</code></td>+<td><code>urlOrProps</code></td><td><code>string</code> | <code>Object</code></td></tr><tr></tr></tbody>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 01eb88741..ce139243f 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -24,6 +24,7 @@ await puppeteer.injectJQuery(page);* [`puppeteer`](#puppeteer) : `object`+ * [`.enqueueLinksByClickingElements`](#puppeteer.enqueueLinksByClickingElements) ⇒ `Promise<Array<QueueOperationInfo>>`* [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ `Promise`* [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ `Promise`* [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ `Promise<Response>`@@ -34,6 +35,126 @@ await puppeteer.injectJQuery(page);* ~~[`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ `Promise`~~* [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ `function`+<a name="puppeteer.enqueueLinksByClickingElements"></a>++## `puppeteer.enqueueLinksByClickingElements` ⇒ `Promise<Array<QueueOperationInfo>>`+The function finds elements matching a specific CSS selector in a Puppeteer page,+clicks all those elements using a mouse move and a left mouse button click and intercepts+all the navigation requests that are subsequently produced by the page. The intercepted+requests, including their methods, headers and payloads are then enqueued to a provided+[`RequestQueue`](requestqueue). This is useful to crawl JavaScript heavy pages where links are not available+in `href` elements, but rather navigations are triggered in click handlers.+If you're looking to find URLs in `href` attributes of the page, see [`enqueueLinks()`](utils#utils.enqueueLinks).++Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](pseudourl) objects+and override settings of the enqueued [`Request`](request) objects.++*IMPORTANT*: To be able to do this, this function uses various mutations on the page,+such as changing the Z-index of elements being clicked and their visibility. Therefore,+it is recommended to only use this function as the last operation in the page.++*USING HEADFUL BROWSER*: When using a headful browser, this function will only be able to click elements+in the focused tab, effectively limiting concurrency to 1. In headless mode, full concurrency can be achieved.++*PERFORMANCE*: Clicking elements with a mouse and intercepting requests is not a low level operation+that takes nanoseconds. It's not very CPU intensive, but it takes time. We strongly recommend limiting+the scope of the clicking as much as possible by using a specific selector that targets only the elements+that you assume or know will produce a navigation. You can certainly click everything by using+the `*` selector, but be prepared to wait minutes to get results on a large and complex page.++**Example usage**++```javascript+const Apify = require('apify');++const browser = await Apify.launchPuppeteer();+const page = await browser.goto('https://www.example.com');+const requestQueue = await Apify.openRequestQueue();++await Apify.utils.enqueueLinksByClickingElements({+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: [+ 'https://www.example.com/handbags/[.*]'+ 'https://www.example.com/purses/[.*]'+ ],+});+```++**Returns**: `Promise<Array<QueueOperationInfo>>` - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.+<table>+<thead>+<tr>+<th>Param</th><th>Type</th><th>Default</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>All <code>enqueueLinksByClickingElements()</code> parameters are passed+ via an options object with the following keys:</p>+</td></tr><tr>+<td><code>options.page</code></td><td><code>Page</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>+</td></tr><tr>+<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>+</td></tr><tr>+<td><code>options.selector</code></td><td><code>String</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>A CSS selector matching elements to be clicked on. Unlike in <a href="utils#utils.enqueueLinks"><code>enqueueLinks()</code></a>, there is no default+ value. This is to prevent suboptimal use of this function by using it too broadly.</p>+</td></tr><tr>+<td><code>[options.pseudoUrls]</code></td><td><code>Array<Object></code> | <code>Array.<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,+ or an array of strings or RegExps or plain Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.</p>+<p> The plain objects must include at least the <code>purl</code> property, which holds the pseudo-URL string or RegExp.+ All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor,+ which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>+<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function+ enqueues all links found on the page.</p>+</td></tr><tr>+<td><code>[options.updateRequestFunction]</code></td><td><code>function</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Just before a new <a href="request"><code>Request</code></a> is enqueued to the <a href="requestqueue"><code>RequestQueue</code></a>, this function can be used to modify+ its contents such as <code>userData</code>, <code>payload</code> or, most importantly <code>uniqueKey</code>. This is useful when necessary+ to enqueue multiple <code>Requests</code> to the queue that share the same URL, but differ in methods or payloads.</p>+<p> For example: by generating your own <code>uniqueKey</code> from a combination of <code>url</code>, <code>method</code> and <code>payload</code> you enable crawling+ of websites that navigate using form submits (POST requests).</p>+</td></tr><tr>+<td><code>[options.waitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>1</code></td>+</tr>+<tr>+<td colspan="3"><p>Clicking in the page triggers various asynchronous operations that lead to new URLs being shown+ by the browser. It could be a simple JavaScript redirect or opening of a new tab in the browser.+ These events often happen only some time after the actual click. Requests typically take milliseconds+ while new tabs open in hundreds of milliseconds.</p>+<p> To be able to capture all those events, the <code>enqueueLinksByClickingElements()</code> function repeatedly waits+ for the <code>waitForPageIdleSecs</code>. By repeatedly we mean that whenever a relevant event is triggered, the timer+ is restarted. As long as new events keep coming, the function will not return, unless+ the below <code>maxWaitForPageIdleSecs</code> timeout is reached.</p>+<p> You may want to reduce this for example when you're sure that your clicks do not open new tabs,+ or increase when you're not getting all the expected URLs.</p>+</td></tr><tr>+<td><code>[options.maxWaitForPageIdleSecs]</code></td><td><code>number</code></td><td><code>5</code></td>+</tr>+<tr>+<td colspan="3"><p>This is the maximum period for which the function will keep tracking events, even if more events keep coming.+ Its purpose is to prevent a deadlock in the page by periodic events, often unrelated to the clicking itself.+ See <code>waitForPageIdleSecs</code> above for an explanation.</p>+</td></tr></tbody>+</table><a name="puppeteer.addInterceptRequestHandler"></a>## `puppeteer.addInterceptRequestHandler` ⇒ `Promise`diff --git a/docs/api/utils.md b/docs/api/utils.mdindex a8e2cc911..96b3e1b37 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -25,7 +25,7 @@ await Apify.utils.sleep(1500);* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)* [`.isDocker()`](#utils.isDocker) ⇒ `Promise`* [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`- * [`.extractUrls(string, [urlRegExp])`](#utils.extractUrls) ⇒ `Array<String>`+ * [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`* [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`@@ -34,14 +34,14 @@ await Apify.utils.sleep(1500);## `utils.enqueueLinks` ⇒ `Promise<Array<QueueOperationInfo>>`The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)either in a Puppeteer page, or in a Cheerio object (parsed HTML),-and enqueues the corresponding links to the provided [`RequestQueue`](requestqueue).+and enqueues the URLs in their `href` attributes to the provided [`RequestQueue`](requestqueue).+If you're looking to find URLs in JavaScript heavy pages where links are not available+in `href` elements, but rather navigations are triggered in click handlers+see [`enqueueLinksByClickingElements()`](puppeteer#puppeteer.enqueueLinksByClickingElements).+Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](pseudourl) objectsand override settings of the enqueued [`Request`](request) objects.-*IMPORTANT*: This is a work in progress. Currently the function only supports elements with-`href` attribute pointing to a URL. However, in the future the function will also support-JavaScript links, buttons and form submissions when used with a Puppeteer Page.-**Example usage**```javascript@@ -111,7 +111,7 @@ await Apify.utils.enqueueLinks({<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,or an array of strings or RegExps or plain Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.</p><p> The plain objects must include at least the <code>purl</code> property, which holds the pseudo-URL string or RegExp.- All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.+ All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor,which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p><p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the functionenqueues all links found on the page.</p>@@ -223,7 +223,7 @@ Optionally, custom regular expression and encoding may be provided.</table><a name="utils.extractUrls"></a>-## `utils.extractUrls(string, [urlRegExp])` ⇒ `Array<String>`+## `utils.extractUrls(options)` ⇒ `Array<String>`Collects all URLs in an arbitrary string to an array, optionally using a custom regular expression.<table>@@ -234,11 +234,15 @@ Collects all URLs in an arbitrary string to an array, optionally using a custom</thead><tbody><tr>-<td><code>string</code></td><td><code>String</code></td><td></td>+<td><code>options</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+</tr><tr>+<td><code>options.string</code></td><td><code>String</code></td><td></td></tr><tr></tr><tr>-<td><code>[urlRegExp]</code></td><td><code>RegExp</code></td><td><code>Apify.utils.URL_NO_COMMAS_REGEX</code></td>+<td><code>[options.urlRegExp]</code></td><td><code>RegExp</code></td><td><code>Apify.utils.URL_NO_COMMAS_REGEX</code></td></tr><tr></tr></tbody>