Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update and build docs
Ondra Urban7e75b2db831f
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex 4c0131a9b..8a0204808 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -53,11 +53,11 @@ await dataset.pushData([* [Dataset](dataset)* [`.pushData(data)`](#Dataset+pushData) ⇒ <code>Promise</code>- * [`.getData([options])`](#Dataset+getData) ⇒ <code>Promise<(Array\|String\|Buffer)></code>+ * [`.getData([options])`](#Dataset+getData) ⇒ <code>Promise<(Array|String|Buffer)></code>* [`.getInfo()`](#Dataset+getInfo) ⇒ <code>Promise<Object></code>* [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ <code>Promise</code>* [`.map(iteratee, options)`](#Dataset+map) ⇒ <code>Promise<Array></code>- * [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ <code>Promise<\*></code>+ * [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ <code>Promise<*></code>* [`.delete()`](#Dataset+delete) ⇒ <code>Promise</code><a name="Dataset+pushData"></a>@@ -101,7 +101,7 @@ The objects must be serializable to JSON and the JSON representation of each obj</table><a name="Dataset+getData"></a>-## `dataset.getData([options])` ⇒ <code>Promise<(Array\|String\|Buffer)></code>+## `dataset.getData([options])` ⇒ <code>Promise<(Array|String|Buffer)></code>Returns items in the dataset based on the provided parameters.<table>@@ -329,7 +329,7 @@ If `iteratee` returns a `Promise` then it's awaited before a next call.</table><a name="Dataset+reduce"></a>-## `dataset.reduce(iteratee, memo, options)` ⇒ <code>Promise<\*></code>+## `dataset.reduce(iteratee, memo, options)` ⇒ <code>Promise<*></code>Reduces a list of values down to a single value.Memo is the initial state of the reduction, and each successive step of it should be returned by `iteratee()`.diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex 804f7759f..f9b7a8f46 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -64,14 +64,14 @@ await store.delete('some-key');* [KeyValueStore](keyvaluestore)- * [`.getValue(key)`](#KeyValueStore+getValue) ⇒ <code>Promise<(Object\|String\|Buffer)></code>+ * [`.getValue(key)`](#KeyValueStore+getValue) ⇒ <code>Promise<(Object|String|Buffer)></code>* [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ <code>Promise</code>* [`.delete()`](#KeyValueStore+delete) ⇒ <code>Promise</code>* [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ <code>Promise</code><a name="KeyValueStore+getValue"></a>-## `keyValueStore.getValue(key)` ⇒ <code>Promise<(Object\|String\|Buffer)></code>+## `keyValueStore.getValue(key)` ⇒ <code>Promise<(Object|String|Buffer)></code>Gets a value from the key-value store.The function returns a `Promise` that resolves to the record value,@@ -94,7 +94,7 @@ const store = await Apify.openKeyValueStore('my-screenshots');const buffer = await store.getValue('screenshot1.png');```-**Returns**: <code>Promise<(Object\|String\|Buffer)></code> - Returns a promise that resolves to an object, string+**Returns**: <code>Promise<(Object|String|Buffer)></code> - Returns a promise that resolves to an object, stringor <a href="https://nodejs.org/api/buffer.html" target="_blank"><code>Buffer</code></a>, dependingon the MIME content type of the record.<table>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 435dcec34..ef54a23b6 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -28,7 +28,6 @@ await puppeteer.injectJQuery(page);* [`.injectFile(page, filePath)`](#puppeteer.injectFile) ⇒ <code>Promise</code>* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ <code>Promise</code>* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ <code>Promise</code>- * [`.enqueueLinks(options)`](#puppeteer.enqueueLinks) ⇒ <code>Promise<Array<QueueOperationInfo>></code>* [`.blockResources(page, [resourceTypes])`](#puppeteer.blockResources) ⇒ <code>Promise</code>* [`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ <code>Promise</code>* [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ <code>function</code>@@ -158,104 +157,6 @@ const escapedHtml = await page.evaluate(() => {<td colspan="3"><p>Puppeteer <a href="https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#class-page">Page</a> object.</p></td></tr></tbody></table>-<a name="puppeteer.enqueueLinks"></a>--## `puppeteer.enqueueLinks(options)` ⇒ <code>Promise<Array<QueueOperationInfo>></code>-The function finds HTML anchor (`<a>`) elements matching a specific CSS selector on a Puppeteer page,-and enqueues the corresponding links to the provided [`RequestQueue`](requestqueue).-Optionally, the function allows you to filter the target links URLs using an array of [`PseudoUrl`](pseudourl) objects-and override settings of the enqueued [`Request`](request) objects.--*IMPORTANT*: This is a work in progress. Currently the function only supports `<a>` elements with-`href` attribute point to some URL. However, in the future the function will also support-JavaScript links, buttons and form submissions.--**Example usage**--```javascript-const Apify = require('apify');--const browser = await Apify.launchPuppeteer();-const page = await browser.goto('https://www.example.com');-const requestQueue = await Apify.openRequestQueue();--await Apify.utils.puppeteer.enqueueLinks({- page,- requestQueue,- selector: 'a.product-detail',- pseudoUrls: [- 'https://www.example.com/handbags/[.*]'- 'https://www.example.com/purses/[.*]'- ],-});-```--**Returns**: <code>Promise<Array<QueueOperationInfo>></code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.-<table>-<thead>-<tr>-<th>Param</th><th>Type</th><th>Default</th>-</tr>-</thead>-<tbody>-<tr>-<td><code>options</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"></td></tr><tr>-<td><code>options.page</code></td><td><code>Page</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>-</td></tr><tr>-<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>-</td></tr><tr>-<td><code>[options.selector]</code></td><td><code>String</code></td><td><code>'a'</code></td>-</tr>-<tr>-<td colspan="3"><p>A CSS selector matching links to be enqueued.</p>-</td></tr><tr>-<td><code>[options.pseudoUrls]</code></td><td><code><a href="pseudourl">Array<PseudoUrl></a></code> | <code>Array<Object></code> | <code>Array<String></code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,- or an array of strings or objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.- The objects must include at least the <code>purl</code> property, which holds the pseudo-URL string.- All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.- which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>-<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function- enqueues all links found on the page.</p>-</td></tr><tr>-<td><code>[options.userData]</code></td><td><code>Object</code></td><td></td>-</tr>-<tr>-<td colspan="3"><p>An object that will be merged with the new <a href="request"><code>Request</code></a>'s <code>userData</code>, overriding any values that- were set via templating from <code>pseudoUrls</code>. This is useful when you need to override generic- <code>userData</code> set by the <a href="pseudourl"><code>PseudoUrl</code></a> template in specific use cases.- <strong>Example:</strong></p>-<pre><code> // pseudoUrl.userData- {- name: 'John',- surname: 'Doe',- }-- // userData- {- name: 'Albert',- age: 31- }-- // enqueued request.userData- {- name: 'Albert',- surname: 'Doe',- age: 31,- }-</code></pre></td></tr></tbody>-</table><a name="puppeteer.blockResources"></a>## `puppeteer.blockResources(page, [resourceTypes])` ⇒ <code>Promise</code>diff --git a/docs/api/social.md b/docs/api/social.mdindex 830b231aa..12382d43c 100644--- a/docs/api/social.md+++ b/docs/api/social.md@@ -31,11 +31,11 @@ const emails = Apify.utils.social.emailsFromText('[email protected] bob@example.* [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ <code>Array<String></code>* [`.phonesFromText(text)`](#social.phonesFromText) ⇒ <code>Array<String></code>* [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ <code>Array<String></code>- * [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ <code>\*</code>+ * [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ <code>*</code><a name="social.LINKEDIN_REGEX"></a>-## `social.LINKEDIN\_REGEX` : <code>RegExp</code>+## `social.LINKEDIN_REGEX` : <code>RegExp</code>Regular expression to exactly match a single LinkedIn profile URL.It has the following form: `/^...$/i` and matches URLs such as:```@@ -59,7 +59,7 @@ if (Apify.utils.social.LINKEDIN_REGEX.test('https://www.linkedin.com/in/alan-tur<a name="social.LINKEDIN_REGEX_GLOBAL"></a>-## `social.LINKEDIN\_REGEX\_GLOBAL` : <code>RegExp</code>+## `social.LINKEDIN_REGEX_GLOBAL` : <code>RegExp</code>Regular expression to find multiple LinkedIn profile URLs in a text or HTML.It has the following form: `/.../ig` and matches URLs such as:```@@ -86,7 +86,7 @@ if (matches) console.log(`${matches.length} LinkedIn profiles found!`);<a name="social.INSTAGRAM_REGEX"></a>-## `social.INSTAGRAM\_REGEX` : <code>RegExp</code>+## `social.INSTAGRAM_REGEX` : <code>RegExp</code>Regular expression to exactly match a single Instagram profile URL.It has the following form: `/^...$/i` and matches URLs such as:```@@ -110,7 +110,7 @@ if (Apify.utils.social.INSTAGRAM_REGEX_STRING.test('https://www.instagram.com/ol<a name="social.INSTAGRAM_REGEX_GLOBAL"></a>-## `social.INSTAGRAM\_REGEX\_GLOBAL` : <code>RegExp</code>+## `social.INSTAGRAM_REGEX_GLOBAL` : <code>RegExp</code>Regular expression to find multiple Instagram profile URLs in a text or HTML.It has the following form: `/.../ig` and matches URLs such as:```@@ -137,7 +137,7 @@ if (matches) console.log(`${matches.length} Instagram profiles found!`);<a name="social.TWITTER_REGEX"></a>-## `social.TWITTER\_REGEX` : <code>RegExp</code>+## `social.TWITTER_REGEX` : <code>RegExp</code>Regular expression to exactly match a single Twitter profile URL.It has the following form: `/^...$/i` and matches URLs such as:```@@ -160,7 +160,7 @@ if (Apify.utils.social.TWITTER_REGEX_STRING.test('https://www.twitter.com/apify'<a name="social.TWITTER_REGEX_GLOBAL"></a>-## `social.TWITTER\_REGEX\_GLOBAL` : <code>RegExp</code>+## `social.TWITTER_REGEX_GLOBAL` : <code>RegExp</code>Regular expression to find multiple Twitter profile URLs in a text or HTML.It has the following form: `/.../ig` and matches URLs such as:```@@ -186,7 +186,7 @@ if (matches) console.log(`${matches.length} Twitter profiles found!`);<a name="social.FACEBOOK_REGEX"></a>-## `social.FACEBOOK\_REGEX` : <code>RegExp</code>+## `social.FACEBOOK_REGEX` : <code>RegExp</code>Regular expression to exactly match a single Facebook profile URL.It has the following form: `/^...$/i` and matches URLs such as:```@@ -211,7 +211,7 @@ if (Apify.utils.social.FACEBOOK_REGEX_STRING.test('https://www.facebook.com/apif<a name="social.FACEBOOK_REGEX_GLOBAL"></a>-## `social.FACEBOOK\_REGEX\_GLOBAL` : <code>RegExp</code>+## `social.FACEBOOK_REGEX_GLOBAL` : <code>RegExp</code>Regular expression to find multiple Facebook profile URLs in a text or HTML.It has the following form: `/.../ig` and matches URLs such as:```@@ -238,13 +238,13 @@ if (matches) console.log(`${matches.length} Facebook profiles found!`);<a name="social.EMAIL_REGEX"></a>-## `social.EMAIL\_REGEX` : <code>RegExp</code>+## `social.EMAIL_REGEX` : <code>RegExp</code>Regular expression to exactly match a single email address.It has the following form: `/^...$/i`.<a name="social.EMAIL_REGEX_GLOBAL"></a>-## `social.EMAIL\_REGEX\_GLOBAL` : <code>RegExp</code>+## `social.EMAIL_REGEX_GLOBAL` : <code>RegExp</code>Regular expression to find multiple email addresses in a text.It has the following form: `/.../ig`.@@ -340,7 +340,7 @@ If no phone numbers are found, the function returns an empty array.</table><a name="social.parseHandlesFromHtml"></a>-## `social.parseHandlesFromHtml(html, data)` ⇒ <code>\*</code>+## `social.parseHandlesFromHtml(html, data)` ⇒ <code>*</code>The function attempts to extract emails, phone numbers and social profile URLs from a HTML document,specifically LinkedIn, Twitter, Instagram and Facebook profile URLs.The function removes duplicates from the resulting arrays and sorts the items alphabetically.@@ -377,7 +377,7 @@ console.log('Social handles:');console.dir(result);```-**Returns**: <code>\*</code> - An object with the social handles.+**Returns**: <code>*</code> - An object with the social handles.<table><thead><tr>diff --git a/docs/api/utils.md b/docs/api/utils.mdindex cb9ca6082..79f7f9f3f 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -19,6 +19,7 @@ await Apify.utils.sleep(1500);* [`utils`](#utils) : <code>object</code>+ * [`.enqueueLinks`](#utils.enqueueLinks) ⇒ <code>Promise<Array<QueueOperationInfo>></code>* [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)* [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)* [`.isDocker()`](#utils.isDocker) ⇒ <code>Promise</code>@@ -28,15 +29,121 @@ await Apify.utils.sleep(1500);* [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ <code>String</code>* [`.htmlToText(html)`](#utils.htmlToText) ⇒ <code>String</code>+<a name="utils.enqueueLinks"></a>++## `utils.enqueueLinks` ⇒ <code>Promise<Array<QueueOperationInfo>></code>+The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)+either in a Puppeteer page, or in a Cheerio object (parsed HTML),+and enqueues the corresponding links to the provided [`RequestQueue`](requestqueue).+Optionally, the function allows you to filter the target links' URLs using an array of [`PseudoUrl`](pseudourl) objects+and override settings of the enqueued [`Request`](request) objects.++*IMPORTANT*: This is a work in progress. Currently the function only supports elements with+`href` attribute pointing to a URL. However, in the future the function will also support+JavaScript links, buttons and form submissions when used with a Puppeteer Page.++**Example usage**++```javascript+const Apify = require('apify');++const browser = await Apify.launchPuppeteer();+const page = await browser.goto('https://www.example.com');+const requestQueue = await Apify.openRequestQueue();++await Apify.utils.enqueueLinks({+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: [+ 'https://www.example.com/handbags/[.*]'+ 'https://www.example.com/purses/[.*]'+ ],+});+```++**Returns**: <code>Promise<Array<QueueOperationInfo>></code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.+<table>+<thead>+<tr>+<th>Param</th><th>Type</th><th>Default</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>options</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>All <code>enqueueLinks()</code> parameters are passed+ via an options object with the following keys:</p>+</td></tr><tr>+<td><code>options.page</code></td><td><code>Page</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.+ Either <code>page</code> or <code>$</code> option must be provided.</p>+</td></tr><tr>+<td><code>options.$</code></td><td><code>Cheerio</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p><a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.+ Either <code>page</code> or <code>$</code> option must be provided.</p>+</td></tr><tr>+<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>+</td></tr><tr>+<td><code>[options.selector]</code></td><td><code>String</code></td><td><code>'a'</code></td>+</tr>+<tr>+<td colspan="3"><p>A CSS selector matching links to be enqueued.</p>+</td></tr><tr>+<td><code>[options.pseudoUrls]</code></td><td><code><a href="pseudourl">Array<PseudoUrl></a></code> | <code>Array<Object></code> | <code>Array<String></code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,+ or an array of strings or objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.+ The objects must include at least the <code>purl</code> property, which holds the pseudo-URL string.+ All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.+ which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>+<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function+ enqueues all links found on the page.</p>+</td></tr><tr>+<td><code>[options.userData]</code></td><td><code>Object</code></td><td></td>+</tr>+<tr>+<td colspan="3"><p>An object that will be merged with the new <a href="request"><code>Request</code></a>'s <code>userData</code>, overriding any values that+ were set via templating from <code>pseudoUrls</code>. This is useful when you need to override generic+ <code>userData</code> set by the <a href="pseudourl"><code>PseudoUrl</code></a> template in specific use cases.</p>+<p> <strong>Example:</strong></p>+<pre><code>// pseudoUrl.userData+{+ name: 'John',+ surname: 'Doe',+}+</code></pre><pre><code>// userData+{+ name: 'Albert',+ age: 31+}+</code></pre><pre><code>// enqueued request.userData+{+ name: 'Albert',+ surname: 'Doe',+ age: 31,+}+</code></pre></td></tr></tbody>+</table><a name="utils.URL_NO_COMMAS_REGEX"></a>-## `utils.URL\_NO\_COMMAS\_REGEX`+## `utils.URL_NO_COMMAS_REGEX`Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL charactersand does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).<a name="utils.URL_WITH_COMMAS_REGEX"></a>-## `utils.URL\_WITH\_COMMAS\_REGEX`+## `utils.URL_WITH_COMMAS_REGEX`Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.mdindex 79d478c74..27100a4f1 100644--- a/docs/typedefs/LaunchPuppeteerOptions.md+++ b/docs/typedefs/LaunchPuppeteerOptions.md@@ -24,7 +24,7 @@ function and in addition, all the options available below.<tr><td colspan="3"><p>You can use any of the<a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>- options too.</p>+ options.</p></td></tr><tr><td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td></tr>diff --git a/docs/typedefs/QueueOperationInfo.md b/docs/typedefs/QueueOperationInfo.mdindex 09dd26c14..7c47130d0 100644--- a/docs/typedefs/QueueOperationInfo.md+++ b/docs/typedefs/QueueOperationInfo.md@@ -6,7 +6,7 @@ title: QueueOperationInfoA helper class that is used to report results from various[`RequestQueue`](../api/requestqueue) functions as well as-[`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).+[`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).**Properties**<table>diff --git a/src/enqueue_links.js b/src/enqueue_links.jsindex d7a515e5e..a8e7ea9ae 100644--- a/src/enqueue_links.js+++ b/src/enqueue_links.js@@ -74,7 +74,7 @@ export const extraxtUrlsFromCheerio = ($, selector) => {let logDeprecationWarning = true;/**- * The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)+ * The function finds elements matching a specific CSS selector (HTML anchor (`<a>`) by default)* either in a Puppeteer page, or in a Cheerio object (parsed HTML),* and enqueues the corresponding links to the provided {@link RequestQueue}.* Optionally, the function allows you to filter the target links' URLs using an array of {@link PseudoUrl} objects@@ -93,7 +93,7 @@ let logDeprecationWarning = true;* const page = await browser.goto('https://www.example.com');* const requestQueue = await Apify.openRequestQueue();*- * await Apify.utils.puppeteer.enqueueLinks({+ * await Apify.utils.enqueueLinks({* page,* requestQueue,* selector: 'a.product-detail',@@ -105,11 +105,13 @@ let logDeprecationWarning = true;* ```** @param {Object} options+ * All `enqueueLinks()` parameters are passed+ * via an options object with the following keys:* @param {Page} options.page* Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.* Either `page` or `$` option must be provided.* @param {Cheerio} options.$- * <a href="https://www.npmjs.com/package/cheerio" target="_blank"><code>Cheerio</code></a> object.+ * <a href="https://github.com/cheeriojs/cheerio" target="_blank"><code>Cheerio</code></a> object with loaded HTML.* Either `page` or `$` option must be provided.* @param {RequestQueue} options.requestQueue* A request queue to which the URLs will be enqueued.@@ -128,30 +130,34 @@ let logDeprecationWarning = true;* An object that will be merged with the new {@link Request}'s `userData`, overriding any values that* were set via templating from `pseudoUrls`. This is useful when you need to override generic* `userData` set by the {@link PseudoUrl} template in specific use cases.- * **Example:**- * ```- * // pseudoUrl.userData- * {- * name: 'John',- * surname: 'Doe',- * }- *- * // userData- * {- * name: 'Albert',- * age: 31- * }*- * // enqueued request.userData- * {- * name: 'Albert',- * surname: 'Doe',- * age: 31,- * }- * ```+ * **Example:**+ * ```+ * // pseudoUrl.userData+ * {+ * name: 'John',+ * surname: 'Doe',+ * }+ * ```+ * ```+ * // userData+ * {+ * name: 'Albert',+ * age: 31+ * }+ * ```+ * ```+ * // Enqueued request.userData+ * {+ * name: 'Albert',+ * surname: 'Doe',+ * age: 31,+ * }+ * ```* @return {Promise<QueueOperationInfo[]>}* Promise that resolves to an array of {@link QueueOperationInfo} objects.* @memberOf utils+ * @name enqueueLinks*/export const enqueueLinks = async (...args) => {// TODO: Remove after v1.0.0 gets released.diff --git a/src/request_queue.js b/src/request_queue.jsindex 14749b1f5..84d0409b3 100644--- a/src/request_queue.js+++ b/src/request_queue.js@@ -116,7 +116,7 @@ const getRequestId = (uniqueKey) => {/*** A helper class that is used to report results from various* [`RequestQueue`](../api/requestqueue) functions as well as- * [`Apify.utils.puppeteer.enqueueLinks()`](../api/puppeteer#puppeteer.enqueueLinks).+ * [`Apify.utils.enqueueLinks()`](../api/utils#utils.enqueueLinks).** @typedef {Object} QueueOperationInfo* @property {Boolean} wasAlreadyPresent Indicates if request was already present in the queue.diff --git a/website/sidebars.json b/website/sidebars.jsonindex a26ae655d..3d8ce1a4f 100644--- a/website/sidebars.json+++ b/website/sidebars.json@@ -23,6 +23,7 @@"api/requestlist","api/requestqueue","api/utils",+ "api/log","api/puppeteer","api/social"]diff --git a/website/tools/helpers.js b/website/tools/helpers.jsindex 919d61dd4..d6be11d21 100644--- a/website/tools/helpers.js+++ b/website/tools/helpers.js@@ -14,3 +14,8 @@ exports.inlineLinks = (text, options) => {}return text;};++exports.escape = function escape(input) {+ if (typeof input !== 'string') return null;+ return input.replace(/([*|])/g, '$1');+};