Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Update and build docs

Ondra Urbanf736fefc0b75
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index 6adc46dbd..1c533be0d 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -725,7 +725,7 @@ a small state object is regularly persisted to keep track of the crawling status
For more details and code examples, see the [`RequestList`](requestlist) class.
-**Example Usage:**
+**Example usage:**
```javascript
const sources = [
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index a709b93b6..435dcec34 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -161,12 +161,34 @@ const escapedHtml = await page.evaluate(() => {
<a name="puppeteer.enqueueLinks"></a>
## `puppeteer.enqueueLinks(options)` ⇒ <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code>
-Finds HTML elements matching a CSS selector, clicks the elements and if a redirect is triggered and destination URL matches
-one of the provided [`PseudoUrl`](pseudourl)s, then the function enqueues that URL to a given request queue.
-To create a Request object function uses `requestTemplate` from a matching [`PseudoUrl`](pseudourl).
+The function finds HTML anchor (`&lt;a&gt;`) elements matching a specific CSS selector on a Puppeteer page,
+and enqueues the corresponding links to the provided [`RequestQueue`](requestqueue).
+Optionally, the function allows you to filter the target links URLs using an array of [`PseudoUrl`](pseudourl) objects
+and override settings of the enqueued [`Request`](request) objects.
-*WARNING*: This is work in progress. Currently the function doesn't click elements and only takes their `href` attribute and so
- is working only for link (`a`) elements, but not for buttons or JavaScript links.
+*IMPORTANT*: This is a work in progress. Currently the function only supports `&lt;a&gt;` elements with
+`href` attribute point to some URL. However, in the future the function will also support
+JavaScript links, buttons and form submissions.
+
+**Example usage**
+
+```javascript
+const Apify = require('apify');
+
+const browser = await Apify.launchPuppeteer();
+const page = await browser.goto('https://www.example.com');
+const requestQueue = await Apify.openRequestQueue();
+
+await Apify.utils.puppeteer.enqueueLinks({
+ page,
+ requestQueue,
+ selector: 'a.product-detail',
+ pseudoUrls: [
+ 'https://www.example.com/handbags/[.*]'
+ 'https://www.example.com/purses/[.*]'
+ ],
+});
+```
**Returns**: <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code> - Promise that resolves to an array of [`QueueOperationInfo`](../typedefs/queueoperationinfo) objects.
<table>
@@ -189,29 +211,30 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse
<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p><a href="requestqueue"><code>RequestQueue</code></a> instance where URLs will be enqueued.</p>
+<td colspan="3"><p>A request queue to which the URLs will be enqueued.</p>
</td></tr><tr>
<td><code>[options.selector]</code></td><td><code>String</code></td><td><code>&#x27;a&#x27;</code></td>
</tr>
<tr>
-<td colspan="3"><p>CSS selector matching elements to be clicked.</p>
+<td colspan="3"><p>A CSS selector matching links to be enqueued.</p>
</td></tr><tr>
<td><code>[options.pseudoUrls]</code></td><td><code><a href="pseudourl">Array&lt;PseudoUrl&gt;</a></code> | <code>Array&lt;Object&gt;</code> | <code>Array&lt;String&gt;</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,
- or an array of Strings or Objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s should be constructed
- The Objects must include at least a <code>purl</code> property, which holds a pseudoUrl string.
+ or an array of strings or objects from which the <a href="pseudourl"><code>PseudoUrl</code></a>s can be constructed.
+ The objects must include at least the <code>purl</code> property, which holds the pseudo-URL string.
All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.
- If <code>pseudoUrls</code> is an empty array, null or undefined, then the function
+ which lets you specify special properties for the enqueued <a href="request"><code>Request</code></a> objects.</p>
+<p> If <code>pseudoUrls</code> is an empty array, <code>null</code> or <code>undefined</code>, then the function
enqueues all links found on the page.</p>
</td></tr><tr>
<td><code>[options.userData]</code></td><td><code>Object</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Object that will be merged with the new Request&#39;s userData, overriding any values that
- were set via templating from pseudoUrls. This is useful when you need to override generic
- userData set by PseudoURL template in specific use cases.
+<td colspan="3"><p>An object that will be merged with the new <a href="request"><code>Request</code></a>&#39;s <code>userData</code>, overriding any values that
+ were set via templating from <code>pseudoUrls</code>. This is useful when you need to override generic
+ <code>userData</code> set by the <a href="pseudourl"><code>PseudoUrl</code></a> template in specific use cases.
<strong>Example:</strong></p>
<pre><code> // pseudoUrl.userData
{
diff --git a/docs/api/social.md b/docs/api/social.md
index 2bd0ca1fe..830b231aa 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -365,8 +365,11 @@ extracted from the plain text, which might be very inaccurate.
**Example usage:**
```javascript
-const puppeteer = await Apify.launchPuppeteer();
-await puppeteer.goto('http://www.example.com');
+const Apify = require('apify');
+
+const browser = await Apify.launchPuppeteer();
+const page = await browser.newPage();
+await page.goto('http://www.example.com');
const html = await puppeteer.content();
const result = Apify.utils.social.parseHandlesFromHtml(html);
diff --git a/docs/examples/PuppeteerCrawler.md b/docs/examples/PuppeteerCrawler.md
index e9ffdc7f6..1e62bf287 100644
--- a/docs/examples/PuppeteerCrawler.md
+++ b/docs/examples/PuppeteerCrawler.md
@@ -72,17 +72,14 @@ Apify.main(async () => {
// Store the results to the default dataset.
await Apify.pushData(data);
- // Find the link to the next page using Puppeteer functions.
- let nextHref;
- try {
- nextHref = await page.$eval('.morelink', el => el.href);
- } catch (err) {
- console.log(`${request.url} is the last page!`);
- return;
- }
-
- // Enqueue the link to the RequestQueue
- await requestQueue.addRequest(new Apify.Request({ url: nextHref }));
+ // Find a link to the next page and enqueue it if it exists.
+ const infos = await Apify.utils.puppeteer.enqueueLinks({
+ page,
+ requestQueue,
+ selector: '.morelink',
+ });
+
+ if (infos.length === 0) console.log(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/typedefs/LaunchPuppeteerOptions.md b/docs/typedefs/LaunchPuppeteerOptions.md
index 747481cd1..79d478c74 100644
--- a/docs/typedefs/LaunchPuppeteerOptions.md
+++ b/docs/typedefs/LaunchPuppeteerOptions.md
@@ -4,11 +4,11 @@ title: LaunchPuppeteerOptions
---
<a name="LaunchPuppeteerOptions"></a>
-An object representing options passed to the
-[`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
-function. In this object, you can pass any options supported by the
-[`puppeteer.launch()`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions),
-as well as the additional options provided by Apify SDK listed below.
+Apify extends the launch options of Puppeteer.
+You can use any of the
+<a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
+function and in addition, all the options available below.
**Properties**
<table>
@@ -19,6 +19,13 @@ as well as the additional options provided by Apify SDK listed below.
</thead>
<tbody>
<tr>
+<td><code>...</code></td><td></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>You can use any of the
+ <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+ options too.</p>
+</td></tr><tr>
<td><code>[proxyUrl]</code></td><td><code>String</code></td><td></td>
</tr>
<tr>
diff --git a/src/puppeteer.js b/src/puppeteer.js
index 173625df8..34fcd6a59 100644
--- a/src/puppeteer.js
+++ b/src/puppeteer.js
@@ -21,13 +21,17 @@ const LAUNCH_PUPPETEER_DEFAULT_VIEWPORT = {
};
/**
- * An object representing options passed to the
- * [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
- * function. In this object, you can pass any options supported by the
- * [`puppeteer.launch()`](https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions),
- * as well as the additional options provided by Apify SDK listed below.
+ * Apify extends the launch options of Puppeteer.
+ * You can use any of the
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+ * options in the [`Apify.launchPuppeteer()`](../api/apify#module_Apify.launchPuppeteer)
+ * function and in addition, all the options available below.
*
* @typedef {Object} LaunchPuppeteerOptions
+ * @property ...
+ * You can use any of the
+ * <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank"><code>puppeteer.launch()</code></a>
+ * options.
* @property {String} [proxyUrl]
* URL to a HTTP proxy server. It must define the port number,
* and it may also contain proxy username and password.