Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Update changelog, bump version, build docs

Ondra Urban66b342950a4d
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 90b7f4ac3..1699de4f4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,10 @@
-xxx
+0.9.15 / 2018-11-30
===================
- Upgraded Puppeteer to 1.11.0
+- **DEPRECATION WARNING:** `Apify.utils.puppeteer.enqueueLinks()` now uses an options object instead of individual parameters
+ and supports passing of `userData` to the enqueued `request`. Previously: `enqueueLinks(page, selector, requestQueue, pseudoUrls)`
+ Now: `enqueueLinks({ page, selector, requestQueue, pseudoUrls, userData })`. Using individual parameters is **DEPRECATED**.
+
0.9.14 / 2018-11-27
===================
diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.md
index ce2a54f9d..61f42fddf 100644
--- a/docs/api/puppeteer.md
+++ b/docs/api/puppeteer.md
@@ -27,7 +27,7 @@ await puppeteer.injectJQuery(page);
* [`.injectFile(page, filePath)`](#puppeteer.injectFile) ⇒ <code>Promise</code>
* [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ <code>Promise</code>
* [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ <code>Promise</code>
- * [`.enqueueLinks(page, selector, requestQueue, [pseudoUrls])`](#puppeteer.enqueueLinks) ⇒ <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code>
+ * [`.enqueueLinks(options)`](#puppeteer.enqueueLinks) ⇒ <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code>
* [`.blockResources(page, resourceTypes)`](#puppeteer.blockResources) ⇒ <code>Promise</code>
* [`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ <code>Promise</code>
* [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ <code>function</code>
@@ -137,7 +137,7 @@ libraries included by the page that use the same variable.
</table>
<a name="puppeteer.enqueueLinks"></a>
-## `puppeteer.enqueueLinks(page, selector, requestQueue, [pseudoUrls])` ⇒ <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code>
+## `puppeteer.enqueueLinks(options)` ⇒ <code>Promise&lt;Array&lt;QueueOperationInfo&gt;&gt;</code>
Finds HTML elements matching a CSS selector, clicks the elements and if a redirect is triggered and destination URL matches
one of the provided [`PseudoUrl`](pseudourl)s, then the function enqueues that URL to a given request queue.
To create a Request object function uses `requestTemplate` from a matching [`PseudoUrl`](pseudourl).
@@ -149,27 +149,31 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse
<table>
<thead>
<tr>
-<th>Param</th><th>Type</th>
+<th>Param</th><th>Type</th><th>Default</th>
</tr>
</thead>
<tbody>
<tr>
-<td><code>page</code></td><td><code>Page</code></td>
+<td><code>options</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"></td></tr><tr>
+<td><code>options.page</code></td><td><code>Page</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Puppeteer <a href="https://pptr.dev/#?product=Puppeteer&show=api-class-page" target="_blank"><code>Page</code></a> object.</p>
</td></tr><tr>
-<td><code>selector</code></td><td><code>String</code></td>
+<td><code>options.requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>CSS selector matching elements to be clicked.</p>
+<td colspan="3"><p><a href="requestqueue"><code>RequestQueue</code></a> instance where URLs will be enqueued.</p>
</td></tr><tr>
-<td><code>requestQueue</code></td><td><code><a href="requestqueue">RequestQueue</a></code></td>
+<td><code>[options.selector]</code></td><td><code>String</code></td><td><code>&#x27;a&#x27;</code></td>
</tr>
<tr>
-<td colspan="3"><p><a href="requestqueue"><code>RequestQueue</code></a> instance where URLs will be enqueued.</p>
+<td colspan="3"><p>CSS selector matching elements to be clicked.</p>
</td></tr><tr>
-<td><code>[pseudoUrls]</code></td><td><code><a href="pseudourl">Array&lt;PseudoUrl&gt;</a></code> | <code>Array&lt;Object&gt;</code> | <code>Array&lt;String&gt;</code></td>
+<td><code>[options.pseudoUrls]</code></td><td><code><a href="pseudourl">Array&lt;PseudoUrl&gt;</a></code> | <code>Array&lt;Object&gt;</code> | <code>Array&lt;String&gt;</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>An array of <a href="pseudourl"><code>PseudoUrl</code></a>s matching the URLs to be enqueued,
@@ -178,7 +182,33 @@ To create a Request object function uses `requestTemplate` from a matching [`Pse
All remaining keys will be used as the <code>requestTemplate</code> argument of the <a href="pseudourl"><code>PseudoUrl</code></a> constructor.
If <code>pseudoUrls</code> is an empty array, null or undefined, then the function
enqueues all links found on the page.</p>
-</td></tr></tbody>
+</td></tr><tr>
+<td><code>[options.userData]</code></td><td><code>Object</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>Object that will be merged with the new Request&#39;s userData, overriding any values that
+ were set via templating from pseudoUrls. This is useful when you need to override generic
+ userData set by PseudoURL template in specific use cases.
+ <strong>Example:</strong></p>
+<pre><code> // pseudoUrl.userData
+ {
+ name: &#39;John&#39;,
+ surname: &#39;Doe&#39;,
+ }
+
+ // userData
+ {
+ name: &#39;Albert&#39;,
+ age: 31
+ }
+
+ // enqueued request.userData
+ {
+ name: &#39;Albert&#39;,
+ surname: &#39;Doe&#39;,
+ age: 31,
+ }
+</code></pre></td></tr></tbody>
</table>
<a name="puppeteer.blockResources"></a>
diff --git a/docs/api/social.md b/docs/api/social.md
index 0eba462a6..0eada551e 100644
--- a/docs/api/social.md
+++ b/docs/api/social.md
@@ -34,7 +34,7 @@ const emails = Apify.utils.social.emailsFromText('[email protected] bob@example.
<a name="social.LINKEDIN_REGEX"></a>
-## `social.LINKEDIN_REGEX` : <code>RegExp</code>
+## `social.LINKEDIN\_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single LinkedIn profile URL, without any additional
subdirectories or query parameters. The regular expression has the following form: `/^...$/i`.
@@ -45,55 +45,55 @@ TODO
<a name="social.LINKEDIN_REGEX_GLOBAL"></a>
-## `social.LINKEDIN_REGEX_GLOBAL` : <code>RegExp</code>
+## `social.LINKEDIN\_REGEX\_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple LinkedIn profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.INSTAGRAM_REGEX"></a>
-## `social.INSTAGRAM_REGEX` : <code>RegExp</code>
+## `social.INSTAGRAM\_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Instagram profile URL.
It has the following form: `/^...$/i`.
<a name="social.INSTAGRAM_REGEX_GLOBAL"></a>
-## `social.INSTAGRAM_REGEX_GLOBAL` : <code>RegExp</code>
+## `social.INSTAGRAM\_REGEX\_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.TWITTER_REGEX"></a>
-## `social.TWITTER_REGEX` : <code>RegExp</code>
+## `social.TWITTER\_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Instagram profile URL.
It has the following form: `/^...$/i`.
<a name="social.TWITTER_REGEX_GLOBAL"></a>
-## `social.TWITTER_REGEX_GLOBAL` : <code>RegExp</code>
+## `social.TWITTER\_REGEX\_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.FACEBOOK_REGEX"></a>
-## `social.FACEBOOK_REGEX` : <code>RegExp</code>
+## `social.FACEBOOK\_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single Facebook user profile URL.
It has the following form: `/^...$/i`.
<a name="social.FACEBOOK_REGEX_GLOBAL"></a>
-## `social.FACEBOOK_REGEX_GLOBAL` : <code>RegExp</code>
+## `social.FACEBOOK\_REGEX\_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple Instagram profile URLs in a text.
It has the following form: `/.../ig`.
<a name="social.EMAIL_REGEX"></a>
-## `social.EMAIL_REGEX` : <code>RegExp</code>
+## `social.EMAIL\_REGEX` : <code>RegExp</code>
Regular expression to exactly match a single email address.
It has the following form: `/^...$/i`.
<a name="social.EMAIL_REGEX_GLOBAL"></a>
-## `social.EMAIL_REGEX_GLOBAL` : <code>RegExp</code>
+## `social.EMAIL\_REGEX\_GLOBAL` : <code>RegExp</code>
Regular expression to find multiple email addresses in a text.
It has the following form: `/.../ig`.
diff --git a/docs/api/utils.md b/docs/api/utils.md
index 2dd1421fb..f1abe4c24 100644
--- a/docs/api/utils.md
+++ b/docs/api/utils.md
@@ -28,13 +28,13 @@ await Apify.utils.sleep(1500);
<a name="utils.URL_NO_COMMAS_REGEX"></a>
-## `utils.URL_NO_COMMAS_REGEX`
+## `utils.URL\_NO\_COMMAS\_REGEX`
Default regular expression to match URLs in a string that may be plain text, JSON, CSV or other. It supports common URL characters
and does not support URLs containing commas or spaces. The URLs also may contain Unicode letters (not symbols).
<a name="utils.URL_WITH_COMMAS_REGEX"></a>
-## `utils.URL_WITH_COMMAS_REGEX`
+## `utils.URL\_WITH\_COMMAS\_REGEX`
Regular expression that, in addition to the default regular expression `URL_NO_COMMAS_REGEX`, supports matching commas in URL path and query.
Note, however, that this may prevent parsing URLs from comma delimited lists, or the URLs may become malformed.
diff --git a/package.json b/package.json
index 824301411..9e526f45c 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.9.14",
+ "version": "0.9.15",
"description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=8.0.0"
diff --git a/website/package.json b/website/package.json
index 81c12c17d..919da3609 100644
--- a/website/package.json
+++ b/website/package.json
@@ -10,7 +10,7 @@
"generate": "node ./tools/build_docs.js"
},
"devDependencies": {
- "docusaurus": "^1.5.0",
+ "docusaurus": "^1.5.1",
"jsdoc-to-markdown": "^4.0.1"
}
}