Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Bump version, update Changelog, build docs

Ondra Urbanf8d63f87a9db
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e00f9f3d2..501959acd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,9 +1,4 @@
-next
-===================
-- Added back the `recycleDiskCache` option to `PuppeteerPool` now that it is supported
- even in headless mode ([read more](https://bugs.chromium.org/p/chromium/issues/detail?id=882431))
-
-xxx
+0.13.0 / 2019-03-14
===================
- **BREAKING CHANGE:** Added `puppeteerOperationTimeoutSecs` option to `PuppeteerPool`.
It defaults to 15 seconds and all Puppeteer operations such as `browser.newPage()`
@@ -16,6 +11,8 @@ xxx
their `handlePageTimeoutSecs`. This is a precaution that should keep requests from hanging forever.
- Added `options.prepareRequestFunction()` to `CheerioCrawler` constructor to enable modification
of `Request` before the HTTP request is made to the target URL.
+- Added back the `recycleDiskCache` option to `PuppeteerPool` now that it is supported
+ even in headless mode ([read more](https://bugs.chromium.org/p/chromium/issues/detail?id=882431))
0.12.4 / 2019-03-05
===================
diff --git a/docs/api/Apify.md b/docs/api/Apify.md
index b9d1b3bbd..796fc366a 100644
--- a/docs/api/Apify.md
+++ b/docs/api/Apify.md
@@ -316,7 +316,7 @@ The following table shows all currently emitted events:
in which case the <code>isMigrating</code> flag is set to <code>true</code>. Otherwise the flag is <code>false</code>.
<br><br>
Note that the <code>persistState</code> event is provided merely for user convenience,
- you can achieve the same effect using `setInterval()` and listening for the `migrating` event.
+ you can achieve the same effect using <code>setInterval()</code> and listening for the <code>migrating</code> event.
</td>
</tr>
</tbody>
@@ -534,7 +534,7 @@ The `launchPuppeteer()` function alters the following Puppeteer options:
<li>
Sets <code>defaultViewport</code> Puppeteer option (if not already set)
to a more reasonable default for screenshots and debugging.
- You can set `options.defaultViewport` to `null` if you prefer to let Puppeteer
+ You can set <code>options.defaultViewport</code> to <code>null</code> if you prefer to let Puppeteer
choose the default viewport size.
</li>
</ul>
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index bc666712f..c5e17ca5d 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -109,6 +109,11 @@ await crawler.run();
<td colspan="3"><p>Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
Either <code>requestList</code> or <code>requestQueue</code> option must be provided (or both).</p>
</td></tr><tr>
+<td><code>[options.handleRequestTimeoutSecs]</code></td><td><code>number</code></td><td><code>60</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>Timeout in which the function passed as <code>options.handleRequestFunction</code> needs to finish, in seconds.</p>
+</td></tr><tr>
<td><code>[options.handleFailedRequestFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index d3bd9e5ba..c1e224557 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -137,6 +137,18 @@ await crawler.run();
- `strictSSL`: use `options.ignoreSslErrors`
- `proxy`: use `options.useApifyProxy` or `options.proxyUrls`
</code></pre></td></tr><tr>
+<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
+</tr>
+<tr>
+<td colspan="3"><p>A function that executes before the HTTP request is made to the target resource.
+ This function is suitable for setting dynamic properties such as cookies to the <a href="request"><code>Request</code></a>.</p>
+<p> The function receives the following object as an argument:</p>
+<pre><code>{
+ request: Request
+}
+</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
+<p> The function should always return <a href="request"><code>Request</code></a>.</p>
+</td></tr><tr>
<td><code>[options.handlePageTimeoutSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
</tr>
<tr>
@@ -198,18 +210,6 @@ await crawler.run();
<p> See <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L13">source code</a>
for the default implementation of this function.</p>
</td></tr><tr>
-<td><code>[options.prepareRequestFunction]</code></td><td><code>function</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>A function that executes before calling the request.
- This function is suitable for setting dynamic properties such as cookies to <a href="request"><code>Request</code></a>.</p>
-<p> The function receives the following object as an argument:</p>
-<pre><code>{
- request: Request
-}
-</code></pre><p> where the <a href="request"><code>Request</code></a> instance corresponds to the initialized request.</p>
-<p> The function Should always return <a href="request"><code>Request</code></a>.</p>
-</td></tr><tr>
<td><code>[options.maxRequestRetries]</code></td><td><code>Number</code></td><td><code>3</code></td>
</tr>
<tr>
diff --git a/docs/api/Dataset.md b/docs/api/Dataset.md
index 00fea4df8..7461b3c8c 100644
--- a/docs/api/Dataset.md
+++ b/docs/api/Dataset.md
@@ -204,14 +204,14 @@ API endpoint.
**Example:**
```
{
- "id": "WkzbQMuFYuamGv3YF",
- "name": "my-dataset",
- "userId": "wRsJZtadYvn4mBZmm",
- "createdAt": new Date("2015-12-12T07:34:14.202Z"),
- "modifiedAt": new Date("2015-12-13T08:36:13.202Z"),
- "accessedAt": new Date("2015-12-14T08:36:13.202Z"),
- "itemCount": 14,
- "cleanItemCount": 10
+ id: "WkzbQMuFYuamGv3YF",
+ name: "my-dataset",
+ userId: "wRsJZtadYvn4mBZmm",
+ createdAt: new Date("2015-12-12T07:34:14.202Z"),
+ modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
+ accessedAt: new Date("2015-12-14T08:36:13.202Z"),
+ itemCount: 14,
+ cleanItemCount: 10
}
```
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 5b67becd2..1e5439f7e 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -166,48 +166,17 @@ await crawler.run();
Always set this value in order to prevent infinite loops in misconfigured crawlers.
Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.</p>
</td></tr><tr>
-<td><code>[options.maxOpenPagesPerInstance]</code></td><td><code>Number</code></td><td><code>50</code></td>
+<td><code>[options.puppeteerPoolOptions]</code></td><td><code>Object</code></td><td></td>
</tr>
<tr>
-<td colspan="3"><p>Maximum number of opened tabs per browser. If this limit is reached then a new
- browser instance is started. See <code>maxOpenPagesPerInstance</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.retireInstanceAfterRequestCount]</code></td><td><code>Number</code></td><td><code>100</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Maximum number of requests that can be processed by a single browser instance.
- After the limit is reached the browser will be retired and new requests will
- be handled by a new browser instance.
- See <code>retireInstanceAfterRequestCount</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.instanceKillerIntervalMillis]</code></td><td><code>Number</code></td><td><code>60000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.
- See <code>instanceKillerIntervalMillis</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.killInstanceAfterMillis]</code></td><td><code>Number</code></td><td><code>300000</code></td>
-</tr>
-<tr>
-<td colspan="3"><p>If Puppeteer instance reaches the <code>options.retireInstanceAfterRequestCount</code> limit then
- it is considered retired and no more tabs will be opened. After the last tab is closed
- the whole browser is closed too. This parameter defines a time limit for inactivity
- after which the browser is closed even if there are pending tabs. See
- <code>killInstanceAfterMillis</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.</p>
-</td></tr><tr>
-<td><code>[options.proxyUrls]</code></td><td><code>Array&lt;String&gt;</code></td><td></td>
-</tr>
-<tr>
-<td colspan="3"><p>An array of custom proxy URLs to be used by the <a href="puppeteerpool"><code>PuppeteerPool</code></a> instance.
- The provided custom proxies&#39; order will be randomized and the resulting list rotated.
- Custom proxies are not compatible with Apify Proxy and an attempt to use both
- configuration options will cause an error to be thrown on startup.</p>
+<td colspan="3"><p>Custom options passed to the underlying <a href="puppeteerpool"><code>PuppeteerPool</code></a> constructor.
+ You can tweak those to fine-tune browser management.</p>
</td></tr><tr>
<td><code>[options.launchPuppeteerFunction]</code></td><td><code>function</code></td><td></td>
</tr>
<tr>
<td colspan="3"><p>Overrides the default function to launch a new Puppeteer instance.
- See <code>launchPuppeteerFunction</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a>.
+ Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option.
See source code on
<a href="https://github.com/apifytech/apify-js/blob/master/src/puppeteer_pool.js#L28" target="_blank">GitHub</a>
for default behavior.</p>
@@ -216,7 +185,7 @@ await crawler.run();
</tr>
<tr>
<td colspan="3"><p>Options used by <a href="apify#module_Apify.launchPuppeteer"><code>Apify.launchPuppeteer()</code></a> to start new Puppeteer instances.
- See <code>launchPuppeteerOptions</code> parameter of <a href="puppeteerpool"><code>PuppeteerPool</code></a> and <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
+ Shortcut to the corresponding <a href="puppeteerpool"><code>PuppeteerPool</code></a> option. See <a href="../typedefs/launchpuppeteeroptions"><code>LaunchPuppeteerOptions</code></a>.</p>
</td></tr><tr>
<td><code>[options.autoscaledPoolOptions]</code></td><td><code>Object</code></td><td></td>
</tr>
diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.md
index 884be3bb7..ea43a7a3c 100644
--- a/docs/api/PuppeteerPool.md
+++ b/docs/api/PuppeteerPool.md
@@ -76,12 +76,19 @@ await puppeteerPool.destroy();
After the limit is reached, the browser is retired and new requests are
handled by a new browser instance.</p>
</td></tr><tr>
-<td><code>[options.instanceKillerIntervalMillis]</code></td><td><code>Number</code></td><td><code>60000</code></td>
+<td><code>[options.puppeteerOperationTimeoutSecs]</code></td><td><code>Number</code></td><td><code>15</code></td>
+</tr>
+<tr>
+<td colspan="3"><p>All browser management operations such as launching a new browser, opening a new page
+ or closing a page will timeout after the set number of seconds and the connected
+ browser will be retired.</p>
+</td></tr><tr>
+<td><code>[options.instanceKillerIntervalSecs]</code></td><td><code>Number</code></td><td><code>60</code></td>
</tr>
<tr>
<td colspan="3"><p>Indicates how often are the open Puppeteer instances checked whether they can be closed.</p>
</td></tr><tr>
-<td><code>[options.killInstanceAfterMillis]</code></td><td><code>Number</code></td><td><code>300000</code></td>
+<td><code>[options.killInstanceAfterSecs]</code></td><td><code>Number</code></td><td><code>300</code></td>
</tr>
<tr>
<td colspan="3"><p>When Puppeteer instance reaches the <code>options.retireInstanceAfterRequestCount</code> limit then
diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.md
index 09ca151d4..d83e1b8d1 100644
--- a/docs/api/RequestQueue.md
+++ b/docs/api/RequestQueue.md
@@ -227,12 +227,12 @@ API endpoint.
**Example:**
```
{
- "id": "WkzbQMuFYuamGv3YF",
- "name": "my-queue",
- "userId": "wRsJZtadYvn4mBZmm",
- "createdAt": new Date("2015-12-12T07:34:14.202Z"),
- "modifiedAt": new Date("2015-12-13T08:36:13.202Z"),
- "accessedAt": new Date("2015-12-14T08:36:13.202Z"),
+ id: "WkzbQMuFYuamGv3YF",
+ name: "my-queue",
+ userId: "wRsJZtadYvn4mBZmm",
+ createdAt: new Date("2015-12-12T07:34:14.202Z"),
+ modifiedAt: new Date("2015-12-13T08:36:13.202Z"),
+ accessedAt: new Date("2015-12-14T08:36:13.202Z"),
totalRequestCount: 0,
handledRequestCount: 0,
pendingRequestCount: 0,
diff --git a/package.json b/package.json
index 7c095a5e2..2bd4f45ea 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "apify",
- "version": "0.12.4",
+ "version": "0.13.0",
"description": "The scalable web crawling and scraping library for JavaScript. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.",
"engines": {
"node": ">=8.14.0"