Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Documented BasicCrawler.autoscaledPool property, and added CheerioCra… (#578)

Jan Čurnd7311d8acb81
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f0ad1fe41..a044a31a1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,8 @@ NEXT
====================
- Improved `Apify.launchPuppeteer()` code to prevent triggering bugs in Puppeteer by passing
more than required options to `puppeteer.launch()`.
+- Documented `BasicCrawler.autoscaledPool` property,
+ and added `CheerioCrawler.autoscaledPool` and `PuppeteerCrawler.autoscaledPool` properties
0.19.1 / 2020-01-30
====================
diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.md
index 1ad927bcc..d7b339ce0 100644
--- a/docs/api/BasicCrawler.md
+++ b/docs/api/BasicCrawler.md
@@ -54,6 +54,27 @@ const crawler = new Apify.BasicCrawler({
await crawler.run();
```
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.
+ Note that this property is only initialized after calling the <a href="#BasicCrawler+run"><code>run</code></a> function.
+ You can use it to change the concurrency settings on the fly,
+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>
+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>
+</td></tr></tbody>
+</table>
+
- [BasicCrawler](basiccrawler)
- [`new BasicCrawler(options)`](#new_BasicCrawler_new)
- [`.run()`](#BasicCrawler+run) ⇒ `Promise<void>`
diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.md
index 594ae2f1b..b842a3a2e 100644
--- a/docs/api/CheerioCrawler.md
+++ b/docs/api/CheerioCrawler.md
@@ -76,6 +76,27 @@ const crawler = new Apify.CheerioCrawler({
await crawler.run();
```
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.
+ Note that this property is only initialized after calling the <a href="#CheerioCrawler+run"><code>run</code></a> function.
+ You can use it to change the concurrency settings on the fly,
+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>
+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>
+</td></tr></tbody>
+</table>
+
- [CheerioCrawler](cheeriocrawler)
- [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)
- [`.run()`](#CheerioCrawler+run) ⇒ `Promise`
diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.md
index 184df6cb5..e3aca8226 100644
--- a/docs/api/PuppeteerCrawler.md
+++ b/docs/api/PuppeteerCrawler.md
@@ -61,6 +61,27 @@ const crawler = new Apify.PuppeteerCrawler({
await crawler.run();
```
+**Properties**
+
+<table>
+<thead>
+<tr>
+<th>Param</th><th>Type</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>
+</tr>
+<tr>
+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.
+ Note that this property is only initialized after calling the <a href="#PuppeteerCrawler+run"><code>run</code></a> function.
+ You can use it to change the concurrency settings on the fly,
+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>
+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>
+</td></tr></tbody>
+</table>
+
- [PuppeteerCrawler](puppeteercrawler)
- [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)
- [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise<void>`
diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.js
index 12d9085fb..3d2dfda28 100644
--- a/src/crawlers/basic_crawler.js
+++ b/src/crawlers/basic_crawler.js
@@ -173,6 +173,13 @@ const DEFAULT_OPTIONS = {
*
* await crawler.run();
* ```
+ *
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
class BasicCrawler {
/**
diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.js
index c7cb78b16..087e1abd1 100644
--- a/src/crawlers/cheerio_crawler.js
+++ b/src/crawlers/cheerio_crawler.js
@@ -311,6 +311,12 @@ const DEFAULT_OPTIONS = {
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
class CheerioCrawler {
/**
@@ -420,6 +426,8 @@ class CheerioCrawler {
if (this.isRunningPromise) return this.isRunningPromise;
this.isRunningPromise = this.basicCrawler.run();
+ this.autoscaledPool = this.basicCrawler.autoscaledPool;
+
await this.isRunningPromise;
}
diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.js
index 97a80c8d8..61034c34a 100644
--- a/src/crawlers/puppeteer_crawler.js
+++ b/src/crawlers/puppeteer_crawler.js
@@ -197,6 +197,12 @@ import { SessionPoolOptions } from '../session_pool/session_pool';
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
class PuppeteerCrawler {
/**
@@ -296,6 +302,8 @@ class PuppeteerCrawler {
this.puppeteerPool = new PuppeteerPool(this.puppeteerPoolOptions);
try {
this.isRunningPromise = this.basicCrawler.run();
+ this.autoscaledPool = this.basicCrawler.autoscaledPool;
+
await this.isRunningPromise;
} finally {
if (this.useSessionPool) {
diff --git a/test/crawlers/basic_crawler.test.js b/test/crawlers/basic_crawler.test.js
index b3a244afe..b95122207 100644
--- a/test/crawlers/basic_crawler.test.js
+++ b/test/crawlers/basic_crawler.test.js
@@ -51,6 +51,7 @@ describe('BasicCrawler', () => {
await requestList.initialize();
await basicCrawler.run();
+ expect(basicCrawler.autoscaledPool.minConcurrency).toBe(25);
expect(processed).toEqual(sources);
expect(await requestList.isFinished()).toBe(true);
expect(await requestList.isEmpty()).toBe(true);
diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.js
index 9368683c5..ba3a92686 100644
--- a/test/crawlers/cheerio_crawler.test.js
+++ b/test/crawlers/cheerio_crawler.test.js
@@ -195,6 +195,7 @@ describe('CheerioCrawler', () => {
await requestList.initialize();
await cheerioCrawler.run();
+ expect(cheerioCrawler.autoscaledPool.minConcurrency).toBe(2);
expect(processed).toHaveLength(6);
expect(failed).toHaveLength(0);
diff --git a/test/crawlers/puppeteer_crawler.test.js b/test/crawlers/puppeteer_crawler.test.js
index b3ca6297a..7627a795a 100644
--- a/test/crawlers/puppeteer_crawler.test.js
+++ b/test/crawlers/puppeteer_crawler.test.js
@@ -56,6 +56,7 @@ describe('PuppeteerCrawler', () => {
await requestList.initialize();
await puppeteerCrawler.run();
+ expect(puppeteerCrawler.autoscaledPool.minConcurrency).toBe(1);
expect(processed).toHaveLength(6);
expect(failed).toHaveLength(0);
diff --git a/types/crawlers/basic_crawler.d.ts b/types/crawlers/basic_crawler.d.ts
index d27191baa..2fd2a42c9 100644
--- a/types/crawlers/basic_crawler.d.ts
+++ b/types/crawlers/basic_crawler.d.ts
@@ -242,6 +242,13 @@ export type HandleFailedRequestInput = {
*
* await crawler.run();
* ```
+ *
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
declare class BasicCrawler {
/**
diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.ts
index 6ee5c4de3..591814408 100644
--- a/types/crawlers/cheerio_crawler.d.ts
+++ b/types/crawlers/cheerio_crawler.d.ts
@@ -506,6 +506,12 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
declare class CheerioCrawler {
/**
@@ -534,6 +540,7 @@ declare class CheerioCrawler {
* @return {Promise}
*/
run(): Promise<any>;
+ autoscaledPool: AutoscaledPool;
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
*
diff --git a/types/crawlers/puppeteer_crawler.d.ts b/types/crawlers/puppeteer_crawler.d.ts
index 7881a3250..2205eb4c6 100644
--- a/types/crawlers/puppeteer_crawler.d.ts
+++ b/types/crawlers/puppeteer_crawler.d.ts
@@ -369,6 +369,12 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse
*
* await crawler.run();
* ```
+ * @property {AutoscaledPool} autoscaledPool
+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.
+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.
+ * You can use it to change the concurrency settings on the fly,
+ * to pause the crawler by calling {@link AutoscaledPool#pause}
+ * or to abort it by calling {@link AutoscaledPool#abort}.
*/
declare class PuppeteerCrawler {
/**
@@ -456,6 +462,7 @@ declare class PuppeteerCrawler {
run(): Promise<void>;
sessionPool: import("../session_pool/session_pool").SessionPool;
isRunningPromise: Promise<void>;
+ autoscaledPool: AutoscaledPool;
/**
* Wrapper around handlePageFunction that opens and closes pages etc.
*