Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Documented BasicCrawler.autoscaledPool property, and added CheerioCra… (#578)
Jan Čurnd7311d8acb81
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f0ad1fe41..a044a31a1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,8 @@ NEXT====================- Improved `Apify.launchPuppeteer()` code to prevent triggering bugs in Puppeteer by passingmore than required options to `puppeteer.launch()`.+- Documented `BasicCrawler.autoscaledPool` property,+ and added `CheerioCrawler.autoscaledPool` and `PuppeteerCrawler.autoscaledPool` properties0.19.1 / 2020-01-30====================diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex 1ad927bcc..d7b339ce0 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -54,6 +54,27 @@ const crawler = new Apify.BasicCrawler({await crawler.run();```+**Properties**++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>+</tr>+<tr>+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.+ Note that this property is only initialized after calling the <a href="#BasicCrawler+run"><code>run</code></a> function.+ You can use it to change the concurrency settings on the fly,+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>+</td></tr></tbody>+</table>+- [BasicCrawler](basiccrawler)- [`new BasicCrawler(options)`](#new_BasicCrawler_new)- [`.run()`](#BasicCrawler+run) ⇒ `Promise<void>`diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 594ae2f1b..b842a3a2e 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -76,6 +76,27 @@ const crawler = new Apify.CheerioCrawler({await crawler.run();```+**Properties**++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>+</tr>+<tr>+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.+ Note that this property is only initialized after calling the <a href="#CheerioCrawler+run"><code>run</code></a> function.+ You can use it to change the concurrency settings on the fly,+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>+</td></tr></tbody>+</table>+- [CheerioCrawler](cheeriocrawler)- [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)- [`.run()`](#CheerioCrawler+run) ⇒ `Promise`diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 184df6cb5..e3aca8226 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -61,6 +61,27 @@ const crawler = new Apify.PuppeteerCrawler({await crawler.run();```+**Properties**++<table>+<thead>+<tr>+<th>Param</th><th>Type</th>+</tr>+</thead>+<tbody>+<tr>+<td><code>autoscaledPool</code></td><td><code><a href="autoscaledpool">AutoscaledPool</a></code></td>+</tr>+<tr>+<td colspan="3"><p>A reference to the underlying <a href="autoscaledpool"><code>AutoscaledPool</code></a> class that manages the concurrency of the crawler.+ Note that this property is only initialized after calling the <a href="#PuppeteerCrawler+run"><code>run</code></a> function.+ You can use it to change the concurrency settings on the fly,+ to pause the crawler by calling <a href="#AutoscaledPool+pause"><code>pause</code></a>+ or to abort it by calling <a href="#AutoscaledPool+abort"><code>abort</code></a>.</p>+</td></tr></tbody>+</table>+- [PuppeteerCrawler](puppeteercrawler)- [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)- [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise<void>`diff --git a/src/crawlers/basic_crawler.js b/src/crawlers/basic_crawler.jsindex 12d9085fb..3d2dfda28 100644--- a/src/crawlers/basic_crawler.js+++ b/src/crawlers/basic_crawler.js@@ -173,6 +173,13 @@ const DEFAULT_OPTIONS = {** await crawler.run();* ```+ *+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/class BasicCrawler {/**diff --git a/src/crawlers/cheerio_crawler.js b/src/crawlers/cheerio_crawler.jsindex c7cb78b16..087e1abd1 100644--- a/src/crawlers/cheerio_crawler.js+++ b/src/crawlers/cheerio_crawler.js@@ -311,6 +311,12 @@ const DEFAULT_OPTIONS = {** await crawler.run();* ```+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/class CheerioCrawler {/**@@ -420,6 +426,8 @@ class CheerioCrawler {if (this.isRunningPromise) return this.isRunningPromise;this.isRunningPromise = this.basicCrawler.run();+ this.autoscaledPool = this.basicCrawler.autoscaledPool;+await this.isRunningPromise;}diff --git a/src/crawlers/puppeteer_crawler.js b/src/crawlers/puppeteer_crawler.jsindex 97a80c8d8..61034c34a 100644--- a/src/crawlers/puppeteer_crawler.js+++ b/src/crawlers/puppeteer_crawler.js@@ -197,6 +197,12 @@ import { SessionPoolOptions } from '../session_pool/session_pool';** await crawler.run();* ```+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/class PuppeteerCrawler {/**@@ -296,6 +302,8 @@ class PuppeteerCrawler {this.puppeteerPool = new PuppeteerPool(this.puppeteerPoolOptions);try {this.isRunningPromise = this.basicCrawler.run();+ this.autoscaledPool = this.basicCrawler.autoscaledPool;+await this.isRunningPromise;} finally {if (this.useSessionPool) {diff --git a/test/crawlers/basic_crawler.test.js b/test/crawlers/basic_crawler.test.jsindex b3a244afe..b95122207 100644--- a/test/crawlers/basic_crawler.test.js+++ b/test/crawlers/basic_crawler.test.js@@ -51,6 +51,7 @@ describe('BasicCrawler', () => {await requestList.initialize();await basicCrawler.run();+ expect(basicCrawler.autoscaledPool.minConcurrency).toBe(25);expect(processed).toEqual(sources);expect(await requestList.isFinished()).toBe(true);expect(await requestList.isEmpty()).toBe(true);diff --git a/test/crawlers/cheerio_crawler.test.js b/test/crawlers/cheerio_crawler.test.jsindex 9368683c5..ba3a92686 100644--- a/test/crawlers/cheerio_crawler.test.js+++ b/test/crawlers/cheerio_crawler.test.js@@ -195,6 +195,7 @@ describe('CheerioCrawler', () => {await requestList.initialize();await cheerioCrawler.run();+ expect(cheerioCrawler.autoscaledPool.minConcurrency).toBe(2);expect(processed).toHaveLength(6);expect(failed).toHaveLength(0);diff --git a/test/crawlers/puppeteer_crawler.test.js b/test/crawlers/puppeteer_crawler.test.jsindex b3ca6297a..7627a795a 100644--- a/test/crawlers/puppeteer_crawler.test.js+++ b/test/crawlers/puppeteer_crawler.test.js@@ -56,6 +56,7 @@ describe('PuppeteerCrawler', () => {await requestList.initialize();await puppeteerCrawler.run();+ expect(puppeteerCrawler.autoscaledPool.minConcurrency).toBe(1);expect(processed).toHaveLength(6);expect(failed).toHaveLength(0);diff --git a/types/crawlers/basic_crawler.d.ts b/types/crawlers/basic_crawler.d.tsindex d27191baa..2fd2a42c9 100644--- a/types/crawlers/basic_crawler.d.ts+++ b/types/crawlers/basic_crawler.d.ts@@ -242,6 +242,13 @@ export type HandleFailedRequestInput = {** await crawler.run();* ```+ *+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link BasicCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class BasicCrawler {/**diff --git a/types/crawlers/cheerio_crawler.d.ts b/types/crawlers/cheerio_crawler.d.tsindex 6ee5c4de3..591814408 100644--- a/types/crawlers/cheerio_crawler.d.ts+++ b/types/crawlers/cheerio_crawler.d.ts@@ -506,6 +506,12 @@ export type CheerioHandlePage = (inputs: CheerioHandlePageInputs) => Promise<voi** await crawler.run();* ```+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link CheerioCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class CheerioCrawler {/**@@ -534,6 +540,7 @@ declare class CheerioCrawler {* @return {Promise}*/run(): Promise<any>;+ autoscaledPool: AutoscaledPool;/*** Wrapper around handlePageFunction that opens and closes pages etc.*diff --git a/types/crawlers/puppeteer_crawler.d.ts b/types/crawlers/puppeteer_crawler.d.tsindex 7881a3250..2205eb4c6 100644--- a/types/crawlers/puppeteer_crawler.d.ts+++ b/types/crawlers/puppeteer_crawler.d.ts@@ -369,6 +369,12 @@ export type LaunchPuppeteer = (inputs: LaunchPuppeteerOptions) => Promise<Browse** await crawler.run();* ```+ * @property {AutoscaledPool} autoscaledPool+ * A reference to the underlying {@link AutoscaledPool} class that manages the concurrency of the crawler.+ * Note that this property is only initialized after calling the {@link PuppeteerCrawler#run} function.+ * You can use it to change the concurrency settings on the fly,+ * to pause the crawler by calling {@link AutoscaledPool#pause}+ * or to abort it by calling {@link AutoscaledPool#abort}.*/declare class PuppeteerCrawler {/**@@ -456,6 +462,7 @@ declare class PuppeteerCrawler {run(): Promise<void>;sessionPool: import("../session_pool/session_pool").SessionPool;isRunningPromise: Promise<void>;+ autoscaledPool: AutoscaledPool;/*** Wrapper around handlePageFunction that opens and closes pages etc.*