Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Adding example

Marek Trunkat1a60268b1823
diff --git a/README.md b/README.md
index 872ae837a..ce00423c2 100644
--- a/README.md
+++ b/README.md
@@ -1,15 +1,18 @@
# apify: Apify Actor runtime for JavaScript
+<!-- Mirror this part to src/index.js -->
[![npm version](https://badge.fury.io/js/apify.svg)](http://badge.fury.io/js/apify)
[![Build Status](https://travis-ci.org/apifytech/apify-js.svg)](https://travis-ci.org/apifytech/apify-js)
-The apify NPM package simplifies development of acts in Apify Actor - a serverless computing platform that enables execution of arbitrary pieces of code in the cloud. The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts without it.
+The apify NPM package simplifies development of acts in <a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a>
+- a serverless computing platform that enables execution of arbitrary pieces of code in the cloud. The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts without it.
Complete documentation of this package is available at https://www.apify.com/docs/sdk/apify-runtime-js/latest
For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor
## Common use-cases
+<!-- Mirror this part to src/index.js -->
Main goal of this package is to help with implementation of web scraping and automation projects. Some of the
most common use-cases are:
@@ -33,6 +36,7 @@ most common use-cases are:
</ul>
## Puppeteer
+<!-- Mirror this part to src/index.js -->
For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser)
we have few helper classes and functions:
@@ -49,4 +53,8 @@ we have few helper classes and functions:
<a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#PuppeteerCrawler">PuppeteerCrawler</a> helps to crawl a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#RequestList">RequestList</a>
in a autoscaled pool.
</li>
-</ul>
\ No newline at end of file
+</ul>
+
+## Examples
+
+Directory `/examples` of this repo contains examples of different usages of this package.
\ No newline at end of file
diff --git a/examples/example_autoscaled_pool.js b/examples/example_autoscaled_pool.js
new file mode 100644
index 000000000..c8d2b27b7
--- /dev/null
+++ b/examples/example_autoscaled_pool.js
@@ -0,0 +1,28 @@
+const Apify = require('../build');
+
+Apify.main(async () => {
+ let counter = 0;
+
+ const workerFunction = () => {
+ const current = counter++;
+
+ if (current >= 1000) return null;
+
+ const randomWaitMillis = Math.round(Math.random() * 100);
+
+ return new Promise((resolve) => {
+ setTimeout(() => {
+ console.log(`${counter} is done!`);
+ resolve();
+ }, randomWaitMillis);
+ });
+ };
+
+ const pool = new Apify.AutoscaledPool({
+ minConcurrency: 3,
+ maxConcurrency: 50,
+ workerFunction,
+ });
+
+ await pool.run();
+});
diff --git a/src/autoscaled_pool.js b/src/autoscaled_pool.js
index 25af23795..c657b5b30 100644
--- a/src/autoscaled_pool.js
+++ b/src/autoscaled_pool.js
@@ -36,7 +36,7 @@ const humanReadable = bytes => `${Math.round(bytes / 1024 / 1024)} MB`;
* ```javascript
* const pool = new Apify.AutoscaledPool({
* maxConcurrency: 50,
- * workerFunction: async () => {
+ * workerFunction: () => {
* // ... do some intensive asynchronous operations here and return a promise ...
* },
* });
diff --git a/src/index.js b/src/index.js
index 5c6c32858..ed6d60cd6 100644
--- a/src/index.js
+++ b/src/index.js
@@ -53,6 +53,7 @@ if (!isProduction() || process.env[ENV_VARS.LOG_LEVEL] === 'DEBUG') log.isDebugM
* ```
*
* <h2>Common use-cases</h2>
+ * <!-- Mirror this part to README.md -->
*
* Main goal of this package is to help with implementation of web scraping and automation projects. Some of the
* most common use-cases are:
@@ -76,6 +77,7 @@ if (!isProduction() || process.env[ENV_VARS.LOG_LEVEL] === 'DEBUG') log.isDebugM
* </ul>
*
* <h2>Puppeteer</h2>
+ * <!-- Mirror this part to README.md -->
*
* For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser)
* we have few helper classes and functions: