Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updating readme

Marek Trunkatd4913ff380bc
diff --git a/README.md b/README.md
index 488caab86..872ae837a 100644
--- a/README.md
+++ b/README.md
@@ -7,4 +7,46 @@ The apify NPM package simplifies development of acts in Apify Actor - a serverle
Complete documentation of this package is available at https://www.apify.com/docs/sdk/apify-runtime-js/latest
-For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor
\ No newline at end of file
+For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor
+
+## Common use-cases
+
+Main goal of this package is to help with implementation of web scraping and automation projects. Some of the
+most common use-cases are:
+
+<ul>
+ <li>
+ If you need to process high volume of <strong>asynchronous tasks in parallel</strong> then take a
+ look at <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#AutoscaledPool">AutoscaledPool</a>. This class executes defined tasks in a pool
+ which size is scaled based on available memory and CPU.
+ </li>
+ <li>
+ If you want to <strong>crawl</strong> a list of urls using for example <a href="https://www.npmjs.com/package/request" target="_blank">
+ Request</a> package then import those url as a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#RequestList">RequestList</a> and then use
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#BasicCrawler">BasicCrawler</a> to process them in a pool.
+ </li>
+ <li>
+ If you want to crawl a list of urls but you need a real <strong>browser</strong>. Then use
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#PuppeteerCrawler">PuppeteerCrawler</a> which helps you to process a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#RequestList">RequestList</a>
+ using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser).
+ </li>
+</ul>
+
+## Puppeteer
+
+For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser)
+we have few helper classes and functions:
+
+<ul>
+ <li>
+ `Apify.launchPuppeteer()` function starts new instance of Puppeteer browser and returns its browser object.
+ </li>
+ <li>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#PuppeteerPool">PuppeteerPool</a> helps to mantain a pool of Puppeteer instances. This is usefull
+ when you need to restart browser after certain number of requests to rotate proxy servers.
+ </li>
+ <li>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#PuppeteerCrawler">PuppeteerCrawler</a> helps to crawl a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/beta#RequestList">RequestList</a>
+ in a autoscaled pool.
+ </li>
+</ul>
\ No newline at end of file
diff --git a/src/actor.js b/src/actor.js
index 98aa34825..51483ae79 100644
--- a/src/actor.js
+++ b/src/actor.js
@@ -363,7 +363,8 @@ export const call = (actId, input, opts = {}) => {
/**
* Returns a URL of Apify Proxy that can be used from Actor acts, web browsers or any other HTTP
- * proxy-enabled applications.
+ * proxy-enabled applications. For more info about Apify Proxy see
+ * <a href="https://www.apify.com/docs/proxy" target="_blank">https://www.apify.com/docs/proxy</a>.
*
* @param {Object} opts
* @param {String} opts.password User proxy password. By default, it is taken from the `APIFY_PROXY_PASSWORD` environment variable.
diff --git a/src/index.js b/src/index.js
index 9c6ce4657..5c6c32858 100644
--- a/src/index.js
+++ b/src/index.js
@@ -52,7 +52,7 @@ if (!isProduction() || process.env[ENV_VARS.LOG_LEVEL] === 'DEBUG') log.isDebugM
* });
* ```
*
- * <h2>Main use-cases</h2>
+ * <h2>Common use-cases</h2>
*
* Main goal of this package is to help with implementation of web scraping and automation projects. Some of the
* most common use-cases are:
diff --git a/src/request.js b/src/request.js
index cb2e12b52..2e14aec97 100644
--- a/src/request.js
+++ b/src/request.js
@@ -25,15 +25,16 @@ export const computeUniqueKey = (url, keepUrlFragment) => normalizeUrl(url, keep
* const foo = request.userData.foo;
* ```
*
- * @param {String} url
- * @param {String} [uniqueKey] Unique key identifying request. In not provided then it is computed as normalized URL.
- * @param {String} [method='GET']
- * @param {String|Buffer} [payload] Request payload. If method='GET' then the payload is not allowed.
- * @param {Number} [retryCount=0] How many times the url was retried in a case of exception.
- * @param {String} [errorMessages] Array of error messages from request processing.
- * @param {String} [headers={}] HTTP headers.
- * @param {Object} [userData={}] Custom data that user can assign to request.
- * @param {Boolean} [keepUrlFragment=false] If false then hash part is removed from url when computing `uniqueKey`.
+ * @param {object} opts
+ * @param {String} opts.url
+ * @param {String} [opts.uniqueKey] Unique key identifying request. In not provided then it is computed as normalized URL.
+ * @param {String} [opts.method='GET']
+ * @param {String|Buffer} [opts.payload] Request payload. If method='GET' then the payload is not allowed.
+ * @param {Number} [opts.retryCount=0] How many times the url was retried in a case of exception.
+ * @param {String} [opts.errorMessages] Array of error messages from request processing.
+ * @param {String} [opts.headers={}] HTTP headers.
+ * @param {Object} [opts.userData={}] Custom data that user can assign to request.
+ * @param {Boolean} [opts.keepUrlFragment=false] If false then hash part is removed from url when computing `uniqueKey`.
*/
export default class Request {
constructor({
diff --git a/src/utils.js b/src/utils.js
index bdf82d15e..16273b93f 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -99,7 +99,6 @@ const createIsDockerPromise = () => {
/**
* Returns promise that resolves to true if the code is running in a Docker container.
- * See https://github.com/sindresorhus/is-docker
*
* @return {Promise}
*