Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Feature/better docs2 (#175)

Jan Čurn1bbd7326b496
diff --git a/README.md b/README.md
index 2c3926ec2..9da8e96de 100644
--- a/README.md
+++ b/README.md
@@ -70,8 +70,11 @@ But eventually things will get complicated, for example when you try to:
* ...
In Python you can use [Scrapy](https://scrapy.org/) for these tasks, but there was no
-such library for JavaScript, **THE language of the web**.
-The goal of Apify SDK is to provide a toolbox
+such library for **JavaScript, the language of the web**.
+The use of JavaScript is natural,
+since the same language is used to write the scripts as well as the data extraction code running in browser.
+
+The goal of Apify SDK is to fill this gap and provide a toolbox
for generic web scraping, crawling and automation tasks in JavaScript.
Don't reinvent the wheel every time you need data from the web,
and focus on writing the code specific to the target website, rather than developing commonalities.
@@ -82,45 +85,45 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify">
<ul>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler">BasicCrawler</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler"><code>BasicCrawler</code></a>
- Provides a simple framework for parallel crawling of web pages,
whose URLs are fed either from a static list or from a dynamic queue of URLs.
This class serves as a base for more complex crawlers (see below).
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#CheerioCrawler">CheerioCrawler</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#CheerioCrawler"><code>CheerioCrawler</code></a>
- Enables parallel crawling of large number of web pages
using <a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>
HTML parser.
This is the most efficient web crawler, but it does not work on websites that require JavaScript.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler">PuppeteerCrawler</a>
- - Enables parallel crawling of large number of web pages using headless Chrome browser
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler"><code>PuppeteerCrawler</code></a>
+ - Enables parallel crawling of large number of web pages using headless Chrome
and <a href="https://github.com/GoogleChrome/puppeteer">Puppeteer</a>.
- The pool of Chrome processes is automatically scaled up and down based on available system resources.
+ The pool of Chrome browsers is automatically scaled up and down based on available system resources.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool">PuppeteerPool</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool"><code>PuppeteerPool</code></a>
- Provides web browser tabs for user jobs
from an automatically-managed pool of Chrome browser instances, with configurable browser recycling and retirement policies.
Supports reuse of the disk cache to speed up crawling of websites and reduce proxy bandwidth.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList">RequestList</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList"><code>RequestList</code></a>
- Represents a list of URLs to crawl. The URLs can be passed in code or in a text file hosted on the web.
The list persists its state so that the crawling can resume
when the Node.js process restarts.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue">RequestQueue</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue"><code>RequestQueue</code></a>
- Represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud.
The queue is used for deep crawling of websites, where you start with
several URLs and then recursively follow links to other pages.
The data structure supports both breadth-first and depth-first crawling orders.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset">Dataset</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset"><code>Dataset</code></a>
- Provides a store for structured data and enables their
export to formats like JSON, JSONL, CSV, Excel or HTML.
The data is stored on local filesystem or in the cloud.
@@ -128,19 +131,19 @@ The Apify SDK is available as the <a href="https://www.npmjs.com/package/apify">
like list of products or real estate offers.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#KeyValueStore">KeyValueStore</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#KeyValueStore"><code>KeyValueStore</code></a>
- A simple key-value store for arbitrary data records or files, along with their MIME content type.
It is ideal for saving screenshots of web pages, PDFs or to persist state of your crawlers.
The data is stored on local filesystem or in the cloud.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool" target="_blank">AutoscaledPool</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool"><code>AutoscaledPool</code></a>
- Runs asynchronous background tasks, while automatically adjusting the concurrency
based on free system memory and CPU usage. This is useful for running web scraping tasks
at maximum capacity of the system.
</li>
<li>
- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerUtils" target="_blank">PuppeteerUtils</a>
+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerUtils"><code>PuppeteerUtils</code></a>
- Provides several helper functions useful for web scraping. For example, to inject jQuery to the web pages
or to hide browser origin.
</li>
@@ -189,7 +192,7 @@ The following table shows basic environment variables used by Apify SDK:
<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#KeyValueStore">key-value stores</a>,
<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList">request lists</a>
and <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue">request queues</a> store their data.
- Typically it's set to `./apify_storage`.
+ Typically it is set to <code>./apify_storage</code>.
If omitted, you should define
the <code>APIFY_TOKEN</code> environment variable instead.
</td>
@@ -197,7 +200,7 @@ The following table shows basic environment variables used by Apify SDK:
<tr>
<td><code>APIFY_TOKEN</code></td>
<td>
- The API token for your Apify account. It is used to access Apify API, e.g. to access cloud storage or to run an actor in the cloud.
+ The API token for your Apify Account. It is used to access Apify API, e.g. to access cloud storage or to run an actor in the cloud.
You can find your API token on the <a href="https://my.apify.com/account#intergrations">Account - Integrations</a> page.
If omitted, you should define the <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
</td>
@@ -206,7 +209,7 @@ The following table shows basic environment variables used by Apify SDK:
<td><code>APIFY_PROXY_PASSWORD</code></td>
<td>
Optional password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address rotation.
- If you have have an Apify account, you can find the password on the
+ If you have have an Apify Account, you can find the password on the
<a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.
This feature is optional. You can use your own proxies or no proxies at all.
</td>
@@ -268,7 +271,7 @@ and [Apify Actor](https://www.apify.com/docs/actor) documentation.
### Usage on the Apify cloud platform
You can also develop your web scraping project
-in an online code editor directly on the Apify cloud. You'll need to have Apify account.
+in an online code editor directly on the Apify cloud. You'll need to have Apify Account.
Go to [Actors](https://my.apify.com/actors)
page in the app, click <i>Create new</i> and then go to
<i>Source</i> tab and start writing your code or paste there one of the code examples below.
@@ -283,10 +286,10 @@ It can run for a few seconds, hours or even infinitely.
An actor can perform anything from a simple action such as filling out a web form or sending an email,
to complex operations such as crawling an entire website and removing duplicates from a large dataset.
-To run an actor, you need to have an [Apify account](https://my.apify.com/).
+To run an actor, you need to have an [Apify Account](https://my.apify.com/).
The actor execution is charged towards your account and the cost depends on
-the amount of memory allocated for the run.
-The actors can be shared in the [library](https://www.apify.com/library?&type=acts)
+the amount of memory allocated for the run and its duration.
+The actors can be shared in the [Apify Library](https://www.apify.com/library?&type=acts)
so that other people can use them. Note that if you share your actor in the library,
you're not charged when other people use it - its use is charged towards their account.
@@ -312,7 +315,7 @@ node basic_crawler.js
```
Note that for production projects you should set either the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable in order
-to tell the SDK how to store its data and crawling state. See [above](#local-stand-alone-usage) for details.
+to tell the SDK how to store its data and crawling state. See [Local stand-alone usage](#local-stand-alone-usage) above for details.
Alternatively, if you're using the [Apify CLI](#local-usage-with-apify-command-line-interface-cli),
you can copy and paste the source code of each of the examples into the `main.js`
@@ -472,10 +475,10 @@ Apify.main(async () => {
This example demonstrates how to use [PuppeteerCrawler](https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler)
in combination with [RequestList](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList)
and [RequestQueue](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue) to recursively scrape
-Hacker News website (https://news.ycombinator.com) using headless Chrome / Puppeteer.
+[Hacker News](https://news.ycombinator.com) website using headless Chrome / Puppeteer.
The crawlers starts with a single URL, finds links to next pages,
enqueues them and continues until no more desired links are available.
-The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`
+The results are stored to the default dataset. In local configuration, the results are represented as JSON files in `./apify_storage/datasets/default`
```javascript
const Apify = require('apify');
@@ -626,7 +629,7 @@ Apify.main(async () => {
This example demonstrates how to load pages in headless Chrome / Puppeteer
over [Apify Proxy](https://www.apify.com/docs/proxy).
-To make it work, you'll need an Apify account
+To make it work, you'll need an Apify Account
that has access to the proxy.
The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app.
Just set it to the `APIFY_PROXY_PASSWORD` environment variable
diff --git a/examples/puppeteer_proxy.js b/examples/puppeteer_proxy.js
index 9e66535fd..433ba5533 100644
--- a/examples/puppeteer_proxy.js
+++ b/examples/puppeteer_proxy.js
@@ -1,7 +1,7 @@
/**
* This example demonstrates how to load pages in headless Chrome / Puppeteer
* over [Apify Proxy](https://www.apify.com/docs/proxy).
- * To make it work, you'll need an Apify account
+ * To make it work, you'll need an Apify Account
* that has access to the proxy.
* The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app.
* Just set it to the `APIFY_PROXY_PASSWORD` environment variable
diff --git a/src/autoscaling/system_status.js b/src/autoscaling/system_status.js
index 12ddc2390..0f167fcb9 100644
--- a/src/autoscaling/system_status.js
+++ b/src/autoscaling/system_status.js
@@ -68,7 +68,7 @@ export default class SystemStatus {
/**
* Returns true if the system has not been overloaded in the last
* currentHistorySecs seconds, otherwise returns false.
- * @return {boolean}
+ * @return {Boolean}
* @ignore
*/
getCurrentStatus() {
@@ -78,7 +78,7 @@ export default class SystemStatus {
/**
* Returns true if the system has not been overloaded in the full
* history of the snapshotter (which is configurable in the snapshotter).
- * @return {boolean}
+ * @return {Boolean}
* @ignore
*/
getHistoricalStatus() {
@@ -90,7 +90,7 @@ export default class SystemStatus {
* in the last sampleDurationMillis.
*
* @param {Number} [sampleDurationMillis]
- * @return {boolean}
+ * @return {Boolean}
* @ignore
*/
_isSystemIdle(sampleDurationMillis) {
diff --git a/src/basic_crawler.js b/src/basic_crawler.js
index 5ee684a11..b3a256082 100644
--- a/src/basic_crawler.js
+++ b/src/basic_crawler.js
@@ -17,19 +17,25 @@ const DEFAULT_OPTIONS = {
/**
* Provides a simple framework for parallel crawling of web pages,
- * whose URLs are fed either from a static list (using the `RequestList` class)
- * or from a dynamic queue of URLs (using the `RequestQueue` class).
+ * whose URLs are fed either from a static list
+ * or from a dynamic queue of URLs.
*
- * `BasicCrawler` invokes `handleRequestFunction` for each `Request` object fetched from `options.requestList` or `options.requestQueue`,
- * as long as any of them is not empty. New requests are only handled if there is enough free CPU and memory available,
- * using the functionality provided by the `AutoscaledPool` class. See <a href="#AutoscaledPool">AutoscaledPool documentation</a>.
- *
- * All `AutoscaledPool` configuration options can be passed to the `autoscaledPoolOptions` parameter
- * of the `CheerioCrawler` constructor. The `minConcurrency` and `maxConcurrency` options are available directly.
+ * `BasicCrawler` invokes the user-provided `handleRequestFunction` for each {@link Request|`Request`}
+ * object, which corresponds to a single URL to crawl.
+ * The `Request` objects are fed from the {@link RequestList|`RequestList`} or {@link RequestQueue|`RequestQueue`}
+ * instances provided by the `requestList` or `requestQueue` constructor options, respectively.
*
* If both `requestList` and `requestQueue` is used, the instance first
- * processes URLs from the `requestList` and automatically enqueues all of them to `requestQueue` before it starts
- * their processing. This guarantees that a single URL is not crawled multiple times.
+ * processes URLs from the `RequestList` and automatically enqueues all of them to `RequestQueue` before it starts
+ * their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes if there are no more `Request` objects to crawl.
+ *
+ * New requests are only launched if there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool|`AutoscaledPool`} class.
+ * All `AutoscaledPool` configuration options can be passed to the `autoscaledPoolOptions` parameter
+ * of the `CheerioCrawler` constructor.
+ * For user convenience, the `minConcurrency` and `maxConcurrency` options are available directly in the constructor.
*
* Example usage:
*
@@ -62,9 +68,16 @@ const DEFAULT_OPTIONS = {
* ```
*
* @param {Object} options
- * @param {Function} options.handleRequestFunction
- * Function that processes a single `Request` object. It must return a promise.
+ * @param {Function} [options.handleRequestFunction]
+ * User-provided function that performs the logic of the crawler. It is called for each URL to crawl.
*
+ * The function that receives an object as argument, with the following field:
+ *
+ * <ul>
+ * <li>`request`: the {@link Request|`Request`} object representing the URL to crawl</li>
+ * </ul>
+ *
+ * The function must return a promise.
* @param {RequestList} [options.requestList]
* Static list of URLs to be processed.
* @param {RequestQueue} [options.requestQueue]
@@ -79,13 +92,16 @@ const DEFAULT_OPTIONS = {
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
* @param {Object} [options.autoscaledPoolOptions]
- * Configures the AutoscaledPool. See <a href="#AutoscaledPool">AutoscaledPool documentation</a>.
- * runTaskFunction(), isTaskReadyFunction() and isFinishedFunction() are provided by BasicCrawler.
- * @param {Object} [options.minConcurrency=1]
- * Sets the minimal concurrency (parallelism) for the crawl. Shorthand to the AutoscaledPool option.
- * @param {Object} [options.maxConcurrency=1000]
- * Sets the maximal concurrency (parallelism) for the crawl. Shorthand to the AutoscaledPool option.
-
+ * Custom options passed to the underlying {@link AutoscaledPool|`AutoscaledPool`} instance constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `BasicCrawler` and cannot be overridden.
+ * @param {Object} [options.minConcurrency]
+ * Sets the minimum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.
+ * @param {Object} [options.maxConcurrency]
+ * Sets the maximum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.
+ *
+ * @see {@link CheerioCrawler}
+ * @see {@link PuppeteerCrawler}
*/
export default class BasicCrawler {
constructor(opts) {
@@ -270,7 +286,7 @@ export default class BasicCrawler {
* @param {Request} request
* @param {RequestList|RequestQueue} source
* @return {Boolean} willBeRetried
- * @ingore
+ * @ignore
*/
async _requestFunctionErrorHandler(error, request, source) {
// Handles case where the crawler was aborted.
diff --git a/src/cheerio_crawler.js b/src/cheerio_crawler.js
index 19c208756..81d605860 100644
--- a/src/cheerio_crawler.js
+++ b/src/cheerio_crawler.js
@@ -18,25 +18,32 @@ const DEFAULT_OPTIONS = {
};
/**
- * Builds upon the `BasicCrawler` with functionality specifically designed for parsing raw HTML
- * of web pages. It uses the `cheerio` npm package to provide the user with a pre-parsed HTML
- * document, that may be manipulated in a same way one would manipulate the DOM in the browser
- * with jQuery.
+ * Provides a framework for parallel crawling of web pages using plain HTTP requests and
+ * [cheerio](https://www.npmjs.com/package/cheerio) HTML parser.
*
- * Unlike `BasicCrawler` which uses a `handleRequestFunction`, the `CheerioCrawler` uses a `handlePageFunction`.
- * This function gets invoked only after the HTTP response is received and the page's HTML has been parsed.
- * To customize the HTTP request that is used to fetch the raw HTML, see the `requestFunction` option.
+ * `CheerioCrawler` downloads each URL using a plain HTTP request,
+ * parses the HTML content using cheerio and then
+ * invokes the user-provided `handlePageFunction` to extract page data
+ * using a [jQuery](https://jquery.com/)-like interface to parsed HTML DOM.
*
- * `CheerioCrawler` invokes `handlePageFunction` for each `Request` object fetched from `options.requestList` or `options.requestQueue`,
- * as long as none of them is empty. New requests are only handled if there is enough free CPU and memory available,
- * using the functionality provided by the `AutoscaledPool` class. See <a href="#AutoscaledPool">AutoscaledPool documentation</a>.
+ * The source URLs are represented using `Request` objects that
+ * are fed from the {@link RequestList|`RequestList`} or {@link RequestQueue|`RequestQueue`}
+ * instances provided by the `requestList` or `requestQueue` constructor options, respectively.
*
- * All `AutoscaledPool` configuration options can be passed to the `autoscaledPoolOptions` parameter
- * of the `CheerioCrawler` constructor. The `minConcurrency` and `maxConcurrency` options are available directly.
+ * If both `requestList` and `requestQueue` is used, the instance first
+ * processes URLs from the `RequestList` and automatically enqueues all of them to `RequestQueue` before it starts
+ * their processing. This ensures that a single URL is not crawled multiple times.
+ *
+ * The crawler finishes if there are no more `Request` objects to crawl.
*
- * If both `requestList` and `requestQueue` are used, the instance first
- * processes URLs from the `requestList` and automatically enqueues all of them to `requestQueue` before it starts
- * their processing. This guarantees that a single URL is not crawled multiple times.
+ * By default, `CheerioCrawler` downloads HTML using the [request-promise](https://www.npmjs.com/package/request-promise) NPM package.
+ * You can override this behavior by setting the `requestFunction` option.
+ *
+ * New requests are only started if there is enough free CPU and memory available,
+ * using the functionality provided by the {@link AutoscaledPool|`AutoscaledPool`} class.
+ * All `AutoscaledPool` configuration options can be passed to the `autoscaledPoolOptions` parameter
+ * of the `CheerioCrawler` constructor.
+ * For user convenience, the `minConcurrency` and `maxConcurrency` options are available directly.
*
* Example usage:
*
@@ -75,17 +82,27 @@ const DEFAULT_OPTIONS = {
* ```
*
* @param {Object} options
- * @param {Function} options.handlePageFunction
- * A function that receives three arguments: the Cheerio object `$`, the raw HTML and the `Request` object and does all the document manipulation.
- * If it returns a promise, it is awaited.
+ * @param {Function} [options.handlePageFunction]
+ * User-provided function that performs the logic of the crawler. It is called for each page
+ * loaded and parsed by the crawler.
+ *
+ * The function that receives an object as argument, with the following three fields:
+ *
+ * <ul>
+ * <li>`$`: the Cheerio object</li>
+ * <li>`html`: the raw HTML</li>
+ * <li>`request`: the {@link Request|`Request`} object representing the URL to crawl</li>
+ * </ul>
+ *
+ * If the function returns a promise, it is awaited.
*
- * @param {Function} [options.requestFunction]
- * Overrides the function that performs the HTTP request to get the raw HTML needed for Cheerio.
- * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L246">GitHub</a> for default behavior.
* @param {RequestList} [options.requestList]
* Static list of URLs to be processed.
* @param {RequestQueue} [options.requestQueue]
* Dynamic queue of URLs to be processed. This is useful for recursive crawling of websites.
+ * @param {Function} [options.requestFunction]
+ * Overrides the function that performs the HTTP request to get the raw HTML needed for Cheerio.
+ * See source code on <a href="https://github.com/apifytech/apify-js/blob/master/src/cheerio_crawler.js#L246">GitHub</a> for default behavior.
* @param {Number} [options.handlePageTimeoutSecs=300]
* Timeout in which the function passed as `options.handlePageFunction` needs to finish, given in seconds.
* @param {Number} [options.requestTimeoutSecs=30]
@@ -103,11 +120,16 @@ const DEFAULT_OPTIONS = {
* Always set this value in order to prevent infinite loops in misconfigured crawlers.
* Note that in cases of parallel crawling, the actual number of pages visited might be slightly higher than this value.
* @param {Object} [options.autoscaledPoolOptions]
- * Configures the AutoscaledPool. See <a href="#AutoscaledPool">AutoscaledPool documentation</a>.
+ * Custom options passed to the underlying {@link AutoscaledPool|`AutoscaledPool`} instance constructor.
+ * Note that the `runTaskFunction`, `isTaskReadyFunction` and `isFinishedFunction` options
+ * are provided by `CheerioCrawler` and cannot be overridden.
* @param {Object} [options.minConcurrency]
- * Sets the minimal concurrency (parallelism) for the crawl. Shorthand to the AutoscaledPool option.
+ * Sets the minimum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.
* @param {Object} [options.maxConcurrency]
- * Sets the maximal concurrency (parallelism) for the crawl. Shorthand to the AutoscaledPool option.
+ * Sets the maximum concurrency (parallelism) for the crawl. hortcut to the corresponding `AutoscaledPool` option.
+ *
+ * @see {@link BasicCrawler}
+ * @see {@link PuppeteerCrawler}
*/
export default class CheerioCrawler {
constructor(opts = {}) {
diff --git a/src/dataset.js b/src/dataset.js
index 0e7811bef..2851691db 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -105,13 +105,29 @@ export const chunkBySize = (items, limitBytes) => {
*/
/**
- * The `Dataset` class represents an append-only data storage that is useful for saving sequential or tabular data,
- * such as a list of e-commerce products.
- * To create an instance of the `Dataset` class, call the [Apify.openDataset()](#module-Apify-openDataset) function.
+ * The `Dataset` class represents a store for structured data where each object stored has the same attributes,
+ * such as online store products or real estate offers. You can imagine it as a table,
+ * where each object is a row and its attributes are columns.
+ * Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove existing records.
+ * Typically it is used to store crawling results.
*
- * The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
- * in the Apify cloud (see [Dataset storage documentation](https://www.apify.com/docs/storage#dataset) when actor is running on Apify
- * platform or if `APIFY_TOKEN` environment variable is set.
+ * Do not instantiate this class directly, use the
+ * {@link Apify#openDataset|`Apify.openDataset()`} function instead.
+ *
+ * `Dataset` stores its data either on local disk or in the Apify cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable is set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the data is stored in
+ * the local directory in the following files:
+ * ```
+ * [APIFY_LOCAL_STORAGE_DIR]/datasets/[DATASET_ID]/[INDEX].json
+ * ```
+ * Note that `[DATASET_ID]` is the name or ID of the dataset. The default dataset has ID `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_DATASET_ID` environment variable.
+ * Each dataset item is stored as a separate JSON file, where `[INDEX]` is a zero-based index of the item in the dataset.
+ *
+ * If the `APIFY_TOKEN` environment variable is provided instead, the data is stored
+ * in the [Apify Dataset](https://www.apify.com/docs/storage#dataset) cloud storage.
*
* Example usage:
*
@@ -131,8 +147,7 @@ export const chunkBySize = (items, limitBytes) => {
* { col3: 123 },
* ]);
* ```
- *
- * @param {String} datasetId - ID of the dataset.
+ * @hideconstructor
*/
export class Dataset {
constructor(datasetId) {
@@ -142,10 +157,11 @@ export class Dataset {
}
/**
- * Stores an object or an array of objects to the default dataset of the current actor run.
- * The function has no result, but throws on invalid args or other errors.
+ * Stores an object or an array of objects to the dataset.
+ * The function returns a promise that resolves when the operation finishes.
+ * It has no result, but throws on invalid args or other errors.
*
- * **IMPORTANT**: Do not forget to use the `await` keyword when calling `Apify.pushData()`,
+ * **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,
* otherwise the actor process might finish before the data is stored!
*
* The size of the data is limited by the receiving API and therefore `pushData` will only
@@ -197,27 +213,42 @@ export class Dataset {
/**
* Returns items in the dataset based on the provided parameters.
*
- * If format is `json` then doesn't return an array of records but <a href="#PaginationList">PaginationList</a> instead.
+ * If format is `json` then the function doesn't return an array of records but {@linkcode PaginationList} instead.
*
* @param {Object} options
- * @param {String} [options.format='json'] - Format of the items, possible values are: json, csv, xlsx, html, xml and rss.
- * @param {Number} [options.offset=0] - Number of array elements that should be skipped at the start.
- * @param {Number} [options.limit=250000] - Maximum number of array elements to return.
- * @param {Number} [options.desc] - If 1 then the objects are sorted by createdAt in descending order.
- * @param {Array} [options.fields] - If provided then returned objects will only contain specified keys
- * @param {String} [options.unwind] - If provided then objects will be unwound based on provided field.
- * @param {Boolean} [options.disableBodyParser] - If true then response from API will not be parsed
- * @param {Number} [options.attachment] - If 1 then the response will define the Content-Disposition: attachment header, forcing a web
- * browser to download the file rather than to display it. By default this header is not present.
- * @param {String} [options.delimiter=','] - A delimiter character for CSV files, only used if format=csv. You might need to URL-encode
- * the character (e.g. use %09 for tab or %3B for semicolon).
- * @param {Number} [options.bom] - All responses are encoded in UTF-8 encoding. By default, the csv files are prefixed with the UTF-8 Byte
- * Order Mark (BOM), while json, jsonl, xml, html and rss files are not. If you want to override this default
- * behavior, specify bom=1 query parameter to include the BOM or bom=0 to skip it.
- * @param {String} [options.xmlRoot] - Overrides default root element name of xml output. By default the root element is results.
- * @param {String} [options.xmlRow] - Overrides default element name that wraps each page or page function result object in xml output.
- * By default the element name is page or result based on value of simplified parameter.
- * @param {Number} [options.skipHeaderRow] - If set to `1` then header row in csv format is skipped.
+ * @param {String} [options.format='json']
+ * Format of the items, possible values are: `json`, `csv`, `xlsx`, `html`, `xml` and `rss`.
+ * @param {Number} [options.offset=0]
+ * Number of array elements that should be skipped at the start.
+ * @param {Number} [options.limit=250000]
+ * Maximum number of array elements to return.
+ * @param {Boolean} [options.desc]
+ * If `true` then the objects are sorted by `createdAt` in descending order.
+ * Otherwise they are sorted in ascending order.
+ * @param {Array} [options.fields]
+ * An array of field names that will be included in the result. If omitted, all fields are included in the results.
+ * @param {String} [options.unwind]
+ * Specifies a name of the field in the result objects that will be used to unwind the resulting objects.
+ * By default, the results are returned as they are.
+ * @param {Boolean} [options.disableBodyParser]
+ * If `true` then response from API will not be parsed.
+ * @param {Number} [options.attachment]
+ * If `true` then the response will define the `Content-Disposition: attachment` HTTP header, forcing a web
+ * browser to download the file rather than to display it. By default, this header is not present.
+ * @param {String} [options.delimiter=',']
+ * A delimiter character for CSV files, only used if `format` is `csv`.
+ * You might need to URL-encode the character (e.g. use `%09` for tab or `%3B` for semicolon).
+ * @param {Number} [options.bom]
+ * All responses are encoded in UTF-8 encoding. By default, the CSV files are prefixed with the UTF-8 Byte
+ * Order Mark (BOM), while JSON, JSONL, XML, HTML and RSS files are not. If you want to override this default
+ * behavior, set `bom` option to `true` to include the BOM, or set `bom` to `false` to skip it.
+ * @param {String} [options.xmlRoot]
+ * Overrides the default root element name of the XML output. By default, the root element is `results`.
+ * @param {String} [options.xmlRow]
+ * Overrides the default element name that wraps each page or page function result object in XML output.
+ * By default, the element name is `page` or `result`, depending on the value of the `simplified` option.
+ * @param {Number} [options.skipHeaderRow]
+ * If set to `1` then header row in csv format is skipped.
* @return {Promise}
*/
getData(opts = {}) {
@@ -252,15 +283,15 @@ export class Dataset {
}
/**
- * Iterates over the all dataset items, yielding each in turn to an iteratee function.
- * Each invocation of iteratee is called with three arguments: (element, index).
+ * Iterates over dataset items, yielding each in turn to an `iteratee` function.
+ * Each invocation of `iteratee` is called with three arguments: `(element, index)`.
*
- * If iteratee returns a Promise then it's awaited before a next call.
+ * If `iteratee` returns a Promise then it is awaited before a next call.
*
* @param {Function} iteratee
* @param {Opts} opts
* @param {Number} [options.offset=0] - Number of array elements that should be skipped at the start.
- * @param {Number} [options.desc] - If 1 then the objects are sorted by createdAt in descending order.
+ * @param {Number} [options.desc] - If `1` then the objects are sorted by `createdAt` in descending order.
* @param {Array} [options.fields] - If provided then returned objects will only contain specified keys
* @param {String} [options.unwind] - If provided then objects will be unwound based on provided field.
* @param {Number} [options.limit=250000] - How many items to load in one request.
@@ -291,10 +322,10 @@ export class Dataset {
}
/**
- * Produces a new array of values by mapping each value in list through a transformation function (iteratee).
- * Each invocation of iteratee is called with three arguments: (element, index).
+ * Produces a new array of values by mapping each value in list through a transformation function (`iteratee`).
+ * Each invocation of `iteratee` is called with three arguments: `(element, index)`.
*
- * If iteratee returns a Promise then it's awaited before a next call.
+ * If `iteratee` returns a `Promise` then it's awaited before a next call.
*
* @param {Function} iteratee
* @param {Opts} opts
@@ -322,13 +353,15 @@ export class Dataset {
}
/**
- * Memo is the initial state of the reduction, and each successive step of it should be returned by iteratee.
- * The iteratee is passed three arguments: the memo, then the value and index of the iteration.
+ * Boils down a list of values into a single value.
*
- * If no memo is passed to the initial invocation of reduce, the iteratee is not invoked on the first element of the list.
- * The first element is instead passed as the memo in the invocation of the iteratee on the next element in the list.
+ * Memo is the initial state of the reduction, and each successive step of it should be returned by `iteratee`.
+ * The `iteratee` is passed three arguments: the `memo`, then the value and index of the iteration.
*
- * If iteratee returns a Promise then it's awaited before a next call.
+ * If no `memo` is passed to the initial invocation of reduce, the `iteratee` is not invoked on the first element of the list.
+ * The first element is instead passed as the memo in the invocation of the `iteratee` on the next element in the list.
+ *
+ * If `iteratee` returns a `Promise` then it's awaited before a next call.
*
* @param {Function} iteratee
* @param {*} memo
@@ -363,7 +396,8 @@ export class Dataset {
}
/**
- * Deletes the dataset.
+ * Removes the dataset either from the Apify cloud storage or from the local directory,
+ * depending on the mode of operation.
*
* @return {Promise}
*/
@@ -577,36 +611,19 @@ const getOrCreateDataset = (datasetIdOrName) => {
/**
- * Opens a dataset and returns a promise resolving to an instance of the [Dataset](#Dataset) class.
- *
- * Dataset is an append-only data storage that is useful for saving sequential or tabular data,
- * such as a list of e-commerce products.
- * The data can be written to the dataset using the [Dataset.pushData()](#Dataset-pushData) function.
- *
- * Example usage:
+ * Opens a dataset and returns a promise resolving to an instance of the {@link Dataset|`Dataset`} class.
*
- * ```javascript
- * // Opens dataset called 'some-name'.
- * const dataset = await Apify.openDataset('some-name');
+ * Datasets are used to store structured data where each object stored has the same attributes,
+ * such as online store products or real estate offers.
+ * The actual data is stored either on local filesystem or in the cloud.
*
- * // Write a single row
- * await dataset.pushData({ foo: 'bar' });
- *
- * // Write multiple rows
- * await dataset.pushData([
- * { foo: 'bar2', col2: 'val2' },
- * { col3: 123 },
- * ]);
- * ```
+ * For more details and code examples, see the {@link Dataset|`Dataset`} class.
*
- * @param {String} datasetIdOrName
- * ID or name of the dataset to be opened. If no value is provided
- * then the function opens the default dataset associated with the actor run,
- * identified by the `APIFY_DEFAULT_DATASET_ID` environment variable.
- * The full name must be specified as `username/dataset-name`.
+ * @param {string} [datasetIdOrName]
+ * ID or name of the dataset to be opened. If `null` or `undefined`,
+ * the function returns the default dataset associated with the actor run.
* @returns {Promise<Dataset>}
- * Returns a promise that resolves to an instance of the [Dataset](#Dataset) class.
- *
+ * Returns a promise that resolves to an instance of the `Dataset` class.
* @memberof module:Apify
* @name openDataset
* @instance
@@ -622,11 +639,10 @@ export const openDataset = (datasetIdOrName) => {
};
/**
- * Stores an object or an array of objects to the default dataset of the current actor run.
- * The ID of the default dataset is taken from the `APIFY_DEFAULT_DATASET_ID` environment variable.
- * The function has no result, but throws on invalid args or other errors.
+ * Stores an object or an array of objects to the default {@linkcode Dataset} of the current actor run.
*
- * Calling
+ * This is just a convenient shortcut for {@link Dataset#pushData|`Dataset.pushData()`}.
+ * For example, calling the following code:
* ```javascript
* await Apify.pushData({ myValue: 123 });
* ```
@@ -637,19 +653,15 @@ export const openDataset = (datasetIdOrName) => {
* await dataset.pushData({ myValue: 123 });
* ```
*
- * The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
- * in the Apify cloud (see [Dataset storage documentation](https://www.apify.com/docs/storage#dataset) when actor is running on Apify
- * platform or if `APIFY_TOKEN` environment variable is set.
- *
- * For more information, see
- * [Apify.openDataset()](#module-Apify-openDataset) and [Dataset.pushData()](#Dataset-pushData).
+ * For more information, see {@link Apify.openDataset|`Apify.openDataset()`} and {@linkcode Dataset#pushData|`Dataset.pushData()`}
*
- * **IMPORTANT**: Do not forget to use the `await` keyword when calling `Apify.pushData()`,
+ * **IMPORTANT**: Make sure to use the `await` keyword when calling `pushData()`,
* otherwise the actor process might finish before the data is stored!
*
* @param {Object|Array} data Object or array of objects containing data to be stored in the default dataset.
* The objects must be serializable to JSON and the JSON representation of each object must be smaller than 9MB.
* @returns {Promise} Returns a promise that resolves once the data is saved.
+ * @see {@linkcode Dataset}
*
* @memberof module:Apify
* @name pushData
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 0f0e1635a..1f31930e4 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -127,6 +127,7 @@ export const maybeStringify = (value, options) => {
* ```
*
* @param {String} storeId - ID of the key-value store.
+ * @hideconstructor
*/
export class KeyValueStore {
constructor(storeId) {
@@ -184,7 +185,8 @@ export class KeyValueStore {
}
/**
- * Deletes the store.
+ * Removes the key-value store either from the Apify cloud storage or from the local directory,
+ * depending on the mode of operation.
*
* @return {Promise}
*/
@@ -452,7 +454,7 @@ export const getValue = key => openKeyValueStore().then(store => store.getValue(
* will get extension based on it's content type. This feature is useful for local development and debugging
* of your actors.
*
- * **IMPORTANT:** Do not forget to use the `await` keyword when calling `Apify.setValue()`,
+ * **IMPORTANT:** Make sure to use the `await` keyword when calling `Apify.setValue()`,
* otherwise the actor process might finish before the value is stored!
*
* @param key Key of the record
diff --git a/src/puppeteer_crawler.js b/src/puppeteer_crawler.js
index 631bb26e6..b0a493a91 100644
--- a/src/puppeteer_crawler.js
+++ b/src/puppeteer_crawler.js
@@ -124,6 +124,9 @@ const PAGE_CLOSE_TIMEOUT_MILLIS = 30000;
* @param {LaunchPuppeteerOptions} [options.launchPuppeteerOptions]
* Options used by `Apify.launchPuppeteer()` to start new Puppeteer instances.
* See `launchPuppeteerOptions` parameter of `PuppeteerPool`.
+ *
+ * @see {@link CheerioCrawler}
+ * @see {@link BasicCrawler}
*/
export default class PuppeteerCrawler {
constructor(opts) {
diff --git a/src/puppeteer_pool.js b/src/puppeteer_pool.js
index d75ad0d14..483b03a9a 100644
--- a/src/puppeteer_pool.js
+++ b/src/puppeteer_pool.js
@@ -65,10 +65,16 @@ class PuppeteerInstance {
}
/**
- * Manages a pool of Chrome browser instances controlled by [Puppeteer](https://github.com/GoogleChrome/puppeteer).
- * `PuppeteerPool` rotates Chrome instances to change proxies
- * and other settings, in order to prevent detection of your web scraping bot,
+ * Manages a pool of Chrome browser instances controlled using [Puppeteer](https://github.com/GoogleChrome/puppeteer).
+ * `PuppeteerPool` reuses Chrome instances and tabs using specific
+ * browser rotation and retirement policies.
+ * This is useful in order to facilitate rotation of proxies, cookies
+ * or other settings in order to prevent detection of your web scraping bot,
* access web pages from various countries etc.
+ * Additionally, the reuse of browser instances instances speeds up crawling,
+ * and the retirement of instances helps mitigate effects of memory leaks in Chrome.
+ *
+ * `PuppeteerPool` is internally used by the {@link PuppeteerCrawler|`PuppeteerCrawler`} class.
*
* Example usage:
*
@@ -86,7 +92,7 @@ class PuppeteerInstance {
* const page2 = await puppeteerPool.newPage();
* const page3 = await puppeteerPool.newPage();
*
- * // ... do something with pages ...
+ * // ... do something with the pages ...
*
* // Close all browsers.
* await puppeteerPool.destroy();
diff --git a/src/request_list.js b/src/request_list.js
index 12e5e303c..38c9a5ec6 100644
--- a/src/request_list.js
+++ b/src/request_list.js
@@ -21,14 +21,24 @@ const ensureUniqueKeyValid = (uniqueKey) => {
};
/**
- * Provides way to handle a list of URLs to be crawled.
- * Each URL is represented using an instance of the `Request` class.
+ * Represents a static list of URLs to crawl.
+ * The URLs can be provided either in code or parsed from a text file hosted on the web.
+ * Each URL is represented using an instance of the {@link Request|`Request`} class.
*
- * `RequestList` has an internal state where it remembers handled requests, requests in progress and also reclaimed requests.
- * The state might be persisted in a key-value store as shown in the example below so if an actor is restarted (due to internal
- * error or restart of the host machine) then the crawling can continue where it left off.
+ * Once you create an instance of `RequestList`, you need to call {@link RequestList#initialize|`initialize()`}
+ * before the instance can be used. After that, no more URLs can be added to the list.
*
- * Basic usage of `RequestList`:
+ * `RequestList` is used by {@link BasicCrawler|`BasicCrawler`}, {@link CheerioCrawler|`CheerioCrawler`}
+ * and {@link PuppeteerCrawler|`PuppeteerCrawler`} as a source of URLs to crawl.
+ * Unlike {@link RequestQueue|`RequestQueue`}, `RequestList` is static but it can contain even millions of URLs.
+ *
+ * `RequestList` has an internal state where it stores information which requests were handled,
+ * which are in progress or which were reclaimed.
+ * The state might be automatically persisted to the default key-value store by setting the `persistStateKey` option
+ * so that if the Node.js process is restarted,
+ * the crawling can continue where it left off. For more details, see {@link KeyValueStore|`KeyValueStore`}.
+ *
+ * Example usage:
*
* ```javascript
* const requestList = new Apify.RequestList({
@@ -44,7 +54,8 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* persistStateKey: 'my-crawling-state'
* });
*
- * await requestList.initialize(); // Load requests.
+ * // This call loads and parses the URLs from the remote file.
+ * await requestList.initialize();
*
* // Get requests from list
* const request1 = await requestList.fetchNextRequest();
@@ -59,25 +70,30 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* ```
*
* @param {Object} options
- * @param {Array} options.sources An array of sources for the `RequestList`. Its contents can either be just plain objects,
- * defining at least the 'url' property, or instances of the `Request` class. See the
- * <a href="#Request">Request class documentation</a> for available properties.
- * Additionally a `requestsFromUrl` property may be used instead of `url`,
- * which will instruct the `RequestList` to download the sources from the given URL.
- * URLs will be parsed from the received response.
+ * @param {Array} options.sources
+ * An array of sources for the `RequestList`. Its contents can either be just plain objects,
+ * defining at least the 'url' property or instances of the {@link Request|`Request`} class.
+ * Additionally a `requestsFromUrl` property may be used instead of `url`,
+ * which will instruct the `RequestList` to download the sources from the given remote location.
+ * The URLs will be parsed from the received response.
+ *
* ```javascript
* [
* // One URL
* { method: 'GET', url: 'http://example.com/a/b' },
- * // Batch import of URLa from a file hosted on the web
+ * // Batch import of URLs from a file hosted on the web
* { method: 'POST', requestsFromUrl: 'http://example.com/urls.txt' },
* ]
* ```
- * @param {String} [options.persistStateKey] Key-value store key under which the `RequestList` persists its state. If this is set then `RequestList`
- * persists its state in regular intervals and loads the state from there in a case that's restarted
- * due to some error or migration to another worker machine.
- * @param {Object} [options.state] The state object that the `RequestList` will be initialized from.
- * It is in the form returned by `requestList.getState()`, such as follows:
+ * @param {String} [options.persistStateKey]
+ * Identifies the key in the default key-value store under which the `RequestList` persists its state.
+ * If this is set then `RequestList`
+ * persists its state in regular intervals and loads the state from there in case it is restarted
+ * due to an error or system reboot.
+ * @param {Object} [options.state]
+ * The state object that the `RequestList` will be initialized from.
+ * It is in the form as returned by `RequestList.getState()`, such as follows:
+ *
* ```javascript
* {
* nextIndex: 5,
@@ -88,17 +104,20 @@ const ensureUniqueKeyValid = (uniqueKey) => {
* },
* }
* ```
- * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList` is to use the `persistStateKey` parameter instead.
*
- * @param {Boolean} [options.keepDuplicateUrls=false] By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
- * on the `uniqueKey` property of passed sources. If not present, it is generated by normalizing
- * the URL. If present, it is kept intact. In any case, only one request per `uniqueKey` is added
- * to the `RequestList` resulting in removing of duplicate URLs / unique keys.
- * Setting `keepDuplicateUrls` to true will append an additional identifier to the `uniqueKey`
- * of each request that does not already include a `uniqueKey`. Therefore, duplicate
- * URLs will be kept in the list. It does not protect the user from having duplicates in user set
- * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys,
- * if they wish to keep more than just a single copy in the `RequestList`.
+ * Note that the preferred (and simpler) way to persist the state of crawling of the `RequestList`
+ * is to use the `persistStateKey` parameter instead.
+ *
+ * @param {Boolean} [options.keepDuplicateUrls=false]
+ * By default, `RequestList` will deduplicate the provided URLs. Default deduplication is based
+ * on the `uniqueKey` property of passed sources. If not present, it is generated by normalizing
+ * the URL. If present, it is kept intact. In any case, only one request per `uniqueKey` is added
+ * to the `RequestList` resulting in removing of duplicate URLs / unique keys.
+ * Setting `keepDuplicateUrls` to true will append an additional identifier to the `uniqueKey`
+ * of each request that does not already include a `uniqueKey`. Therefore, duplicate
+ * URLs will be kept in the list. It does not protect the user from having duplicates in user set
+ * `uniqueKey`s however. It is the user's responsibility to ensure uniqueness of their unique keys,
+ * if they wish to keep more than just a single copy in the `RequestList`.
*/
export default class RequestList {
constructor(opts = {}) {
@@ -148,7 +167,8 @@ export default class RequestList {
}
/**
- * Loads all sources specified.
+ * Loads all remote sources of URLs and potentially starts periodic state persistence.
+ * This function must be called before you can start using the instance in a meaningful way.
*
* @returns {Promise}
*/
@@ -239,7 +259,8 @@ export default class RequestList {
}
/**
- * Returns an object representing the state of the RequestList instance. Do not alter the resulting object!
+ * Returns an object representing the internal state of the `RequestList` instance.
+ * Note that the objects fields can change in future releases.
*
* @returns Object
*/
@@ -256,10 +277,11 @@ export default class RequestList {
}
/**
- * Returns `true` if the next call to `fetchNextRequest()` will return null, otherwise it returns `false`.
+ * Resolves to `true` if the next call to {@link RequestList#fetchNextRequest|`fetchNextRequest()`}
+ * will return `null`, otherwise it resolves to `false`.
* Note that even if the list is empty, there might be some pending requests currently being processed.
*
- * @returns {Promise<boolean>}
+ * @returns {Promise<Boolean>}
*/
isEmpty() {
return Promise
@@ -288,7 +310,7 @@ export default class RequestList {
/**
* Gets the next `Request` to process. First, the function gets a request previously reclaimed
- * using `reclaimRequest()` function, if there is any.
+ * using {@link RequestList#reclaimRequest|`reclaimRequest()`} function, if there is any.
* Otherwise it gets a next request from the sources.
*
* The function gets `null` if there are no more
@@ -324,7 +346,7 @@ export default class RequestList {
}
/**
- * Marks request handled after successfull processing.
+ * Marks request as handled after successful processing.
*
* @param {Request} request
*
@@ -413,7 +435,8 @@ export default class RequestList {
/**
* Adds given request.
- * If opts parameter is a plain object and not an instance of a Request then creates it.
+ * If the `opts` parameter is a plain object and not an instance of a `Request`, then the function
+ * creates a `Request` instance.
*
* @ignore
*/
diff --git a/src/request_queue.js b/src/request_queue.js
index 08287a5e2..f7646ad8b 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -117,11 +117,34 @@ const getRequestId = (uniqueKey) => {
*/
/**
- * Provides a simple interface to the [Apify Request Queue](https://www.apify.com/docs/storage#queue)
- * storage, which is used to manage a dynamic queue of web pages to crawl.
+ * Represents a queue of URLs to crawl, which is used for deep crawling of websites
+ * where you start with several URLs and then recursively
+ * follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.
+ * Each URL is represented as an instance of the {@link Request|`Request`} class.
*
- * You should not instantiate this class directly, but use the
- * [Apify.openRequestQueue()](#module-Apify-openRequestQueue) function.
+ * Do not instantiate this class directly, use the
+ * {@link Apify#openRequestQueue|`Apify.openRequestQueue()`} function instead.
+ *
+ * `RequestQueue` is used by {@link BasicCrawler|`BasicCrawler`}, {@link CheerioCrawler|`CheerioCrawler`}
+ * and {@link PuppeteerCrawler|`PuppeteerCrawler`} as a source of URLs to crawl.
+ * Unlike {@link RequestList|`RequestList`}, `RequestQueue` supports dynamic adding and removing of requests.
+ * On the other hand, the queue is not optimized for operations that add or remove a large number of URLs in a batch.
+ *
+ * `RequestQueue` stores its data either on local disk or in the Apify cloud,
+ * depending on whether the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable is set.
+ *
+ * If the `APIFY_LOCAL_STORAGE_DIR` environment variable is set, the queue data is stored in
+ * that local directory as follows:
+ * ```
+ * [APIFY_LOCAL_STORAGE_DIR]/request_queues/[QUEUE_ID]/[STATE]/[NUMBER].json
+ * ```
+ * Note that `[QUEUE_ID]` is the name or ID of the request queue. The default queue has ID `default`,
+ * unless you override it by setting the `APIFY_DEFAULT_REQUEST_QUEUE_ID` environment variable.
+ * Each request in the queue is stored as a separate JSON file, where `[STATE]` is either `handled` or `pending`,
+ * and `[NUMBER]` is an integer indicating the position of the request in the queue.
+ *
+ * If the `APIFY_TOKEN` environment variable is provided instead, the data is stored
+ * in the [Apify Request Queue](https://www.apify.com/docs/storage#queue) cloud storage.
*
* Example usage:
*
@@ -145,11 +168,10 @@ const getRequestId = (uniqueKey) => {
* // Mark a request as handled
* await queue.markRequestHandled(request1);
*
- * // If processing fails then reclaim the request back to the queue, so that it's crawled again
- * await queue.reclaimRequest(request2);
+ * // If processing of a request fails then reclaim it back to the queue, so that it's crawled again
+ * await queue.reclaimRequest(request2);
* ```
- *
- * @param {String} queueId - ID of the request queue.
+ * @hideconstructor
*/
export class RequestQueue {
constructor(queueId) {
@@ -213,10 +235,10 @@ export class RequestQueue {
}
/**
- * Gets a request from the queue.
+ * Gets the request from the queue specified by ID.
*
* @param {String} requestId Request ID
- * @return {Request}
+ * @return {Promise<Request>}
*/
getRequest(requestId) {
validateGetRequestParams(requestId);
@@ -230,9 +252,9 @@ export class RequestQueue {
}
/**
- * Returns next upcoming request.
+ * Returns next request in the queue to be processed.
*
- * @returns {Request}
+ * @returns {Promise<Request>}
*/
fetchNextRequest() {
return this
@@ -264,7 +286,7 @@ export class RequestQueue {
* Marks request handled after successfull processing.
*
* @param {Request} request
- * @return {RequestOperationInfo}
+ * @return {Promise<RequestOperationInfo>}
*/
markRequestHandled(request) {
validateMarkRequestHandledParams(request);
@@ -289,13 +311,15 @@ export class RequestQueue {
}
/**
- * Reclaims request after unsuccessfull operation. Requests gets returned into the queue.
+ * Reclaims failed request back to the queue,
+ * so that it can be processed later again.
*
* @param {Request} request
* @param {Object} [opts]
- * @param {Boolean} [opts.forefront=false] If true then requests gets returned to the begining of the queue
- * and to the back of the queue otherwise.
- * @return {RequestOperationInfo}
+ * @param {Boolean} [opts.forefront=false]
+ * If true then requests gets returned to the begining of the queue
+ * and to the back of the queue otherwise.
+ * @return {Promise<RequestOperationInfo>}
*/
reclaimRequest(request, opts = {}) {
const { forefront } = validateReclaimRequestParams(request, opts);
@@ -317,12 +341,13 @@ export class RequestQueue {
}
/**
- * Returns `true` if the next call to `fetchNextRequest()` will return `null`, otherwise it returns `false`.
+ * Resolves to `true` if the next call to `fetchNextRequest()` will return `null`, otherwise it resolves to `false`.
* Note that even if the queue is empty, there might be some pending requests currently being processed.
*
- * The function might occasionally return a false negative, but it should never return a false positive!
+ * Due to the nature of distributed storage systems,
+ * the function might occasionally return a false negative, but it should never return a false positive!
*
- * @returns {boolean}
+ * @returns {Promise<Boolean>}
*/
isEmpty() {
return this
@@ -331,10 +356,11 @@ export class RequestQueue {
}
/**
- * Returns `true` if all requests were already handled and there are no more left.
- * The function might occasionally return a false negative, but it should never return a false positive!
+ * Resolves to `true` if all requests were already handled and there are no more left.
+ * Due to the nature of distributed storage systems,
+ * the function might occasionally return a false negative, but it will never return a false positive.
*
- * @returns {boolean}
+ * @returns {Promise<Boolean>}
*/
isFinished() {
return this
@@ -470,7 +496,8 @@ export class RequestQueue {
}
/**
- * Deletes the queue.
+ * Removes the queue either from the Apify cloud storage or from the local directory,
+ * depending on the mode of operation.
*
* @return {Promise}
*/
@@ -499,7 +526,7 @@ const filePathToQueueOrderNo = (filepath) => {
};
/**
- * Local implementation of the `RequestQueue` class.
+ * Local directory-based implementation of the `RequestQueue` class.
*
* @ignore
*/
@@ -757,43 +784,23 @@ const getOrCreateQueue = (queueIdOrName) => {
/**
* Opens a request queue and returns a promise resolving to an instance
- * of the [RequestQueue](#RequestQueue) class.
+ * of the {@link RequestQueue|`RequestQueue`} class.
*
- * Example usage:
+ * `RequestQueue` represents a queue of URLs to crawl, which is stored either on local filesystem or in the cloud.
+ * The queue is used for deep crawling of websites, where you start with several URLs and then
+ * recursively follow links to other pages. The data structure supports both breadth-first
+ * and depth-first crawling orders.
*
- * ```javascript
- * const queue = await Apify.openRequestQueue('my-queue-id');
- *
- * // Enqueue some requests
- * await queue.addRequest(new Apify.Request({ url: 'http://example.com/aaa'});
- * await queue.addRequest(new Apify.Request({ url: 'http://example.com/bbb'});
- * await queue.addRequest(new Apify.Request({ url: 'http://example.com/foo/bar'}, { forefront: true });
- *
- * // Get requests from queue
- * const request1 = queue.fetchNextRequest();
- * const request2 = queue.fetchNextRequest();
- * const request3 = queue.fetchNextRequest();
- *
- * // Mark some of them as handled
- * queue.markRequestHandled(request1);
- *
- * // If processing fails then reclaim it back to the queue
- * queue.reclaimRequest(request2);
- * ```
- *
- * The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
- * in the Apify cloud (see [Request queue documentation](https://www.apify.com/docs/storage#queue) when actor is running on Apify
- * platform or if `APIFY_TOKEN` environment variable is set.
- *
- * @param {string} queueIdOrName ID or name of the request queue to be opened.
- * @returns {Promise<RequestQueue>} Returns a promise that resolves to a `RequestQueue` object. If no value is provided
- * then the function opens the default request queue associated with the actor run.
+ * For more details and code examples, see the {@link RequestQueue|`RequestQueue`} class.
*
+ * @param {string} [queueIdOrName]
+ * ID or name of the request queue to be opened. If `null` or `undefined`,
+ * the function returns the default request queue associated with the actor run.
+ * @returns {Promise<RequestQueue>}
+ * Returns a promise that resolves to an instance of the `RequestQueue` class.
* @memberof module:Apify
* @name openRequestQueue
* @instance
- *
- * @ignore
*/
export const openRequestQueue = (queueIdOrName) => {
checkParamOrThrow(queueIdOrName, 'queueIdOrName', 'Maybe String');
diff --git a/src/utils.js b/src/utils.js
index 6c761a007..06a972781 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -159,9 +159,10 @@ export const avg = arr => sum(arr) / arr.length;
/**
* Computes a weighted average of an array of numbers, complemented by an array of weights.
+ *
* @param {Array} arrValues
* @param {Array} arrWeights
- * @return {number}
+ * @return {Number}
*/
export const weightedAvg = (arrValues, arrWeights) => {
const result = arrValues.map((value, i) => {