Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Udpating env vars docs in readme

Marek Trunkat3e75cb02cb5d
diff --git a/README.md b/README.md
index d1556714c..2f1cffc9a 100644
--- a/README.md
+++ b/README.md
@@ -159,8 +159,7 @@ You can use Apify SDK in any Node.js project by running:
npm install apify
```
-SDK will use local directory `./apify_storage` for key-value stores, request lists and request queues to
-store their data.
+The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or in the Apify cloud when actor is running on Apify platform or if `APIFY_TOKEN` environment variable is set.
However, to make the package work at its full potential (use
<a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a>, call another actors, etc.),
@@ -168,41 +167,37 @@ you'll need to set one or more of the following environment variables
for your Node.js process, depending on your circumstances:
<table class="table table-bordered table-condensed">
- <thead>
- <tr>
- <th>Environment variable</th>
- <th>Local default value</th>
- <th>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>APIFY_PLATFORM_STORAGE</code></td>
- <td></td>
- <td>
- Defines if key-value stores, request lists and request queues should store data in Apify cloud or local
- directory. By default local directory is used. Use <code>APIFY_PLATFORM_STORAGE=1</code>
- environment variable for Apify cloud.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_TOKEN</code></td>
- <td></td>
- <td>
- The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
- You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
- If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_PROXY_PASSWORD</code></td>
- <td></td>
- <td>
- Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
- rotation. If you have have an Apify account, you can find the password on the
- <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.</td>
- </tr>
- </tbody>
+ <thead>
+ <tr>
+ <th>Environment variable</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>
+ <td>
+ If provided then key-value stores, request queues and datasets use local directory defined by this variable to store the data.
+ Otherwise the data get stored in the Apify cloud which requires additional <code>APIFY_TOKEN</code> environment variable to be set.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_TOKEN</code></td>
+ <td>
+ The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
+ You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
+ If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_PROXY_PASSWORD</code></td>
+ <td>
+ Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
+ rotation. If you have have an Apify account, you can find the password on the
+ <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.
+ </td>
+ </tr>
+ </tbody>
</table>
There are more environment variables you may want to customize. For the full list, please see
@@ -655,103 +650,92 @@ when you are developing your actor locally as it takes care of authentication to
`APIFY_TOKEN` and `APIFY_PROXY_PASSWORD` environment variables.
<table class="table table-bordered table-condensed">
- <thead>
- <tr>
- <th>Environment variable</th>
- <th>Local default value</th>
- <th>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>APIFY_PLATFORM_STORAGE</code></td>
- <td></td>
- <td>
- Defines if key-value stores, request lists and request queues should store data in Apify cloud or local
- directory. By default local directory is used. Use `APIFY_PLATFORM_STORAGE=1` environment variable
- for Apify cloud.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>
- <td><code>apify_storage</code></td>
- <td>
- Defines the path to a local directory where key-value stores, request lists and request queues store
- their data. If omitted, the package will try to use cloud storage instead and will expect that the
- <code>APIFY_TOKEN</code> environment variable is defined.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_TOKEN</code></td>
- <td></td>
- <td>
- The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
- You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
- If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_PROXY_PASSWORD</code></td>
- <td></td>
- <td>
- Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
- rotation. If you have have an Apify account, you can find the password on the
- <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.</td>
- </tr>
- <tr>
- <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>
- <td><code>default</code></td>
- <td>
- ID of the default key-value store, where the
- <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then each value is stored as a file at
- <code>[APIFY_LOCAL_STORAGE_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,
- where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the
- value.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_DEFAULT_DATASET_ID</code></td>
- <td><code>default</code></td>
- <td>
- ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then dataset items are stored as files at
- <code>[APIFY_LOCAL_STORAGE_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,
- where <code>[INDEX]</code> is a zero-based index of the item.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>
- <td><code>default</code></td>
- <td>
- ID of the default request queue, where the <code>Apify.to_do()</code> function stores the data.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then request queue records are stored as files at
- <code>[APIFY_LOCAL_STORAGE_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,
- where <code>[INDEX]</code> is a zero-based index of the item.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_CONTAINER_PORT</code></td>
- <td><code>localhost</code></td>
- <td>
- TODO
- </td>
- </tr>
- <tr>
- <td><code>APIFY_CONTAINER_URL</code></td>
- <td><code>http://localhost:4321</code></td>
- <td>
- TODO
- </td>
- </tr>
- </tbody>
+ <thead>
+ <tr>
+ <th>Environment variable</th>
+ <th>Local default value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>
+ <td>
+ If provided then key-value stores, request queues and datasets use local directory defined by this variable to store the data.
+ Otherwise the data get stored in the Apify cloud which requires additional <code>APIFY_TOKEN</code> environment variable to be set.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_TOKEN</code></td>
+ <td></td>
+ <td>
+ The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
+ You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
+ If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_PROXY_PASSWORD</code></td>
+ <td></td>
+ <td>
+ Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
+ rotation. If you have have an Apify account, you can find the password on the
+ <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default key-value store, where the
+ <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then each value is stored as a file at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,
+ where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the
+ value.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_DATASET_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then dataset items are stored as files at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,
+ where <code>[INDEX]</code> is a zero-based index of the item.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default request queue, where the <code>Apify.to_do()</code> function stores the data.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then request queue records are stored as files at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,
+ where <code>[INDEX]</code> is a zero-based index of the item.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_CONTAINER_PORT</code></td>
+ <td><code>localhost</code></td>
+ <td>
+ TODO
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_CONTAINER_URL</code></td>
+ <td><code>http://localhost:4321</code></td>
+ <td>
+ TODO
+ </td>
+ </tr>
+ </tbody>
</table>
For the full list of environment variables used by Apify SDK, please see the
<a href="https://www.apify.com/docs/actor#environment-variabes" target="_blank">Environment variables</a>
section of the Apify actor documentation.
-
## Examples
Directory <a href="https://github.com/apifytech/apify-js/tree/master/examples" target="_blank">examples</a> of this repository demonstrates different usages of this package.
diff --git a/src/dataset.js b/src/dataset.js
index b4de4d030..e59be303f 100644
--- a/src/dataset.js
+++ b/src/dataset.js
@@ -111,7 +111,7 @@ export const chunkBySize = (items, limitBytes) => {
*
* The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
* in the Apify cloud (see [Dataset storage documentation](https://www.apify.com/docs/storage#dataset) when actor is running on Apify
- * platform if `APIFY_TOKEN` environment variable is set.
+ * platform or if `APIFY_TOKEN` environment variable is set.
*
* Example usage:
*
@@ -636,7 +636,7 @@ export const openDataset = (datasetIdOrName) => {
*
* The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
* in the Apify cloud (see [Dataset storage documentation](https://www.apify.com/docs/storage#dataset) when actor is running on Apify
- * platform if `APIFY_TOKEN` environment variable is set.
+ * platform or if `APIFY_TOKEN` environment variable is set.
*
* For more information, see
* [Apify.openDataset()](#module-Apify-openDataset) and [Dataset.pushData()](#Dataset-pushData).
diff --git a/src/key_value_store.js b/src/key_value_store.js
index 1cabaefcd..26592e4be 100644
--- a/src/key_value_store.js
+++ b/src/key_value_store.js
@@ -100,7 +100,7 @@ export const maybeStringify = (value, options) => {
*
* The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
* in the Apify cloud (see [Key-value stores documentation](https://www.apify.com/docs/storage#key-value-store) when actor is running on Apify
- * platform if `APIFY_TOKEN` environment variable is set.
+ * platform or if `APIFY_TOKEN` environment variable is set.
*
* Example usage:
*
@@ -354,7 +354,7 @@ const getOrCreateKeyValueStore = (storeIdOrName) => {
*
* The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
* in the Apify cloud (see [Key-value stores documentation](https://www.apify.com/docs/storage#key-value-store) when actor is running on Apify
- * platform if `APIFY_TOKEN` environment variable is set.
+ * platform or if `APIFY_TOKEN` environment variable is set.
*
* Example usage:
*
diff --git a/src/request_queue.js b/src/request_queue.js
index 4f6017e13..1b476e685 100644
--- a/src/request_queue.js
+++ b/src/request_queue.js
@@ -768,7 +768,7 @@ const getOrCreateQueue = (queueIdOrName) => {
*
* The actual data is either stored on the local disk in the directory defined by `APIFY_LOCAL_STORAGE_DIR` environment variable if provided or
* in the Apify cloud (see [Request queue documentation](https://www.apify.com/docs/storage#queue) when actor is running on Apify
- * platform if `APIFY_TOKEN` environment variable is set.
+ * platform or if `APIFY_TOKEN` environment variable is set.
*
* @param {string} queueIdOrName ID or name of the request queue to be opened.
* @returns {Promise<RequestQueue>} Returns a promise that resolves to a `RequestQueue` object. If no value is provided