Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updating docs

Marek Trunkate5327619c73b
diff --git a/README.md b/README.md
index 64fc0ca39..326ea2be9 100644
--- a/README.md
+++ b/README.md
@@ -48,6 +48,7 @@
+ [Puppeteer Pool](#puppeteer-pool)
+ [Puppeteer Live View](#puppeteer-live-view)
- [Promises vs. callbacks](#promises-vs-callbacks)
+- [Environment variables](#environment-variables)
- [Examples](#examples)
* [Recursive crawling](#recursive-crawling)
* [Crawling url list](#crawling-url-list)
@@ -158,8 +159,8 @@ You can use Apify SDK in any Node.js project by running:
npm install apify
```
-SDK will use local directory `./apify_storage` for key-value stores, request lists and request queues store
-their data.
+SDK will use local directory `./apify_storage` for key-value stores, request lists and request queues to
+store their data.
However, to make the package work at its full potential (use Apify Proxy, call another actors, etc.),
you'll need to set one or more of the following environment variables
@@ -174,27 +175,18 @@ for your Node.js process, depending on your circumstances:
</tr>
</thead>
<tbody>
- <tr>
+ <tr>
<td><code>APIFY_PLATFORM_STORAGE</code></td>
- <th></th>
+ <td></td>
<td>
Defines if key-value stores, request lists and request queues should store data in Apify cloud or local
directory. By default local directory is used. Use `APIFY_PLATFORM_STORAGE=1` environment variable
for Apify cloud.
</td>
- </tr>
- <tr>
- <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>
- <th><code>apify_storage</code></th>
- <td>
- Defines the path to a local directory where key-value stores, request lists and request queues store
- their data. If omitted, the package will try to use cloud storage instead and will expect that the
- <code>APIFY_TOKEN</code> environment variable is defined.
- </td>
- </tr>
- <tr>
+ </tr>
+ <tr>
<td><code>APIFY_TOKEN</code></td>
- <th></th>
+ <td></td>
<td>
The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
@@ -203,50 +195,17 @@ for your Node.js process, depending on your circumstances:
</tr>
<tr>
<td><code>APIFY_PROXY_PASSWORD</code></td>
- <th></th>
+ <td></td>
<td>
Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
rotation. If you have have an Apify account, you can find the password on the
<a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.</td>
</tr>
- <tr>
- <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>
- <th>default</th>
- <td>
- ID of the default key-value store, where the
- <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then each value is stored as a file at
- <code>[APIFY_LOCAL_STORAGE_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,
- where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the
- value.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_DEFAULT_DATASET_ID</code></td>
- <th>default</th>
- <td>
- ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then dataset items are stored as files at
- <code>[APIFY_LOCAL_STORAGE_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,
- where <code>[INDEX]</code> is a zero-based index of the item.
- </td>
- </tr>
- <tr>
- <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>
- <th>default</th>
- <td>
- ID of the default request queue, where the <code>Apify.to_do()</code> function stores the data.
- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then request queue records are stored as files at
- <code>[APIFY_LOCAL_STORAGE_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,
- where <code>[INDEX]</code> is a zero-based index of the item.
- </td>
- </tr>
</tbody>
</table>
-For the full list of environment variables used by Apify SDK, please see the
-<a href="https://www.apify.com/docs/actor#environment-variabes" target="_blank">Environment variables</a>
-section of the Apify actor documentation.
+There are more environment variables you may want to customize. For the full list, please see
+[Environment variables](#environment-variables) section below.
### Local usage with Apify command-line interface (CLI)
@@ -685,6 +644,113 @@ Apify.launchPuppeteer().asCallback((err, browser) => {
```
+## Environment variables
+
+Here are the environment variables used by the SDK. When you are running you actor locally then some of the
+environment variables have predefined value.
+
+We recommend you to use [Apify command-line interface (CLI)](#local-usage-with-apify-command-line-interface-cli)
+when you are developing your actor locally as it takes care of authentication to Apify and configures
+`APIFY_TOKEN` and `APIFY_PROXY_PASSWORD` environment variables.
+
+<table class="table table-bordered table-condensed">
+ <thead>
+ <tr>
+ <th>Environment variable</th>
+ <th>Local default value</th>
+ <th>Description</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td><code>APIFY_PLATFORM_STORAGE</code></td>
+ <td></td>
+ <td>
+ Defines if key-value stores, request lists and request queues should store data in Apify cloud or local
+ directory. By default local directory is used. Use `APIFY_PLATFORM_STORAGE=1` environment variable
+ for Apify cloud.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>
+ <td><code>apify_storage</code></td>
+ <td>
+ Defines the path to a local directory where key-value stores, request lists and request queues store
+ their data. If omitted, the package will try to use cloud storage instead and will expect that the
+ <code>APIFY_TOKEN</code> environment variable is defined.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_TOKEN</code></td>
+ <td></td>
+ <td>
+ The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.
+ You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.
+ If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_PROXY_PASSWORD</code></td>
+ <td></td>
+ <td>
+ Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address
+ rotation. If you have have an Apify account, you can find the password on the
+ <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.</td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default key-value store, where the
+ <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then each value is stored as a file at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,
+ where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the
+ value.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_DATASET_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then dataset items are stored as files at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,
+ where <code>[INDEX]</code> is a zero-based index of the item.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>
+ <td><code>default</code></td>
+ <td>
+ ID of the default request queue, where the <code>Apify.to_do()</code> function stores the data.
+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then request queue records are stored as files at
+ <code>[APIFY_LOCAL_STORAGE_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,
+ where <code>[INDEX]</code> is a zero-based index of the item.
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_CONTAINER_PORT</code></td>
+ <td><code>localhost</code></td>
+ <td>
+ TODO
+ </td>
+ </tr>
+ <tr>
+ <td><code>APIFY_CONTAINER_URL</code></td>
+ <td><code>http://localhost:4321</code></td>
+ <td>
+ TODO
+ </td>
+ </tr>
+ </tbody>
+</table>
+
+For the full list of environment variables used by Apify SDK, please see the
+<a href="https://www.apify.com/docs/actor#environment-variabes" target="_blank">Environment variables</a>
+section of the Apify actor documentation.
+
+
## Examples
Directory <a href="https://github.com/apifytech/apify-js/tree/master/examples" target="_blank">examples</a> of this repository demonstrates different usages of this package.