Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Merge branch 'develop' into feature/env_vars
Marek Trunkátdd295fbae0dc
diff --cc README.mdindex 114707c16,5243eaf99..c1be2084b--- a/README.md+++ b/README.md@@@ -639,103 -655,6 +639,103 @@@ Apify.launchPuppeteer().asCallback((err```+## Environment variables++Here are the environment variables used by the SDK. When you are running you actor locally then some of the+environment variables have predefined value.++We recommend you to use [Apify command-line interface (CLI)](#local-usage-with-apify-command-line-interface-cli)+when you are developing your actor locally as it takes care of authentication to Apify and configures+`APIFY_TOKEN` and `APIFY_PROXY_PASSWORD` environment variables.++<table class="table table-bordered table-condensed">+ <thead>+ <tr>+ <th>Environment variable</th>+ <th>Local default value</th>+ <th>Description</th>+ </tr>+ </thead>+ <tbody>+ <tr>+ <td><code>APIFY_LOCAL_STORAGE_DIR</code></td>+ <td></td>+ <td>+ If provided then key-value stores, request queues and datasets use local directory defined by this variable to store the data.+ Otherwise the data get stored in the Apify cloud which requires additional <code>APIFY_TOKEN</code> environment variable to be set.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_TOKEN</code></td>+ <td></td>+ <td>+ The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.+ You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.+ If omitted, you should define <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable instead.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_PROXY_PASSWORD</code></td>+ <td></td>+ <td>+ Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address+ rotation. If you have have an Apify account, you can find the password on the+ <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>+ <td><code>default</code></td>+ <td>+ ID of the default key-value store, where the+ <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then each value is stored as a file at- <code>[APIFY_LOCAL_STORAGE_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,++ <code>[APIFY_LOCAL_STORAGE_DIR]/key_value_stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,+ where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the+ value.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_DATASET_ID</code></td>+ <td><code>default</code></td>+ <td>+ ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.+ If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then dataset items are stored as files at+ <code>[APIFY_LOCAL_STORAGE_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,+ where <code>[INDEX]</code> is a zero-based index of the item.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>+ <td><code>default</code></td>+ <td>- ID of the default request queue, where the <code>Apify.to_do()</code> function stores the data.- If you defined <code>APIFY_LOCAL_STORAGE_DIR</code>, then request queue records are stored as files at- <code>[APIFY_LOCAL_STORAGE_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,++ ID of the default request queue (request queue opened using <code>Apify.openRequestQueue()</code> function).++ If you defined <code>APIFY_LOCAL_EMULATION_DIR</code>, then request queue records are stored as files at++ <code>[APIFY_LOCAL_EMULATION_DIR]/request_queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[INDEX].json</code>,+ where <code>[INDEX]</code> is a zero-based index of the item.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_CONTAINER_URL</code></td>+ <td><code>http://localhost:4321</code></td>+ <td rowspan="2">+ Defines a HTTP port and URL under which actor may run a publicly available web server. This is used for+ <a href="#puppeteer-live-view">Puppeteer Live View</a>. Check+ <a href="https://www.apify.com/docs/actor#container-web-server" target=_blank">documentation</a> for more information on+ how to run custom web server in actor.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_CONTAINER_PORT</code></td>+ <td><code>localhost</code></td>+ </tr>+ </tbody>+</table>++For the full list of environment variables used by Apify SDK, please see the+<a href="https://www.apify.com/docs/actor#environment-variabes" target="_blank">Environment variables</a>+section of the Apify actor documentation.+## ExamplesDirectory <a href="https://github.com/apifytech/apify-js/tree/master/examples" target="_blank">examples</a> of this repository demonstrates different usages of this package.