Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Apify.main() now sets the APIFY_LOCAL_STORAGE_DIR by default
Jan Curn3bb378cbe215
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7b2cca34c..f8ee885c0 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,8 @@ xxx==================- Upgraded Puppeteer to 1.8.0- Upgraded NPM dependencies, fixed lint errors+- `Apify.main()` now sets the `APIFY_LOCAL_STORAGE_DIR` env var to a default value+ if neither `APIFY_LOCAL_STORAGE_DIR` nor `APIFY_TOKEN` is defined0.7.0 / 2018-09-11==================diff --git a/README.md b/README.mdindex b2e14ad22..659dd3c64 100644--- a/README.md+++ b/README.md@@ -167,6 +167,8 @@ npm install apify --saveYou'll need to specify where the SDK should store the crawling data.Either define the `APIFY_LOCAL_STORAGE_DIR` environment variable to store the data locally on your diskor define `APIFY_TOKEN` to store the data to Apify cloud platform.+If neither of these variables is defined, by default Apify SDK sets `APIFY_LOCAL_STORAGE_DIR`+to `./apify_storage` in the current working directory and prints a warning.The following table shows basic environment variables used by Apify SDK:@@ -245,8 +247,7 @@ cd my-hello-worldapify run```-By default, the crawling data will be stored in a local directory at `./apify_storage`-(the CLI sets the `APIFY_LOCAL_STORAGE_DIR` environment variable).+By default, the crawling data will be stored in a local directory at `./apify_storage`.For example, the input JSON file for the actor is expected to be in the default key-value storein `./apify_storage/key_value_stores/default/INPUT.json`.@@ -305,10 +306,10 @@ To run the examples, just copy them into the directory where you installed Apify`npm install apify` and then run them by calling:```-node APIFY_LOCAL_STORAGE_DIR=./apify_storage basic_crawler.js+node basic_crawler.js```-Note that it is necessary to set either the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables in order+Note that for production projects you should set either the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable in orderto tell the SDK how to store its data and crawling state. See [above](#local-stand-alone-usage) for details.Alternatively, if you're using the [Apify CLI](#local-usage-with-apify-command-line-interface-cli),@@ -754,7 +755,7 @@ Apify.main(async () => {Apify SDK has several data storage types that are useful for specific tasks.The data is stored either on local disk to a directory defined by the `APIFY_LOCAL_STORAGE_DIR` environment variable,or on the Apify cloud under user account identified by the API token defined by the `APIFY_TOKEN` environment variable.-Only one of these variables should always be set.+One of these variables should always be set.Typically, you will be developing the code on your local computer and thus set the `APIFY_LOCAL_STORAGE_DIR` environment variable.Once the code is ready, you will deploy it to Apify cloud where it will automatically@@ -783,7 +784,7 @@ In order to simplify access to the default key-value store, the SDK also provide<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-getValue"><code>Apify.getValue()</code></a>and <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-setValue"><code>Apify.setValue()</code></a> functions.-In local configuration, the data is stored in the directory specified in the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:+In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:```[APIFY_LOCAL_STORAGE_DIR]/key_value_stores/[STORE_ID]/[KEY].[EXT]@@ -838,7 +839,7 @@ In Apify SDK, the dataset is represented by the [Dataset](https://www.apify.com/In order to simplify writes to the default dataset, the SDK also provides the<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-pushData"><code>Apify.pushData()</code></a> function.-In local configuration, the data is stored in the directory specified in the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:+In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:```[APIFY_LOCAL_STORAGE_DIR]/datasets/[DATASET_ID]/[INDEX].json@@ -885,7 +886,7 @@ Typically it is used to store URLs to crawl in the specific actor run. Its usageIn Apify SDK, the request queue is represented by the [RequestQueue](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue) class.-In local configuration, the request queue data is stored in the directory specified in the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:+In local configuration, the request queue data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:```[APIFY_LOCAL_STORAGE_DIR]/request_queues/[QUEUE_ID]/[STATE]/[NUMBER].jsondiff --git a/src/actor.js b/src/actor.jsindex f8a5884f2..a7ce648e1 100644--- a/src/actor.js+++ b/src/actor.js@@ -1,3 +1,4 @@+import path from 'path';import _ from 'underscore';import Promise from 'bluebird';import log from 'apify-shared/log';@@ -98,10 +99,15 @@ export const getEnv = () => {* The function performs the following actions:** <ol>- * <li>When running on the Apify platform, it sets up a connection to listen for platform events.+ * <li>When running on the Apify platform (i.e. <code>APIFY_IS_AT_HOME</code> environment variable is set),+ * it sets up a connection to listen for platform events.* For example, to get a notification about an imminent migration to another server.* See <a href="#module-Apify-events"><code>Apify.events</code></a> for details.* </li>+ * <li>It checks that either <code>APIFY_TOKEN</code> or <code>APIFY_LOCAL_STORAGE_DIR</code> environment variable+ * is defined. If not, the functions sets <code>APIFY_LOCAL_STORAGE_DIR</code> to <code>./apify_storage</code>+ * inside the current working directory. This is to simplify running code examples.+ * </li>* <li>It invokes the user function passed as the `userFunc` parameter.</li>* <li>If the user function returned a promise, waits for it to resolve.</li>* <li>If the user function throws an exception or some other error is encountered,@@ -155,7 +161,9 @@ export const main = (userFunc) => {}if (!process.env[ENV_VARS.LOCAL_STORAGE_DIR] && !process.env[ENV_VARS.TOKEN]) {- throw new Error(`Neither ${ENV_VARS.LOCAL_STORAGE_DIR} nor ${ENV_VARS.TOKEN} environment variable is set. You need to set one these variables in order to enable data storage.`); // eslint-disable-line max-len+ const dir = path.join(process.cwd(), './apify_storage');+ process.env[ENV_VARS.LOCAL_STORAGE_DIR] = dir;+ log.warning(`Neither ${ENV_VARS.LOCAL_STORAGE_DIR} nor ${ENV_VARS.TOKEN} environment variable is set, defaulting to ${ENV_VARS.LOCAL_STORAGE_DIR}="${dir}"`); // eslint-disable-line max-len}// This is to enable unit tests where process.exit() is mocked and doesn't really exit the processdiff --git a/test/actor.js b/test/actor.jsindex 9f61505a6..48ab548a3 100644--- a/test/actor.js+++ b/test/actor.js@@ -1,3 +1,4 @@+import path from 'path';import _ from 'underscore';import chai, { expect } from 'chai';import chaiAsPromised from 'chai-as-promised';@@ -170,11 +171,24 @@ describe('Apify.main()', () => {it('works with simple user function', () => {return testMain({userFunc: () => {},- env: {},exitCode: 0,});});+ it('sets default APIFY_LOCAL_STORAGE_DIR', async () => {+ delete process.env[ENV_VARS.LOCAL_STORAGE_DIR];+ delete process.env[ENV_VARS.TOKEN];++ await testMain({+ userFunc: () => {+ expect(process.env[ENV_VARS.LOCAL_STORAGE_DIR]).to.eql(path.join(process.cwd(), './apify_storage'));+ },+ exitCode: 0,+ });++ delete process.env[ENV_VARS.LOCAL_STORAGE_DIR];+ });+it('works with promised user function', () => {let called = false;return testMain({@@ -187,7 +201,6 @@ describe('Apify.main()', () => {}, 20);});},- env: {},exitCode: 0,}).then(() => {@@ -200,7 +213,6 @@ describe('Apify.main()', () => {userFunc: () => {throw new Error('Test exception I');},- env: {},exitCode: 91,});});@@ -213,7 +225,6 @@ describe('Apify.main()', () => {throw new Error('Text exception II');});},- env: {},exitCode: 91,});});