Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Renamed Apify.getContext() to Apify.getEnv(), the context object is no longer passed to user main function, updated README.md
Jan Curnf5ca6b9d2ab0
diff --git a/README.md b/README.mdindex 6602b907f..fca8ba9a7 100644--- a/README.md+++ b/README.md@@ -30,122 +30,100 @@ const Apify = require('apify');### Main function-To simplify development of acts, the Actor runtime provides the `Apify.main()` function which does the following:+To simplify development of acts, the runtime provides the `Apify.main(func)` function which does the following:-*TODO: this needs to be changed, call getValue() / setValue() explicitely*+1) Invokes the user function `func`-1) Prepares the execution context object by calling `Apify.getContext()`+2) If the function returned a promise, waits for it to resolve-2) Fetches act run input by calling `Apify.getValue('INPUT')`--3) Waits for the user function to finish--4) Stores the return value of the user function as act output by calling `Apify.setOutput()`--5) Exits the process+3) Exits the processIf the user function throws an exception or some other error is encountered,-then `Apify.main()` prints the details to console so that it's saved into log file.+then `Apify.main()` prints the details to console so that they are stored to the log file.-`Apify.main()` accepts a single argument - a user function that performs the act.+`Apify.main()` accepts a single argument - the user function that performs the operation of the act.In the simplest case, the user function is synchronous:```javascript-Apify.main((context) => {+Apify.main(() => {// my synchronous function that returns immediately- console.dir(context.input);- return 'Output from my act';});```-The user function accepts a single argument called `context`, which is a context object-holding information about the act run. It is described in detail in the next section.--The return value from the user function is stored as output of the act.-It must be an object that can be stringified to JSON, otherwise the act will fail.-The output record will have the MIME content type set to `application/json; charset=utf-8`.-If the user function returns a promise, it is considered as asynchronous:```javascript-Apify.main((context) => {+Apify.main(() => {// my asynchronous function that returns a promisereturn Promise.resolve().then(() => {- return 'Output from my act';+ return request('http://www.example.com');+ })+ .then((html) => {+ console.log(html);});});```-In this case, the return value of the last function in the promise chain is considered as the act output.--To simplify your code, you can also take advantage of the async/await keywords:+To simplify your code, you can take advantage of the `async`/`await` keywords:```javascriptconst request = require('request-promise');-Apify.main(async (context) => {- const result = await request('http://www.example.com');- return result;+Apify.main(async () => {+ const html = await request('http://www.example.com');+ console.log(html);});```Note that the `Apify.main()` function does not need to be used at all,-it is provided merely for user convenience. The same activity-can be performed using the lower-level functions described in the following sections.+it is provided merely for a user convenience.-### Context+### Environment-The user function passed to `Apify.main()` accepts a single-argument called `context` which is an object such as:+On the Apify platform, the act process is executed with several environment variables.+To simplify access to these variables, you can use the `Apify.getEnv()` function,+which returns an object with the following properties:```javascript{- // ID of the act+ // ID of the act.+ // Environment variable: APIFY_ACT_IDactId: String,// ID of the act run+ // Environment variable: APIFY_ACT_RUN_IDactRunId: String,// ID of the user who started the act (might be different than the owner of the act)+ // Environment variable: APIFY_USER_IDuserId: String,// Authentication token representing privileges given to the act run,// it can be passed to various Apify APIs.+ // Environment variable: APIFY_TOKENtoken: String,// Date when the act was started+ // Environment variable: APIFY_STARTED_ATstartedAt: Date,// Date when the act will time out+ // Environment variable: APIFY_TIMEOUT_ATtimeoutAt: Date,// ID of the key-value store where input and output data of this act is stored+ // Environment variable: APIFY_DEFAULT_KEY_VALUE_STORE_IDdefaultKeyValueStoreId: String,- // Input data for the act as provided by Apify.getInput()- input: {- body: Object,- contentType: String,- },-// Port on which the act's internal web server is listening.// This is still work in progress, stay tuned.+ // Environment variable: APIFY_INTERNAL_PORTinternalPort: Number,}```-The values of the objects are determined from process environment variables,-such as `APIFY_INTERNAL_PORT` or `APIFY_STARTED_AT`, and the input is obtained by calling the-`Apify.getInput()` function.--The `context` object can also be obtained as follows:--```javascript-const context = await Apify.getContext();-console.dir(context);-```### Input and output@@ -156,11 +134,12 @@ under keys named `INPUT` and `OUTPUT`, respectively.The ID of the key-value store is provided by the Actor runtime as the `APIFY_DEFAULT_KEY_VALUE_STORE_ID`environment variable.-To obtain the input of the act, use the following code:+Use the following code to obtain the input of the act:```javascriptconst input = await Apify.getValue('INPUT');-console.dir(`My input is: ${input}``);+console.log('My input:');+console.dir(input);```If the input data has the `application/json` content type, it is automatically parsed into a JavaScript object,@@ -175,9 +154,8 @@ const output = {await Apify.setValue('OUTPUT', output);```-This is especially useful if the output of the act is not a JSON,-because return value from `Apify.main()` is always converted to JSON.-In such case, make sure the main function returns `undefined` or `null`.+*Don't forget to use the `await` keyword, otherwise the act's process might finish before the output is stored!*+### Browserdiff --git a/src/actor.js b/src/actor.jsindex 173502f80..e5a5487f2 100644--- a/src/actor.js+++ b/src/actor.js@@ -161,7 +161,7 @@ export const setValue = (key, value, contentType, callback = null) => {/**- * Generates a context object which contains meta-data about this act run such as:+ * Generates an object which contains parsed environment variables:* ```javascript* {* actId: String,@@ -175,11 +175,11 @@ export const setValue = (key, value, contentType, callback = null) => {* }* ```* All the information is generated from the APIFY_XXX environment variables.- * If some of the variables is not defined or is invalid, the corresponding value in the context object will be null;+ * If some of the variables is not defined or is invalid, the corresponding value in the resulting object will be null;* an error is not thrown in such a case in order to simplify local development and debugging of acts.- * @return Object Context object.+ * @return Object*/-export const getContext = () => {+export const getEnv = () => {const env = process.env || {};return {actId: env[ENV_VARS.ACT_ID] || null,@@ -213,8 +213,7 @@ export const main = (userFunc) => {try {newPromise().then(() => {- const context = getContext();- return userFunc(context);+ return userFunc();}).catch((err) => {if (!exited) {diff --git a/src/constants.js b/src/constants.jsindex f00a85d3a..470f44547 100644--- a/src/constants.js+++ b/src/constants.js@@ -14,7 +14,6 @@ export const EXIT_CODES = {* Dictionary of APIFY_XXX environment variable names.*/export const ENV_VARS = {- INTERNAL_PORT: 'APIFY_INTERNAL_PORT',ACT_ID: 'APIFY_ACT_ID',ACT_RUN_ID: 'APIFY_ACT_RUN_ID',USER_ID: 'APIFY_USER_ID',@@ -25,6 +24,7 @@ export const ENV_VARS = {WATCH_FILE: 'APIFY_WATCH_FILE',API_BASE_URL: 'APIFY_API_BASE_URL',HEADLESS: 'APIFY_HEADLESS',+ INTERNAL_PORT: 'APIFY_INTERNAL_PORT',};diff --git a/src/index.js b/src/index.jsindex c73074577..7857de019 100644--- a/src/index.js+++ b/src/index.js@@ -1,7 +1,7 @@import EventEmitter from 'events';import { setPromisesDependency, getPromisesDependency } from './utils';-import { main, readyFreddy, getContext, getValue, setValue, apifyClient } from './actor';+import { main, readyFreddy, getEnv, getValue, setValue, apifyClient } from './actor';import { browse } from './browser';/* globals module */@@ -9,7 +9,7 @@ import { browse } from './browser';// Publicly available functionsconst Apify = {main,- getContext,+ getEnv,getValue,setValue,readyFreddy,diff --git a/test/actor.js b/test/actor.jsindex c7841c8ea..aa2d924e0 100644--- a/test/actor.js+++ b/test/actor.js@@ -128,7 +128,7 @@ const testMain = (method, bodyRaw, contentType, userFunc, expectedExitCode = 0)* Helper function that enables testing of Apify.main()* @return Promise*/-const testMain = ({ userFunc, context, exitCode }) => {+const testMain = ({ userFunc, exitCode }) => {// Mock process.exit() to check exit code and prevent process exitconst processMock = sinon.mock(process);const exitExpectation = processMock@@ -137,19 +137,14 @@ const testMain = ({ userFunc, context, exitCode }) => {.once().returns();- // Mock APIFY_ environment variables- _.defaults(context, getEmptyContext());- setContextToEnv(context);-let error = null;return Promise.resolve().then(() => {return new Promise((resolve, reject) => {// Invoke main() function, the promise resolves after the user function is run- Apify.main((realContext) => {+ Apify.main(() => {try {- expect(realContext).to.eql(context);// Wait for all tasks in Node.js event loop to finishresolve();} catch (err) {@@ -157,7 +152,7 @@ const testMain = ({ userFunc, context, exitCode }) => {return;}// Call user func to test other behavior (note that it can throw)- if (userFunc) return userFunc(realContext);+ if (userFunc) return userFunc();});});})@@ -188,7 +183,7 @@ const testMain = ({ userFunc, context, exitCode }) => {};-const getEmptyContext = () => {+const getEmptyEnv = () => {return {internalPort: null,actId: null,@@ -201,7 +196,7 @@ const getEmptyContext = () => {};};-const setContextToEnv = (context) => {+const setEnv = (env) => {delete process.env.APIFY_INTERNAL_PORT;delete process.env.APIFY_ACT_ID;delete process.env.APIFY_ACT_RUN_ID;@@ -211,27 +206,27 @@ const setContextToEnv = (context) => {delete process.env.APIFY_TIMEOUT_AT;delete process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID;- if (context.internalPort) process.env.APIFY_INTERNAL_PORT = context.internalPort.toString();- if (context.actId) process.env.APIFY_ACT_ID = context.actId;- if (context.actRunId) process.env.APIFY_ACT_RUN_ID = context.actRunId;- if (context.userId) process.env.APIFY_USER_ID = context.userId;- if (context.token) process.env.APIFY_TOKEN = context.token;- if (context.startedAt) process.env.APIFY_STARTED_AT = context.startedAt.toISOString();- if (context.timeoutAt) process.env.APIFY_TIMEOUT_AT = context.timeoutAt.toISOString();- if (context.defaultKeyValueStoreId) process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = context.defaultKeyValueStoreId;+ if (env.internalPort) process.env.APIFY_INTERNAL_PORT = env.internalPort.toString();+ if (env.actId) process.env.APIFY_ACT_ID = env.actId;+ if (env.actRunId) process.env.APIFY_ACT_RUN_ID = env.actRunId;+ if (env.userId) process.env.APIFY_USER_ID = env.userId;+ if (env.token) process.env.APIFY_TOKEN = env.token;+ if (env.startedAt) process.env.APIFY_STARTED_AT = env.startedAt.toISOString();+ if (env.timeoutAt) process.env.APIFY_TIMEOUT_AT = env.timeoutAt.toISOString();+ if (env.defaultKeyValueStoreId) process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = env.defaultKeyValueStoreId;};-describe('Apify.getContext()', () => {+describe('Apify.getEnv()', () => {it('works with null values', () => {- const expectedContext = getEmptyContext();- setContextToEnv(expectedContext);+ const expectedEnv = getEmptyEnv();+ setEnv(expectedEnv);- const context = Apify.getContext();- expect(context).to.eql(expectedContext);+ const env = Apify.getEnv();+ expect(env).to.eql(expectedEnv);});it('works with with non-null values', () => {- const expectedContext = _.extend(getEmptyContext(), {+ const expectedEnv = _.extend(getEmptyEnv(), {internalPort: 12345,actId: 'test actId',actRunId: 'test actId',@@ -241,10 +236,10 @@ describe('Apify.getContext()', () => {timeoutAt: new Date(),defaultKeyValueStoreId: 'some store',});- setContextToEnv(expectedContext);+ setEnv(expectedEnv);- const context = Apify.getContext();- expect(context).to.eql(expectedContext);+ const env = Apify.getEnv();+ expect(env).to.eql(expectedEnv);});});@@ -259,7 +254,7 @@ describe('Apify.main()', () => {it('works with simple user function', () => {return testMain({userFunc: () => {},- context: {},+ env: {},exitCode: 0,});});@@ -276,7 +271,7 @@ describe('Apify.main()', () => {}, 20);});},- context: {},+ env: {},exitCode: 0,}).then(() => {@@ -289,7 +284,7 @@ describe('Apify.main()', () => {userFunc: () => {throw new Error('Test exception I');},- context: {},+ env: {},exitCode: 91,});});@@ -304,7 +299,7 @@ describe('Apify.main()', () => {throw new Error('Text exception II');});},- context: {},+ env: {},exitCode: 91,});});