Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Updated README.md
Jan Curn03e836a70653
diff --git a/README.md b/README.mdindex e2cc6f5a6..8ee1473e9 100644--- a/README.md+++ b/README.md@@ -1,8 +1,9 @@# apifier-sdk-js [](https://travis-ci.org/Apifier/apifier-sdk-js) [](http://badge.fury.io/js/apifier)-Apifier SDK for JavaScript+Apifier Actor runtime for JavaScript.-This package is still a work in progress, stay tuned.+This is a helper package that simplifies development of Apifier acts.+It's still a work in progress, stay tuned.## Installation@@ -13,18 +14,36 @@ npm install apifier --save## Usage inside acts-Import the package to your source code+Import the package to your act.```javascriptconst Apifier = require('apifier');```-And use `Apifier.main(func)` to pass a user function that will be invoked by the Apifier actor runtime.+### Main function++To simplify development of acts, the Actor runtime provides the `Apifier.main()` function which does the following:++1) Prepares the execution context object by calling `Apifier.getContext()`++2) Fetches act run input by calling `Apifier.getInput()`++3) Waits for the user function to finish++4) Stores the output of the user function by calling `Apifier.setOutput()`++5) Exits the process++If the user function throws an exception or some other error is encountered,+then `Apifier.main()` prints the details to console so that it's saved into log file.++`Apifier.main()` accepts a single argument - a user function that performs the act.+The simples use case is a syncronous user function.```javascript-Apifier.main( (options) => {- // my synchronous function that returns immediatelly- console.dir(`Received: ${options.input}`);+Apifier.main( (context) => {+ // my synchronous function that returns immediately+ console.dir(context);return 'Hello world from actor!';});```@@ -32,41 +51,119 @@ Apifier.main( (options) => {If the user function returns a promise, it is considered as asynchronous.```javascript-Apifier.main( (options) => {+Apifier.main( (context) => {// my asynchronous function that returns a promise- console.dir(`Received: ${options.input}`);+ console.dir(context);return Promise.resolve()- .then(() => {- // do something async here- }).then(() => {return 'Hello world from asynchronous actor!';});});```-The user function has a single argument `options` which is an object such as:+You can also take advantage of the async/await keywords:++```javascript+const request = require('request-promise');++Apifier.main( async (context) => {+ const result = await request('http://www.example.com');+ return result;+});+```++Note that you don't need to use `Apifier.main()` function at all,+it is provided merely for user convenience. The same activity+can be performed using the lower-level functions described in the following text.+++### Context++The user function passed to `Apifier.main()` accepts a single+argument called `context` which is an object such as:+```javascript{+ // Internal port on which the web server is listening+ internalPort: Number,++ // ID of the act+ actId: String,++ // ID of the act run+ actRunId: String,++ // Date when the act was started+ startedAt: Date,++ // Date when the act will time out+ timeoutAt: Date,++ // ID of the key-value store where input and output data of this act is stored+ defaultKeyValueStoreId: String,++ // Input data for the act, as provided by Apifier.getInput()input: {- // HTTP request body parsed using body-parser middleware:- // * For 'application/json' content type the body is parsed from JSON to any JavaScript object- // * For 'text/*' content types the body is passed as String- // * For any other content type the body is passed as raw Buffer object- // For HTTP GET request, body is null.- body: Object|String|Buffer,-- // HTTP method used to pass the data- method: String-- // Content-Type HTTP header- contentType: String+ body: String/Buffer,+ contentType: String,}}```-Instead of the `Apifier.main()` helper function,-you can run a web server inside the act and handle the requests all by yourself.+The values of the objects are determined from process environment variables,+such as 'APIFY_INTERNAL_PORT' or 'APIFY_STARTED_AT', and the input is obtained by calling the+`Apifier.getInput()` function.++The `context` object can be obtained independently as follows:++```javascript+Apifier.getContext().then( (context) => {+ console.dir(context);+});+```++### Input and output++Each act can have input and output data.++To only obtain only the input of the act, use the following code:++```javascript+Apifier.getInput().then( (input) => {+ console.log(`Input in ${input.contentType}:`);+ console.dir(input.body);+});+```++Similarly,++```javascript+const output = {+ body: 'test output from act',+ contentType: 'application/text'+};+Apifier.setOutput(output).then( () => {+ console.log('Output saved!');+});+```++### Promises++Note that the `getContext`, `getInput` and `setOutput` also accept a Node.js-style callback parameter.+If the callback is not provided, they return a promise.+To set a promise dependency from an external library, use the following code:+```javascript+const Promise = require('bluebird');+Apifier.setPromisesDependency(Promise);+```+Otherwise, the SDK defaults to native promises.+++### Internal web server++**TODO: this is still not finished**++You can run a web server inside the act and handle the requests all by yourself.```javascriptconst http = require('http');@@ -92,12 +189,6 @@ Note that by calling `Apifier.readyFreddy()` you tell the Actor runtime that youreceiving HTTP requests over the port specified by the `APIFY_INTERNAL_PORT` environment variable.-To set a promise dependency from an external library, use the following code:-```javascript-const Promise = require('bluebird');-Apifier.setPromisesDependency(Promise);-```-Otherwise, the SDK defaults to native promises.## Package maintenance