Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Renamed the package from 'apifier' to 'apify', bumped package.json version to 0.4.0 to avoid conflicts with the previous package 0.3.4, renamed "Apifier" in code to "Apify", partially updated README.md
Jan Curn9f1578099d68
diff --git a/README.md b/README.mdindex b0a20488f..19225d46a 100644--- a/README.md+++ b/README.md@@ -1,15 +1,20 @@-# apify-runtime-js [](https://travis-ci.org/Apifier/apify-runtime-js) [](http://badge.fury.io/js/apifier)+# apify: Apify runtime for JavaScript-Apifier Actor runtime for JavaScript.+[](http://badge.fury.io/js/apify)+[](https://travis-ci.org/Apifier/apify-runtime-js)-This is a helper package that simplifies development of Apifier acts.-It is still a work in progress, stay tuned.+A a helper package that simplifies development of Apify acts. See https://www.apifier.com for details.+This is still work in progress, things might change and break.++If you're looking for the API documentation package previously published as *apify*,+please go to [jsdocify](https://www.npmjs.com/package/jsdocify).+Big kudos to [jbalsas](https://www.npmjs.com/~jbalsas) for giving away this package name!## Installation```bash-npm install apifier --save+npm install apify --save```This package requires Node.js 6 or higher.@@ -20,33 +25,33 @@ It might work with lower versions too, but they are neither tested nor supportedImport the package to your act.```javascript-const Apifier = require('apifier');+const Apify = require('apify');```### Main function-To simplify development of acts, the Actor runtime provides the `Apifier.main()` function which does the following:+To simplify development of acts, the Actor runtime provides the `Apify.main()` function which does the following:*TODO: this needs to be changed, call getValue() / setValue() explicitely*-1) Prepares the execution context object by calling `Apifier.getContext()`+1) Prepares the execution context object by calling `Apify.getContext()`-2) Fetches act run input by calling `Apifier.getValue('INPUT')`+2) Fetches act run input by calling `Apify.getValue('INPUT')`3) Waits for the user function to finish-4) Stores the return value of the user function as act output by calling `Apifier.setOutput()`+4) Stores the return value of the user function as act output by calling `Apify.setOutput()`5) Exits the processIf the user function throws an exception or some other error is encountered,-then `Apifier.main()` prints the details to console so that it's saved into log file.+then `Apify.main()` prints the details to console so that it's saved into log file.-`Apifier.main()` accepts a single argument - a user function that performs the act.+`Apify.main()` accepts a single argument - a user function that performs the act.In the simplest case, the user function is synchronous:```javascript-Apifier.main((context) => {+Apify.main((context) => {// my synchronous function that returns immediatelyconsole.dir(context.input);return 'Output from my act';@@ -63,7 +68,7 @@ The output record will have the MIME content type set to `application/json; charIf the user function returns a promise, it is considered as asynchronous:```javascript-Apifier.main((context) => {+Apify.main((context) => {// my asynchronous function that returns a promisereturn Promise.resolve().then(() => {@@ -79,20 +84,20 @@ To simplify your code, you can also take advantage of the async/await keywords:```javascriptconst request = require('request-promise');-Apifier.main(async (context) => {+Apify.main(async (context) => {const result = await request('http://www.example.com');return result;});```-Note that the `Apifier.main()` function does not need to be used at all,+Note that the `Apify.main()` function does not need to be used at all,it is provided merely for user convenience. The same activitycan be performed using the lower-level functions described in the following sections.### Context-The user function passed to `Apifier.main()` accepts a single+The user function passed to `Apify.main()` accepts a singleargument called `context` which is an object such as:```javascript@@ -107,7 +112,7 @@ argument called `context` which is an object such as:userId: String,// Authentication token representing privileges given to the act run,- // it can be passed to various Apifier APIs.+ // it can be passed to various Apify APIs.token: String,// Date when the act was started@@ -119,7 +124,7 @@ argument called `context` which is an object such as:// ID of the key-value store where input and output data of this act is storeddefaultKeyValueStoreId: String,- // Input data for the act as provided by Apifier.getInput()+ // Input data for the act as provided by Apify.getInput()input: {body: Object,contentType: String,@@ -133,12 +138,12 @@ argument called `context` which is an object such as:The values of the objects are determined from process environment variables,such as `APIFY_INTERNAL_PORT` or `APIFY_STARTED_AT`, and the input is obtained by calling the-`Apifier.getInput()` function.+`Apify.getInput()` function.The `context` object can also be obtained as follows:```javascript-const context = await Apifier.getContext();+const context = await Apify.getContext();console.dir(context);```@@ -146,7 +151,7 @@ console.dir(context);Each act can have an input and output data record, which is raw datawith a specific MIME content type.-Both input and output is stored in the Apifier key-value store created specifically for the act run,+Both input and output is stored in the Apify key-value store created specifically for the act run,under keys named `INPUT` and `OUTPUT`, respectively.The ID of the key-value store is provided by the Actor runtime as the `APIFY_DEFAULT_KEY_VALUE_STORE_ID`environment variable.@@ -154,7 +159,7 @@ environment variable.To obtain the input of the act, use the following code:```javascript-const input = await Apifier.getValue('INPUT');+const input = await Apify.getValue('INPUT');console.dir(`My input is: ${input}``);```@@ -167,29 +172,29 @@ Similarly, the output can be stored as follows:const output = {someValue: 123};-await Apifier.setValue('OUTPUT', output);+await Apify.setValue('OUTPUT', output);```This is especially useful if the output of the act is not a JSON,-because return value from `Apifier.main()` is always converted to JSON.+because return value from `Apify.main()` is always converted to JSON.In such case, make sure the main function returns `undefined` or `null`.### Browser-Apifier runtime optionally depends on+Apify runtime optionally depends onthe [selenium-webdriver](https://www.npmjs.com/package/selenium-webdriver) package that enablesautomation of a web browser.-The simplest way to launch a new web browser is using the `Apifier.browse([url,] [options,] [callback])`+The simplest way to launch a new web browser is using the `Apify.browse([url,] [options,] [callback])`function. For example:```javascript-const browser = await Apifier.browse('https://www.example.com/');+const browser = await Apify.browse('https://www.example.com/');```or```javascript-const browser = await Apifier.browse({+const browser = await Apify.browse({url: 'https://www.example.com/',userAgent: 'MyCrawlingBot/1.23',});@@ -199,7 +204,7 @@ The `options` parameter controls settings of the web browser and it has the foll```javascript{- // Initial URL to open. Note that the url argument in Apifier.browse() overrides this value.+ // Initial URL to open. Note that the url argument in Apify.browse() overrides this value.// The default value is 'about:blank'url: String,@@ -223,10 +228,10 @@ The `options` parameter controls settings of the web browser and it has the foll}```-The result of the `Apifier.browse()` is a new instance of the `Browser` class,+The result of the `Apify.browse()` is a new instance of the `Browser` class,which represents a web browser instance (possibly with multiple windows or tabs).If you pass a Node.js-style callback the `Browser` instance is passed to it,-otherwise the `Apifier.browse()` function returns a promise that resolves to the `Browser` instance.+otherwise the `Apify.browse()` function returns a promise that resolves to the `Browser` instance.The `Browser` class has the following properties:@@ -267,24 +272,24 @@ To set a promise dependency from an external library, use a code such as:```javascriptconst Promise = require('bluebird');-Apifier.setPromisesDependency(Promise);+Apify.setPromisesDependency(Promise);```-If `Apifier.setPromisesDependency()` is not called, the runtime defaults to+If `Apify.setPromisesDependency()` is not called, the runtime defaults tonative promises if they are available, or it throws an error.### Miscellaneous-The `Apifier.client` property contains a reference to the `ApifyClient` instance+The `Apify.client` property contains a reference to the `ApifyClient` instance(from the [apify-client](https://www.npmjs.com/package/apify-client) NPM package),-that is used for all underlying calls to the Apifier API.-The instance is created when the `apifier` package is first imported+that is used for all underlying calls to the Apify API.+The instance is created when the `apify` package is first importedand it is configured using the `APIFY_API_BASE_URL`, `APIFY_USER_ID` and `APIFY_TOKEN`environment variables.-The default settings of the instance can be overridden by calling `Apifier.client.setOptions()` function.+The default settings of the instance can be overridden by calling `Apify.client.setOptions()` function.-`Apifier.events` property contains a reference to an `EventEmitter` instance+`Apify.events` property contains a reference to an `EventEmitter` instancethat is used by Actor runtime to notify your process about various events.This will be used in the future.@@ -311,11 +316,11 @@ server.listen(process.env.APIFY_INTERNAL_PORT|0, (err) => {process.exit(1);}console.log('Hey I am ready');- Apifier.readyFreddy();+ Apify.readyFreddy();});```-Note that by calling `Apifier.readyFreddy()` you tell the Actor runtime that your server is ready to start+Note that by calling `Apify.readyFreddy()` you tell the Actor runtime that your server is ready to startreceiving HTTP requests over the port specified by the `APIFY_INTERNAL_PORT` environment variable.diff --git a/package.json b/package.jsonindex b57ac7f0a..e5eb36bd9 100644--- a/package.json+++ b/package.json@@ -1,7 +1,7 @@{- "name": "apifier",- "version": "0.0.52",- "description": "Apifier runtime for JavaScript",+ "name": "apify",+ "version": "0.4.0",+ "description": "Apify runtime for JavaScript","main": "build/index.js","keywords": ["apify",diff --git a/src/actor.js b/src/actor.jsindex 571606158..8d2ea943b 100644--- a/src/actor.js+++ b/src/actor.js@@ -249,7 +249,7 @@ export const setValue = (key, value, contentType, callback = null) => {* }* }* ```- * The information is generate using the APIFY_XXX environment variables and the input data is fetched from Apifier API.+ * The information is generate using the APIFY_XXX environment variables and the input data is fetched from Apify API.* If some of the variables is not defined or is invalid, the corresponding value in the context object will be null;* an error is not thrown in such a case in order to simplify local development and debugging of acts.* @param callback Optional callback.@@ -343,7 +343,7 @@ export const main = (userFunc) => {};/**- * Notifies Apifier runtime that act is listening on port specified by the APIFY_INTERNAL_PORT environment+ * Notifies Apify runtime that act is listening on port specified by the APIFY_INTERNAL_PORT environment* variable and is ready to receive a HTTP request with act input.*/export const readyFreddy = () => {diff --git a/src/browser.js b/src/browser.jsindex b04970cc8..39a8e93d5 100644--- a/src/browser.js+++ b/src/browser.js@@ -14,7 +14,7 @@ import { ProxyChain } from './proxy_chain';// logging.installConsoleHandler();// logging.getLogger('webdriver.http').setLevel(logging.Level.ALL);-// TODO: on first use of Apifier.browse(), print out the version of Chrome and ChromeDriver+// TODO: on first use of Apify.browse(), print out the version of Chrome and ChromeDriver/**@@ -175,7 +175,7 @@ export class Browser {}/**- * Normalizes arguments for Apifier.browse(), fills correctly default values.+ * Normalizes arguments for Apify.browse(), fills correctly default values.* The function is exported to allow unit testing.* @param url Optional string* @param options Optional object@@ -203,7 +203,7 @@ export const processBrowseArgs = (url, options, callback) => {/**- * Opens a new web browser, which is attached to Apifier debugger so that snapshots are sent to Run console (TODO).+ * Opens a new web browser, which is attached to Apify debugger so that snapshots are sent to Run console (TODO).* Internally, this function calls Selenium WebDrivers's Builder command to create a new WebDriver instance.* (see http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_Builder.html)* The result of the function is a new instance of the Browser class.diff --git a/src/index.js b/src/index.jsindex ce62ab95d..68bb4012c 100644--- a/src/index.js+++ b/src/index.js@@ -7,7 +7,7 @@ import { browse } from './browser';/* globals module */// Publicly available functions-const Apifier = {+const Apify = {main,getContext,getInput,@@ -23,5 +23,5 @@ const Apifier = {};// export this way so that we can import using:-// const Apifier = require('apifier');-module.exports = Apifier;+// const Apify = require('apify');+module.exports = Apify;diff --git a/src/utils.js b/src/utils.jsindex 4ee5984fc..1ced629ab 100644--- a/src/utils.js+++ b/src/utils.js@@ -21,7 +21,7 @@ export const setPromisesDependency = (dep) => {};/**- * Gets the promise dependency set by `Apifier.setPromisesDependency`.+ * Gets the promise dependency set by `Apify.setPromisesDependency`.*/export const getPromisesDependency = () => {return PromisesDependency;@@ -35,12 +35,12 @@ export const getPromisesDependency = () => {export const getPromisePrototype = () => {if (PromisesDependency) {if (typeof (PromisesDependency.resolve) !== 'function') {- throw new Error('The promise dependency set using Apifier.setPromisesDependency() does not define resolve() function.');+ throw new Error('The promise dependency set using Apify.setPromisesDependency() does not define resolve() function.');}return PromisesDependency;}if (typeof Promise === 'function') return Promise;- throw new Error('Native promises are not available, please call Apifier.setPromisesDependency() to set a promise library.');+ throw new Error('Native promises are not available, please call Apify.setPromisesDependency() to set a promise library.');};/**diff --git a/test/actor.js b/test/actor.jsindex c65e2d83f..073612225 100644--- a/test/actor.js+++ b/test/actor.js@@ -6,7 +6,7 @@ import tmp from 'tmp';import Promise from 'bluebird';// NOTE: test use of require() here because this is how its done in acts-const Apifier = require('../build/index');+const Apify = require('../build/index');/* global process, describe, it */@@ -86,7 +86,7 @@ const testMain = (method, bodyRaw, contentType, userFunc, expectedExitCode = 0)});}, 20);- Apifier.main((opts) => {+ Apify.main((opts) => {// console.dir(opts);try {expect(opts.input.method).to.equal(method);@@ -125,7 +125,7 @@ const testMain = (method, bodyRaw, contentType, userFunc, expectedExitCode = 0)*//**- * Helper function that enables testing of Apifier.main()+ * Helper function that enables testing of Apify.main()* @return Promise*/const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputException, expectedOutput }) => {@@ -137,8 +137,8 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputE.once().returns();- // Mock Apifier.client.keyValueStores.getRecord() to test act input- const kvStoresMock = sinon.mock(Apifier.client.keyValueStores);+ // Mock Apify.client.keyValueStores.getRecord() to test act input+ const kvStoresMock = sinon.mock(Apify.client.keyValueStores);if (mockInputException) {kvStoresMock.expects('getRecord')@@ -148,7 +148,7 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputE.expects('getRecord').withExactArgs({storeId: context.defaultKeyValueStoreId,- promise: Apifier.getPromisesDependency(),+ promise: Apify.getPromisesDependency(),key: 'INPUT',}).returns(Promise.resolve(context.input))@@ -159,7 +159,7 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputE.never();}- // Mock Apifier.client.keyValueStores.putRecord() to test act output+ // Mock Apify.client.keyValueStores.putRecord() to test act outputif (mockOutputException) {kvStoresMock.expects('putRecord')@@ -169,7 +169,7 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputE.expects('putRecord').withExactArgs({storeId: context.defaultKeyValueStoreId,- promise: Apifier.getPromisesDependency(),+ promise: Apify.getPromisesDependency(),key: 'OUTPUT',contentType: expectedOutput.contentType,body: expectedOutput.body,@@ -194,7 +194,7 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputE// Invoke main() function, the promise resolves after the user function is run// Note that if mockInputException is set, then user function will never get called!if (!mockInputException) {- Apifier.main((realContext) => {+ Apify.main((realContext) => {try {expect(realContext).to.eql(context);// Wait for all tasks in Node.js event loop to finish@@ -207,7 +207,7 @@ const testMain = ({ userFunc, context, exitCode, mockInputException, mockOutputEif (userFunc) return userFunc(realContext);});} else {- Apifier.main(() => {});+ Apify.main(() => {});resolve();}});@@ -275,12 +275,12 @@ const setContextToEnv = (context) => {if (context.defaultKeyValueStoreId) process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = context.defaultKeyValueStoreId;};-describe('Apifier.getContext()', () => {+describe('Apify.getContext()', () => {it('works with null values', () => {const expectedContext = getEmptyContext();setContextToEnv(expectedContext);- return Apifier.getContext()+ return Apify.getContext().then((context) => {expect(context).to.eql(expectedContext);});@@ -300,7 +300,7 @@ describe('Apifier.getContext()', () => {});setContextToEnv(expectedContext);- return Apifier.getContext()+ return Apify.getContext().then((context) => {expect(context).to.eql(expectedContext);});@@ -317,7 +317,7 @@ describe('Apifier.getContext()', () => {setContextToEnv(expectedContext);return new Promise((resolve, reject) => {- Apifier.getContext((err, context) => {+ Apify.getContext((err, context) => {if (err) reject(err);resolve(context);});@@ -344,12 +344,12 @@ describe('Apifier.getContext()', () => {};setContextToEnv(expectedContext);- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);const expectation = mock.expects('getRecord');expectation.withExactArgs({storeId: expectedContext.defaultKeyValueStoreId,- promise: Apifier.getPromisesDependency(),+ promise: Apify.getPromisesDependency(),key: 'INPUT',}).once()@@ -357,7 +357,7 @@ describe('Apifier.getContext()', () => {return Promise.resolve().then(() => {- return Apifier.getContext();+ return Apify.getContext();}).then((context) => {expect(context).to.eql(expectedContext);@@ -369,9 +369,9 @@ describe('Apifier.getContext()', () => {});});-describe('Apifier.getInput()', () => {+describe('Apify.getInput()', () => {it('supports both promises and callbacks (on success)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').twice().returns(Promise.resolve(null));@@ -379,7 +379,7 @@ describe('Apifier.getInput()', () => {return Promise.resolve().then(() => {// test promise- return Apifier.getInput();+ return Apify.getInput();}).then((input) => {expect(input).to.be.eql(null);@@ -387,7 +387,7 @@ describe('Apifier.getInput()', () => {.then(() => {// test callbackreturn new Promise((resolve, reject) => {- return Apifier.getInput((err, input) => {+ return Apify.getInput((err, input) => {if (err) return reject(err);resolve(input);});@@ -404,7 +404,7 @@ describe('Apifier.getInput()', () => {});it('supports both promises and callbacks (on error)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').twice().throws(new Error('Test error'));@@ -412,7 +412,7 @@ describe('Apifier.getInput()', () => {return Promise.resolve().then(() => {// test promise- return Apifier.getInput();+ return Apify.getInput();}).catch((err) => {expect(err.message).to.be.eql('Test error');@@ -420,7 +420,7 @@ describe('Apifier.getInput()', () => {.then(() => {// test callbackreturn new Promise((resolve, reject) => {- return Apifier.getInput((err, input) => {+ return Apify.getInput((err, input) => {if (err) return reject(err);resolve(input);});@@ -438,14 +438,14 @@ describe('Apifier.getInput()', () => {});it('returns null on undefined keyValueStores.getRecord() result', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').once().returns(Promise.resolve(undefined));return Promise.resolve().then(() => {- return Apifier.getInput();+ return Apify.getInput();}).then((input) => {expect(input).to.be.eql(null);@@ -457,14 +457,14 @@ describe('Apifier.getInput()', () => {});it('fails on invalid keyValueStores.getRecord() result', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').once().returns(Promise.resolve({ invalid: 'bla bla' }));return Promise.resolve().then(() => {- return Apifier.getInput();+ return Apify.getInput();}).then(() => {expect.fail();@@ -481,10 +481,10 @@ describe('Apifier.getInput()', () => {});});-describe('Apifier.main()', () => {+describe('Apify.main()', () => {it('throws on invalid args', () => {expect(() => {- Apifier.main();+ Apify.main();}).to.throw(Error);});@@ -635,49 +635,49 @@ describe('Apifier.main()', () => {});});-describe('Apifier.setOutput()', () => {+describe('Apify.setOutput()', () => {it('throws on invalid args', () => {expect(() => {- Apifier.setOutput();+ Apify.setOutput();}).to.throw(Error);expect(() => {- Apifier.setOutput('bla bla');+ Apify.setOutput('bla bla');}).to.throw(Error);expect(() => {- Apifier.setOutput(1234);+ Apify.setOutput(1234);}).to.throw(Error);expect(() => {- Apifier.setOutput({});+ Apify.setOutput({});}).to.throw(Error);expect(() => {- Apifier.setOutput({ body: undefined, contentType: 'test' });+ Apify.setOutput({ body: undefined, contentType: 'test' });}).to.throw(Error);expect(() => {- Apifier.setOutput({ body: {}, contentType: 456 });+ Apify.setOutput({ body: {}, contentType: 456 });}).to.throw(Error);expect(() => {- Apifier.setOutput({ body: {}, contentType: undefined });+ Apify.setOutput({ body: {}, contentType: undefined });}).to.throw(Error);});});-describe('Apifier.getValue()', () => {+describe('Apify.getValue()', () => {it('throws on invalid args', () => {const keyErrMsg = 'The "key" parameter must be a non-empty string';- expect(() => { Apifier.getValue(); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.getValue({}); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.getValue(''); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.getValue(null); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.getValue(); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.getValue({}); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.getValue(''); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.getValue(null); }).to.throw(Error, keyErrMsg);});it('supports both promises and callbacks (on success)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').twice().returns(Promise.resolve(null));@@ -685,7 +685,7 @@ describe('Apifier.getValue()', () => {return Promise.resolve().then(() => {// test promise- return Apifier.getValue('INPUT');+ return Apify.getValue('INPUT');}).then((input) => {expect(input).to.be.eql(null);@@ -693,7 +693,7 @@ describe('Apifier.getValue()', () => {.then(() => {// test callbackreturn new Promise((resolve, reject) => {- return Apifier.getValue('INPUT', (err, input) => {+ return Apify.getValue('INPUT', (err, input) => {if (err) return reject(err);resolve(input);});@@ -710,7 +710,7 @@ describe('Apifier.getValue()', () => {});it('supports both promises and callbacks (on error)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').twice().throws(new Error('Test error'));@@ -718,7 +718,7 @@ describe('Apifier.getValue()', () => {return Promise.resolve().then(() => {// test promise- return Apifier.getValue('INPUT');+ return Apify.getValue('INPUT');}).catch((err) => {expect(err.message).to.be.eql('Test error');@@ -726,7 +726,7 @@ describe('Apifier.getValue()', () => {.then(() => {// test callbackreturn new Promise((resolve, reject) => {- return Apifier.getValue('INPUT', (err, input) => {+ return Apify.getValue('INPUT', (err, input) => {if (err) return reject(err);resolve(input);});@@ -744,14 +744,14 @@ describe('Apifier.getValue()', () => {});it('returns null on undefined keyValueStores.getRecord() result', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').once().returns(Promise.resolve(undefined));return Promise.resolve().then(() => {- return Apifier.getValue('BLA BLA');+ return Apify.getValue('BLA BLA');}).then((input) => {expect(input).to.be.eql(null);@@ -763,14 +763,14 @@ describe('Apifier.getValue()', () => {});it('fails on invalid keyValueStores.getRecord() result', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('getRecord').once().returns(Promise.resolve({ invalid: 'bla bla' }));return Promise.resolve().then(() => {- return Apifier.getValue('ANYTHING REALLY');+ return Apify.getValue('ANYTHING REALLY');}).then(() => {expect.fail();@@ -787,42 +787,42 @@ describe('Apifier.getValue()', () => {});});-describe('Apifier.setValue()', () => {+describe('Apify.setValue()', () => {it('throws on invalid args', () => {const keyErrMsg = 'The "key" parameter must be a non-empty string';- expect(() => { Apifier.setValue(); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.setValue('', null); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.setValue('', 'some value'); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.setValue({}, 'some value'); }).to.throw(Error, keyErrMsg);- expect(() => { Apifier.setValue(123, 'some value'); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.setValue(); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.setValue('', null); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.setValue('', 'some value'); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.setValue({}, 'some value'); }).to.throw(Error, keyErrMsg);+ expect(() => { Apify.setValue(123, 'some value'); }).to.throw(Error, keyErrMsg);const valueErrMsg = 'The "value" parameter must be a String or Buffer when "contentType" is specified';- expect(() => { Apifier.setValue('key', {}, 'image/png'); }).to.throw(Error, valueErrMsg);- expect(() => { Apifier.setValue('key', 12345, 'image/png'); }).to.throw(Error, valueErrMsg);- expect(() => { Apifier.setValue('key', () => {}, 'image/png'); }).to.throw(Error, valueErrMsg);+ expect(() => { Apify.setValue('key', {}, 'image/png'); }).to.throw(Error, valueErrMsg);+ expect(() => { Apify.setValue('key', 12345, 'image/png'); }).to.throw(Error, valueErrMsg);+ expect(() => { Apify.setValue('key', () => {}, 'image/png'); }).to.throw(Error, valueErrMsg);const circularObj = {};circularObj.xxx = circularObj;const jsonErrMsg = 'The "value" parameter cannot be stringified to JSON';- expect(() => { Apifier.setValue('key', circularObj, null); }).to.throw(Error, jsonErrMsg);- expect(() => { Apifier.setValue('key', undefined); }).to.throw(Error, jsonErrMsg);- expect(() => { Apifier.setValue('key', () => {}); }).to.throw(Error, jsonErrMsg);- expect(() => { Apifier.setValue('key'); }).to.throw(Error, jsonErrMsg);+ expect(() => { Apify.setValue('key', circularObj, null); }).to.throw(Error, jsonErrMsg);+ expect(() => { Apify.setValue('key', undefined); }).to.throw(Error, jsonErrMsg);+ expect(() => { Apify.setValue('key', () => {}); }).to.throw(Error, jsonErrMsg);+ expect(() => { Apify.setValue('key'); }).to.throw(Error, jsonErrMsg);const contTypeRedundantErrMsg = 'The "contentType" parameter must not be used when removing the record';- expect(() => { Apifier.setValue('key', null, 'image/png'); }).to.throw(Error, contTypeRedundantErrMsg);- expect(() => { Apifier.setValue('key', null, ''); }).to.throw(Error, contTypeRedundantErrMsg);- expect(() => { Apifier.setValue('key', null, {}); }).to.throw(Error, contTypeRedundantErrMsg);+ expect(() => { Apify.setValue('key', null, 'image/png'); }).to.throw(Error, contTypeRedundantErrMsg);+ expect(() => { Apify.setValue('key', null, ''); }).to.throw(Error, contTypeRedundantErrMsg);+ expect(() => { Apify.setValue('key', null, {}); }).to.throw(Error, contTypeRedundantErrMsg);const contTypeStringErrMsg = 'The "contentType" parameter must be a non-empty string, null or undefined';- expect(() => { Apifier.setValue('key', 'value', 123); }).to.throw(Error, contTypeStringErrMsg);- expect(() => { Apifier.setValue('key', 'value', {}); }).to.throw(Error, contTypeStringErrMsg);- expect(() => { Apifier.setValue('key', 'value', new Date()); }).to.throw(Error, contTypeStringErrMsg);- expect(() => { Apifier.setValue('key', 'value', ''); }).to.throw(Error, contTypeStringErrMsg);+ expect(() => { Apify.setValue('key', 'value', 123); }).to.throw(Error, contTypeStringErrMsg);+ expect(() => { Apify.setValue('key', 'value', {}); }).to.throw(Error, contTypeStringErrMsg);+ expect(() => { Apify.setValue('key', 'value', new Date()); }).to.throw(Error, contTypeStringErrMsg);+ expect(() => { Apify.setValue('key', 'value', ''); }).to.throw(Error, contTypeStringErrMsg);});it('supports both promises and callbacks (on success)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('putRecord').exactly(4).returns(Promise.resolve(null));@@ -830,16 +830,16 @@ describe('Apifier.setValue()', () => {return Promise.resolve().then(() => {// test promise (no content type)- return Apifier.setValue('mykey', { someValue: 123 });+ return Apify.setValue('mykey', { someValue: 123 });}).then(() => {// test promise (with content type)- return Apifier.setValue('mykey', 'value', 'text/plain');+ return Apify.setValue('mykey', 'value', 'text/plain');}).then(() => {// test callback (no content type)return new Promise((resolve, reject) => {- return Apifier.setValue('mykey', { someValue: 123 }, (err) => {+ return Apify.setValue('mykey', { someValue: 123 }, (err) => {if (err) return reject(err);resolve();});@@ -848,7 +848,7 @@ describe('Apifier.setValue()', () => {.then(() => {// test callback (with content type)return new Promise((resolve, reject) => {- return Apifier.setValue('mykey', 'myvalue', 'text/plain', (err) => {+ return Apify.setValue('mykey', 'myvalue', 'text/plain', (err) => {if (err) return reject(err);resolve();});@@ -863,7 +863,7 @@ describe('Apifier.setValue()', () => {});it('supports both promises and callbacks (on error)', () => {- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('putRecord').twice().throws(new Error('Test error'));@@ -871,7 +871,7 @@ describe('Apifier.setValue()', () => {return Promise.resolve().then(() => {// test promise- return Apifier.setValue('mykey', { someValue: 1 });+ return Apify.setValue('mykey', { someValue: 1 });}).catch((err) => {expect(err.message).to.be.eql('Test error');@@ -879,7 +879,7 @@ describe('Apifier.setValue()', () => {.then(() => {// test callbackreturn new Promise((resolve, reject) => {- return Apifier.setValue('mykey', { someValue: 1 }, (err) => {+ return Apify.setValue('mykey', { someValue: 1 }, (err) => {if (err) return reject(err);resolve();});@@ -903,9 +903,9 @@ describe('Apifier.setValue()', () => {process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = storeId;- Apifier.setPromisesDependency(Promise);+ Apify.setPromisesDependency(Promise);- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('putRecord').once().withArgs({@@ -920,7 +920,7 @@ describe('Apifier.setValue()', () => {return Promise.resolve().then(() => {- return Apifier.setValue(key, value);+ return Apify.setValue(key, value);}).then(() => {mock.verify();@@ -938,9 +938,9 @@ describe('Apifier.setValue()', () => {process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = storeId;- Apifier.setPromisesDependency(Promise);+ Apify.setPromisesDependency(Promise);- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('putRecord').once().withArgs({@@ -955,7 +955,7 @@ describe('Apifier.setValue()', () => {return Promise.resolve().then(() => {- return Apifier.setValue(key, value, contentType);+ return Apify.setValue(key, value, contentType);}).then(() => {mock.verify();@@ -973,9 +973,9 @@ describe('Apifier.setValue()', () => {process.env.APIFY_DEFAULT_KEY_VALUE_STORE_ID = storeId;- Apifier.setPromisesDependency(Promise);+ Apify.setPromisesDependency(Promise);- const mock = sinon.mock(Apifier.client.keyValueStores);+ const mock = sinon.mock(Apify.client.keyValueStores);mock.expects('putRecord').once().withArgs({@@ -990,7 +990,7 @@ describe('Apifier.setValue()', () => {return Promise.resolve().then(() => {- return Apifier.setValue(key, value, contentType);+ return Apify.setValue(key, value, contentType);}).then(() => {mock.verify();@@ -1001,12 +1001,12 @@ describe('Apifier.setValue()', () => {});});-describe('Apifier.events', () => {+describe('Apify.events', () => {it('is there and works as EventEmitter', () => {return new Promise((resolve, reject) => {try {- Apifier.events.on('foo', resolve);- Apifier.events.emit('foo', 'test event');+ Apify.events.on('foo', resolve);+ Apify.events.emit('foo', 'test event');} catch (e) {reject(e);}@@ -1018,10 +1018,10 @@ describe('Apifier.events', () => {});-describe('Apifier.readyFreddy()', () => {+describe('Apify.readyFreddy()', () => {it('it works as expected', () => {process.env.APIFY_WATCH_FILE = createWatchFile();- Apifier.readyFreddy();+ Apify.readyFreddy();return testWatchFileWillBecomeEmpty(process.env.APIFY_WATCH_FILE, 1000);});});diff --git a/test/browser.js b/test/browser.jsindex 9d1453dd2..56ffe6551 100644--- a/test/browser.js+++ b/test/browser.js@@ -8,7 +8,7 @@ import Promise from 'bluebird';// import fs from 'fs';import { processBrowseArgs, getDefaultBrowseOptions } from '../build/browser';-import Apifier from '../build/index';+import Apify from '../build/index';/* globals process */@@ -167,26 +167,26 @@ describe('processBrowseArgs()', () => {});-describe('Apifier.browse()', function () {+describe('Apify.browse()', function () {// Need a large timeout to run unit tests on Travis CIthis.timeout(300 * 1000);it('throws on invalid args', () => {assert.throws(() => {- Apifier.browse('http://www.blabla.bla', { proxyUrl: 'invalidurl' });+ Apify.browse('http://www.blabla.bla', { proxyUrl: 'invalidurl' });}, Error);assert.throws(() => {- Apifier.browse('http://www.blabla.bla', { proxyUrl: 'http://host-without-port' });+ Apify.browse('http://www.blabla.bla', { proxyUrl: 'http://host-without-port' });}, Error);assert.throws(() => {- Apifier.browse('http://www.blabla.bla', { proxyUrl: 'invalid://somehost:1234' });+ Apify.browse('http://www.blabla.bla', { proxyUrl: 'invalid://somehost:1234' });}, Error);});it('opens about:blank with no args', () => {process.env.APIFY_HEADLESS = '1';let browser;- return Apifier.browse()+ return Apify.browse().then((res) => {browser = res;expect(browser.constructor.name).to.eql('Browser');@@ -201,7 +201,7 @@ describe('Apifier.browse()', function () {it('opens https://www.example.com in headless mode', () => {delete process.env.APIFY_HEADLESS;let browser;- return Apifier.browse('https://www.example.com', { headless: true })+ return Apify.browse('https://www.example.com', { headless: true }).then((res) => {browser = res;expect(browser.constructor.name).to.eql('Browser');@@ -218,7 +218,7 @@ describe('Apifier.browse()', function () {return new Promise((resolve, reject) => {try {process.env.APIFY_HEADLESS = '1';- const retVal = Apifier.browse('about:blank', {}, (err, result) => {+ const retVal = Apify.browse('about:blank', {}, (err, result) => {if (err) return reject(err);browser = result;try {@@ -233,7 +233,7 @@ describe('Apifier.browse()', function () {reject(e);}});- assert(!retVal, 'Apifier.browse() with callback should return false-ish value');+ assert(!retVal, 'Apify.browse() with callback should return false-ish value');} catch (e) {reject(e);}@@ -251,7 +251,7 @@ describe('Apifier.browse()', function () {browserName: 'chrome',proxyUrl: `http://${proxyAuth.username}:${proxyAuth.password}@127.0.0.1:${proxyPort}`,};- return Apifier.browse(opts)+ return Apify.browse(opts).then((result) => {browser = result;})@@ -280,7 +280,7 @@ describe('Apifier.browse()', function () {browserName: 'chrome',userAgent: 'MyUserAgent/1234',};- return Apifier.browse(opts)+ return Apify.browse(opts).then((result) => {browser = result;})@@ -297,7 +297,7 @@ describe('Apifier.browse()', function () {/*it('test xxx', () => {// const _ = require('underscore');- // const Apifier = require('apifier');+ // const Apify = require('apify');// const request = require('request-promise');@@ -305,13 +305,13 @@ describe('Apifier.browse()', function () {let browser;- return Apifier.browse({+ return Apify.browse({url: 'https://www.momondo.co.uk/flightsearch/?Search=true&TripType=2&SegNo=2&SO0=LHR&SD0=SGN&SDP0=11-10-2017&SO1=SGN&SD1=LHR&SDP1=18-10-2017&AD=1&TK=ECO&DO=false&NA=false',//url: 'http://www.example.com',userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/60.0.3112.32 Safari/537.36',- proxyUrl: 'http://apifier:[email protected]:60000',+ proxyUrl: 'http://something.com',headless: true,//extraChromeArguments: ['--reduce-security-for-testing', '--disable-web-security'],})diff --git a/test/utils.js b/test/utils.jsindex 035072453..336cfe432 100644--- a/test/utils.js+++ b/test/utils.js@@ -3,34 +3,34 @@ import urlModule from 'url';import BluebirdPromise from 'bluebird';import { expect } from 'chai';import * as utils from '../build/utils';-import Apifier from '../build/index';+import Apify from '../build/index';/* global process, describe, it */// TODO: run tests against build scripts too!-describe('Apifier.xxxPromisesDependency()', () => {+describe('Apify.xxxPromisesDependency()', () => {it('should throw on invalid args', () => {- expect(() => { Apifier.setPromisesDependency('test'); }).to.throw(Error);- expect(() => { Apifier.setPromisesDependency({}); }).to.throw(Error);- expect(() => { Apifier.setPromisesDependency(123); }).to.throw(Error);- expect(() => { Apifier.setPromisesDependency(); }).to.throw(Error);- expect(() => { Apifier.setPromisesDependency(undefined); }).to.throw(Error);+ expect(() => { Apify.setPromisesDependency('test'); }).to.throw(Error);+ expect(() => { Apify.setPromisesDependency({}); }).to.throw(Error);+ expect(() => { Apify.setPromisesDependency(123); }).to.throw(Error);+ expect(() => { Apify.setPromisesDependency(); }).to.throw(Error);+ expect(() => { Apify.setPromisesDependency(undefined); }).to.throw(Error);});it('should work as expected', () => {- Apifier.setPromisesDependency(null);- expect(Apifier.getPromisesDependency()).to.be.a('null');+ Apify.setPromisesDependency(null);+ expect(Apify.getPromisesDependency()).to.be.a('null');expect(utils.newPromise()).to.have.property('then');// Check native promise- Apifier.setPromisesDependency(Promise);- expect(Apifier.getPromisesDependency()).to.equal(Promise);+ Apify.setPromisesDependency(Promise);+ expect(Apify.getPromisesDependency()).to.equal(Promise);expect(utils.newPromise()).to.have.property('then');// Check bluebird- Apifier.setPromisesDependency(BluebirdPromise);- expect(Apifier.getPromisesDependency()).to.equal(BluebirdPromise);+ Apify.setPromisesDependency(BluebirdPromise);+ expect(Apify.getPromisesDependency()).to.equal(BluebirdPromise);expect(utils.newPromise()).to.have.property('then');});});