Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Better default args for Apifier.browse(), added unit tests
Jan Curna91195e802c4
diff --git a/README.md b/README.mdindex 1370d9248..bf21f6aee 100644--- a/README.md+++ b/README.md@@ -181,18 +181,21 @@ because return value from `Apifier.main()` is always converted to JSON.Apifier runtime optionally depends onthe [selenium-webdriver](https://www.npmjs.com/package/selenium-webdriver) package that enablesautomation of a web browser.-The simplest way to launch a new web browser process is to call:+The simplest way to launch a new web browser is using the `Apifier.browse([url,] [options,] [callback])`+function. For example:```javascript-const browser = await Apifier.browse('https://www.example.com/', { browserName: 'chrome' });+const browser = await Apifier.browse('https://www.example.com/');```-The first parameter is the URL of the initial page, by default it is `about:blank`.-The second parameter is an optional object that defines options for the web browser.-It has the following properties:+The `options` parameter controls settings of the web browser and it has the following properties:```javascript{+ // Initial URL to open. Note that the url argument in Apifier.browse() overrides this value.+ // By default it is 'about:blank'+ url: String,+// The type of the web browser to use.// See https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities for possible options.// By default it is 'chrome', which is currently the only fully-supported browser.@@ -212,19 +215,32 @@ It has the following properties:}```-The `Apifier.browse()` function returns a promise resolving to a new instance of the `Browser` class,+The result of the `Apifier.browse()` is a new instance of the `Browser` class,which represents a web browser instance (possibly with multiple windows or tabs).-Currently it only contains the `webDriver` property,-which is an instance of Selenium's-[WebDriver](http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html)-class:+If you pass a Node.js-style callback the `Browser` instance is passed to it,+otherwise the `Apifier.browse()` function returns a promise that resolves to the `Browser` instance.++The `Browser` class has the following properties:```javascript-const ulr = await browser.webDriver.getCurrentUrl();+{+ // An instance of the Selenium's WebDriver class.+ webDriver: Object,++ // A method that closes the web browser and releases associated resources.+ // The method has no arguments and returns a promise that resolves when the browser was closed.+ close: Function,+}++The `webDriver` property can be used to manipulated the web browser:++```javascript+const url = await browser.webDriver.getCurrentUrl();```-Additionally, the `Browser` class defines the `close()` method which closes the web browser process.-It returns a promise that resolves when the browser was closed:+For more information, see [WebDriver documentation](http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html).++When the web browser is no longer needed, it should be closed:```javascriptawait browser.close();diff --git a/src/browser.js b/src/browser.jsindex 690f7cfe3..8c9cd4b0a 100644--- a/src/browser.js+++ b/src/browser.js@@ -3,6 +3,7 @@ import { APIFY_ENV_VARS } from './constants';import { newPromise, nodeifyPromise, parseUrl } from './utils';import { ProxyChain } from './proxy_chain';+/* global process, require */// interesting resources:@@ -82,11 +83,11 @@ export class Browser {* Initializes the browser.* @return Promise*/- initialize() {+ _initialize() {// logging.installConsoleHandler();return newPromise().then(() => {- return this.setupProxy();+ return this._setupProxy();}).then(() => {this.webDriver = this.builder@@ -107,7 +108,7 @@ export class Browser {* @param capabilities* @param chromeOpts*/- setupProxy() {+ _setupProxy() {if (!this.parsedProxyUrl) return null;// NOTE: to view effective proxy settings in Chrome, open chrome://net-internals/#proxy@@ -170,30 +171,57 @@ export class Browser {}}+/**+ * Normalizes arguments for Apifier.browse(), fills correctly default values.+ * The function is exported to allow unit testing.+ * @param url Optional string+ * @param options Optional object+ * @param callback Optional function+ */+export const processBrowseArgs = (url, options, callback) => {+ if (typeof (url) === 'object' || typeof (url) === 'function') {+ callback = options;+ options = url;+ url = null;+ }+ if (typeof (options) === 'function') {+ callback = options;+ options = null;+ }+ options = Object.assign({}, options);+ options.url = url || options.url || 'about:blank';+ callback = callback || null;++ if (typeof (options.url) !== 'string') throw new Error('Invalid "url" provided.');+ if (callback && typeof (callback) !== 'function') throw new Error('Invalid "callback" provided.');++ return { options, callback };+};+/*** Opens a new web browser, which is attached to Apifier debugger so that snapshots are sent to Run console (TODO).* Internally, this function calls Selenium WebDrivers's Builder command to create a new WebDriver instance.* (see http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_Builder.html)* The result of the function is a new instance of the Browser class.- * @param url The start URL to open. Defaults to about:blank- * @param options Configuration options, their defaults are provided by the getDefaultBrowseOptions() function.+ * @param url Optional start URL to open. Defaults to about:blank+ * @param options Optional settings, their defaults are provided by the getDefaultBrowseOptions() function.* @param callback Optional callback.* @return Returns a promise if no callback was provided, otherwise the return value is not defined.*/-export const browse = (url, options = null, callback = null) => {- url = url || 'about:blank';+export const browse = (url, options, callback) => {+ const args = processBrowseArgs(url, options, callback);- const browser = new Browser(options);- const promise = browser.initialize()+ const browser = new Browser(args.options);+ const promise = browser._initialize().then(() => {- return browser.webDriver.get(url);+ return browser.webDriver.get(args.options.url);}).then(() => {return browser;});- return nodeifyPromise(promise, callback);+ return nodeifyPromise(promise, args.callback);};diff --git a/test/browser.js b/test/browser.jsindex 6a22c22ad..169856672 100644--- a/test/browser.js+++ b/test/browser.js@@ -6,7 +6,7 @@ import portastic from 'portastic';import basicAuthParser from 'basic-auth-parser';import Promise from 'bluebird';-import { getDefaultBrowseOptions } from '../build/browser';+import { processBrowseArgs, getDefaultBrowseOptions } from '../build/browser';import Apifier from '../build/index';/* globals process */@@ -78,6 +78,93 @@ describe('getDefaultBrowseOptions()', () => {});});+describe('processBrowseArgs()', () => {+ it('it handles default parameters well', () => {+ const func = () => {};++ expect(processBrowseArgs()).to.eql({+ options: {+ url: 'about:blank',+ },+ callback: null,+ });++ expect(processBrowseArgs(func)).to.eql({+ options: {+ url: 'about:blank',+ },+ callback: func,+ });++ expect(processBrowseArgs('example.com', func)).to.eql({+ options: {+ url: 'example.com',+ },+ callback: func,+ });++ expect(processBrowseArgs('example.com', { opt: true }, func)).to.eql({+ options: {+ url: 'example.com',+ opt: true,+ },+ callback: func,+ });++ expect(processBrowseArgs('example.com', { opt: true })).to.eql({+ options: {+ url: 'example.com',+ opt: true,+ },+ callback: null,+ });++ expect(processBrowseArgs('example.com', { opt: true }, null)).to.eql({+ options: {+ url: 'example.com',+ opt: true,+ },+ callback: null,+ });++ expect(processBrowseArgs('example.com', { url: 'another.com' }, null)).to.eql({+ options: {+ url: 'example.com',+ },+ callback: null,+ });++ expect(processBrowseArgs({}, null)).to.eql({+ options: {+ url: 'about:blank',+ },+ callback: null,+ });++ expect(processBrowseArgs({ url: 'example.com' }, null)).to.eql({+ options: {+ url: 'example.com',+ },+ callback: null,+ });++ expect(processBrowseArgs({ url: 'example.com' }, func)).to.eql({+ options: {+ url: 'example.com',+ },+ callback: func,+ });++ expect(processBrowseArgs({ some: 123 }, func)).to.eql({+ options: {+ url: 'about:blank',+ some: 123,+ },+ callback: func,+ });+ });+});+describe('Apifier.browse()', function () {// Need a large timeout to run unit tests on Travis CI@@ -158,11 +245,12 @@ describe('Apifier.browse()', function () {let browser;wasProxyCalled = false;const opts = {+ url: 'https://www.example.com',headless: false,browserName: 'chrome',proxyUrl: `http://${proxyAuth.username}:${proxyAuth.password}@127.0.0.1:${proxyPort}`,};- return Apifier.browse('https://www.example.com', opts)+ return Apifier.browse(opts).then((result) => {browser = result;})