Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Feature/docs (#62)
Marek Trunkát614c95101f7a
diff --git a/README.md b/README.mdindex f37ee00a1..bf39cfcbf 100644--- a/README.md+++ b/README.md@@ -1,18 +1,50 @@-# apify: Apify: web scraping and automation SDK for JavaScript / Node.js+# Apify: web scraping and automation SDK for JavaScript / Node.js<!-- Mirror this part to src/index.js -->[](http://badge.fury.io/js/apify)[](https://travis-ci.org/apifytech/apify-js)+<div id="include-readme-1">+ The `apify` NPM package enables development of web scrapers, crawlers and web automation projects+ either locally or running on <a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a> -+ a serverless computing platform that enables execution of arbitrary code in the cloud.+ The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts at Apify platform without it.-The `apify` NPM package enables development of web scrapers and crawlers,-either locally or running on <a href="https://www.apify.com/docs/actor" target="_blank">Apify Actor</a> --a serverless computing platform that enables execution of arbitrary code in the cloud.-The package provides helper functions to launch web browsers with proxies, access the storage etc. Note that the usage of the package is optional, you can create acts without it.+ For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor+</div>Complete documentation of this package is available at https://www.apify.com/docs/sdk/apify-runtime-js/latest-For more information about the Apify Actor platform, please see https://www.apify.com/docs/actor+## Table of Contents++<!-- toc -->++- [Common use-cases](#common-use-cases)+- [Quick start](#quick-start)+- [Puppeteer](#puppeteer)+- [Components](#components)+ * [Storage](#storage)+ + [Key-value store](#key-value-store)+ + [Dataset](#dataset)+ + [Request queue](#request-queue)+ * [Helper Classes](#helper-classes)+ + [Autoscaled Pool](#autoscaled-pool)+ + [Basic Crawler](#basic-crawler)+ + [Puppeteer Crawler](#puppeteer-crawler)+ + [Request List](#request-list)+ + [Puppeteer Pool](#puppeteer-pool)+- [Local usage](#local-usage)+- [Promises vs. callbacks](#promises-vs-callbacks)+- [Examples](#examples)+ * [Recursive crawling](#recursive-crawling)+ * [Crawling url list](#crawling-url-list)+ * [Call to another act](#call-to-another-act)+ * [Act used and synchronous API](#act-used-and-synchronous-api)+ * [Other](#other)++<!-- tocstop -->++<div id="include-readme-2">## Common use-cases<!-- Mirror this part to src/index.js -->@@ -22,42 +54,405 @@ most common use-cases are:<ul><li>- If you need to process high volume of <strong>asynchronous tasks in parallel</strong> then take a- look at <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool">AutoscaledPool</a>. This class executes defined tasks in a pool- which size is scaled based on available memory and CPU.+ If you want to <strong>crawl</strong> a website using for example <a href="https://www.npmjs.com/package/request" target="_blank">+ Request</a> package then take a look at <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler" target="_blank">BasicCrawler</a>+ in combination with <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList" target="_blank">RequestList</a> for fix list of urls+ or <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue" target="_blank">RequestQueue</a> for recursive crawl.+ </li>+ <li>+ If you want to crawl a website using a real <strong>browser</strong>. Then use+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler" target="_blank">PuppeteerCrawler</a> which uses+ <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless/non-headless Chrome browser). PuppeteerCrawler supports+ both <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList" target="_blank">RequestList</a> for fix list of urls+ or <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue" target="_blank">RequestQueue</a> for recursive crawl.</li><li>- If you want to <strong>crawl</strong> a list of urls using for example <a href="https://www.npmjs.com/package/request" target="_blank">- Request</a> package then import those url as a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList">RequestList</a> and then use- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler">BasicCrawler</a> to process them in a pool.+ If you need to process high volume of <strong>asynchronous tasks in parallel</strong> then take a+ look at <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool" target="_blank">AutoscaledPool</a>. This class executes defined tasks in a pool which size is scaled based on available memory and CPU.</li><li>- If you want to crawl a list of urls but you need a real <strong>browser</strong>. Then use- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler">PuppeteerCrawler</a> which helps you to process a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList">RequestList</a>- using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser).+ If you want to automate filling of forms or any other web interaction then you can use+ <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless/non-headless Chrome browser).</li></ul>+If you deploy your code to Apify platform then you can set up scheduler or execute your code with web API.++## Quick start++To use Apify SDK you must have <a href="https://nodejs.org/en/" target="_blank">Node JS</a> (version 7.0.0 or newer) and+<a href="https://www.npmjs.com" target="_blank">NPM</a> installed. If you have both then the easiest way how to start is to use+<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify CLI</a> (command line tool).++Install the tool with:++```bash+npm -g install apify-cli+```++and create your project with:++```bash+apify create my_hello_world++cd my_hello_world+```++Apify CLI asks to you choose a template and then creates a directory `my_hello_world` containing:++- `package.json` with Apify SDK as dependency+- `main.js` containing basic code for your project+- `apify_local` directory containing local emultation of <a href="https://www.apify.com/docs/storage" target="_blank">Apify storage types</a>+- files needed for optional deployment to Apify platform (`Dockerfile`, `apify.json`)+- `node_modules` directory containing all the required NPM packages++If you chose template `Puppeteer` then the `main.js` file looks like:++```javascript+const Apify = require('apify');++Apify.main(async () => {+ const input = await Apify.getValue('INPUT');++ if (!input || !input.url) throw new Error('INPUT must contain a url!');++ console.log('Launching Puppeteer...');+ const browser = await Apify.launchPuppeteer();++ console.log(`Opening page ${input.url}...`);+ const page = await browser.newPage();+ await page.goto(input.url);+ const title = await page.title();+ console.log(`Title of the page "${input.url}" is "${title}".`);++ console.log('Closing Puppeteer...');+ await browser.close();++ console.log('Done.');+});+```++It simply takes a `url` field of its input opens that page using+<a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> in Chrome browser and prints its title.+Input is always stored in default key-value store of run. Local emulation of this store you can find in directory+`apify_local/key-value-stores/default`. To create an input simply create a file `apify_local/key-value-stores/default/INPUT.json`+containing:++```javascript+{+ "url": "https://news.ycombinator.com"+}+```++Now can then run you code with:++```bash+apify run+```++and see following output:++```+Launching Puppeteer...++Opening page https://news.ycombinator.com...++Title of the page "https://news.ycombinator.com" is "Hacker News".++Closing Puppeteer...++Done.+```++Check <a href="#examples">examples</a> below to see what you can do with Apify SDK. After you are done with your code+you can deploy your project to Apify platform with following 2 steps:++```+apify login+apify push+```+## Puppeteer<!-- Mirror this part to src/index.js -->-For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser)+For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless/non-headless Chrome browser)we have few helper classes and functions:<ul><li>- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-launchPuppeteer">Apify.launchPuppeteer()</a> function starts new instance of Puppeteer browser and returns its browser object.+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-launchPuppeteer" target="_blank">launchPuppeteer()</a> function starts new instance of Puppeteer browser and returns its browser object.</li><li>- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool">PuppeteerPool</a> helps to mantain a pool of Puppeteer instances. This is usefull+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool" target="_blank">PuppeteerPool</a> helps to mantain a pool of Puppeteer instances. This is usefullwhen you need to restart browser after certain number of requests to rotate proxy servers.</li><li>- <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler">PuppeteerCrawler</a> helps to crawl a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList">RequestList</a>- in a autoscaled pool.+ <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler" target="_blank">PuppeteerCrawler</a> helps to crawl a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList" target="_blank">RequestList</a> or <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue" target="_blank">RequestQueue</a> in parallel using autoscaled pool.</li></ul>+```javascript+const url = 'https://news.ycombinator.com';++const browser = await Apify.launchPuppeteer();+const page = await browser.newPage();+await page.goto(url);+const title = await page.title();++console.log(`Title of the page "${url}" is "${title}".`);+```++For more information on Puppeteer see its <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">documenation</a>.++</div>++## Components++### Storage++Apify package provides 3 storage types for commons use cases both locally and at Apify platform.++#### Key-value store++Key value store is simple storage that can be used for string or file (buffer) records.++```javascript+// Save value to key-value store.+await Apify.setValue('my-key', { foo: 'bar' });++// Get value from key-value store.+const value = await Apify.getValue('my-key');+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#KeyValueStore" target="_blank">documentation</a>.++#### Dataset++The dataset is a storage that enables saving and retrieval of sequential data objects — typically results of some long running operation such as scraping or data extraction.+Dataset is immutable and allows only storing and retrieving of its items.++```javascript+// Push some data to dataset.+await Apify.pushData({ foo: 'bar' });+await Apify.pushData({ myArray: [1, 2, 3] });+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset" target="_blank">documentation</a>.++#### Request queue++Request queue is used to manage a dynamic queue of web pages to crawl.++```javascript+const queue = await Apify.openRequestQueue('my-queue-id');++await queue.addRequest(new Apify.Request({ url: 'http://example.com/aaa'});+await queue.addRequest(new Apify.Request({ url: 'http://example.com/bbb'});++// Get requests from queue to be processed.+const request1 = queue.fetchNextRequest();+const request2 = queue.fetchNextRequest();++// Mark one of them as handled.+queue.markRequestHandled(request1);++// If processing fails then reclaim it back to the queue.+request2.pushErrorMessage('Request failed for network error!');+queue.reclaimRequest(request2);+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue" target="_blank">documentation</a>.++### Helper Classes++#### Autoscaled Pool++This class manages a pool of asynchronous resource-intensive tasks that are executed in parallel. The pool only starts new tasks if there is enough free memory and CPU capacity. The information about the CPU and memory usage is obtained either from the local system or from the Apify cloud infrastructure in case the process is running on the Apify Actor platform.++```javascript+const pool = new Apify.AutoscaledPool({+ maxConcurrency: 50,+ runTaskFunction: () => {+ // Run some resource-intensive asynchronous operation here and return a promise...+ },+});++await pool.run();+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#AutoscaledPool" target="_blank">documentation</a>.++#### Basic Crawler++Provides a simple framework for parallel crawling of web pages from a list of URLs managed by the RequestList class or dynamically enqueued URLs managed by RequestQueue.++For examples on how to use it see <a href="#examples">examples</a> section below and also check+<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler" target="_blank">documentation</a>.++#### Puppeteer Crawler++Provides a simple framework for parallel crawling of web pages using the Chrome with Puppeteer. The URLs of pages to visit are given by Request objects that are provided by the RequestList class or a dynamically enqueued requests provided by the RequestQueue class.++For examples on how to use it see <a href="#examples">examples</a> section below and also check+<a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler" target="_blank">documentation</a>.++#### Request List++Provides way to handle a list of URLs to be crawled. Each URL is reprented using an instance of the Request class.++```javascript+const requestList = new Apify.RequestList({+ sources: [+ { url: 'http://www.example.com/page-1' },+ { url: 'http://www.example.com/page-2' },+ ],+});++// Get requests from list to be processed.+const request1 = requestList.fetchNextRequest();+const request2 = requestList.fetchNextRequest();++// Mark one of them as handled.+requestList.markRequestHandled(request1);++// If processing fails then reclaim it back to the queue.+request2.pushErrorMessage('Request failed for network error!');+requestList.reclaimRequest(request2);+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList" target="_blank">documentation</a>.++#### Puppeteer Pool++Provides a pool of Puppeteer (Chrome browser) instances. The class rotates the instances based on its configuration in order to change proxies.++```javascript+const puppeteerPool = new PuppeteerPool({ groups: 'some-proxy-group' });++// Open browser pages.+const page1 = await puppeteerPool.newPage();+const page2 = await puppeteerPool.newPage();+const page3 = await puppeteerPool.newPage();++// ... do something with pages ...++// Close all the browsers.+await puppeteerPool.destroy();+```++For more information see complete <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerPool" target="_blank">documentation</a>.++<div id="include-readme-3">++## Local usage++The easiest way how to use apify locally is with <a href="https://github.com/apifytech/apify-cli" target="_blank">Apify CLI</a> as shown+in <a href="#quick-start">quick start</a> section. Other way is to manually define required environment variables:++<table class="table table-bordered table-condensed">+ <thead>+ <tr>+ <th>Environment variable</th>+ <th>Description</th>+ </tr>+ </thead>+ <tbody>+ <tr>+ <td><code>APIFY_LOCAL_EMULATION_DIR</code></td>+ <td>+ Directory where apify package locally emulates Apify storages - key-value store and dataset.+ Key-value stores will be emulated in directory+ <code>[APIFY_LOCAL_EMULATION_DIR]/key-value-stores/[STORE_ID]</code>+ and datasets in directory+ <code>[APIFY_LOCAL_EMULATION_DIR]/datasets/[DATESET_ID]</code>.+ </td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>+ <td>ID of default key-value store.</td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_DATASET_ID</code></td>+ <td>ID of default dataset.</td>+ </tr>+ <tr>+ <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>+ <td>ID of default request queue.</td>+ </tr>+ </tbody>+</table>++Apify will then store key-value store records in files named <code>[KEY].[EXT]</code> where <code>[KEY]</code>+is the record key and <code>[EXT]</code> is based on the record content type. Dataset items will be stored+in files named <code>[ID].json</code> where <code>[ID]</code> is sequence number of your dataset item.+ *+If you want to use <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> locally+then you must define an environment variable <code>PROXY_PASSWORD</code> with password you find at+<a href="https://my.apify.com/proxy" target="_blank">https://my.apify.com/proxy</a>.++## Promises vs. callbacks++By default, all asynchronous functions provided by this package return a promise.+But Apify uses a <a href="http://bluebirdjs.com/" target="_blank">Bluebird</a>+promise implementation so you can easily convert any function that returns a Promise+into callback style function.+See <a href="http://bluebirdjs.com/docs/api/promise.promisify.html" target="_blank">Bluebird documentation</a>+for more information.+## Examples-Directory `/examples` of this repo contains examples of different usages of this package.\ No newline at end of file+Directory <a href="https://github.com/apifytech/apify-js/tree/master/examples" target="_blank">examples</a> of this repository demonstrates different usages of this package.++### Recursive crawling++Following 2 examples demonstrate recursive crawling of <a href="https://news.ycombinator.com" target="_blank">https://news.ycombinator.com</a>.+Crawler starts at https://news.ycombinator.com and in each step enqueues a new page linked by "more" button at the bottom of the page+and stores posts from the opened page in a <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset" target="_blank">Dataset</a>. As a queue crawler uses <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue" target="_blank">Request Queue</a>.++Former example crawls page simply using NPM <a href="https://www.npmjs.com/package/request" target="_blank">Request</a> and+<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> packages and former one uses+<a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> that provides full Chrome browser.++- <a href="https://github.com/apifytech/apify-js/tree/master/examples/crawler_puppeteer.js">Recursive crawl with Puppeteer</a>+- <a href="https://github.com/apifytech/apify-js/tree/master/examples/crawler_cheerio.js">Recursive crawl with Cheerio and Request NPM packages</a>++### Crawling url list++These examples show how to scrape data from a fix list of urls using+<a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> or+<a href="https://www.npmjs.com/package/request" target="_blank">Request</a> and+<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>.++- <a href="https://github.com/apifytech/apify-js/tree/master/examples/url_list_puppeteer.js">Crawling a url list with Puppeteer</a>+- <a href="https://github.com/apifytech/apify-js/tree/master/examples/url_list_cheerio.js">Crawling a url list with Cheerio and Request NPM packages</a>++### Call to another act++This example shows how to call another act at Apify plarform - in this case `apify/send-mail`+to send email.++<a href="https://github.com/apifytech/apify-js/tree/master/examples/call_another_act.js">Check source code here</a>++### Act used and synchronous API++This example shows an act that has short runtime - just few seconds. It opens a webpage+http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate+bridge, takes a screenshot and saves it as output. This makes act executable at Apify platform+synchronously with a single request that also returns its output.++Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam+so you can easily run it with request to+`https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]`+and get image as response. Then you can for example use it directly in html:++```html+<img src="https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]" />+```++<a href="https://github.com/apifytech/apify-js/tree/master/examples/url_list_cheerio.js">Check source code here</a>++### Other++- <a href="https://github.com/apifytech/apify-js/tree/master/examples/autoscaled_pool.js">Use of AutoscaledPool</a>++</div>\ No newline at end of filediff --git a/examples/example_autoscaled_pool.js b/examples/autoscaled_pool.jssimilarity index 92%rename from examples/example_autoscaled_pool.jsrename to examples/autoscaled_pool.jsindex 4b9461acc..421ee69a4 100644--- a/examples/example_autoscaled_pool.js+++ b/examples/autoscaled_pool.js@@ -7,7 +7,7 @@ Apify.main(async () => {// Size of the pool is auto-scaled based on memory and CPU.// If this function returns null then all tasks are considered// to be done.- const workerFunction = () => {+ const runTaskFunction = () => {const current = counter++;if (current >= 1000) return null;@@ -25,7 +25,7 @@ Apify.main(async () => {const pool = new Apify.AutoscaledPool({minConcurrency: 3,maxConcurrency: 50,- workerFunction,+ runTaskFunction,});await pool.run();diff --git a/examples/call_another_act.js b/examples/call_another_act.jsnew file mode 100644index 000000000..219669078--- /dev/null+++ b/examples/call_another_act.js@@ -0,0 +1,32 @@+/**+ * This example shows how to call another act - in this case apify/send-main to send+ * email.+ *+ * BTC is falling down for few months already. If you don't want to miss the chance of+ * of your live then you can use this code to get current BTC prices from Kraken.com+ * and mail them to your mailbox.+ *+ * If you deploy this act to Apify platform then you can setup a scheduler for early+ * morning.+ */++const Apify = require('apify');++const YOUR_MAIL = '[email protected]';++Apify.main(async () => {+ // Start browser.+ const browser = await Apify.launchPuppeteer();++ // Load Kraken and get last traded price of BTC.+ const page = await browser.newPage();+ await page.goto('https://www.kraken.com/charts');+ const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);++ // Send prices to your email.+ await Apify.call('apify/send-mail', {+ to: YOUR_MAIL,+ subject: 'Kraken.com BTC',+ html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,+ });+});diff --git a/examples/crawler_cheerio.js b/examples/crawler_cheerio.jsnew file mode 100644index 000000000..c4a8abc3c--- /dev/null+++ b/examples/crawler_cheerio.js@@ -0,0 +1,71 @@+/**+ * This is example how to scrape Hacker News site (https://news.ycombinator.com) using Apify SDK+ * with Cheerio and Request NPM packages.+ *+ * Example uses:+ * - Apify BasicCrawler to scrape pages in parallel+ * - Apify Dataset to store data+ * - Apify RequestQueue to manage dynamic queue of pending and handled requests+ * - Request NPM package to request html content of website+ * - Cherio NPM package to parse html and extract data+ */++const Apify = require('apify');+const rp = require('request-promise');+const cheerio = require('cheerio');++Apify.main(async () => {+ // Get queue and enqueue first url.+ const requestQueue = await Apify.openRequestQueue();++ // Enqueue Start url.+ await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));++ // Create crawler.+ const crawler = new Apify.BasicCrawler({+ requestQueue,+ disableProxy: true,++ // This page is executed for each request.+ // If request failes then it's retried 3 times.+ handleRequestFunction: async ({ request }) => {+ console.log(`Processing ${request.url}...`);++ // Request html of page.+ const html = await rp(request.url);++ // Extract data with cheerio.+ const data = [];+ const $ = cheerio.load(html);+ $('.athing').each((index, el) => {+ data.push({+ title: $(el).find('.title a').text(),+ rank: $(el).find('.rank').text(),+ href: $(el).find('.title a').attr('href'),+ });+ });++ // Save data.+ await Apify.pushData(data);++ // Enqueue next page.+ const $moreLink = $('.morelink');+ if ($moreLink.length) {+ const path = $moreLink.attr('href')+ const url = `https://news.ycombinator.com/${path}`;++ await requestQueue.addRequest(new Apify.Request({ url }));+ } else {+ console.log(`Url ${request.url} is the last page!`);+ }+ },++ // If request failed 4 times then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`);+ },+ });++ // Run crawler.+ await crawler.run();+});diff --git a/examples/crawler_puppeteer.js b/examples/crawler_puppeteer.jsnew file mode 100644index 000000000..69a7e8b46--- /dev/null+++ b/examples/crawler_puppeteer.js@@ -0,0 +1,67 @@+/**+ * This is example how to scrape Hacker News site (https://news.ycombinator.com) using Apify SDK and Puppeteer.+ *+ * Example uses:+ * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel+ * - Apify Dataset to store data+ * - Apify RequestQueue to manage dynamic queue of pending and handled requests+ * - Puppeter to controll headless Chrome browser+ */++const Apify = require('apify');++Apify.main(async () => {+ // Get queue and enqueue first url.+ const requestQueue = await Apify.openRequestQueue();++ // Enqueue Start url.+ await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));++ // Create crawler.+ const crawler = new Apify.PuppeteerCrawler({+ requestQueue,+ disableProxy: true,++ // This page is executed for each request.+ // If request failes then it's retried 3 times.+ // Parameter page is Puppeteers page object with loaded page.+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);++ // Extract all posts.+ const pageFunction = ($posts) => {+ const data = [];++ $posts.forEach(($post) => {+ data.push({+ title: $post.querySelector('.title a').innerText,+ rank: $post.querySelector('.rank').innerText,+ href: $post.querySelector('.title a').href,+ });+ });++ return data;+ };+ const data = await page.$$eval('.athing', pageFunction);++ // Save data.+ await Apify.pushData(data);++ // Enqueue next page.+ try {+ const nextHref = await page.$eval('.morelink', el => el.href);+ await requestQueue.addRequest(new Apify.Request({ url: nextHref }));+ } catch (err) {+ console.log(`Url ${request.url} is the last page!`);+ }+ },++ // If request failed 4 times then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`);+ },+ });++ // Run crawler.+ await crawler.run();+});diff --git a/examples/example_basic_crawler.js b/examples/example_basic_crawler.jsdeleted file mode 100644index 330fadf34..000000000--- a/examples/example_basic_crawler.js+++ /dev/null@@ -1,51 +0,0 @@-const Apify = require('apify');-const rp = require('request-promise');--Apify.main(async () => {- // Create a request list.- const requestList = new Apify.RequestList({- sources: [- { url: 'http://www.example.com' },- { url: 'http://www.example.com/?page=2' },- { url: 'http://www.example.com/?page=3' },- { url: 'http://www.example.com/?page=4' },- { url: 'http://www.example.com/?page=5' },- ],- });-- await requestList.initialize();-- const crawler = new Apify.BasicCrawler({- requestList,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- handleRequestFunction: async ({ request }) => {- const pageHtml = await rp(request.url);-- console.log(`Request ${request.url} succeeded with return html of length ${pageHtml.length}`);-- await Apify.pushData({- url: request.url,- html: pageHtml,- errors: null,- })- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);-- console.log(request);-- await Apify.pushData({- url: request.url,- html: null,- errors: request.errorMessages,- })- },- });-- // Run crawler for request list.- await crawler.run();-});diff --git a/examples/example_puppeteer_crawler.js b/examples/example_puppeteer_crawler.jsdeleted file mode 100644index 406b08bed..000000000--- a/examples/example_puppeteer_crawler.js+++ /dev/null@@ -1,47 +0,0 @@-const Apify = require('apify');--Apify.main(async () => {- // Create a request list.- const requestList = new Apify.RequestList({- sources: [- { url: 'http://www.example.com' },- { url: 'http://www.some-nonexisting-domain.com' },- ],- });-- await requestList.initialize();-- const crawler = new Apify.PuppeteerCrawler({- requestList,- disableProxy: true,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- // Parameter page is Puppeteers page object with loaded page.- handlePageFunction: async ({ page, request }) => {- const pageTitle = await page.title();-- console.log(`Request ${request.url} succeeded and it's title is ${pageTitle}`);-- await Apify.pushData({- url: request.url,- pageTitle,- errors: null,- })- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);-- await Apify.pushData({- url: request.url,- pageTitle: null,- errors: request.errorMessages,- })- },- });-- // Run crawler for request list.- await crawler.run();-});diff --git a/examples/synchronous_run.js b/examples/synchronous_run.jsnew file mode 100644index 000000000..b08bb67a7--- /dev/null+++ b/examples/synchronous_run.js@@ -0,0 +1,32 @@+/**+ * This example shows shows an act that has short runtime - just few seconds. It opens a webpage+ * http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate+ * bridge, takes a screenshot and saves it as output. This makes act executable at Apify platform+ * synchronously with a single request that also returns its output.+ *+ * Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam+ * so you can easily run it with request to+ * https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]+ */++const Apify = require('apify');++Apify.main(async () => {+ // Start browser.+ const browser = await Apify.launchPuppeteer();++ // Load http://goldengatebridge75.org/news/webcam.html and get an iframe+ // containing webcam stream.+ const page = await browser.newPage();+ await page.goto('http://goldengatebridge75.org/news/webcam.html');+ const iframe = (await page.frames()).pop();++ // Get webcam image element handle.+ const imageElementHandle = await iframe.$('.VideoColm img');++ // Get a screenshot of that image.+ const imageBuffer = await imageElementHandle.screenshot();++ // Save it as an output.+ await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });+});diff --git a/examples/url_list_cheerio.js b/examples/url_list_cheerio.jsnew file mode 100644index 000000000..cb5ca107a--- /dev/null+++ b/examples/url_list_cheerio.js@@ -0,0 +1,67 @@+/**+ * This example shows how to extract data (title and "see also" links) form a list of Wikipedia articles+ * using Cheerio and Request NPM packages.+ *+ * Example uses:+ * - Apify BasicCrawler to scrape pages using Puppeteer in parallel+ * - Apify Dataset to store data+ * - Apify RequestList to manage a list of urls to be processed+ * - Request NPM package to request html content of website+ * - Cherio NPM package to parse html and extract data+ */++const Apify = require('apify');+const rp = require('request-promise');+const cheerio = require('cheerio');++Apify.main(async () => {+ const sources = [+ { url: 'https://en.wikipedia.org/wiki/Amazon_Web_Services' },+ { url: 'https://en.wikipedia.org/wiki/Google_Cloud_Platform' },+ { url: 'https://en.wikipedia.org/wiki/Microsoft_Azure' },+ { url: 'https://en.wikipedia.org/wiki/Rackspace_Cloud' },+ ];++ // Create a request list.+ const requestList = new Apify.RequestList({ sources });+ await requestList.initialize();++ const crawler = new Apify.BasicCrawler({+ requestList,++ // This page is executed for each request.+ // If request failes then it's retried 3 times.+ handleRequestFunction: async ({ request }) => {+ console.log(`Processing ${request.url}...`);++ // Request html of page.+ const html = await rp(request.url);++ // Extract data with cheerio.+ const $ = cheerio.load(html);+ const $seeAlsoElement = $('#See_also').parent().next();+ const seeAlsoLinks = [];+ $seeAlsoElement.find('a').each((index, el) => {+ seeAlsoLinks.push({+ url: $(el).attr('href'),+ text: $(el).text(),+ });+ });++ // Save data.+ await Apify.pushData({+ url: request.url,+ title: $('h1').text(),+ seeAlsoLinks,+ });+ },++ // If request failed 4 times then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`);+ },+ });++ // Run crawler for request list.+ await crawler.run();+});diff --git a/examples/url_list_puppeteer.js b/examples/url_list_puppeteer.jsnew file mode 100644index 000000000..b52b71650--- /dev/null+++ b/examples/url_list_puppeteer.js@@ -0,0 +1,68 @@+/**+ * This example shows how to extract data (title and "see also" links) form a list of Wikipedia articles using Puppeteer.+ *+ * Example uses:+ * - Apify BasicCrawler to scrape pages in parallel+ * - Apify Dataset to store data+ * - Apify RequestList to manage a list of urls to be processed+ * - Puppeter to controll headless Chrome browser+ */++const Apify = require('apify');+const rp = require('request-promise');+const cheerio = require('cheerio');++Apify.main(async () => {+ const sources = [+ { url: 'https://en.wikipedia.org/wiki/Amazon_Web_Services' },+ { url: 'https://en.wikipedia.org/wiki/Google_Cloud_Platform' },+ { url: 'https://en.wikipedia.org/wiki/Microsoft_Azure' },+ { url: 'https://en.wikipedia.org/wiki/Rackspace_Cloud' },+ ];++ // Create a request list.+ const requestList = new Apify.RequestList({ sources });+ await requestList.initialize();++ const crawler = new Apify.PuppeteerCrawler({+ requestList,++ // This page is executed for each request.+ // If request failes then it's retried 3 times.+ // Parameter page is Puppeteers page object with loaded page.+ handlePageFunction: async ({ request, page }) => {+ console.log(`Processing ${request.url}...`);++ // Extract data with Puppeteer.+ const seeAlsoLinks = await page.$eval('#See_also', (seeAlsoTitleEl) => {+ const seeAlsoListEl = seeAlsoTitleEl.parentNode.nextSibling.nextSibling;+ const seeAlsoLinksEls = seeAlsoListEl.querySelectorAll('a');+ const data = [];++ seeAlsoLinksEls.forEach((linkEl) => {+ data.push({+ title: linkEl.innerText,+ url: linkEl.getAttribute('href'),+ });+ });++ return data;+ });++ // Save data.+ await Apify.pushData({+ url: request.url,+ title: await page.$eval('h1', el => el.innerText),+ seeAlsoLinks,+ });+ },++ // If request failed 4 times then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`);+ },+ });++ // Run crawler for request list.+ await crawler.run();+});diff --git a/package.json b/package.jsonindex b6d5438fa..79425b62a 100644--- a/package.json+++ b/package.json@@ -35,9 +35,10 @@"scripts": {"build": "rm -rf ./build && babel src --out-dir build","build-doc": "npm run clean && npm run build && node ./node_modules/jsdoc/jsdoc.js --package ./package.json -c ./jsdoc/conf.json -d docs",+ "build-toc": "./node_modules/.bin/markdown-toc README.md -i","test": "npm run build && mocha --timeout 60000 --compilers js:babel-core/register --recursive","test-cov": "npm run build && babel-node node_modules/isparta/bin/isparta cover --report html --report text node_modules/.bin/_mocha",- "prepare": "npm run build",+ "prepare": "npm run build-toc && npm run build","prepublishOnly": "(test $RUNNING_FROM_SCRIPT || (echo \"You must use publish.sh instead of 'npm publish' directly!\"; exit 1)) && npm test && npm run lint","clean": "rm -rf build","lint": "npm run build && eslint src test"@@ -71,6 +72,7 @@"isparta": "^4.0.0","jsdoc": "^3.4.3","jsdoc-export-default-interop": "^0.3.1",+ "markdown-toc": "^1.2.0","mocha": "^3.5.3","portastic": "^1.0.1","proxy": "^0.2.4",diff --git a/src/events.js b/src/events.jsindex 14c73d63f..12db10c17 100644--- a/src/events.js+++ b/src/events.js@@ -49,12 +49,12 @@ let persistStateInterval = null;* <th>Event</th>* <th>Name</th>* <th>Constant</th>- * <ht>Message</th>- * <ht>Description</th>+ * <th>Message</th>+ * <th>Description</th>* </thead>* <tbody>* <tr>- * <td>cpuInfo</td>+ * <td>`cpuInfo`</td>* <td>`ACTOR_EVENT_NAMES.CPU_INFO`</td>* <td>`{ "isCpuOverloaded": true }`</td>* <td>@@ -63,7 +63,7 @@ let persistStateInterval = null;* </td>* </tr>* <tr>- * <td>migrating</td>+ * <td>`migrating`</td>* <td>`ACTOR_EVENT_NAMES.MIGRATING`</td>* <td>`null`</td>* <td>@@ -72,7 +72,7 @@ let persistStateInterval = null;* </td>* </tr>* <tr>- * <td>persistState</td>+ * <td>`persistState`</td>* <td>`ACTOR_EVENT_NAMES.PERSIST_STATE`</td>* <td>`{ "isMigrating": true }`</td>* <td>@@ -86,7 +86,7 @@ let persistStateInterval = null;* </table>** See <a href="https://nodejs.org/api/events.html#events_class_eventemitter" target="_blank">NodeJs documentation</a>- * for more information about event emitter use.+ * for more information on event emitter use.** @memberof module:Apify* @name eventsdiff --git a/src/index.js b/src/index.jsindex 41eaa6ede..a01cf9570 100644--- a/src/index.js+++ b/src/index.js@@ -22,140 +22,9 @@ import { browse, launchWebDriver } from './webdriver';if (!isProduction() || process.env[ENV_VARS.LOG_LEVEL] === 'DEBUG') log.isDebugMode = true;/**- * The `apify` NPM package simplifies development of acts in Apify Actor -- * a serverless computing platform that enables execution of arbitrary pieces of code in the cloud.- * The package provides helper functions to launch web browsers with proxies,- * access the storage etc. Note that the usage of the package is optional,- * you can create acts without it.- *- * For more information about the Apify Actor platform, please go to- * <a href="https://www.apify.com/docs/actor" target="_blank">Actor documentation</a>.- *- * The source code of this package is available on <a href="https://github.com/apifytech/apify-js" target="_blank">GitHub</a>.- *- * <h2>Example usage</h2>- *- * ```javascript- * Apify.main(async () => {- * // Get input of the act- * const input = await Apify.getValue('INPUT');- * console.dir(input);- * - * // Do something useful here...- * const browser = await Apify.launchPuppeteer();- * const page = await browser.newPage();- * await page.goto('http://www.example.com');- * const pageTitle = await page.title();- * - * // Save output- * await Apify.setValue('OUTPUT', { pageTitle });- * });- * ```- *- * <h2>Common use-cases</h2>- * <!-- Mirror this part to README.md -->- *- * Main goal of this package is to help with implementation of web scraping and automation projects. Some of the- * most common use-cases are:- *- * <ul>- * <li>- * If you need to process high volume of <strong>asynchronous tasks in parallel</strong> then take a- * look at <a href="#AutoscaledPool">AutoscaledPool</a>. This class executes defined tasks in a pool- * which size is scaled based on available memory and CPU.- * </li>- * <li>- * If you want to <strong>crawl</strong> a list of urls using for example <a href="https://www.npmjs.com/package/request" target="_blank">- * Request</a> package then import those url as a <a href="#RequestList">RequestList</a> and then use- * <a href="#BasicCrawler">BasicCrawler</a> to process them in a pool.- * </li>- * <li>- * If you want to crawl a list of urls but you need a real <strong>browser</strong>. Then use- * <a href="#PuppeteerCrawler">PuppeteerCrawler</a> which helps you to process a <a href="#RequestList">RequestList</a>- * using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser).- * </li>- * </ul>- *- * <h2>Puppeteer</h2>- * <!-- Mirror this part to README.md -->- *- * For those who are using <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> (headless Chrome browser)- * we have few helper classes and functions:- *- * <ul>- * <li>- * `Apify.launchPuppeteer()` function starts new instance of Puppeteer browser and returns its browser object.- * </li>- * <li>- * <a href="#PuppeteerPool">PuppeteerPool</a> helps to mantain a pool of Puppeteer instances. This is usefull- * when you need to restart browser after certain number of requests to rotate proxy servers.- * </li>- * <li>- * <a href="#PuppeteerCrawler">PuppeteerCrawler</a> helps to crawl a <a href="#RequestList">RequestList</a>- * in a autoscaled pool.- * </li>- * </ul>- *- * <h2>Installation</h2>- *- * This package requires Node.js 6 or higher. It might work with lower versions too,- * but they are neither tested nor supported.- *- * To install the package locally, run the following command in your Node.js project:- *- * ```bash- * npm install apify --save- * ```- *- * <h2>Local usage</h2>- *- * You can use Apify package locally. To do that you must define following environment variables- *- * <table class="table table-bordered table-condensed">- * <thead>- * <tr>- * <th>Environment variable</th>- * <th>Description</th>- * </tr>- * </thead>- * <tbody>- * <tr>- * <td><code>APIFY_LOCAL_EMULATION_DIR</code></td>- * <td>- * Directory where apify package locally emulates Apify storages - key-value store and dataset.- * Key-value stores will be emulated in directory- * <code>[APIFY_LOCAL_EMULATION_DIR]/key-value-stores/[STORE_ID]</code>- * and datasets in directory- * <code>[APIFY_LOCAL_EMULATION_DIR]/datasets/[DATESET_ID]</code>.- * </td>- * </tr>- * <tr>- * <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>- * <td>Store ID of default key-value store.</td>- * </tr>- * <tr>- * <td><code>APIFY_DEFAULT_DATASET_ID</code></td>- * <td>Dataset ID of default dataset.</td>- * </tr>- * </tbody>- * </table>- *- * Apify will then store key-value store records in files named <code>[KEY].[EXT]</code> where <code>[KEY]</code>- * is the record key and <code>[EXT]</code> is based on the record content type. Dataset items will be stored- * in files named <code>[ID].json</code> where <code>[ID]</code> is sequence number of your dataset item.- *- * If you want to use <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> locally- * then you must define an environment variable <code>PROXY_PASSWORD</code> with password you find at- * <a href="https://my.apify.com/proxy" target="_blank">https://my.apify.com/proxy</a>.- *- * <h2>Promises vs. callbacks</h2>- *- * By default, all asynchronous functions provided by this package return a promise.- * But Apify uses a <a href="http://bluebirdjs.com/" target="_blank">Bluebird</a>- * promise implementation so you can easily convert any function that returns a Promise- * into callback style function.- * See <a href="http://bluebirdjs.com/docs/api/promise.promisify.html" target="_blank">Bluebird documentation</a>- * for more information.+ *{include-readme-1}+ *{include-readme-2}+ *{include-readme-3}** <h2>Programmer's reference</h2>*diff --git a/test/basic_crawler.js b/test/basic_crawler.jsindex ef15eb354..15ad1ac5d 100644--- a/test/basic_crawler.js+++ b/test/basic_crawler.js@@ -6,6 +6,7 @@ import sinon from 'sinon';import { delayPromise } from 'apify-shared/utilities';import * as Apify from '../build/index';import { RequestQueue, RequestQueueLocal } from '../build/request_queue';+import { LOCAL_EMULATION_DIR } from './_helper';chai.use(chaiAsPromised);@@ -139,7 +140,7 @@ describe('BasicCrawler', () => {it('should also support RequestQueueLocal', () => {const requestQueue = new RequestQueue('xxx');- const requestQueueLocal = new RequestQueueLocal('xxx', 'yyy');+ const requestQueueLocal = new RequestQueueLocal('xxx', LOCAL_EMULATION_DIR);const handleRequestFunction = () => {};expect(() => new Apify.BasicCrawler({ handleRequestFunction, requestQueue })).to.not.throw();