Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update prettier config and lint
Travis CIb5644aa47715
diff --git a/docs/api/Apify.md b/docs/api/Apify.mdindex 38d568bf3..14a7fab98 100644--- a/docs/api/Apify.md+++ b/docs/api/Apify.md@@ -9,27 +9,27 @@ The following section describes all functions and properties provided by the `apexcept individual classes and namespaces that have their separate, detailed, documentation pagesaccessible from the left sidebar.-- [Apify](#module_Apify)- - [`.addWebhook(options)`](#module_Apify.addWebhook) ⇒ `Promise<Object>`- - [`.call(actId, [input], [options])`](#module_Apify.call) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)- - [`.callTask(taskId, [input], [options])`](#module_Apify.callTask) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)- - [`.client`](#module_Apify.client)- - [`.events`](#module_Apify.events)- - [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ `String`- - [`.getEnv()`](#module_Apify.getEnv) ⇒ `Object`- - [`.getInput`](#module_Apify.getInput) ⇒ `Promise<Object>`- - [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ `Promise<Object>`- - [`.getValue(key)`](#module_Apify.getValue) ⇒ `Promise<Object>`- - [`.isAtHome()`](#module_Apify.isAtHome) ⇒ `Boolean`- - [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ `Promise<Browser>`- - [`.main(userFunc)`](#module_Apify.main)- - [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise`- - [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [`Promise<Dataset>`](dataset)- - [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [`Promise<KeyValueStore>`](keyvaluestore)- - [`.openRequestList`](#module_Apify.openRequestList) ⇒ [`Promise<RequestList>`](requestlist)- - [`.openRequestQueue`](#module_Apify.openRequestQueue) ⇒ [`Promise<RequestQueue>`](requestqueue)- - [`.pushData(item)`](#module_Apify.pushData) ⇒ `Promise`- - [`.setValue(key, value, [options])`](#module_Apify.setValue) ⇒ `Promise`+- [Apify](#module_Apify)+ - [`.addWebhook(options)`](#module_Apify.addWebhook) ⇒ `Promise<Object>`+ - [`.call(actId, [input], [options])`](#module_Apify.call) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)+ - [`.callTask(taskId, [input], [options])`](#module_Apify.callTask) ⇒ [`Promise<ActorRun>`](../typedefs/actorrun)+ - [`.client`](#module_Apify.client)+ - [`.events`](#module_Apify.events)+ - [`.getApifyProxyUrl(options)`](#module_Apify.getApifyProxyUrl) ⇒ `String`+ - [`.getEnv()`](#module_Apify.getEnv) ⇒ `Object`+ - [`.getInput`](#module_Apify.getInput) ⇒ `Promise<Object>`+ - [`.getMemoryInfo()`](#module_Apify.getMemoryInfo) ⇒ `Promise<Object>`+ - [`.getValue(key)`](#module_Apify.getValue) ⇒ `Promise<Object>`+ - [`.isAtHome()`](#module_Apify.isAtHome) ⇒ `Boolean`+ - [`.launchPuppeteer([options])`](#module_Apify.launchPuppeteer) ⇒ `Promise<Browser>`+ - [`.main(userFunc)`](#module_Apify.main)+ - [`.metamorph(targetActorId, [input], [options])`](#module_Apify.metamorph) ⇒ `Promise`+ - [`.openDataset([datasetIdOrName], [options])`](#module_Apify.openDataset) ⇒ [`Promise<Dataset>`](dataset)+ - [`.openKeyValueStore([storeIdOrName], [options])`](#module_Apify.openKeyValueStore) ⇒ [`Promise<KeyValueStore>`](keyvaluestore)+ - [`.openRequestList`](#module_Apify.openRequestList) ⇒ [`Promise<RequestList>`](requestlist)+ - [`.openRequestQueue`](#module_Apify.openRequestQueue) ⇒ [`Promise<RequestQueue>`](requestqueue)+ - [`.pushData(item)`](#module_Apify.pushData) ⇒ `Promise`+ - [`.setValue(key, value, [options])`](#module_Apify.setValue) ⇒ `Promise`<a name="module_Apify.addWebhook"></a>@@ -109,7 +109,7 @@ For more information about actors, read the**Example usage:**```javascript-const run = await Apify.call("apify/hello-world", { myInput: 123 });+const run = await Apify.call('apify/hello-world', { myInput: 123 });console.log(`Received message: ${run.output.body.message}`);```@@ -119,7 +119,7 @@ and several other API endpoints to obtain the output.**Throws**:-- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.+- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.<table><thead>@@ -220,7 +220,7 @@ For more information about actor tasks, read the [`documentation`](https://apify**Example usage:**```javascript-const run = await Apify.callTask("bob/some-task");+const run = await Apify.callTask('bob/some-task');console.log(`Received message: ${run.output.body.message}`);```@@ -230,7 +230,7 @@ and several other API endpoints to obtain the output.**Throws**:-- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.+- [`ApifyCallError`](../typedefs/apifycallerror) If the run did not succeed, e.g. if it failed or timed out.<table><thead>@@ -327,8 +327,8 @@ The event emitter is initialized by calling the [`Apify.main()`](#module_Apify.m**Example usage:**```javascript-Apify.events.on("cpuInfo", data => {- if (data.isCpuOverloaded) console.log("Oh no, the CPU is overloaded!");+Apify.events.on('cpuInfo', data => {+ if (data.isCpuOverloaded) console.log('Oh no, the CPU is overloaded!');});```@@ -491,7 +491,7 @@ is equivalent to:```javascriptconst store = await Apify.openKeyValueStore();-await store.getValue("INPUT");+await store.getValue('INPUT');```For more information, see [`Apify.openKeyValueStore()`](#module_Apify.openKeyValueStore)@@ -535,14 +535,14 @@ This is just a convenient shortcut for [`keyValueStore.getValue()`](keyvaluestorFor example, calling the following code:```javascript-const input = await Apify.getValue("my-key");+const input = await Apify.getValue('my-key');```is equivalent to:```javascriptconst store = await Apify.openKeyValueStore();-await store.getValue("my-key");+await store.getValue('my-key');```To store the value to the default-key value store, you can use the [`Apify.setValue()`](#module_Apify.setValue) function.@@ -673,33 +673,33 @@ The user function can be synchronous:```javascriptApify.main(() => {- // My synchronous function that returns immediately- console.log("Hello world from actor!");+ // My synchronous function that returns immediately+ console.log('Hello world from actor!');});```If the user function returns a promise, it is considered asynchronous:```javascript-const request = require("request-promise-native");+const request = require('request-promise-native');Apify.main(() => {- // My asynchronous function that returns a promise- return request("http://www.example.com").then(html => {- console.log(html);- });+ // My asynchronous function that returns a promise+ return request('http://www.example.com').then(html => {+ console.log(html);+ });});```To simplify your code, you can take advantage of the `async`/`await` keywords:```javascript-const request = require("request-promise-native");+const request = require('request-promise-native');Apify.main(async () => {- // My asynchronous function- const html = await request("http://www.example.com");- console.log(html);+ // My asynchronous function+ const html = await request('http://www.example.com');+ console.log(html);});```@@ -856,13 +856,9 @@ For more details and code examples, see the [`RequestList`](requestlist) class.**Example usage:**```javascript-const sources = [- "https://www.example.com",- "https://www.google.com",- "https://www.bing.com"-];+const sources = ['https://www.example.com', 'https://www.google.com', 'https://www.bing.com'];-const requestList = await Apify.openRequestList("my-name", sources);+const requestList = await Apify.openRequestList('my-name', sources);```<table>@@ -993,14 +989,14 @@ This is just a convenient shortcut for [`keyValueStore.setValue()`](keyvaluestorFor example, calling the following code:```javascript-await Apify.setValue("OUTPUT", { foo: "bar" });+await Apify.setValue('OUTPUT', { foo: 'bar' });```is equivalent to:```javascriptconst store = await Apify.openKeyValueStore();-await store.setValue("OUTPUT", { foo: "bar" });+await store.setValue('OUTPUT', { foo: 'bar' });```To get a value from the default-key value store, you can use the [`Apify.getValue()`](#module_Apify.getValue) function.diff --git a/docs/api/AutoscaledPool.md b/docs/api/AutoscaledPool.mdindex e8874aec2..db7eef393 100644--- a/docs/api/AutoscaledPool.md+++ b/docs/api/AutoscaledPool.md@@ -33,37 +33,37 @@ and also in the interval set by the `options.maybeRunIntervalSecs` parameter.```javascriptconst pool = new Apify.AutoscaledPool({- maxConcurrency: 50,- runTaskFunction: async () => {- // Run some resource-intensive asynchronous operation here.- },- isTaskReadyFunction: async () => {- // Tell the pool whether more tasks are ready to be processed.- // Return true or false- },- isFinishedFunction: async () => {- // Tell the pool whether it should finish- // or wait for more tasks to become available.- // Return true or false- }+ maxConcurrency: 50,+ runTaskFunction: async () => {+ // Run some resource-intensive asynchronous operation here.+ },+ isTaskReadyFunction: async () => {+ // Tell the pool whether more tasks are ready to be processed.+ // Return true or false+ },+ isFinishedFunction: async () => {+ // Tell the pool whether it should finish+ // or wait for more tasks to become available.+ // Return true or false+ },});await pool.run();```-- [AutoscaledPool](autoscaledpool)- - [`new AutoscaledPool(options)`](#new_AutoscaledPool_new)- - [`.minConcurrency`](#AutoscaledPool+minConcurrency) ⇒ `number`- - [`.minConcurrency`](#AutoscaledPool+minConcurrency)- - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency) ⇒ `number`- - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency)- - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency) ⇒ `number`- - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency)- - [`.currentConcurrency`](#AutoscaledPool+currentConcurrency) ⇒ `number`- - [`.run()`](#AutoscaledPool+run) ⇒ `Promise`- - [`.abort()`](#AutoscaledPool+abort) ⇒ `Promise`- - [`.pause([timeoutSecs])`](#AutoscaledPool+pause) ⇒ `Promise`- - [`.resume()`](#AutoscaledPool+resume)+- [AutoscaledPool](autoscaledpool)+ - [`new AutoscaledPool(options)`](#new_AutoscaledPool_new)+ - [`.minConcurrency`](#AutoscaledPool+minConcurrency) ⇒ `number`+ - [`.minConcurrency`](#AutoscaledPool+minConcurrency)+ - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency) ⇒ `number`+ - [`.maxConcurrency`](#AutoscaledPool+maxConcurrency)+ - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency) ⇒ `number`+ - [`.desiredConcurrency`](#AutoscaledPool+desiredConcurrency)+ - [`.currentConcurrency`](#AutoscaledPool+currentConcurrency) ⇒ `number`+ - [`.run()`](#AutoscaledPool+run) ⇒ `Promise`+ - [`.abort()`](#AutoscaledPool+abort) ⇒ `Promise`+ - [`.pause([timeoutSecs])`](#AutoscaledPool+pause) ⇒ `Promise`+ - [`.resume()`](#AutoscaledPool+resume)<a name="new_AutoscaledPool_new"></a>diff --git a/docs/api/BasicCrawler.md b/docs/api/BasicCrawler.mdindex ddfeb6ee5..84ac16b96 100644--- a/docs/api/BasicCrawler.md+++ b/docs/api/BasicCrawler.md@@ -35,36 +35,33 @@ parameter of the `BasicCrawler` constructor. For user convenience, the `minConcu**Example usage:**```javascript-const rp = require("request-promise-native");+const rp = require('request-promise-native');// Prepare a list of URLs to crawlconst requestList = new Apify.RequestList({- sources: [- { url: "http://www.example.com/page-1" },- { url: "http://www.example.com/page-2" }- ]+ sources: [{ url: 'http://www.example.com/page-1' }, { url: 'http://www.example.com/page-2' }],});await requestList.initialize();// Crawl the URLsconst crawler = new Apify.BasicCrawler({- requestList,- handleRequestFunction: async ({ request }) => {- // 'request' contains an instance of the Request class- // Here we simply fetch the HTML of the page and store it to a dataset- await Apify.pushData({- url: request.url,- html: await rp(request.url)- });- }+ requestList,+ handleRequestFunction: async ({ request }) => {+ // 'request' contains an instance of the Request class+ // Here we simply fetch the HTML of the page and store it to a dataset+ await Apify.pushData({+ url: request.url,+ html: await rp(request.url),+ });+ },});await crawler.run();```-- [BasicCrawler](basiccrawler)- - [`new BasicCrawler(options)`](#new_BasicCrawler_new)- - [`.run()`](#BasicCrawler+run) ⇒ `Promise`+- [BasicCrawler](basiccrawler)+ - [`new BasicCrawler(options)`](#new_BasicCrawler_new)+ - [`.run()`](#BasicCrawler+run) ⇒ `Promise`<a name="new_BasicCrawler_new"></a>diff --git a/docs/api/CheerioCrawler.md b/docs/api/CheerioCrawler.mdindex 3a08441bd..16a6b2128 100644--- a/docs/api/CheerioCrawler.md+++ b/docs/api/CheerioCrawler.md@@ -45,43 +45,40 @@ parameter of the `CheerioCrawler` constructor. For user convenience, the `minCon```javascript// Prepare a list of URLs to crawlconst requestList = new Apify.RequestList({- sources: [- { url: "http://www.example.com/page-1" },- { url: "http://www.example.com/page-2" }- ]+ sources: [{ url: 'http://www.example.com/page-1' }, { url: 'http://www.example.com/page-2' }],});await requestList.initialize();// Crawl the URLsconst crawler = new Apify.CheerioCrawler({- requestList,- handlePageFunction: async ({ request, response, html, $ }) => {- const data = [];+ requestList,+ handlePageFunction: async ({ request, response, html, $ }) => {+ const data = [];- // Do some data extraction from the page with Cheerio.- $(".some-collection").each((index, el) => {- data.push({- title: $(el)- .find(".some-title")- .text()- });- });+ // Do some data extraction from the page with Cheerio.+ $('.some-collection').each((index, el) => {+ data.push({+ title: $(el)+ .find('.some-title')+ .text(),+ });+ });- // Save the data to dataset.- await Apify.pushData({- url: request.url,- html,- data- });- }+ // Save the data to dataset.+ await Apify.pushData({+ url: request.url,+ html,+ data,+ });+ },});await crawler.run();```-- [CheerioCrawler](cheeriocrawler)- - [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)- - [`.run()`](#CheerioCrawler+run) ⇒ `Promise`+- [CheerioCrawler](cheeriocrawler)+ - [`new CheerioCrawler(options)`](#new_CheerioCrawler_new)+ - [`.run()`](#CheerioCrawler+run) ⇒ `Promise`<a name="new_CheerioCrawler_new"></a>diff --git a/docs/api/Dataset.md b/docs/api/Dataset.mdindex 99220f6fc..be7db518e 100644--- a/docs/api/Dataset.md+++ b/docs/api/Dataset.md@@ -38,26 +38,26 @@ even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.```javascript// Write a single row to the default dataset-await Apify.pushData({ col1: 123, col2: "val2" });+await Apify.pushData({ col1: 123, col2: 'val2' });// Open a named dataset-const dataset = await Apify.openDataset("some-name");+const dataset = await Apify.openDataset('some-name');// Write a single row-await dataset.pushData({ foo: "bar" });+await dataset.pushData({ foo: 'bar' });// Write multiple rows-await dataset.pushData([{ foo: "bar2", col2: "val2" }, { col3: 123 }]);+await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);```-- [Dataset](dataset)- - [`.pushData(data)`](#Dataset+pushData) ⇒ `Promise`- - [`.getData([options])`](#Dataset+getData) ⇒ `Promise<Object>`- - [`.getInfo()`](#Dataset+getInfo) ⇒ `Promise<Object>`- - [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise`- - [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array>`- - [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<*>`- - [`.delete()`](#Dataset+delete) ⇒ `Promise`+- [Dataset](dataset)+ - [`.pushData(data)`](#Dataset+pushData) ⇒ `Promise`+ - [`.getData([options])`](#Dataset+getData) ⇒ `Promise<Object>`+ - [`.getInfo()`](#Dataset+getInfo) ⇒ `Promise<Object>`+ - [`.forEach(iteratee, [options], [index])`](#Dataset+forEach) ⇒ `Promise`+ - [`.map(iteratee, options)`](#Dataset+map) ⇒ `Promise<Array>`+ - [`.reduce(iteratee, memo, options)`](#Dataset+reduce) ⇒ `Promise<*>`+ - [`.delete()`](#Dataset+delete) ⇒ `Promise`<a name="Dataset+pushData"></a>@@ -256,9 +256,9 @@ If it throws an error, the iteration is aborted and the `forEach` function throw**Example usage**```javascript-const dataset = await Apify.openDataset("my-results");+const dataset = await Apify.openDataset('my-results');await dataset.forEach(async (item, index) => {- console.log(`Item at ${index}: ${JSON.stringify(item)}`);+ console.log(`Item at ${index}: ${JSON.stringify(item)}`);});```diff --git a/docs/api/KeyValueStore.md b/docs/api/KeyValueStore.mdindex 8beb4d839..af1f281fe 100644--- a/docs/api/KeyValueStore.md+++ b/docs/api/KeyValueStore.md@@ -48,32 +48,32 @@ even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.```javascript// Get actor input from the default key-value storeconst input = await Apify.getInput();-const otherValue = Apify.getValue("my-key");+const otherValue = Apify.getValue('my-key');// Write actor output to the default key-value store.-await Apify.setValue("OUTPUT", { myResult: 123 });+await Apify.setValue('OUTPUT', { myResult: 123 });// Open a named key-value store-const store = await Apify.openKeyValueStore("some-name");+const store = await Apify.openKeyValueStore('some-name');// Write a record. JavaScript object is automatically converted to JSON,// strings and binary buffers are stored as they are-await store.setValue("some-key", { foo: "bar" });+await store.setValue('some-key', { foo: 'bar' });// Read a record. Note that JSON is automatically parsed to a JavaScript object,// text data returned as a string and other data is returned as binary buffer-const value = await store.getValue("some-key");+const value = await store.getValue('some-key');// Delete record-await store.delete("some-key");+await store.delete('some-key');```-- [KeyValueStore](keyvaluestore)- - [`.getValue(key)`](#KeyValueStore+getValue) ⇒ `Promise<(Object|String|Buffer)>`- - [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise`- - [`.delete()`](#KeyValueStore+delete) ⇒ `Promise`- - [`.getPublicUrl(key)`](#KeyValueStore+getPublicUrl) ⇒ `string`- - [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise`+- [KeyValueStore](keyvaluestore)+ - [`.getValue(key)`](#KeyValueStore+getValue) ⇒ `Promise<(Object|String|Buffer)>`+ - [`.setValue(key, value, [options])`](#KeyValueStore+setValue) ⇒ `Promise`+ - [`.delete()`](#KeyValueStore+delete) ⇒ `Promise`+ - [`.getPublicUrl(key)`](#KeyValueStore+getPublicUrl) ⇒ `string`+ - [`.forEachKey(iteratee, [options])`](#KeyValueStore+forEachKey) ⇒ `Promise`<a name="KeyValueStore+getValue"></a>@@ -97,8 +97,8 @@ To save or delete a value in the key-value store, use the**Example usage:**```javascript-const store = await Apify.openKeyValueStore("my-screenshots");-const buffer = await store.getValue("screenshot1.png");+const store = await Apify.openKeyValueStore('my-screenshots');+const buffer = await store.getValue('screenshot1.png');```**Returns**: `Promise<(Object|String|Buffer)>` - Returns a promise that resolves to an object, string@@ -130,8 +130,8 @@ The function returns a promise that resolves once the record has been saved or d**Example usage:**```javascript-const store = await Apify.openKeyValueStore("my-store");-await store.setValue("RESULTS", "my text data", { contentType: "text/plain" });+const store = await Apify.openKeyValueStore('my-store');+await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });```Beware that the key can be at most 256 characters long and only contain the following characters: `a-zA-Z0-9!-_.'()`@@ -141,8 +141,8 @@ By default, `value` is converted to JSON and stored with theTo store the value with another content type, pass it in the options as follows:```javascript-const store = await Apify.openKeyValueStore("my-store");-await store.setValue("RESULTS", "my text data", { contentType: "text/plain" });+const store = await Apify.openKeyValueStore('my-store');+await store.setValue('RESULTS', 'my text data', { contentType: 'text/plain' });```If you set custom content type, `value` must be either a string or@@ -238,7 +238,7 @@ If it throws an error, the iteration is aborted and the `forEachKey` function th```javascriptconst keyValueStore = await Apify.openKeyValueStore();await keyValueStore.forEachKey(async (key, index, info) => {- console.log(`Key at ${index}: ${key} has size ${info.size}`);+ console.log(`Key at ${index}: ${key} has size ${info.size}`);});```diff --git a/docs/api/LiveViewServer.md b/docs/api/LiveViewServer.mdindex d69c4d0b2..d531e12e5 100644--- a/docs/api/LiveViewServer.md+++ b/docs/api/LiveViewServer.md@@ -11,10 +11,10 @@ pieces of information, the currently opened URL, the content of the page (HTML)```json{- "pageUrl": "https://www.example.com",- "htmlContent": "<html><body> ....",- "screenshotIndex": 3,- "createdAt": "2019-04-18T11:50:40.060Z"+ "pageUrl": "https://www.example.com",+ "htmlContent": "<html><body> ....",+ "screenshotIndex": 3,+ "createdAt": "2019-04-18T11:50:40.060Z"}```@@ -36,13 +36,13 @@ single page improves performance and stability dramatically in high concurrencyWhen running locally, it is often best to use a headful browser for debugging, since it providesa better view into the browser, including DevTools, but `LiveViewServer` works too.-- [LiveViewServer](liveviewserver)- - [`new LiveViewServer([options])`](#new_LiveViewServer_new)- - [`.start()`](#LiveViewServer+start) ⇒ `Promise`- - [`.stop()`](#LiveViewServer+stop) ⇒ `Promise`- - [`.serve(page)`](#LiveViewServer+serve) ⇒ `Promise`- - [`.isRunning()`](#LiveViewServer+isRunning) ⇒ `boolean`- - [`.hasClients()`](#LiveViewServer+hasClients) ⇒ `boolean`+- [LiveViewServer](liveviewserver)+ - [`new LiveViewServer([options])`](#new_LiveViewServer_new)+ - [`.start()`](#LiveViewServer+start) ⇒ `Promise`+ - [`.stop()`](#LiveViewServer+stop) ⇒ `Promise`+ - [`.serve(page)`](#LiveViewServer+serve) ⇒ `Promise`+ - [`.isRunning()`](#LiveViewServer+isRunning) ⇒ `boolean`+ - [`.hasClients()`](#LiveViewServer+hasClients) ⇒ `boolean`<a name="new_LiveViewServer_new"></a>diff --git a/docs/api/PseudoUrl.md b/docs/api/PseudoUrl.mdindex 65ec539a8..86f133115 100644--- a/docs/api/PseudoUrl.md+++ b/docs/api/PseudoUrl.md@@ -47,21 +47,20 @@ you need to escape it with another back backslash, as shown in the example below```javascript// Using a pseudo-URL string-const purl = new Apify.PseudoUrl("http://www.example.com/pages/[(\\w|-)+]", {- userData: { foo: "bar" }+const purl = new Apify.PseudoUrl('http://www.example.com/pages/[(\\w|-)+]', {+ userData: { foo: 'bar' },});// Using a regular expressionconst purl2 = new Apify.PseudoUrl(/http:\/\/www\.example\.com\/pages\/(\w|-)+/);-if (purl.matches("http://www.example.com/pages/my-awesome-page"))- console.log("Match!");+if (purl.matches('http://www.example.com/pages/my-awesome-page')) console.log('Match!');```-- [PseudoUrl](pseudourl)- - [`new PseudoUrl(purl, requestTemplate)`](#new_PseudoUrl_new)- - [`.matches(url)`](#PseudoUrl+matches) ⇒ `Boolean`- - [`.createRequest(urlOrProps)`](#PseudoUrl+createRequest) ⇒ [`Request`](request)+- [PseudoUrl](pseudourl)+ - [`new PseudoUrl(purl, requestTemplate)`](#new_PseudoUrl_new)+ - [`.matches(url)`](#PseudoUrl+matches) ⇒ `Boolean`+ - [`.createRequest(urlOrProps)`](#PseudoUrl+createRequest) ⇒ [`Request`](request)<a name="new_PseudoUrl_new"></a>diff --git a/docs/api/PuppeteerCrawler.md b/docs/api/PuppeteerCrawler.mdindex 07206c8b6..e40b41030 100644--- a/docs/api/PuppeteerCrawler.md+++ b/docs/api/PuppeteerCrawler.md@@ -43,33 +43,33 @@ to [`PuppeteerPool`](puppeteerpool) constructor.```javascriptconst crawler = new Apify.PuppeteerCrawler({- requestList,- handlePageFunction: async ({ page, request }) => {- // This function is called to extract data from a single web page- // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called- // 'request' is an instance of Request class with information about the page to load- await Apify.pushData({- title: await page.title(),- url: request.url,- succeeded: true- });- },- handleFailedRequestFunction: async ({ request }) => {- // This function is called when the crawling of a request failed too many times- await Apify.pushData({- url: request.url,- succeeded: false,- errors: request.errorMessages- });- }+ requestList,+ handlePageFunction: async ({ page, request }) => {+ // This function is called to extract data from a single web page+ // 'page' is an instance of Puppeteer.Page with page.goto(request.url) already called+ // 'request' is an instance of Request class with information about the page to load+ await Apify.pushData({+ title: await page.title(),+ url: request.url,+ succeeded: true,+ });+ },+ handleFailedRequestFunction: async ({ request }) => {+ // This function is called when the crawling of a request failed too many times+ await Apify.pushData({+ url: request.url,+ succeeded: false,+ errors: request.errorMessages,+ });+ },});await crawler.run();```-- [PuppeteerCrawler](puppeteercrawler)- - [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)- - [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise`+- [PuppeteerCrawler](puppeteercrawler)+ - [`new PuppeteerCrawler(options)`](#new_PuppeteerCrawler_new)+ - [`.run()`](#PuppeteerCrawler+run) ⇒ `Promise`<a name="new_PuppeteerCrawler_new"></a>diff --git a/docs/api/PuppeteerPool.md b/docs/api/PuppeteerPool.mdindex 2cfc14998..ffda3b1d5 100644--- a/docs/api/PuppeteerPool.md+++ b/docs/api/PuppeteerPool.md@@ -22,12 +22,12 @@ and the retirement of instances helps mitigate effects of memory leaks in Chrome```javascriptconst puppeteerPool = new PuppeteerPool({- launchPuppeteerFunction: () => {- // Use a new proxy with a new IP address for each new Chrome instance- return Apify.launchPuppeteer({- apifyProxySession: Math.random()- });- }+ launchPuppeteerFunction: () => {+ // Use a new proxy with a new IP address for each new Chrome instance+ return Apify.launchPuppeteer({+ apifyProxySession: Math.random(),+ });+ },});const page1 = await puppeteerPool.newPage();@@ -40,13 +40,13 @@ const page3 = await puppeteerPool.newPage();await puppeteerPool.destroy();```-- [PuppeteerPool](puppeteerpool)- - [`new PuppeteerPool([options])`](#new_PuppeteerPool_new)- - [`.newPage()`](#PuppeteerPool+newPage) ⇒ `Promise<Page>`- - [`.destroy()`](#PuppeteerPool+destroy) ⇒ `Promise`- - [`.retire(browser)`](#PuppeteerPool+retire) ⇒ `Promise`- - [`.recyclePage(page)`](#PuppeteerPool+recyclePage) ⇒ `Promise`- - [`.serveLiveViewSnapshot(page)`](#PuppeteerPool+serveLiveViewSnapshot) ⇒ `Promise`+- [PuppeteerPool](puppeteerpool)+ - [`new PuppeteerPool([options])`](#new_PuppeteerPool_new)+ - [`.newPage()`](#PuppeteerPool+newPage) ⇒ `Promise<Page>`+ - [`.destroy()`](#PuppeteerPool+destroy) ⇒ `Promise`+ - [`.retire(browser)`](#PuppeteerPool+retire) ⇒ `Promise`+ - [`.recyclePage(page)`](#PuppeteerPool+recyclePage) ⇒ `Promise`+ - [`.serveLiveViewSnapshot(page)`](#PuppeteerPool+serveLiveViewSnapshot) ⇒ `Promise`<a name="new_PuppeteerPool_new"></a>diff --git a/docs/api/Request.md b/docs/api/Request.mdindex 8547a8768..74a900134 100644--- a/docs/api/Request.md+++ b/docs/api/Request.md@@ -109,10 +109,10 @@ const foo = request.userData.foo;</td></tr></tbody></table>-- [Request](request)- - [`new Request(options)`](#new_Request_new)- - [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)- - ~~[`.doNotRetry([message])`](#Request+doNotRetry)~~+- [Request](request)+ - [`new Request(options)`](#new_Request_new)+ - [`.pushErrorMessage(errorOrMessage, [options])`](#Request+pushErrorMessage)+ - ~~[`.doNotRetry([message])`](#Request+doNotRetry)~~<a name="new_Request_new"></a>diff --git a/docs/api/RequestList.md b/docs/api/RequestList.mdindex dff501924..9efe76db3 100644--- a/docs/api/RequestList.md+++ b/docs/api/RequestList.md@@ -39,20 +39,17 @@ which will prevent any issues that a live list of URLs might cause.```javascriptconst requestList = new Apify.RequestList({- sources: [- // Separate requests- { url: "http://www.example.com/page-1", method: "GET", headers: {} },- { url: "http://www.example.com/page-2", userData: { foo: "bar" } },-- // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`- // Note that all URLs must start with http:// or https://- {- requestsFromUrl: "http://www.example.com/my-url-list.txt",- userData: { isFromUrl: true }- }- ],- persistStateKey: "my-state",- persistSourcesKey: "my-sources"+ sources: [+ // Separate requests+ { url: 'http://www.example.com/page-1', method: 'GET', headers: {} },+ { url: 'http://www.example.com/page-2', userData: { foo: 'bar' } },++ // Bulk load of URLs from file `http://www.example.com/my-url-list.txt`+ // Note that all URLs must start with http:// or https://+ { requestsFromUrl: 'http://www.example.com/my-url-list.txt', userData: { isFromUrl: true } },+ ],+ persistStateKey: 'my-state',+ persistSourcesKey: 'my-sources',});// This call loads and parses the URLs from the remote file.@@ -70,18 +67,18 @@ await requestList.markRequestHandled(request1);await requestList.reclaimRequest(request2);```-- [RequestList](requestlist)- - [`new exports.RequestList(options)`](#new_RequestList_new)- - [`.initialize()`](#RequestList+initialize) ⇒ `Promise`- - [`.persistState()`](#RequestList+persistState) ⇒ `Promise`- - [`.getState()`](#RequestList+getState) ⇒ `Object`- - [`.isEmpty()`](#RequestList+isEmpty) ⇒ `Promise<Boolean>`- - [`.isFinished()`](#RequestList+isFinished) ⇒ `Promise<Boolean>`- - [`.fetchNextRequest()`](#RequestList+fetchNextRequest) ⇒ [`Promise<Request>`](request)- - [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ `Promise`- - [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ `Promise`- - [`.length()`](#RequestList+length) ⇒ `Number`- - [`.handledCount()`](#RequestList+handledCount) ⇒ `Number`+- [RequestList](requestlist)+ - [`new exports.RequestList(options)`](#new_RequestList_new)+ - [`.initialize()`](#RequestList+initialize) ⇒ `Promise`+ - [`.persistState()`](#RequestList+persistState) ⇒ `Promise`+ - [`.getState()`](#RequestList+getState) ⇒ `Object`+ - [`.isEmpty()`](#RequestList+isEmpty) ⇒ `Promise<Boolean>`+ - [`.isFinished()`](#RequestList+isFinished) ⇒ `Promise<Boolean>`+ - [`.fetchNextRequest()`](#RequestList+fetchNextRequest) ⇒ [`Promise<Request>`](request)+ - [`.markRequestHandled(request)`](#RequestList+markRequestHandled) ⇒ `Promise`+ - [`.reclaimRequest(request)`](#RequestList+reclaimRequest) ⇒ `Promise`+ - [`.length()`](#RequestList+length) ⇒ `Number`+ - [`.handledCount()`](#RequestList+handledCount) ⇒ `Number`<a name="new_RequestList_new"></a>diff --git a/docs/api/RequestQueue.md b/docs/api/RequestQueue.mdindex a11cd7193..708bfae37 100644--- a/docs/api/RequestQueue.md+++ b/docs/api/RequestQueue.md@@ -51,15 +51,12 @@ even if the `APIFY_LOCAL_STORAGE_DIR` variable is set.const queue = await Apify.openRequestQueue();// Open a named request queue-const queueWithName = await Apify.openRequestQueue("some-name");+const queueWithName = await Apify.openRequestQueue('some-name');// Enqueue few requests-await queue.addRequest({ url: "http://example.com/aaa" });-await queue.addRequest({ url: "http://example.com/bbb" });-await queue.addRequest(- { url: "http://example.com/foo/bar" },- { forefront: true }-);+await queue.addRequest({ url: 'http://example.com/aaa' });+await queue.addRequest({ url: 'http://example.com/bbb' });+await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });// Get requests from queueconst request1 = await queue.fetchNextRequest();@@ -73,17 +70,17 @@ await queue.markRequestHandled(request1);await queue.reclaimRequest(request2);```-- [RequestQueue](requestqueue)- - [`.addRequest(request, [options])`](#RequestQueue+addRequest) ⇒ [`QueueOperationInfo`](../typedefs/queueoperationinfo)- - [`.getRequest(requestId)`](#RequestQueue+getRequest) ⇒ [`Promise<Request>`](request)- - [`.fetchNextRequest()`](#RequestQueue+fetchNextRequest) ⇒ [`Promise<Request>`](request)- - [`.markRequestHandled(request)`](#RequestQueue+markRequestHandled) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- - [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)- - [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ `Promise<Boolean>`- - [`.isFinished()`](#RequestQueue+isFinished) ⇒ `Promise<Boolean>`- - [`.delete()`](#RequestQueue+delete) ⇒ `Promise`- - ~~[`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`~~- - [`.getInfo()`](#RequestQueue+getInfo) ⇒ `Promise<Object>`+- [RequestQueue](requestqueue)+ - [`.addRequest(request, [options])`](#RequestQueue+addRequest) ⇒ [`QueueOperationInfo`](../typedefs/queueoperationinfo)+ - [`.getRequest(requestId)`](#RequestQueue+getRequest) ⇒ [`Promise<Request>`](request)+ - [`.fetchNextRequest()`](#RequestQueue+fetchNextRequest) ⇒ [`Promise<Request>`](request)+ - [`.markRequestHandled(request)`](#RequestQueue+markRequestHandled) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)+ - [`.reclaimRequest(request, [options])`](#RequestQueue+reclaimRequest) ⇒ [`Promise<QueueOperationInfo>`](../typedefs/queueoperationinfo)+ - [`.isEmpty()`](#RequestQueue+isEmpty) ⇒ `Promise<Boolean>`+ - [`.isFinished()`](#RequestQueue+isFinished) ⇒ `Promise<Boolean>`+ - [`.delete()`](#RequestQueue+delete) ⇒ `Promise`+ - ~~[`.handledCount()`](#RequestQueue+handledCount) ⇒ `Promise<number>`~~+ - [`.getInfo()`](#RequestQueue+getInfo) ⇒ `Promise<Object>`<a name="RequestQueue+addRequest"></a>diff --git a/docs/api/Snapshotter.md b/docs/api/Snapshotter.mdindex a69fcf600..fffbc01e9 100644--- a/docs/api/Snapshotter.md+++ b/docs/api/Snapshotter.md@@ -28,14 +28,14 @@ Event loop becomes overloaded if it slows down by more than the `maxBlockedMilliClient becomes overloaded when rate limit errors (429 - Too Many Requests),typically received from the request queue, exceed the set limit within the set interval.-- [Snapshotter](snapshotter)- - [`new Snapshotter([options])`](#new_Snapshotter_new)- - [`.start()`](#Snapshotter+start) ⇒ `Promise`- - [`.stop()`](#Snapshotter+stop) ⇒ `Promise`- - [`.getMemorySample([sampleDurationMillis])`](#Snapshotter+getMemorySample) ⇒ `Array`- - [`.getEventLoopSample([sampleDurationMillis])`](#Snapshotter+getEventLoopSample) ⇒ `Array`- - [`.getCpuSample([sampleDurationMillis])`](#Snapshotter+getCpuSample) ⇒ `Array`- - [`.getClientSample(sampleDurationMillis)`](#Snapshotter+getClientSample) ⇒ `Array`+- [Snapshotter](snapshotter)+ - [`new Snapshotter([options])`](#new_Snapshotter_new)+ - [`.start()`](#Snapshotter+start) ⇒ `Promise`+ - [`.stop()`](#Snapshotter+stop) ⇒ `Promise`+ - [`.getMemorySample([sampleDurationMillis])`](#Snapshotter+getMemorySample) ⇒ `Array`+ - [`.getEventLoopSample([sampleDurationMillis])`](#Snapshotter+getEventLoopSample) ⇒ `Array`+ - [`.getCpuSample([sampleDurationMillis])`](#Snapshotter+getCpuSample) ⇒ `Array`+ - [`.getClientSample(sampleDurationMillis)`](#Snapshotter+getClientSample) ⇒ `Array`<a name="new_Snapshotter_new"></a>diff --git a/docs/api/SystemStatus.md b/docs/api/SystemStatus.mdindex 5ae7437ed..53d8368d3 100644--- a/docs/api/SystemStatus.md+++ b/docs/api/SystemStatus.md@@ -25,10 +25,10 @@ returns a boolean that represents the long-term statusof the system. It considers the full snapshot history availablein the [`Snapshotter`](snapshotter) instance.-- [SystemStatus](systemstatus)- - [`new SystemStatus([options])`](#new_SystemStatus_new)- - [`.getCurrentStatus()`](#SystemStatus+getCurrentStatus) ⇒ `Object`- - [`.getHistoricalStatus()`](#SystemStatus+getHistoricalStatus) ⇒ `Object`+- [SystemStatus](systemstatus)+ - [`new SystemStatus([options])`](#new_SystemStatus_new)+ - [`.getCurrentStatus()`](#SystemStatus+getCurrentStatus) ⇒ `Object`+ - [`.getHistoricalStatus()`](#SystemStatus+getHistoricalStatus) ⇒ `Object`<a name="new_SystemStatus_new"></a>diff --git a/docs/api/log.md b/docs/api/log.mdindex d2734f89c..958d86372 100644--- a/docs/api/log.md+++ b/docs/api/log.md@@ -51,15 +51,15 @@ Another very useful way of setting the log level is by setting the `APIFY_LOG_LEenvironment variable, such as `APIFY_LOG_LEVEL=DEBUG`. This way, no code changesare necessary to turn on your debug messages and start debugging right away.-- [`log`](#log) : `object`- - [`.LEVELS`](#log.LEVELS) : `Object`- - [`.setLevel(level)`](#log.setLevel)- - [`.getLevel()`](#log.getLevel)- - [`.debug(message, [data])`](#log.debug)- - [`.info(message, [data])`](#log.info)- - [`.warning(message, [data])`](#log.warning)- - [`.error(message, [data])`](#log.error)- - [`.exception(exception, [message], [data])`](#log.exception)+- [`log`](#log) : `object`+ - [`.LEVELS`](#log.LEVELS) : `Object`+ - [`.setLevel(level)`](#log.setLevel)+ - [`.getLevel()`](#log.getLevel)+ - [`.debug(message, [data])`](#log.debug)+ - [`.info(message, [data])`](#log.info)+ - [`.warning(message, [data])`](#log.warning)+ - [`.error(message, [data])`](#log.error)+ - [`.exception(exception, [message], [data])`](#log.exception)<a name="log.LEVELS"></a>diff --git a/docs/api/puppeteer.md b/docs/api/puppeteer.mdindex 90f49e81f..72645c88e 100644--- a/docs/api/puppeteer.md+++ b/docs/api/puppeteer.md@@ -11,30 +11,30 @@ A namespace that contains various utilities for**Example usage:**```javascript-const Apify = require("apify");+const Apify = require('apify');const { puppeteer } = Apify.utils;// Open https://www.example.com in Puppeteerconst browser = await Apify.launchPuppeteer();const page = await browser.newPage();-await page.goto("https://www.example.com");+await page.goto('https://www.example.com');// Inject jQuery into a pageawait puppeteer.injectJQuery(page);```-- [`puppeteer`](#puppeteer) : `object`- - [`.enqueueLinksByClickingElements`](#puppeteer.enqueueLinksByClickingElements) ⇒ `Promise<Array<QueueOperationInfo>>`- - [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ `Promise`- - [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ `Promise`- - [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ `Promise<Response>`- - [`.infiniteScroll`](#puppeteer.infiniteScroll) ⇒ `Promise`- - [`.injectFile(page, filePath, [options])`](#puppeteer.injectFile) ⇒ `Promise`- - [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ `Promise`- - [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ `Promise`- - [`.blockRequests(page, [options])`](#puppeteer.blockRequests) ⇒ `Promise`- - ~~[`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ `Promise`~~- - [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ `function`+- [`puppeteer`](#puppeteer) : `object`+ - [`.enqueueLinksByClickingElements`](#puppeteer.enqueueLinksByClickingElements) ⇒ `Promise<Array<QueueOperationInfo>>`+ - [`.addInterceptRequestHandler`](#puppeteer.addInterceptRequestHandler) ⇒ `Promise`+ - [`.removeInterceptRequestHandler`](#puppeteer.removeInterceptRequestHandler) ⇒ `Promise`+ - [`.gotoExtended`](#puppeteer.gotoExtended) ⇒ `Promise<Response>`+ - [`.infiniteScroll`](#puppeteer.infiniteScroll) ⇒ `Promise`+ - [`.injectFile(page, filePath, [options])`](#puppeteer.injectFile) ⇒ `Promise`+ - [`.injectJQuery(page)`](#puppeteer.injectJQuery) ⇒ `Promise`+ - [`.injectUnderscore(page)`](#puppeteer.injectUnderscore) ⇒ `Promise`+ - [`.blockRequests(page, [options])`](#puppeteer.blockRequests) ⇒ `Promise`+ - ~~[`.cacheResponses(page, cache, responseUrlRules)`](#puppeteer.cacheResponses) ⇒ `Promise`~~+ - [`.compileScript(scriptString, context)`](#puppeteer.compileScript) ⇒ `function`<a name="puppeteer.enqueueLinksByClickingElements"></a>@@ -189,30 +189,30 @@ to any of the remaining handlers.```javascript// Replace images with placeholder.await addInterceptRequestHandler(page, request => {- if (request.resourceType() === "image") {- return request.respond({- statusCode: 200,- contentType: "image/jpeg",- body: placeholderImageBuffer- });- }- return request.continue();+ if (request.resourceType() === 'image') {+ return request.respond({+ statusCode: 200,+ contentType: 'image/jpeg',+ body: placeholderImageBuffer,+ });+ }+ return request.continue();});// Abort all the scripts.await addInterceptRequestHandler(page, request => {- if (request.resourceType() === "script") return request.abort();- return request.continue();+ if (request.resourceType() === 'script') return request.abort();+ return request.continue();});// Change requests to post.await addInterceptRequestHandler(page, request => {- return request.continue({- method: "POST"- });+ return request.continue({+ method: 'POST',+ });});-await page.goto("http://example.com");+await page.goto('http://example.com');```<table>@@ -387,7 +387,7 @@ The injected jQuery will survive page navigations and reloads.```javascriptawait Apify.utils.puppeteer.injectJQuery(page);const title = await page.evaluate(() => {- return $("head title").text();+ return $('head title').text();});```@@ -426,7 +426,7 @@ The injected Underscore will survive page navigations and reloads.```javascriptawait Apify.utils.puppeteer.injectUnderscore(page);const escapedHtml = await page.evaluate(() => {- return _.escape("<h1>Hello</h1>");+ return _.escape('<h1>Hello</h1>');});```@@ -473,17 +473,17 @@ The function will never block main document loads and their respective redirects**Example usage**```javascript-const Apify = require("apify");+const Apify = require('apify');const browser = await Apify.launchPuppeteer();const page = await browser.newPage();// Block all requests to URLs that include `adsbygoogle.js` and also all defaults.await Apify.utils.puppeteer.blockRequests(page, {- extraUrlPatterns: ["adsbygoogle.js"]+ extraUrlPatterns: ['adsbygoogle.js'],});-await page.goto("https://cnn.com");+await page.goto('https://cnn.com');```<table>diff --git a/docs/api/social.md b/docs/api/social.mdindex 9d8be4ba1..1215fc68a 100644--- a/docs/api/social.md+++ b/docs/api/social.md@@ -11,29 +11,27 @@ from text, URLs and and HTML documents.**Example usage:**```javascript-const Apify = require("apify");--const emails = Apify.utils.social.emailsFromText(-);-```--- [`social`](#social) : `object`- - [`.LINKEDIN_REGEX`](#social.LINKEDIN_REGEX) : `RegExp`- - [`.LINKEDIN_REGEX_GLOBAL`](#social.LINKEDIN_REGEX_GLOBAL) : `RegExp`- - [`.INSTAGRAM_REGEX`](#social.INSTAGRAM_REGEX) : `RegExp`- - [`.INSTAGRAM_REGEX_GLOBAL`](#social.INSTAGRAM_REGEX_GLOBAL) : `RegExp`- - [`.TWITTER_REGEX`](#social.TWITTER_REGEX) : `RegExp`- - [`.TWITTER_REGEX_GLOBAL`](#social.TWITTER_REGEX_GLOBAL) : `RegExp`- - [`.FACEBOOK_REGEX`](#social.FACEBOOK_REGEX) : `RegExp`- - [`.FACEBOOK_REGEX_GLOBAL`](#social.FACEBOOK_REGEX_GLOBAL) : `RegExp`- - [`.EMAIL_REGEX`](#social.EMAIL_REGEX) : `RegExp`- - [`.EMAIL_REGEX_GLOBAL`](#social.EMAIL_REGEX_GLOBAL) : `RegExp`- - [`.emailsFromText(text)`](#social.emailsFromText) ⇒ `Array<String>`- - [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ `Array<String>`- - [`.phonesFromText(text)`](#social.phonesFromText) ⇒ `Array<String>`- - [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ `Array<String>`- - [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ `*`+const Apify = require('apify');++const emails = Apify.utils.social.emailsFromText('[email protected] [email protected]');+```++- [`social`](#social) : `object`+ - [`.LINKEDIN_REGEX`](#social.LINKEDIN_REGEX) : `RegExp`+ - [`.LINKEDIN_REGEX_GLOBAL`](#social.LINKEDIN_REGEX_GLOBAL) : `RegExp`+ - [`.INSTAGRAM_REGEX`](#social.INSTAGRAM_REGEX) : `RegExp`+ - [`.INSTAGRAM_REGEX_GLOBAL`](#social.INSTAGRAM_REGEX_GLOBAL) : `RegExp`+ - [`.TWITTER_REGEX`](#social.TWITTER_REGEX) : `RegExp`+ - [`.TWITTER_REGEX_GLOBAL`](#social.TWITTER_REGEX_GLOBAL) : `RegExp`+ - [`.FACEBOOK_REGEX`](#social.FACEBOOK_REGEX) : `RegExp`+ - [`.FACEBOOK_REGEX_GLOBAL`](#social.FACEBOOK_REGEX_GLOBAL) : `RegExp`+ - [`.EMAIL_REGEX`](#social.EMAIL_REGEX) : `RegExp`+ - [`.EMAIL_REGEX_GLOBAL`](#social.EMAIL_REGEX_GLOBAL) : `RegExp`+ - [`.emailsFromText(text)`](#social.emailsFromText) ⇒ `Array<String>`+ - [`.emailsFromUrls(urls)`](#social.emailsFromUrls) ⇒ `Array<String>`+ - [`.phonesFromText(text)`](#social.phonesFromText) ⇒ `Array<String>`+ - [`.phonesFromUrls(urls)`](#social.phonesFromUrls) ⇒ `Array<String>`+ - [`.parseHandlesFromHtml(html, data)`](#social.parseHandlesFromHtml) ⇒ `*`<a name="social.LINKEDIN_REGEX"></a>@@ -420,15 +418,15 @@ extracted from the plain text, which might be very inaccurate.**Example usage:**```javascript-const Apify = require("apify");+const Apify = require('apify');const browser = await Apify.launchPuppeteer();const page = await browser.newPage();-await page.goto("http://www.example.com");+await page.goto('http://www.example.com');const html = await page.content();const result = Apify.utils.social.parseHandlesFromHtml(html);-console.log("Social handles:");+console.log('Social handles:');console.dir(result);```diff --git a/docs/api/utils.md b/docs/api/utils.mdindex 5bbf7cda3..93e8575ae 100644--- a/docs/api/utils.md+++ b/docs/api/utils.md@@ -18,16 +18,16 @@ const Apify = require('apify');await Apify.utils.sleep(1500);```-- [`utils`](#utils) : `object`- - [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`- - [`.sleep`](#utils.sleep) ⇒ `Promise`- - [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)- - [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)- - [`.isDocker()`](#utils.isDocker) ⇒ `Promise`- - [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`- - [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`- - [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`- - [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`+- [`utils`](#utils) : `object`+ - [`.enqueueLinks`](#utils.enqueueLinks) ⇒ `Promise<Array<QueueOperationInfo>>`+ - [`.sleep`](#utils.sleep) ⇒ `Promise`+ - [`.URL_NO_COMMAS_REGEX`](#utils.URL_NO_COMMAS_REGEX)+ - [`.URL_WITH_COMMAS_REGEX`](#utils.URL_WITH_COMMAS_REGEX)+ - [`.isDocker()`](#utils.isDocker) ⇒ `Promise`+ - [`.downloadListOfUrls(options)`](#utils.downloadListOfUrls) ⇒ `Promise<Array<String>>`+ - [`.extractUrls(options)`](#utils.extractUrls) ⇒ `Array<String>`+ - [`.getRandomUserAgent()`](#utils.getRandomUserAgent) ⇒ `String`+ - [`.htmlToText(html)`](#utils.htmlToText) ⇒ `String`<a name="utils.enqueueLinks"></a>@@ -46,20 +46,17 @@ and override settings of the enqueued [`Request`](request) objects.**Example usage**```javascript-const Apify = require("apify");+const Apify = require('apify');const browser = await Apify.launchPuppeteer();-const page = await browser.goto("https://www.example.com");+const page = await browser.goto('https://www.example.com');const requestQueue = await Apify.openRequestQueue();await Apify.utils.enqueueLinks({- page,- requestQueue,- selector: "a.product-detail",- pseudoUrls: [- "https://www.example.com/handbags/[.*]",- "https://www.example.com/purses/[.*]"- ]+ page,+ requestQueue,+ selector: 'a.product-detail',+ pseudoUrls: ['https://www.example.com/handbags/[.*]', 'https://www.example.com/purses/[.*]'],});```@@ -265,7 +262,7 @@ to avoid merging distinct parts of text and thus enable extraction of data from**Example usage**```javascript-const text = htmlToText("<html><body>Some text</body></html>");+const text = htmlToText('<html><body>Some text</body></html>');console.log(text);```@@ -275,8 +272,8 @@ an existing Cheerio object to the function instead of the HTML text. The HTML shwith the `decodeEntities` option set to `true`. For example:```javascript-const cheerio = require("cheerio");-const html = "<html><body>Some text</body></html>";+const cheerio = require('cheerio');+const html = '<html><body>Some text</body></html>';const text = htmlToText(cheerio.load(html, { decodeEntities: true }));```diff --git a/docs/examples/BasicCrawler.md b/docs/examples/BasicCrawler.mdindex cd9a4903b..cb5e38fb9 100644--- a/docs/examples/BasicCrawler.md+++ b/docs/examples/BasicCrawler.md@@ -16,49 +16,49 @@ To run this example on the Apify Platform, select the `Node.js 8 on Alpine Linuxon the source tab of your actor configuration.```javascript-const Apify = require("apify");-const requestPromise = require("request-promise");+const Apify = require('apify');+const requestPromise = require('request-promise');// Apify.main() function wraps the crawler logic (it is optional).Apify.main(async () => {- // Create and initialize an instance of the RequestList class that contains- // a list of URLs to crawl. Here we use just a few hard-coded URLs.- const requestList = new Apify.RequestList({- sources: [- { url: "http://www.google.com/" },- { url: "http://www.example.com/" },- { url: "http://www.bing.com/" },- { url: "http://www.wikipedia.com/" }- ]- });- await requestList.initialize();+ // Create and initialize an instance of the RequestList class that contains+ // a list of URLs to crawl. Here we use just a few hard-coded URLs.+ const requestList = new Apify.RequestList({+ sources: [+ { url: 'http://www.google.com/' },+ { url: 'http://www.example.com/' },+ { url: 'http://www.bing.com/' },+ { url: 'http://www.wikipedia.com/' },+ ],+ });+ await requestList.initialize();- // Create a BasicCrawler - the simplest crawler that enables- // users to implement the crawling logic themselves.- const crawler = new Apify.BasicCrawler({- // Let the crawler fetch URLs from our list.- requestList,+ // Create a BasicCrawler - the simplest crawler that enables+ // users to implement the crawling logic themselves.+ const crawler = new Apify.BasicCrawler({+ // Let the crawler fetch URLs from our list.+ requestList,- // This function will be called for each URL to crawl.- // The 'request' option is an instance of the Request class, which contains- // information such as URL and HTTP method, as supplied by the RequestList.- handleRequestFunction: async ({ request }) => {- console.log(`Processing ${request.url}...`);+ // This function will be called for each URL to crawl.+ // The 'request' option is an instance of the Request class, which contains+ // information such as URL and HTTP method, as supplied by the RequestList.+ handleRequestFunction: async ({ request }) => {+ console.log(`Processing ${request.url}...`);- // Fetch the page HTML- const html = await requestPromise(request.url);+ // Fetch the page HTML+ const html = await requestPromise(request.url);- // Store the HTML and URL to the default dataset.- await Apify.pushData({- url: request.url,- html- });- }- });+ // Store the HTML and URL to the default dataset.+ await Apify.pushData({+ url: request.url,+ html,+ });+ },+ });- // Run the crawler and wait for it to finish.- await crawler.run();+ // Run the crawler and wait for it to finish.+ await crawler.run();- console.log("Crawler finished.");+ console.log('Crawler finished.');});```diff --git a/docs/examples/CallActor.md b/docs/examples/CallActor.mdindex 99f4222a8..02953b132 100644--- a/docs/examples/CallActor.md+++ b/docs/examples/CallActor.md@@ -20,36 +20,33 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- // Launch the web browser.- const browser = await Apify.launchPuppeteer();-- console.log("Obtaining email address...");- const user = await Apify.client.users.getUser();-- // Load Kraken.com charts and get last traded price of BTC- console.log("Extracting data from kraken.com...");- const page = await browser.newPage();- await page.goto("https://www.kraken.com/charts");- const tradedPricesHtml = await page.$eval(- "#ticker-top ul",- el => el.outerHTML- );-- // Send prices to your email. For that, you can use an actor we already- // have available on the platform under the name: apify/send-mail.- // The second parameter to the Apify.call() invocation is the actor's- // desired input. You can find the required input parameters by checking- // the actor's documentation page: https://apify.com/apify/send-mail- console.log(`Sending email to ${user.email}...`);- await Apify.call("apify/send-mail", {- to: user.email,- subject: "Kraken.com BTC",- html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`- });-- console.log("Email sent. Good luck!");+ // Launch the web browser.+ const browser = await Apify.launchPuppeteer();++ console.log('Obtaining email address...');+ const user = await Apify.client.users.getUser();++ // Load Kraken.com charts and get last traded price of BTC+ console.log('Extracting data from kraken.com...');+ const page = await browser.newPage();+ await page.goto('https://www.kraken.com/charts');+ const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);++ // Send prices to your email. For that, you can use an actor we already+ // have available on the platform under the name: apify/send-mail.+ // The second parameter to the Apify.call() invocation is the actor's+ // desired input. You can find the required input parameters by checking+ // the actor's documentation page: https://apify.com/apify/send-mail+ console.log(`Sending email to ${user.email}...`);+ await Apify.call('apify/send-mail', {+ to: user.email,+ subject: 'Kraken.com BTC',+ html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,+ });++ console.log('Email sent. Good luck!');});```diff --git a/docs/examples/CheerioCrawler.md b/docs/examples/CheerioCrawler.mdindex e5e88ac1d..b80ad8782 100644--- a/docs/examples/CheerioCrawler.md+++ b/docs/examples/CheerioCrawler.md@@ -12,7 +12,7 @@ To run this example on the Apify Platform, select the `Node.js 8 on Alpine Linuxon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');// Apify.utils contains various utilities, e.g. for logging.// Here we turn off the logging of unimportant messages.@@ -20,72 +20,71 @@ const { log } = Apify.utils;log.setLevel(log.LEVELS.WARNING);// A link to a list of Fortune 500 companies' websites available on GitHub.-const CSV_LINK =- "https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/f1000.csv";+const CSV_LINK = 'https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/f1000.csv';// Apify.main() function wraps the crawler logic (it is optional).Apify.main(async () => {- // Create an instance of the RequestList class that contains a list of URLs to crawl.- // Here we download and parse the list of URLs from an external file.- const requestList = new Apify.RequestList({- sources: [{ requestsFromUrl: CSV_LINK }]- });- await requestList.initialize();-- // Create an instance of the CheerioCrawler class - a crawler- // that automatically loads the URLs and parses their HTML using the cheerio library.- const crawler = new Apify.CheerioCrawler({- // Let the crawler fetch URLs from our list.- requestList,-- // The crawler downloads and processes the web pages in parallel, with a concurrency- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).- // Here we define some hard limits for the concurrency.- minConcurrency: 10,- maxConcurrency: 50,-- // On error, retry each page at most once.- maxRequestRetries: 1,-- // Increase the timeout for processing of each page.- handlePageTimeoutSecs: 60,-- // This function will be called for each URL to crawl.- // It accepts a single parameter, which is an object with the following fields:- // - request: an instance of the Request class with information such as URL and HTTP method- // - html: contains raw HTML of the page- // - $: the cheerio object containing parsed HTML- handlePageFunction: async ({ request, html, $ }) => {- console.log(`Processing ${request.url}...`);-- // Extract data from the page using cheerio.- const title = $("title").text();- const h1texts = [];- $("h1").each((index, el) => {- h1texts.push({- text: $(el).text()- });- });-- // Store the results to the default dataset. In local configuration,- // the data will be stored as JSON files in ./apify_storage/datasets/default- await Apify.pushData({- url: request.url,- title,- h1texts,- html- });- },-- // This function is called if the page processing failed more than maxRequestRetries+1 times.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed twice.`);- }- });-- // Run the crawler and wait for it to finish.- await crawler.run();-- console.log("Crawler finished.");+ // Create an instance of the RequestList class that contains a list of URLs to crawl.+ // Here we download and parse the list of URLs from an external file.+ const requestList = new Apify.RequestList({+ sources: [{ requestsFromUrl: CSV_LINK }],+ });+ await requestList.initialize();++ // Create an instance of the CheerioCrawler class - a crawler+ // that automatically loads the URLs and parses their HTML using the cheerio library.+ const crawler = new Apify.CheerioCrawler({+ // Let the crawler fetch URLs from our list.+ requestList,++ // The crawler downloads and processes the web pages in parallel, with a concurrency+ // automatically managed based on the available system memory and CPU (see AutoscaledPool class).+ // Here we define some hard limits for the concurrency.+ minConcurrency: 10,+ maxConcurrency: 50,++ // On error, retry each page at most once.+ maxRequestRetries: 1,++ // Increase the timeout for processing of each page.+ handlePageTimeoutSecs: 60,++ // This function will be called for each URL to crawl.+ // It accepts a single parameter, which is an object with the following fields:+ // - request: an instance of the Request class with information such as URL and HTTP method+ // - html: contains raw HTML of the page+ // - $: the cheerio object containing parsed HTML+ handlePageFunction: async ({ request, html, $ }) => {+ console.log(`Processing ${request.url}...`);++ // Extract data from the page using cheerio.+ const title = $('title').text();+ const h1texts = [];+ $('h1').each((index, el) => {+ h1texts.push({+ text: $(el).text(),+ });+ });++ // Store the results to the default dataset. In local configuration,+ // the data will be stored as JSON files in ./apify_storage/datasets/default+ await Apify.pushData({+ url: request.url,+ title,+ h1texts,+ html,+ });+ },++ // This function is called if the page processing failed more than maxRequestRetries+1 times.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed twice.`);+ },+ });++ // Run the crawler and wait for it to finish.+ await crawler.run();++ console.log('Crawler finished.');});```diff --git a/docs/examples/HelloWorld.md b/docs/examples/HelloWorld.mdindex 842669d70..90f6c2327 100644--- a/docs/examples/HelloWorld.md+++ b/docs/examples/HelloWorld.md@@ -9,29 +9,24 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://www.iana.org/" });- const pseudoUrls = [new Apify.PseudoUrl("https://www.iana.org/[.*]")];+ const requestQueue = await Apify.openRequestQueue();+ await requestQueue.addRequest({ url: 'https://www.iana.org/' });+ const pseudoUrls = [new Apify.PseudoUrl('https://www.iana.org/[.*]')];- const crawler = new Apify.PuppeteerCrawler({- requestQueue,- handlePageFunction: async ({ request, page }) => {- const title = await page.title();- console.log(`Title of ${request.url}: ${title}`);- await Apify.utils.enqueueLinks({- page,- selector: "a",- pseudoUrls,- requestQueue- });- },- maxRequestsPerCrawl: 100,- maxConcurrency: 10- });+ const crawler = new Apify.PuppeteerCrawler({+ requestQueue,+ handlePageFunction: async ({ request, page }) => {+ const title = await page.title();+ console.log(`Title of ${request.url}: ${title}`);+ await Apify.utils.enqueueLinks({ page, selector: 'a', pseudoUrls, requestQueue });+ },+ maxRequestsPerCrawl: 100,+ maxConcurrency: 10,+ });- await crawler.run();+ await crawler.run();});```diff --git a/docs/examples/PuppeteerCrawler.md b/docs/examples/PuppeteerCrawler.mdindex f3043ee00..8cf46ee4c 100644--- a/docs/examples/PuppeteerCrawler.md+++ b/docs/examples/PuppeteerCrawler.md@@ -14,79 +14,79 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- // Apify.openRequestQueue() is a factory to get a preconfigured RequestQueue instance.- // We add our first request to it - the initial page the crawler will visit.- const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://news.ycombinator.com/" });-- // Create an instance of the PuppeteerCrawler class - a crawler- // that automatically loads the URLs in headless Chrome / Puppeteer.- const crawler = new Apify.PuppeteerCrawler({- requestQueue,-- // Here you can set options that are passed to the Apify.launchPuppeteer() function.- launchPuppeteerOptions: {- // For example, by adding "slowMo" you'll slow down Puppeteer operations to simplify debugging- // slowMo: 500,- },-- // Stop crawling after several pages- maxRequestsPerCrawl: 10,-- // This function will be called for each URL to crawl.- // Here you can write the Puppeteer scripts you are familiar with,- // with the exception that browsers and pages are automatically managed by the Apify SDK.- // The function accepts a single parameter, which is an object with the following fields:- // - request: an instance of the Request class with information such as URL and HTTP method- // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)- handlePageFunction: async ({ request, page }) => {- console.log(`Processing ${request.url}...`);-- // A function to be evaluated by Puppeteer within the browser context.- const pageFunction = $posts => {- const data = [];-- // We're getting the title, rank and URL of each post on Hacker News.- $posts.forEach($post => {- data.push({- title: $post.querySelector(".title a").innerText,- rank: $post.querySelector(".rank").innerText,- href: $post.querySelector(".title a").href- });- });-- return data;- };- const data = await page.$$eval(".athing", pageFunction);-- // Store the results to the default dataset.- await Apify.pushData(data);-- // Find a link to the next page and enqueue it if it exists.- const infos = await Apify.utils.enqueueLinks({- page,+ // Apify.openRequestQueue() is a factory to get a preconfigured RequestQueue instance.+ // We add our first request to it - the initial page the crawler will visit.+ const requestQueue = await Apify.openRequestQueue();+ await requestQueue.addRequest({ url: 'https://news.ycombinator.com/' });++ // Create an instance of the PuppeteerCrawler class - a crawler+ // that automatically loads the URLs in headless Chrome / Puppeteer.+ const crawler = new Apify.PuppeteerCrawler({requestQueue,- selector: ".morelink"- });- if (infos.length === 0) console.log(`${request.url} is the last page!`);- },-- // This function is called if the page processing failed more than maxRequestRetries+1 times.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed too many times`);- await Apify.pushData({- "#debug": Apify.utils.createRequestDebugInfo(request)- });- }- });-- // Run the crawler and wait for it to finish.- await crawler.run();-- console.log("Crawler finished.");+ // Here you can set options that are passed to the Apify.launchPuppeteer() function.+ launchPuppeteerOptions: {+ // For example, by adding "slowMo" you'll slow down Puppeteer operations to simplify debugging+ // slowMo: 500,+ },++ // Stop crawling after several pages+ maxRequestsPerCrawl: 10,++ // This function will be called for each URL to crawl.+ // Here you can write the Puppeteer scripts you are familiar with,+ // with the exception that browsers and pages are automatically managed by the Apify SDK.+ // The function accepts a single parameter, which is an object with the following fields:+ // - request: an instance of the Request class with information such as URL and HTTP method+ // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)+ handlePageFunction: async ({ request, page }) => {+ console.log(`Processing ${request.url}...`);++ // A function to be evaluated by Puppeteer within the browser context.+ const pageFunction = $posts => {+ const data = [];++ // We're getting the title, rank and URL of each post on Hacker News.+ $posts.forEach($post => {+ data.push({+ title: $post.querySelector('.title a').innerText,+ rank: $post.querySelector('.rank').innerText,+ href: $post.querySelector('.title a').href,+ });+ });++ return data;+ };+ const data = await page.$$eval('.athing', pageFunction);++ // Store the results to the default dataset.+ await Apify.pushData(data);++ // Find a link to the next page and enqueue it if it exists.+ const infos = await Apify.utils.enqueueLinks({+ page,+ requestQueue,+ selector: '.morelink',+ });++ if (infos.length === 0) console.log(`${request.url} is the last page!`);+ },++ // This function is called if the page processing failed more than maxRequestRetries+1 times.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed too many times`);+ await Apify.pushData({+ '#debug': Apify.utils.createRequestDebugInfo(request),+ });+ },+ });++ // Run the crawler and wait for it to finish.+ await crawler.run();++ console.log('Crawler finished.');});```diff --git a/docs/examples/PuppeteerSitemap.md b/docs/examples/PuppeteerSitemap.mdindex 68ed50425..6ab693bf6 100644--- a/docs/examples/PuppeteerSitemap.md+++ b/docs/examples/PuppeteerSitemap.md@@ -12,29 +12,27 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- const requestList = new Apify.RequestList({- sources: [- { requestsFromUrl: "https://edition.cnn.com/sitemaps/cnn/news.xml" }- ]- });- await requestList.initialize();+ const requestList = new Apify.RequestList({+ sources: [{ requestsFromUrl: 'https://edition.cnn.com/sitemaps/cnn/news.xml' }],+ });+ await requestList.initialize();- const crawler = new Apify.PuppeteerCrawler({- requestList,- handlePageFunction: async ({ page, request }) => {- console.log(`Processing ${request.url}...`);- await Apify.pushData({- url: request.url,- title: await page.title(),- html: await page.content()- });- }- });+ const crawler = new Apify.PuppeteerCrawler({+ requestList,+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);+ await Apify.pushData({+ url: request.url,+ title: await page.title(),+ html: await page.content(),+ });+ },+ });- await crawler.run();- console.log("Done.");+ await crawler.run();+ console.log('Done.');});```diff --git a/docs/examples/PuppeteerWithProxy.md b/docs/examples/PuppeteerWithProxy.mdindex 00c920d54..8acfb5588 100644--- a/docs/examples/PuppeteerWithProxy.md+++ b/docs/examples/PuppeteerWithProxy.md@@ -15,29 +15,29 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- // Apify.launchPuppeteer() is similar to Puppeteer's launch() function.- // It accepts the same parameters and returns a preconfigured Puppeteer.Browser instance.- // Moreover, it accepts several additional options, such as useApifyProxy.- const options = {- useApifyProxy: true- };- const browser = await Apify.launchPuppeteer(options);-- console.log("Running Puppeteer script...");-- // Proceed with a plain Puppeteer script.- const page = await browser.newPage();- const url = "https://en.wikipedia.org/wiki/Main_Page";- await page.goto(url);- const title = await page.title();-- console.log(`Page title: ${title}`);-- // Cleaning up after yourself is always good.- await browser.close();- console.log("Puppeteer closed.");+ // Apify.launchPuppeteer() is similar to Puppeteer's launch() function.+ // It accepts the same parameters and returns a preconfigured Puppeteer.Browser instance.+ // Moreover, it accepts several additional options, such as useApifyProxy.+ const options = {+ useApifyProxy: true,+ };+ const browser = await Apify.launchPuppeteer(options);++ console.log('Running Puppeteer script...');++ // Proceed with a plain Puppeteer script.+ const page = await browser.newPage();+ const url = 'https://en.wikipedia.org/wiki/Main_Page';+ await page.goto(url);+ const title = await page.title();++ console.log(`Page title: ${title}`);++ // Cleaning up after yourself is always good.+ await browser.close();+ console.log('Puppeteer closed.');});```diff --git a/docs/examples/Screenshots.md b/docs/examples/Screenshots.mdindex a244b8111..8fb10d78f 100644--- a/docs/examples/Screenshots.md+++ b/docs/examples/Screenshots.md@@ -16,12 +16,7 @@ In local configuration, the input is stored in the default key-value store's dir`./apify_storage/key_value_stores/default/INPUT.json`. You need to create the file and set it with the following content:```json-{- "sources": [- { "url": "https://www.google.com" },- { "url": "https://www.duckduckgo.com" }- ]-}+{ "sources": [{ "url": "https://www.google.com" }, { "url": "https://www.duckduckgo.com" }] }```On the Apify cloud, the input can be either set manually@@ -34,41 +29,41 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- // Read the actor input configuration containing the URLs for the screenshot.- // By convention, the input is present in the actor's default key-value store under the "INPUT" key.- const input = await Apify.getInput();- if (!input) throw new Error("Have you passed the correct INPUT ?");+ // Read the actor input configuration containing the URLs for the screenshot.+ // By convention, the input is present in the actor's default key-value store under the "INPUT" key.+ const input = await Apify.getInput();+ if (!input) throw new Error('Have you passed the correct INPUT ?');- const { sources } = input;+ const { sources } = input;- const requestList = new Apify.RequestList({ sources });- await requestList.initialize();+ const requestList = new Apify.RequestList({ sources });+ await requestList.initialize();- const crawler = new Apify.PuppeteerCrawler({- requestList,- handlePageFunction: async ({ page, request }) => {- console.log(`Processing ${request.url}...`);+ const crawler = new Apify.PuppeteerCrawler({+ requestList,+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);- // This is a Puppeteer function that takes a screenshot of the page and returns its buffer.- const screenshotBuffer = await page.screenshot();+ // This is a Puppeteer function that takes a screenshot of the page and returns its buffer.+ const screenshotBuffer = await page.screenshot();- // The record key may only include the following characters: a-zA-Z0-9!-_.'()- const key = request.url.replace(/[:/]/g, "_");+ // The record key may only include the following characters: a-zA-Z0-9!-_.'()+ const key = request.url.replace(/[:/]/g, '_');- // Save the screenshot. Choosing the right content type will automatically- // assign the local file the right extension, in this case .png.- // The screenshots will be stored in ./apify_storage/key_value_stores/default/- await Apify.setValue(key, screenshotBuffer, { contentType: "image/png" });- console.log(`Screenshot of ${request.url} saved.`);- }- });+ // Save the screenshot. Choosing the right content type will automatically+ // assign the local file the right extension, in this case .png.+ // The screenshots will be stored in ./apify_storage/key_value_stores/default/+ await Apify.setValue(key, screenshotBuffer, { contentType: 'image/png' });+ console.log(`Screenshot of ${request.url} saved.`);+ },+ });- // Run crawler.- await crawler.run();+ // Run crawler.+ await crawler.run();- console.log("Crawler finished.");+ console.log('Crawler finished.');});```diff --git a/docs/examples/SynchronousRun.md b/docs/examples/SynchronousRun.mdindex 356ba4655..35990edd4 100644--- a/docs/examples/SynchronousRun.md+++ b/docs/examples/SynchronousRun.md@@ -22,32 +22,32 @@ To run this example on the Apify Platform, select the `Node.js 8 + Chrome on Debon the source tab of your actor configuration.```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {- // Launch web browser.- const browser = await Apify.launchPuppeteer();-- // Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam stream- console.log("Opening web page...");- const page = await browser.newPage();- await page.goto("http://goldengatebridge75.org/news/webcam.html");- const iframe = (await page.frames()).pop();-- // Get webcam image element handle.- const imageElementHandle = await iframe.$(".VideoColm img");-- // Give the webcam image some time to load.- console.log("Waiting for page to load...");- await Apify.utils.sleep(3000);-- // Get a screenshot of that image.- const imageBuffer = await imageElementHandle.screenshot();- console.log("Screenshot captured.");-- // Save the screenshot as the actor's output. By convention, similarly to "INPUT",- // the actor's output is stored in the default key-value store under the "OUTPUT" key.- await Apify.setValue("OUTPUT", imageBuffer, { contentType: "image/jpeg" });- console.log("Actor finished.");+ // Launch web browser.+ const browser = await Apify.launchPuppeteer();++ // Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam stream+ console.log('Opening web page...');+ const page = await browser.newPage();+ await page.goto('http://goldengatebridge75.org/news/webcam.html');+ const iframe = (await page.frames()).pop();++ // Get webcam image element handle.+ const imageElementHandle = await iframe.$('.VideoColm img');++ // Give the webcam image some time to load.+ console.log('Waiting for page to load...');+ await Apify.utils.sleep(3000);++ // Get a screenshot of that image.+ const imageBuffer = await imageElementHandle.screenshot();+ console.log('Screenshot captured.');++ // Save the screenshot as the actor's output. By convention, similarly to "INPUT",+ // the actor's output is stored in the default key-value store under the "OUTPUT" key.+ await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });+ console.log('Actor finished.');});```diff --git a/docs/guides/data_storage.md b/docs/guides/data_storage.mdindex 57af6a997..eda2a8060 100644--- a/docs/guides/data_storage.md+++ b/docs/guides/data_storage.md@@ -3,16 +3,14 @@ id: datastoragetitle: Data Storage----The Apify SDK has several data storage types that are useful for specific tasks.-The data is stored either on local disk to a directory defined by the `APIFY_LOCAL_STORAGE_DIR` environment variable,-or on the <a href="https://apify.com" target="_blank">Apify Cloud</a> under the user account identified by the API token defined by the `APIFY_TOKEN` environment variable.-If neither of these variables is defined, by default Apify SDK sets `APIFY_LOCAL_STORAGE_DIR`-to `./apify_storage` in the current working directory and prints a warning.+The Apify SDK has several data storage types that are useful for specific tasks. The data is stored either on local disk to a directory defined by the+`APIFY_LOCAL_STORAGE_DIR` environment variable, or on the <a href="https://apify.com" target="_blank">Apify Cloud</a> under the user account+identified by the API token defined by the `APIFY_TOKEN` environment variable. If neither of these variables is defined, by default Apify SDK sets+`APIFY_LOCAL_STORAGE_DIR` to `./apify_storage` in the current working directory and prints a warning.-Typically, you will be developing the code on your local computer and thus set the `APIFY_LOCAL_STORAGE_DIR` environment variable.-Once the code is ready, you will deploy it to the Apify Cloud, where it will automatically-set the `APIFY_TOKEN` environment variable and thus use cloud storage.-No code changes are needed.+Typically, you will be developing the code on your local computer and thus set the `APIFY_LOCAL_STORAGE_DIR` environment variable. Once the code is+ready, you will deploy it to the Apify Cloud, where it will automatically set the `APIFY_TOKEN` environment variable and thus use cloud storage. No+code changes are needed.**Related links**@@ -22,21 +20,16 @@ No code changes are needed.## Key-value store-The key-value store is used for saving and reading data records or files.-Each data record is represented by a unique key and associated with a MIME content type.-Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.+The key-value store is used for saving and reading data records or files. Each data record is represented by a unique key and associated with a MIME+content type. Key-value stores are ideal for saving screenshots of web pages, PDFs or to persist the state of crawlers.-Each actor run is associated with a **default key-value store**, which is created exclusively for the actor run.-By convention, the actor run input and output is stored in the default key-value store-under the `INPUT` and `OUTPUT` key, respectively. Typically the input and output is a JSON file,+Each actor run is associated with a **default key-value store**, which is created exclusively for the actor run. By convention, the actor run input+and output is stored in the default key-value store under the `INPUT` and `OUTPUT` key, respectively. Typically the input and output is a JSON file,although it can be any other format.-In the Apify SDK, the key-value store is represented by the-[`KeyValueStore`](../api/keyvaluestore)-class.-In order to simplify access to the default key-value store, the SDK also provides-[`Apify.getValue()`](../api/apify#module_Apify.getValue)-and [`Apify.setValue()`](../api/apify#module_Apify.setValue) functions.+In the Apify SDK, the key-value store is represented by the [`KeyValueStore`](../api/keyvaluestore) class. In order to simplify access to the default+key-value store, the SDK also provides [`Apify.getValue()`](../api/apify#module_Apify.getValue) and+[`Apify.setValue()`](../api/apify#module_Apify.setValue) functions.In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:@@ -44,10 +37,8 @@ In local configuration, the data is stored in the directory specified by the `AP{APIFY_LOCAL_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}```-Note that `{STORE_ID}` is the name or ID of the key-value store.-The default key value store has ID `default`, unless you override it by setting the `APIFY_DEFAULT_KEY_VALUE_STORE_ID`-environment variable.-The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the+Note that `{STORE_ID}` is the name or ID of the key-value store. The default key value store has ID `default`, unless you override it by setting the+`APIFY_DEFAULT_KEY_VALUE_STORE_ID` environment variable. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of thedata value.The following code demonstrates basic operations of key-value stores:@@ -57,47 +48,42 @@ The following code demonstrates basic operations of key-value stores:const input = await Apify.getInput();// Write actor output to the default key-value store.-await Apify.setValue("OUTPUT", { myResult: 123 });+await Apify.setValue('OUTPUT', { myResult: 123 });// Open a named key-value store-const store = await Apify.openKeyValueStore("some-name");+const store = await Apify.openKeyValueStore('some-name');// Write record. JavaScript object is automatically converted to JSON,// strings and binary buffers are stored as they are-await store.setValue("some-key", { foo: "bar" });+await store.setValue('some-key', { foo: 'bar' });// Read record. Note that JSON is automatically parsed to a JavaScript object,// text data returned as a string and other data is returned as binary buffer-const value = await store.getValue("some-key");+const value = await store.getValue('some-key');// Delete record-await store.setValue("some-key", null);+await store.setValue('some-key', null);```-To see a real-world example of how to get the input from the key-value store, see the-[Screenshots](../examples/screenshots) example.+To see a real-world example of how to get the input from the key-value store, see the [Screenshots](../examples/screenshots) example.## Dataset-Datasets are used to store structured data-where each object stored has the same attributes, such as online store products or real estate offers.-You can imagine a dataset as a table, where each object is a row and its attributes are columns.-Dataset is an append-only storage - you can only add new records to it but you cannot modify or remove-existing records.+Datasets are used to store structured data where each object stored has the same attributes, such as online store products or real estate offers. You+can imagine a dataset as a table, where each object is a row and its attributes are columns. Dataset is an append-only storage - you can only add new+records to it but you cannot modify or remove existing records.-When the dataset is stored in the <a href="https://apify.com" target="_blank">Apify Cloud</a>,-you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS.-The datasets are displayed on the actor run details page and in the <a href="https://my.apify.com/storage" target="_blank">Storage</a>-section in the Apify app. The actual data is exported using the-<a href="https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items" target="_blank">Get dataset items</a> Apify API endpoint.-This way you can easily share crawling results.+When the dataset is stored in the <a href="https://apify.com" target="_blank">Apify Cloud</a>, you can export its data to the following formats: HTML,+JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the+<a href="https://my.apify.com/storage" target="_blank">Storage</a> section in the Apify app. The actual data is exported using the+<a href="https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items" target="_blank">Get dataset items</a> Apify API endpoint. This+way you can easily share crawling results.-Each actor run is associated with a **default dataset**, which is created exclusively for the actor run.-Typically, it is used to store crawling results specific for the actor run. Its usage is optional.+Each actor run is associated with a **default dataset**, which is created exclusively for the actor run. Typically, it is used to store crawling+results specific for the actor run. Its usage is optional.-In the Apify SDK, the dataset is represented by the [`Dataset`](../api/dataset) class.-In order to simplify writes to the default dataset, the SDK also provides the-[`Apify.pushData()`](../api/apify#module_Apify.pushData) function.+In the Apify SDK, the dataset is represented by the [`Dataset`](../api/dataset) class. In order to simplify writes to the default dataset, the SDK+also provides the [`Apify.pushData()`](../api/apify#module_Apify.pushData) function.In local configuration, the data is stored in the directory specified by the `APIFY_LOCAL_STORAGE_DIR` environment variable as follows:@@ -105,40 +91,35 @@ In local configuration, the data is stored in the directory specified by the `AP{APIFY_LOCAL_STORAGE_DIR}/datasets/{DATASET_ID}/{INDEX}.json```-Note that `{DATASET_ID}` is the name or ID of the dataset.-The default dataset has ID `default`, unless you override it by setting the `APIFY_DEFAULT_DATASET_ID`-environment variable.-Each dataset item is stored as a separate JSON file,-where `{INDEX}` is a zero-based index of the item in the dataset.+Note that `{DATASET_ID}` is the name or ID of the dataset. The default dataset has ID `default`, unless you override it by setting the+`APIFY_DEFAULT_DATASET_ID` environment variable. Each dataset item is stored as a separate JSON file, where `{INDEX}` is a zero-based index of the+item in the dataset.The following code demonstrates basic operations of the dataset:```javascript// Write a single row to the default dataset-await Apify.pushData({ col1: 123, col2: "val2" });+await Apify.pushData({ col1: 123, col2: 'val2' });// Open a named dataset-const dataset = await Apify.openDataset("some-name");+const dataset = await Apify.openDataset('some-name');// Write a single row-await dataset.pushData({ foo: "bar" });+await dataset.pushData({ foo: 'bar' });// Write multiple rows-await dataset.pushData([{ foo: "bar2", col2: "val2" }, { col3: 123 }]);+await dataset.pushData([{ foo: 'bar2', col2: 'val2' }, { col3: 123 }]);```-To see how to use the dataset to store crawler results, see the-[Cheerio Crawler](../examples/cheeriocrawler) example.+To see how to use the dataset to store crawler results, see the [Cheerio Crawler](../examples/cheeriocrawler) example.## Request queue-The request queue is a storage of URLs to crawl.-The queue is used for the deep crawling of websites, where you start with-several URLs and then recursively follow links to other pages.-The data structure supports both breadth-first and depth-first crawling orders.+The request queue is a storage of URLs to crawl. The queue is used for the deep crawling of websites, where you start with several URLs and then+recursively follow links to other pages. The data structure supports both breadth-first and depth-first crawling orders.-Each actor run is associated with a **default request queue**, which is created exclusively for the actor run.-Typically, it is used to store URLs to crawl in the specific actor run. Its usage is optional.+Each actor run is associated with a **default request queue**, which is created exclusively for the actor run. Typically, it is used to store URLs to+crawl in the specific actor run. Its usage is optional.In Apify SDK, the request queue is represented by the [`RequestQueue`](../api/requestqueue) class.@@ -148,12 +129,9 @@ In local configuration, the request queue data is stored in the directory specif{APIFY_LOCAL_STORAGE_DIR}/request_queues/{QUEUE_ID}/{STATE}/{NUMBER}.json```-Note that `{QUEUE_ID}` is the name or ID of the request queue.-The default queue has ID `default`, unless you override it by setting the `APIFY_DEFAULT_REQUEST_QUEUE_ID`-environment variable.-Each request in the queue is stored as a separate JSON file,-where `{STATE}` is either `handled` or `pending`,-and `{NUMBER}` is an integer indicating the position of the request in the queue.+Note that `{QUEUE_ID}` is the name or ID of the request queue. The default queue has ID `default`, unless you override it by setting the+`APIFY_DEFAULT_REQUEST_QUEUE_ID` environment variable. Each request in the queue is stored as a separate JSON file, where `{STATE}` is either+`handled` or `pending`, and `{NUMBER}` is an integer indicating the position of the request in the queue.The following code demonstrates basic operations of the request queue:@@ -162,15 +140,12 @@ The following code demonstrates basic operations of the request queue:const queue = await Apify.openRequestQueue();// Open a named request queue-const queueWithName = await Apify.openRequestQueue("some-name");+const queueWithName = await Apify.openRequestQueue('some-name');// Enqueue few requests-await queue.addRequest({ url: "http://example.com/aaa" });-await queue.addRequest({ url: "http://example.com/bbb" });-await queue.addRequest(- { url: "http://example.com/foo/bar" },- { forefront: true }-);+await queue.addRequest({ url: 'http://example.com/aaa' });+await queue.addRequest({ url: 'http://example.com/bbb' });+await queue.addRequest({ url: 'http://example.com/foo/bar' }, { forefront: true });// Get requests from queueconst request1 = await queue.fetchNextRequest();@@ -184,5 +159,4 @@ await queue.markRequestHandled(request1);await queue.reclaimRequest(request2);```-To see how to use the request queue with a crawler, see the-[Puppeteer Crawler](../examples/puppeteercrawler) example.+To see how to use the request queue with a crawler, see the [Puppeteer Crawler](../examples/puppeteercrawler) example.diff --git a/docs/guides/getting_started.md b/docs/guides/getting_started.mdindex 574722181..39754845c 100644--- a/docs/guides/getting_started.md+++ b/docs/guides/getting_started.md@@ -3,24 +3,31 @@ id: gettingstartedtitle: Getting Started----Without the right tools, crawling and scraping the web can be a difficult thing. At the very least, you need an HTTP client to make the necessary requests, but that only gets you raw HTML and sometimes not even that. Then you have to read this HTML and extract the data you're interested in. Once extracted, it must be stored in a machine readable format and easily accessible for further processing, because it is the processed data that hold value.+Without the right tools, crawling and scraping the web can be a difficult thing. At the very least, you need an HTTP client to make the necessary+requests, but that only gets you raw HTML and sometimes not even that. Then you have to read this HTML and extract the data you're interested in. Once+extracted, it must be stored in a machine readable format and easily accessible for further processing, because it is the processed data that hold+value.-Apify SDK covers the process end-to-end. From crawling the web for links and scraping the raw data to storing it in various machine readable formats, ready for processing. With this guide in hand, you should have your own data extraction solutions up and running in a few hours.+Apify SDK covers the process end-to-end. From crawling the web for links and scraping the raw data to storing it in various machine readable formats,+ready for processing. With this guide in hand, you should have your own data extraction solutions up and running in a few hours.## Intro-The goal of this getting started guide is to provide a step-by-step introduction to all the features of the Apify SDK. It will walk you through creating the simplest of crawlers that only print text to console, all the way up to complex systems that crawl pages, interact with them as if a real user were sitting in front of a real browser and output structured data.+The goal of this getting started guide is to provide a step-by-step introduction to all the features of the Apify SDK. It will walk you through+creating the simplest of crawlers that only print text to console, all the way up to complex systems that crawl pages, interact with them as if a real+user were sitting in front of a real browser and output structured data.-Since Apify SDK is usable both locally on any computer and on the-<a href="https://my.apify.com" target="_blank">Apify Platform</a>,-you will be able to use the source code in both environments interchangeably. Nevertheless, some initial setup is still required, so choose your preferred starting environment and let's get into it.+Since Apify SDK is usable both locally on any computer and on the <a href="https://my.apify.com" target="_blank">Apify Platform</a>, you will be able+to use the source code in both environments interchangeably. Nevertheless, some initial setup is still required, so choose your preferred starting+environment and let's get into it.## Setting up locallyTo run Apify SDK on your own computer, you need to meet the following pre-requisites first:1. Have Node.js version 8.14 or higher installed.- - Visit <a href="https://nodejs.org/en/download/" target="_blank">Node.js website</a> to download or use <a href="https://github.com/creationix/nvm" target="_blank">nvm</a>+ - Visit <a href="https://nodejs.org/en/download/" target="_blank">Node.js website</a> to download or use+ <a href="https://github.com/creationix/nvm" target="_blank">nvm</a>2. Have NPM installed.- NPM comes bundled with Node.js so you should already have it. If not, reinstall Node.js.@@ -37,8 +44,9 @@ npm -v### Creating a new projectThe fastest and best way to create new projects with the Apify SDK is to use our own-<a href="https://www.npmjs.com/package/apify-cli" target="_blank">Apify CLI</a>.-This command line tool allows you to create, run and manage Apify projects with ease, including their deployment to the <a href="https://my.apify.com" target="_blank">Apify Platform</a> if you wish to run them in the cloud after developing them locally.+<a href="https://www.npmjs.com/package/apify-cli" target="_blank">Apify CLI</a>. This command line tool allows you to create, run and manage Apify+projects with ease, including their deployment to the <a href="https://my.apify.com" target="_blank">Apify Platform</a> if you wish to run them in the+cloud after developing them locally.Let's install the Apify CLI with the following command:@@ -52,7 +60,9 @@ Once the installation finishes, all you need to do to set up an Apify SDK projecapify create my-new-project```-A prompt will be shown, asking to choose a template. Disregard the different options for now and choose the template labeled `Hello world`. The command will now create a new directory in your current working directory, called `my-new-project`, create a `package.json` in this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.+A prompt will be shown, asking to choose a template. Disregard the different options for now and choose the template labeled `Hello world`. The+command will now create a new directory in your current working directory, called `my-new-project`, create a `package.json` in this folder and install+all the necessary dependencies. It will also add example source code that you can immediately run.Let's try that!@@ -64,9 +74,14 @@ cd my-new-projectapify run -p```-> The `-p` flag is great to remember, because it stands for `--purge` and it clears out your persistent storages before starting the actor. `INPUT.json` and named storages are kept. Whenever you're just restarting your actor and you're not interested in the data of the previous run, you should use `apify run -p` to prevent the old state from messing with your current run. If this is confusing, don't worry. You'll learn about storages and `INPUT.json` soon.+> The `-p` flag is great to remember, because it stands for `--purge` and it clears out your persistent storages before starting the actor.+> `INPUT.json` and named storages are kept. Whenever you're just restarting your actor and you're not interested in the data of the previous run, you+> should use `apify run -p` to prevent the old state from messing with your current run. If this is confusing, don't worry. You'll learn about+> storages and `INPUT.json` soon.-You should start seeing log messages in the terminal as the system boots up and after a second, a Chromium browser window should pop up. In the window, you'll see quickly changing pages and back in the terminal, you should see the titles (contents of the `<title>` HTML tags) of the pages printed.+You should start seeing log messages in the terminal as the system boots up and after a second, a Chromium browser window should pop up. In the+window, you'll see quickly changing pages and back in the terminal, you should see the titles (contents of the `<title>` HTML tags) of the pages+printed.You can always terminate the crawl with a keypress in the terminal:@@ -78,51 +93,78 @@ Did you see all that? If you did, congratulations! You're ready to go!## Setting up on the Apify Platform-Maybe you don't have Node.js installed and don't want the hassle. Or you can't install anything on your computer because you're using a company provided one. Or perhaps you'd just prefer to start working in the cloud right away. Well, no worries, we've got you covered.+Maybe you don't have Node.js installed and don't want the hassle. Or you can't install anything on your computer because you're using a company+provided one. Or perhaps you'd just prefer to start working in the cloud right away. Well, no worries, we've got you covered.-The <a href="https://my.apify.com" target="_blank">Apify Platform</a> is the foundational product of <a href="https://apify.com" target="_blank">Apify</a>. It's a serverless cloud computing platform, specifically designed for any web automation jobs, that may include crawling and scraping, but really works amazing for any batch jobs and long running tasks.+The <a href="https://my.apify.com" target="_blank">Apify Platform</a> is the foundational product of+<a href="https://apify.com" target="_blank">Apify</a>. It's a serverless cloud computing platform, specifically designed for any web automation jobs,+that may include crawling and scraping, but really works amazing for any batch jobs and long running tasks.-It comes with a free account, so let's go to our-<a href="https://my.apify.com/sign-up" target="_blank">sign-up page</a>-and create one, if you haven't already. Don't forget to verify your email. Without it, you won't be able to run any projects.+It comes with a free account, so let's go to our <a href="https://my.apify.com/sign-up" target="_blank">sign-up page</a> and create one, if you+haven't already. Don't forget to verify your email. Without it, you won't be able to run any projects.-Once you're in, you might be prompted by our in-app help to walk through a step-by-step guide into some of our new features. Feel free to finish that, if you'd like, but once you're done, click on the **Actors** tab in the left menu. To read more about **Actors**, see: [What is an Actor](./whatisanactor).+Once you're in, you might be prompted by our in-app help to walk through a step-by-step guide into some of our new features. Feel free to finish that,+if you'd like, but once you're done, click on the **Actors** tab in the left menu. To read more about **Actors**, see:+[What is an Actor](./whatisanactor).### Creating a new project-In the page that shows after clicking on Actors in the left menu, choose **Create new**. Give it a name in the form that opens, let's say, `my-new-actor`. Disregard all the available options for now and save your changes.+In the page that shows after clicking on Actors in the left menu, choose **Create new**. Give it a name in the form that opens, let's say,+`my-new-actor`. Disregard all the available options for now and save your changes.-Now click on the **Sources** tab at the top. Disregard the version and environment variables inputs for now and proceed directly to **Source code**. This is where you develop the actor, if you choose not to do it locally. Just press **Run** below the **Source code** panel. It will automatically build and run the example source code. You should start seeing log messages that represent the build and after the build is complete, log messages of the running actor. Feel free to check out the other **Run** tabs, such as **Info**, where you can find useful information about the run, or **Key-value-store**, where the actor's **INPUT** and **OUTPUT** are stored.+Now click on the **Sources** tab at the top. Disregard the version and environment variables inputs for now and proceed directly to **Source code**.+This is where you develop the actor, if you choose not to do it locally. Just press **Run** below the **Source code** panel. It will automatically+build and run the example source code. You should start seeing log messages that represent the build and after the build is complete, log messages of+the running actor. Feel free to check out the other **Run** tabs, such as **Info**, where you can find useful information about the run, or+**Key-value-store**, where the actor's **INPUT** and **OUTPUT** are stored.Good job. You're now ready to run your own source code on the Apify Platform. For more information visit the-<a href="https://apify.com/docs/actor" target="_blank">Actor documentation page</a>,-where you'll find everything about the platform's various options.+<a href="https://apify.com/docs/actor" target="_blank">Actor documentation page</a>, where you'll find everything about the platform's various+options.## First crawler-Whether you've chosen to develop locally or in the cloud, it's time to start writing some actual source code. But before we do, let me just briefly introduce all the Apify SDK classes necessary to make it happen.+Whether you've chosen to develop locally or in the cloud, it's time to start writing some actual source code. But before we do, let me just briefly+introduce all the Apify SDK classes necessary to make it happen.### The general idea-There are 3 crawler classes available for use in the Apify SDK. [`BasicCrawler`](../api/basiccrawler), [`CheerioCrawler`](../api/cheeriocrawler) and [`PuppeteerCrawler`](../api/puppeteercrawler). We'll talk about their differences later. Now, let's talk about what they have in common.+There are 3 crawler classes available for use in the Apify SDK. [`BasicCrawler`](../api/basiccrawler), [`CheerioCrawler`](../api/cheeriocrawler) and+[`PuppeteerCrawler`](../api/puppeteercrawler). We'll talk about their differences later. Now, let's talk about what they have in common.-All the crawlers' general idea is to go to a web page, open it, do some stuff there, save some results and continue to the next page, until it's done its job. So each time the crawler needs to find answers to two questions: **Where should I go?** and **What should I do there?**. Answering those two questions is the only setup mandatory to run the crawlers.+All the crawlers' general idea is to go to a web page, open it, do some stuff there, save some results and continue to the next page, until it's done+its job. So each time the crawler needs to find answers to two questions: **Where should I go?** and **What should I do there?**. Answering those two+questions is the only setup mandatory to run the crawlers.### The Where - `Request`, `RequestList` and `RequestQueue`-All crawlers use instances of the [`Request`](../api/request) class to determine where they need to go. Each request may hold a lot of information, but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. We need to either have a pre-existing list of our own URLs that we wish to visit, perhaps a thousand, or a million, or we need to build this list dynamically as we crawl, adding more and more URLs to the list as we progress.+All crawlers use instances of the [`Request`](../api/request) class to determine where they need to go. Each request may hold a lot of information,+but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. We need to either have a+pre-existing list of our own URLs that we wish to visit, perhaps a thousand, or a million, or we need to build this list dynamically as we crawl,+adding more and more URLs to the list as we progress.-A representation of the pre-existing list is an instance of the [`RequestList`](../api/requestlist) class. It is a static, immutable list of URLs and other metadata (see the [`Request`](../api/request) object) that the crawler will visit, one by one, retrying whenever an error occurs, until there are no more `Requests` to process.+A representation of the pre-existing list is an instance of the [`RequestList`](../api/requestlist) class. It is a static, immutable list of URLs and+other metadata (see the [`Request`](../api/request) object) that the crawler will visit, one by one, retrying whenever an error occurs, until there+are no more `Requests` to process.-[`RequestQueue`](../api/requestqueue) on the other hand, represents a dynamic queue of `Requests`. One that can be updated at runtime by adding more pages - `Requests` to process. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain, add them to the queue (called _enqueuing_) and repeat this process to build a queue of tens of thousands or more URLs while knowing only a single one at the beginning.+[`RequestQueue`](../api/requestqueue) on the other hand, represents a dynamic queue of `Requests`. One that can be updated at runtime by adding more+pages - `Requests` to process. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain,+add them to the queue (called _enqueuing_) and repeat this process to build a queue of tens of thousands or more URLs while knowing only a single one+at the beginning.-`RequestList` and `RequestQueue` are essential for the crawler's operation. There is no other way to supply `Requests` = "pages to crawl" to the crawlers. At least one of them always needs to be provided while setting up. You can also use both at the same time, if you wish.+`RequestList` and `RequestQueue` are essential for the crawler's operation. There is no other way to supply `Requests` = "pages to crawl" to the+crawlers. At least one of them always needs to be provided while setting up. You can also use both at the same time, if you wish.### The What - `handlePageFunction`-The `handlePageFunction` is the brain of the crawler. It tells it what to do at each and every page it visits. Generally it handles extraction of data from the page, processing the data, saving it, calling APIs, doing calculations and whatever else you need it to do, really.+The `handlePageFunction` is the brain of the crawler. It tells it what to do at each and every page it visits. Generally it handles extraction of data+from the page, processing the data, saving it, calling APIs, doing calculations and whatever else you need it to do, really.-The `handlePageFunction` is provided by you, the user, and invoked automatically by the crawler for each `Request` from either the `RequestList` or `RequestQueue`. It always receives a single argument and that is a plain `Object`. Its properties change depending on the used crawler class, but it always includes at least the `request` property, which represents the currently crawled `Request` instance (i.e. the URL the crawler is visiting and related metadata) and the `autoscaledPool` property, which is an instance of the [`AutoscaledPool`](../api/autoscaledpool) class and we'll talk about it in detail later.+The `handlePageFunction` is provided by you, the user, and invoked automatically by the crawler for each `Request` from either the `RequestList` or+`RequestQueue`. It always receives a single argument and that is a plain `Object`. Its properties change depending on the used crawler class, but it+always includes at least the `request` property, which represents the currently crawled `Request` instance (i.e. the URL the crawler is visiting and+related metadata) and the `autoscaledPool` property, which is an instance of the [`AutoscaledPool`](../api/autoscaledpool) class and we'll talk about+it in detail later.```js// The object received as a single argument by the handlePageFunction@@ -134,54 +176,63 @@ The `handlePageFunction` is provided by you, the user, and invoked automatically### Putting it all together-Enough theory! Let's put some of those hard learned facts into practice. We learned above that we need `Requests` and a `handlePageFunction` to setup a crawler. We will also use the [`Apify.main()`](../api/apify#module_Apify.main) function. It's not mandatory, but it makes our life easier. We'll learn about it in detail later on.+Enough theory! Let's put some of those hard learned facts into practice. We learned above that we need `Requests` and a `handlePageFunction` to setup+a crawler. We will also use the [`Apify.main()`](../api/apify#module_Apify.main) function. It's not mandatory, but it makes our life easier. We'll+learn about it in detail later on.Let's start super easy. Visit one page, get its title and close. First of all we need to require Apify, to make all of its features available to us:```js-const Apify = require("apify");+const Apify = require('apify');```-Easy right? It doesn't get much more difficult, trust me. For the purposes of this tutorial, we'll be scraping our own webpage <a href="https://apify.com" target="_blank">https://apify.com</a>. Now, to get there, we need a `Request` with the page's URL in one of our sources, `RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.+Easy right? It doesn't get much more difficult, trust me. For the purposes of this tutorial, we'll be scraping our own webpage+<a href="https://apify.com" target="_blank">https://apify.com</a>. Now, to get there, we need a `Request` with the page's URL in one of our sources,+`RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.```js-const Apify = require("apify");+const Apify = require('apify');// This is how you use the Apify.main() function.Apify.main(async () => {// First we create the request queue instance.const requestQueue = await Apify.openRequestQueue();// And then we add a request to it.- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });});```-> If you're not familiar with the `async` and `await` keywords used in the example, trust that it is a native syntax in modern JavaScript and you can [learn more about it here](https://nikgrozev.com/2017/10/01/async-await/).+> If you're not familiar with the `async` and `await` keywords used in the example, trust that it is a native syntax in modern JavaScript and you can+> [learn more about it here](https://nikgrozev.com/2017/10/01/async-await/).-The [`requestQueue.addRequest()`](../api/requestqueue#RequestQueue+addRequest) function automatically converts the plain object we passed to it to a `Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://apify.com`. Now we need the `handlePageFunction`.+The [`requestQueue.addRequest()`](../api/requestqueue#RequestQueue+addRequest) function automatically converts the plain object we passed to it to a+`Request` instance, so now we have a `requestQueue` that holds one `request` which points to `https://apify.com`. Now we need the+`handlePageFunction`.```js// We'll define the function separately so it's more obvious.const handlePageFunction = async ({ request, $ }) => {// This should look familiar if you ever worked with jQuery.// We're just getting the text content of the <title> HTML element.- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);};```-Wait, where did the `$` come from? Remember what we learned about the `handlePageFunction` earlier. It expects a plain `Object` as an argument that will always have a `request` property, but it will also have other properties, depending on the chosen crawler class. Well, `$` is a property provided by the `CheerioCrawler` class which we'll set up right now.+Wait, where did the `$` come from? Remember what we learned about the `handlePageFunction` earlier. It expects a plain `Object` as an argument that+will always have a `request` property, but it will also have other properties, depending on the chosen crawler class. Well, `$` is a property provided+by the `CheerioCrawler` class which we'll set up right now.```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);};@@ -189,54 +240,75 @@ Apify.main(async () => {// Set up the crawler, passing a single options object as an argument.const crawler = new Apify.CheerioCrawler({requestQueue,- handlePageFunction+ handlePageFunction,});await crawler.run();});```-And we're done! You just created your first crawler from scratch. It will download the HTML of `https://apify.com`, find the `<title>` element, get its text content and print it to console. Good job!+And we're done! You just created your first crawler from scratch. It will download the HTML of `https://apify.com`, find the `<title>` element, get+its text content and print it to console. Good job!-To run the code locally, copy and paste the code, if you haven't already typed it in yourself, to the `main.js` file in the `my-new-project` we created earlier and run `apify run` from that project's directory.+To run the code locally, copy and paste the code, if you haven't already typed it in yourself, to the `main.js` file in the `my-new-project` we+created earlier and run `apify run` from that project's directory.To run the code on Apify Platform, just replace the original example with your new code and hit Run.-Whichever environment you choose, you should see the message `The title of "https://apify.com" is: Web Scraping, Data Extraction and Automation - Apify.` printed to the screen. If you do, congratulations and let's move onto some bigger challenges! And if you feel like you don't really know what just happened there, no worries, it will all become clear when you learn more about the `CheerioCrawler`.+Whichever environment you choose, you should see the message+`The title of "https://apify.com" is: Web Scraping, Data Extraction and Automation - Apify.` printed to the screen. If you do, congratulations and+let's move onto some bigger challenges! And if you feel like you don't really know what just happened there, no worries, it will all become clear when+you learn more about the `CheerioCrawler`.## CheerioCrawler aka jQuery crawler-This is the crawler that we used in our earlier example. Our simplest and also the fastest crawling solution. If you're familiar with `jQuery`, you'll understand [`CheerioCrawler`](../api/cheeriocrawler) in minutes. <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> is essentially `jQuery` for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would `jQuery`, for manipulating the DOM of a HTML page. In crawling, you'll mostly use it to select the right elements and extract their text values - the data you're interested in. But `jQuery` runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of [`CheerioCrawler`](../api/cheeriocrawler) comes in.+This is the crawler that we used in our earlier example. Our simplest and also the fastest crawling solution. If you're familiar with `jQuery`, you'll+understand [`CheerioCrawler`](../api/cheeriocrawler) in minutes. <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> is+essentially `jQuery` for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would `jQuery`, for manipulating+the DOM of a HTML page. In crawling, you'll mostly use it to select the right elements and extract their text values - the data you're interested in.+But `jQuery` runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of+[`CheerioCrawler`](../api/cheeriocrawler) comes in.### Overview-[`CheerioCrawler`](../api/cheeriocrawler) crawls by making plain HTTP requests to the provided URLs. As you remember from the previous section, the URLs are fed to the crawler using either the [`RequestList`](../api/cheeriocrawler) or the [`RequestQueue`](../api/cheeriocrawler). The HTTP responses it gets back are HTML pages, the same pages you would get in your browser when you first load a URL.+[`CheerioCrawler`](../api/cheeriocrawler) crawls by making plain HTTP requests to the provided URLs. As you remember from the previous section, the+URLs are fed to the crawler using either the [`RequestList`](../api/cheeriocrawler) or the [`RequestQueue`](../api/cheeriocrawler). The HTTP responses+it gets back are HTML pages, the same pages you would get in your browser when you first load a URL.-> Note, however, that modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards and together they create the final page. See our [`PuppeteerCrawler`](../api/puppeteercrawler) to crawl those.+> Note, however, that modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to+> other resources such as CSS and JavaScript that get downloaded afterwards and together they create the final page. See our+> [`PuppeteerCrawler`](../api/puppeteercrawler) to crawl those.-Once the page's HTML is retrieved, the crawler will pass it to <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> for parsing. The result is the typical `$` function, which should be familiar to `jQuery` users. You can use this `$` to do all sorts of lookups and manipulations over the page's HTML, but in scraping, we will mostly use it to find specific HTML elements and extract their data.+Once the page's HTML is retrieved, the crawler will pass it to <a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a> for+parsing. The result is the typical `$` function, which should be familiar to `jQuery` users. You can use this `$` to do all sorts of lookups and+manipulations over the page's HTML, but in scraping, we will mostly use it to find specific HTML elements and extract their data.Example use of Cheerio and its `$` function in comparison to browser JavaScript:```js// Return the text content of the <title> element.-document.querySelector("title").textContent; // plain JS-$("title").text(); // Cheerio+document.querySelector('title').textContent; // plain JS+$('title').text(); // Cheerio// Return an array of all 'href' links on the page.-Array.from(document.querySelectorAll("[href]")).map(el => el.href); // plain JS-$("[href]")- .map((i, el) => $(el).attr("href"))+Array.from(document.querySelectorAll('[href]')).map(el => el.href); // plain JS+$('[href]')+ .map((i, el) => $(el).attr('href')).get(); // Cheerio```-> This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides. Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is our best bet to do the parsing.+> This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides.+> Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is our best bet to do the parsing.### When to use [`CheerioCrawler`](../api/cheeriocrawler)-Even though using `CheerioCrawler` is extremely easy, it probably will not be your first choice for most kinds of crawling or scraping in production environments. Since most websites nowadays use modern JavaScript to create rich, responsive and data driven user experiences, the plain HTTP requests the crawler uses may just fall short of your needs.+Even though using `CheerioCrawler` is extremely easy, it probably will not be your first choice for most kinds of crawling or scraping in production+environments. Since most websites nowadays use modern JavaScript to create rich, responsive and data driven user experiences, the plain HTTP requests+the crawler uses may just fall short of your needs.-But `CheerioCrawler` is far from useless! It really shines when you need to do extremely high workloads. With just 4 GBs of memory and a single CPU core, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_ To get this high with a full browser scraper, such as the [`PuppeteerCrawler`](../api/puppeteercrawler), you'd need significantly more computing power.+But `CheerioCrawler` is far from useless! It really shines when you need to do extremely high workloads. With just 4 GBs of memory and a single CPU+core, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_ To get this high with a full browser+scraper, such as the [`PuppeteerCrawler`](../api/puppeteercrawler), you'd need significantly more computing power.**Advantages:**@@ -254,21 +326,23 @@ But `CheerioCrawler` is far from useless! It really shines when you need to do e### Basic use of [`CheerioCrawler`](../api/cheeriocrawler)-Now that we have an idea of the crawler's inner workings, let's build one. We'll use the example from the previous section and improve on it by letting it truly crawl the page, finding new links as it goes, enqueuing them into the `RequestQueue` and then scraping them.+Now that we have an idea of the crawler's inner workings, let's build one. We'll use the example from the previous section and improve on it by+letting it truly crawl the page, finding new links as it goes, enqueuing them into the `RequestQueue` and then scraping them.#### Refresher-Just to refresh your memory, in the previous section we built a very simple crawler that downloads HTML of a single page, reads its title and prints it to the console. This is the original source code:+Just to refresh your memory, in the previous section we built a very simple crawler that downloads HTML of a single page, reads its title and prints+it to the console. This is the original source code:```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);};@@ -276,7 +350,7 @@ Apify.main(async () => {// Set up the crawler, passing a single options object as an argument.const crawler = new Apify.CheerioCrawler({requestQueue,- handlePageFunction+ handlePageFunction,});await crawler.run();@@ -294,72 +368,75 @@ So let's get to it!#### Finding new links-There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute. For example `<a href="https://apify.com/store>This is a link to Apify Store</a>`. To do this, we need to update our Cheerio function.+There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain+the `href` attribute. For example `<a href="https://apify.com/store>This is a link to Apify Store</a>`. To do this, we need to update our Cheerio+function.```js-const links = $("a[href]")- .map((i, el) => $(el).attr("href"))+const links = $('a[href]')+ .map((i, el) => $(el).attr('href')).get();```-Our new function finds all the `<a>` elements that contain the `href` attribute and extracts the attributes into an array of strings. There is a problem though. There can be relative links in the list and those can't be used on their own. We need to resolve them using our domain as base URL and we will use one of Node.js' standard libraries to do this.+Our new function finds all the `<a>` elements that contain the `href` attribute and extracts the attributes into an array of strings. There is a+problem though. There can be relative links in the list and those can't be used on their own. We need to resolve them using our domain as base URL and+we will use one of Node.js' standard libraries to do this.```js// At the top of the file:-const { URL } = require("url");+const { URL } = require('url');// ...-const ourDomain = "https://apify.com";+const ourDomain = 'https://apify.com';const absoluteUrls = links.map(link => new URL(link, ourDomain));```#### Filtering links to same domain-Websites typically contain a lot of links that lead away from the original page. This is normal, but when crawling a website, we usually want to crawl this one site and do not let our crawler wander away to Google, Facebook and Twitter. Therefore, we need to filter out the off-domain links and only keep the ones that lead to the same domain.+Websites typically contain a lot of links that lead away from the original page. This is normal, but when crawling a website, we usually want to crawl+this one site and do not let our crawler wander away to Google, Facebook and Twitter. Therefore, we need to filter out the off-domain links and only+keep the ones that lead to the same domain.> Don't worry, we'll learn how to do this with a single function call using Apify in a few moments.```js// At the top of the file:-const { URL } = require("url");+const { URL } = require('url');// ...-const links = $("a[href]")- .map((i, el) => $(el).attr("href"))+const links = $('a[href]')+ .map((i, el) => $(el).attr('href')).get();-const ourDomain = "apify.com";+const ourDomain = 'apify.com';const absoluteUrls = links.map(link => new URL(link, ourDomain));-const sameDomainLinks = absoluteUrls.filter(url =>- url.href.startsWith(ourDomain)-);+const sameDomainLinks = absoluteUrls.filter(url => url.href.startsWith(ourDomain));// ...```#### Enqueueing links to `RequestQueue`-This should be easy, because we already did that [earlier](#puttingitalltogether), remember? Just call `requestQueue.addRequest()` for all the new links. This will add them to the end of the queue for processing.+This should be easy, because we already did that [earlier](#puttingitalltogether), remember? Just call `requestQueue.addRequest()` for all the new+links. This will add them to the end of the queue for processing.```js// At the top of the file:-const { URL } = require("url");+const { URL } = require('url');// ...-const links = $("a[href]")- .map((i, el) => $(el).attr("href"))+const links = $('a[href]')+ .map((i, el) => $(el).attr('href')).get();-const ourDomain = "https://apify.com";+const ourDomain = 'https://apify.com';const absoluteUrls = links.map(link => new URL(link, ourDomain));-const sameDomainLinks = absoluteUrls.filter(url =>- url.href.startsWith(ourDomain)-);+const sameDomainLinks = absoluteUrls.filter(url => url.href.startsWith(ourDomain));// Add the requests in series. There's of course room for speed// improvement by parallelization. Try to implement it, if you wish.@@ -373,47 +450,51 @@ for (const url of sameDomainLinks) {#### Scrape the newly enqueued links-And we're approching the finishing line. All we need to do now is just integrate the new code into our original crawler. It will be easy, because almost everything needs to go into the `handlePageFunction`. But just before we do that, let me introduce the first crawler configuration option, that is not a `handlePageFunction` or `requestQueue`. It's called `maxRequestsPerCrawl`.+And we're approching the finishing line. All we need to do now is just integrate the new code into our original crawler. It will be easy, because+almost everything needs to go into the `handlePageFunction`. But just before we do that, let me introduce the first crawler configuration option, that+is not a `handlePageFunction` or `requestQueue`. It's called `maxRequestsPerCrawl`.##### The `maxRequestsPerCrawl` limit-This configuration option is available in all crawler classes and you can use it to limit the number of `Requests` the crawler should process. It's very useful when you're just testing your code or when your crawler could potentially crawl millions of pages and you want to save resources. You can add it to the crawler options like this:+This configuration option is available in all crawler classes and you can use it to limit the number of `Requests` the crawler should process. It's+very useful when you're just testing your code or when your crawler could potentially crawl millions of pages and you want to save resources. You can+add it to the crawler options like this:```jsconst crawler = new Apify.CheerioCrawler({maxRequestsPerCrawl: 20,requestQueue,- handlePageFunction+ handlePageFunction,});```-This limits the number of successfully handled `Requests` to 20. Bear in mind that the actual number of processed requests might be a little higher and that's because usually there are multiple `Requests` processed at the same time and once the 20th `Request` finishes, the other running `Requests` will be allowed to finish too.+This limits the number of successfully handled `Requests` to 20. Bear in mind that the actual number of processed requests might be a little higher+and that's because usually there are multiple `Requests` processed at the same time and once the 20th `Request` finishes, the other running `Requests`+will be allowed to finish too.#### Putting it all together```js-const { URL } = require("url"); // <------ This is new.-const Apify = require("apify");+const { URL } = require('url'); // <------ This is new.+const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);// Here starts the new part of handlePageFunction.- const links = $("a[href]")- .map((i, el) => $(el).attr("href"))+ const links = $('a[href]')+ .map((i, el) => $(el).attr('href')).get();- const ourDomain = "https://apify.com";+ const ourDomain = 'https://apify.com';const absoluteUrls = links.map(link => new URL(link, ourDomain));- const sameDomainLinks = absoluteUrls.filter(url =>- url.href.startsWith(ourDomain)- );+ const sameDomainLinks = absoluteUrls.filter(url => url.href.startsWith(ourDomain));console.log(`Enqueueing ${sameDomainLinks.length} URLs.`);for (const url of sameDomainLinks) {@@ -424,39 +505,53 @@ Apify.main(async () => {const crawler = new Apify.CheerioCrawler({maxRequestsPerCrawl: 20, // <------ This is new too.requestQueue,- handlePageFunction+ handlePageFunction,});await crawler.run();});```-No matter if you followed along with our coding or just copy pasted the resulting source, try running it now, perhaps even in both environments. You should see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the first enqueued page and so on and so on.+No matter if you followed along with our coding or just copy pasted the resulting source, try running it now, perhaps even in both environments. You+should see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the+first enqueued page and so on and so on.-> If you need help with running the code, refer back to the chapters on environment setup: [Setting up locally](#settinguplocally) and [Setting up on the Apify Platform](#settingupontheapifyplatform).+> If you need help with running the code, refer back to the chapters on environment setup: [Setting up locally](#settinguplocally) and+> [Setting up on the Apify Platform](#settingupontheapifyplatform).## Using Apify SDK to enqueue links like a boss-If you were paying attention carefully in the previous chapter, we said that we would show a way to enqueue new `Requests` with a single function call. You might be wondering why we had to go through the whole process of getting the individual links, filtering the same domain ones and then manually enqueuing them into the `RequestQueue`, when there is a simpler way.+If you were paying attention carefully in the previous chapter, we said that we would show a way to enqueue new `Requests` with a single function+call. You might be wondering why we had to go through the whole process of getting the individual links, filtering the same domain ones and then+manually enqueuing them into the `RequestQueue`, when there is a simpler way.-Well, the obvious reason is practice. This is a tutorial after all. The other reason is to make you think about all the bits and pieces that come together, so that in the end, a new page, not previously entered in by you, can be scraped. We think that by seeing the bigger picture, you will be able to get the most out of Apify SDK.+Well, the obvious reason is practice. This is a tutorial after all. The other reason is to make you think about all the bits and pieces that come+together, so that in the end, a new page, not previously entered in by you, can be scraped. We think that by seeing the bigger picture, you will be+able to get the most out of Apify SDK.### Meet `Apify.utils`-We will talk at length about them later, but in short, `Apify.utils` is a namespace where you can find various helpful functions and constants that make your life easier. One of the available functions is `Apify.utils.enqueueLinks()` which encapsulates the whole enqueueing process and even adds some extra functionality.+We will talk at length about them later, but in short, `Apify.utils` is a namespace where you can find various helpful functions and constants that+make your life easier. One of the available functions is `Apify.utils.enqueueLinks()` which encapsulates the whole enqueueing process and even adds+some extra functionality.### Introduction to `Apify.utils.enqueueLinks()`-Since enqueuing new links to crawl is such an integral part of web crawling, we created a function that attempts to simplify this process as much as possible. With a single function call, it allows you to find all the links on a page that match specified criteria and add them to a `RequestQueue`. It also allows you to modify the resulting `Requests` to match your crawling needs.+Since enqueuing new links to crawl is such an integral part of web crawling, we created a function that attempts to simplify this process as much as+possible. With a single function call, it allows you to find all the links on a page that match specified criteria and add them to a `RequestQueue`.+It also allows you to modify the resulting `Requests` to match your crawling needs.-`enqueueLinks` is quite a powerful function so, like crawlers, it gets its arguments from an options object. This is useful, because you don't have to remember their order! But also because we can easily extend its API and add new features. You can [find the full reference here](../api/utils#utils.enqueuelinks).+`enqueueLinks` is quite a powerful function so, like crawlers, it gets its arguments from an options object. This is useful, because you don't have to+remember their order! But also because we can easily extend its API and add new features. You can+[find the full reference here](../api/utils#utils.enqueuelinks).-We suggest using ES6 destructuring to grab the `enqueueLinks()` function off of the `utils` object, so you don't have to type `Apify.utils` all the time.+We suggest using ES6 destructuring to grab the `enqueueLinks()` function off of the `utils` object, so you don't have to type `Apify.utils` all the+time.```js-const Apify = require("apify");+const Apify = require('apify');const {- utils: { enqueueLinks }+ utils: { enqueueLinks },} = Apify;// Now you can use enqueueLinks like this:@@ -467,7 +562,8 @@ await enqueueLinks({### Basic use of `enqueueLinks()` with `CheerioCrawler`-We already implemented logic that takes care of enqueueing new links to a `RequestQueue` in the previous chapter on `CheerioCrawler`. Let's look at that logic and implement the same functionality using `enqueueLinks()`.+We already implemented logic that takes care of enqueueing new links to a `RequestQueue` in the previous chapter on `CheerioCrawler`. Let's look at+that logic and implement the same functionality using `enqueueLinks()`.We found that the crawler needed to do these 4 things to crawl `apify.com`:@@ -476,20 +572,26 @@ We found that the crawler needed to do these 4 things to crawl `apify.com`:3. Enqueue them to the `RequestQueue`4. Scrape the newly enqueued links-Using `enqueueLinks()` we can squash the first 3 into a single function call, if we set the options correctly. For now, let's just stick to the basics. At the very least, we need a source where to find the links and the queue to enqueue them to. The `$` Cheerio object is one of the sources the function accepts and we already know how to work with it in the `handlePageFunction`. We also know how to get a `requestQueue` instance.+Using `enqueueLinks()` we can squash the first 3 into a single function call, if we set the options correctly. For now, let's just stick to the+basics. At the very least, we need a source where to find the links and the queue to enqueue them to. The `$` Cheerio object is one of the sources the+function accepts and we already know how to work with it in the `handlePageFunction`. We also know how to get a `requestQueue` instance.```js// Assuming previous existence of the '$' and 'requestQueue' variables.await enqueueLinks({ $, requestQueue });```-That's all we need to do to enqueue all `<a href="...">` links from the given page to the given queue. Easy, right? Scratch number 1 and 3 off the list. Only number 2 remains and to tackle this one, we need to talk about yet another new concept, the pseudo-URL.+That's all we need to do to enqueue all `<a href="...">` links from the given page to the given queue. Easy, right? Scratch number 1 and 3 off the+list. Only number 2 remains and to tackle this one, we need to talk about yet another new concept, the pseudo-URL.#### Introduction to pseudo-URLs-Pseudo-URLs are represented by our `PseudoUrl` class and even though the name sounds discouraging, they're a pretty simple concept. They're just URLs with some parts replaced by wildcards (read <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular expressions</a>). They are matched against URLs to find specific links, domains, patterns, file extensions and so on.+Pseudo-URLs are represented by our `PseudoUrl` class and even though the name sounds discouraging, they're a pretty simple concept. They're just URLs+with some parts replaced by wildcards (read <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular+expressions</a>). They are matched against URLs to find specific links, domains, patterns, file extensions and so on.-In scraping, there usually are patterns to be found in the websites' URLs that can be leveraged to scrape only the pages we're interested in. Imagine a typical online store. It has different categories which list different items The URL for might looks something like this:+In scraping, there usually are patterns to be found in the websites' URLs that can be leveraged to scrape only the pages we're interested in. Imagine+a typical online store. It has different categories which list different items The URL for might looks something like this:```https://www.online-store.com/categories@@ -507,11 +609,13 @@ Going to this page would produce a list of offered computers. Then, clicking onhttps://www.online-store.com/items/613804```-As you can see, there's a structure to the links. In the real world, the structure might not always be perfectly obvious, but it's very often there. Pseudo-URLs help to use this structure to select only the relevant links from a given page.+As you can see, there's a structure to the links. In the real world, the structure might not always be perfectly obvious, but it's very often there.+Pseudo-URLs help to use this structure to select only the relevant links from a given page.#### Structure of a pseudo-URL-Pseudo-URL is a URL with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular expressions</a>) enclosed in `[]` brackets. Since we're running Node.js, the regular expressions should follow the JavaScript style.+Pseudo-URL is a URL with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions">regular expressions</a>) enclosed+in `[]` brackets. Since we're running Node.js, the regular expressions should follow the JavaScript style.For example, a pseudo-URL@@ -536,7 +640,8 @@ https://www.online-store.com/items/613804This way, you can easily find just the URLs that you're looking for while ignoring the rest.-A pseudo-URL may include any number of bracketed regular expressions, so you can compose much more complex matching logic. The following Pseudo URL will match the items in the store even if the links use the non-secure `http` protocol, omit the `www` from the hostname or use different TLD.+A pseudo-URL may include any number of bracketed regular expressions, so you can compose much more complex matching logic. The following Pseudo URL+will match the items in the store even if the links use the non-secure `http` protocol, omit the `www` from the hostname or use different TLD.```http[s?]://[(www)?\.]online-store.[com|net|org]/items/[\d+]@@ -562,28 +667,36 @@ www.online-store.org/items/7003#### Using `enqueueLinks()` to filter links-That's been quite a lot of theory and examples. We might as well put it to practice. Going back to our `CheerioCrawler` exercise, we still have number 2 left to cross off the list - filter links pointing to `apify.com`. We've already shown that at the very least, the `enqueueLinks()` function needs two arguments. The source, in our case the `$` object, and the destination - the `requestQueue`. To filter links, we need to add a third argument: `pseudoUrls`.+That's been quite a lot of theory and examples. We might as well put it to practice. Going back to our `CheerioCrawler` exercise, we still have number+2 left to cross off the list - filter links pointing to `apify.com`. We've already shown that at the very least, the `enqueueLinks()` function needs+two arguments. The source, in our case the `$` object, and the destination - the `requestQueue`. To filter links, we need to add a third argument:+`pseudoUrls`.-The `options.pseudoUrls` argument is always an `Array`, but its contents can take on many forms. [See the reference](../api/utils#utils.enqueueLinks) for all of them. Since we just need to filter out same domain links, we'll keep it simple and use a pseudo-URL `string`.+The `options.pseudoUrls` argument is always an `Array`, but its contents can take on many forms. [See the reference](../api/utils#utils.enqueueLinks)+for all of them. Since we just need to filter out same domain links, we'll keep it simple and use a pseudo-URL `string`.```js// Assuming previous existence of the '$' and 'requestQueue' variables.const options = {$,requestQueue,- pseudoUrls: ["http[s?]://apify.com[.*]"]+ pseudoUrls: ['http[s?]://apify.com[.*]'],};await enqueueLinks(options);```-> To break the pseudo-URL string down, we're looking for both `http` and `https` protocols and the links may only lead to `apify.com` domain. The final brackets `[.*]` allow everything, so `apify.com/contact` as well as `apify.com/store` will match. If this is complex to you, we suggest <a href="https://www.regular-expressions.info/tutorial.html" target="_blank">reading a tutorial</a> or two on regular expression syntax.+> To break the pseudo-URL string down, we're looking for both `http` and `https` protocols and the links may only lead to `apify.com` domain. The+> final brackets `[.*]` allow everything, so `apify.com/contact` as well as `apify.com/store` will match. If this is complex to you, we suggest+> <a href="https://www.regular-expressions.info/tutorial.html" target="_blank">reading a tutorial</a> or two on regular expression syntax.#### Resolving relative URLs with `enqueueLinks()`**TLDR;** Just use `baseUrl: request.loadedUrl` when working with `CheerioCrawler`.-This is probably the weirdest and most complicated addition to the list. This is not the place to talk at length about <a href="https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls" target="_blank">absolute and relative paths</a>, but in short, the links we encounter in a page can either be absolute, such as:+This is probably the weirdest and most complicated addition to the list. This is not the place to talk at length about+<a href="https://stackoverflow.com/questions/2005079/absolute-vs-relative-urls" target="_blank">absolute and relative paths</a>, but in short, the+links we encounter in a page can either be absolute, such as:```https://apify.com/john-doe/my-actor@@ -595,49 +708,51 @@ or relative:./john-doe/my-actor```-Browsers handle this automatically, but since we're only using plain HTTP requests, we need to tell the `enqueueLinks()` function how to resolve the relative links to the absolute ones, so we can use them for scraping. This is where the `response.loadedUrl` comes into play, because it returns the correct URL to use as `baseUrl`.+Browsers handle this automatically, but since we're only using plain HTTP requests, we need to tell the `enqueueLinks()` function how to resolve the+relative links to the absolute ones, so we can use them for scraping. This is where the `response.loadedUrl` comes into play, because it returns the+correct URL to use as `baseUrl`.```js// Assuming previous existence of the '$', 'requestQueue' and 'request' variables.const options = {$,requestQueue,- pseudoUrls: ["http[s?]://apify.com[.*]"],- baseUrl: request.loadedUrl+ pseudoUrls: ['http[s?]://apify.com[.*]'],+ baseUrl: request.loadedUrl,};await enqueueLinks(options);```-> Even though it seems possible, we can't use the `request.url` of our `Request` instances, because the page could have been redirected and the final URL would be different from the one we requested.+> Even though it seems possible, we can't use the `request.url` of our `Request` instances, because the page could have been redirected and the final+> URL would be different from the one we requested.#### Integrating `enqueueLinks()` into our crawler-That was fairly easy, wasn't it. That ticks the number 2 off our list and we're done! Let's take a look at the original crawler code, where we enqueued all the links manually.+That was fairly easy, wasn't it. That ticks the number 2 off our list and we're done! Let's take a look at the original crawler code, where we+enqueued all the links manually.```js-const { URL } = require("url"); // <------ This is new.-const Apify = require("apify");+const { URL } = require('url'); // <------ This is new.+const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);// Here starts the new part of handlePageFunction.- const links = $("a[href]")- .map((i, el) => $(el).attr("href"))+ const links = $('a[href]')+ .map((i, el) => $(el).attr('href')).get();- const ourDomain = "https://apify.com";+ const ourDomain = 'https://apify.com';const absoluteUrls = links.map(link => new URL(link, ourDomain));- const sameDomainLinks = absoluteUrls.filter(url =>- url.href.startsWith(ourDomain)- );+ const sameDomainLinks = absoluteUrls.filter(url => url.href.startsWith(ourDomain));console.log(`Enqueueing ${sameDomainLinks.length} URLs.`);for (const url of sameDomainLinks) {@@ -648,7 +763,7 @@ Apify.main(async () => {const crawler = new Apify.CheerioCrawler({maxRequestsPerCrawl: 20, // <------ This is new too.requestQueue,- handlePageFunction+ handlePageFunction,});await crawler.run();@@ -658,25 +773,25 @@ Apify.main(async () => {Since we've already prepared the `enqueueLinks()` options, we can just replace all the above enqueuing logic with a single function call, as promised.```js-const Apify = require("apify");+const Apify = require('apify');const {- utils: { enqueueLinks }+ utils: { enqueueLinks },} = Apify;Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://apify.com" });+ await requestQueue.addRequest({ url: 'https://apify.com' });const handlePageFunction = async ({ request, $ }) => {- const title = $("title").text();+ const title = $('title').text();console.log(`The title of "${request.url}" is: ${title}.`);// Enqueue linksconst enqueued = await enqueueLinks({$,requestQueue,- pseudoUrls: ["http[s?]://apify.com[.*]"],- baseUrl: request.loadedUrl+ pseudoUrls: ['http[s?]://apify.com[.*]'],+ baseUrl: request.loadedUrl,});console.log(`Enqueued ${enqueued.length} URLs.`);};@@ -684,35 +799,55 @@ Apify.main(async () => {const crawler = new Apify.CheerioCrawler({maxRequestsPerCrawl: 20,requestQueue,- handlePageFunction+ handlePageFunction,});await crawler.run();});```-And that's it! No more parsing the links from HTML using Cheerio, filtering them and enqueueing them one by one It all gets done automatically! `enqueueLinks()` is just one example of Apify SDK's powerful helper functions. They're all designed to make your life easier so you can focus on getting your data, while leaving the mundane crawling management to your tools.+And that's it! No more parsing the links from HTML using Cheerio, filtering them and enqueueing them one by one It all gets done automatically!+`enqueueLinks()` is just one example of Apify SDK's powerful helper functions. They're all designed to make your life easier so you can focus on+getting your data, while leaving the mundane crawling management to your tools.-`Apify.utils.enqueueLinks()` has a lot more tricks up its sleeve. Make sure to check out the [reference documentation](../api/utils#utils.enqueueLinks) to see what else it can do for you. Namely the feature to prepopulate the `Request` instances it creates with `userData` of your choice is extremely useful!+`Apify.utils.enqueueLinks()` has a lot more tricks up its sleeve. Make sure to check out the+[reference documentation](../api/utils#utils.enqueueLinks) to see what else it can do for you. Namely the feature to prepopulate the `Request`+instances it creates with `userData` of your choice is extremely useful!## Getting some real world data-> Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine readable format? Because that's why started reading this tutorial in the first place!+> Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real+> data and save it somewhere in a machine readable format? Because that's why started reading this tutorial in the first place!We hear you young padawan! First, learn how to crawl, you must. Only then, save data, you can!-### Making a store crawler-Fortunately, we don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape structured data. The <a href="https://apify.com/store" target="_blank">Apify Store</a> is a store of public actors that anyone can grab and use. You can find ready-made solutions for crawling <a href="https://apify.com/drobnikj/crawler-google-places" target="_blank">Google Places</a>, <a href="https://apify.com/vaclavrut/amazon-crawler" target="_blank">Amazon</a>, <a href="https://apify.com/apify/google-search-scraper" target="_blank">Google Search</a>, <a href="https://apify.com/petr_cermak/booking-scraper" target="_blank">Booking</a>, <a href="https://apify.com/jaroslavhejlek/instagram-scraper" target="_blank">Instagram</a>, <a href="https://apify.com/petrpatek/tripadvisor-scraper" target="_blank">Tripadvisor</a> and many other websites. Feel free to check them out! It also poses a great place to practice our jedi scraping skills since it has categories, lists and details. That's almost like our imaginary `online-store.com` from the previous chapter.+### Making a store crawler Fortunately, we don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape+structured data. The <a href="https://apify.com/store" target="_blank">Apify Store</a> is a store of public actors that anyone can grab and use. You+can find ready-made solutions for crawling <a href="https://apify.com/drobnikj/crawler-google-places" target="_blank">Google Places</a>,+<a href="https://apify.com/vaclavrut/amazon-crawler" target="_blank">Amazon</a>,+<a href="https://apify.com/apify/google-search-scraper" target="_blank">Google Search</a>,+<a href="https://apify.com/petr_cermak/booking-scraper" target="_blank">Booking</a>,+<a href="https://apify.com/jaroslavhejlek/instagram-scraper" target="_blank">Instagram</a>,+<a href="https://apify.com/petrpatek/tripadvisor-scraper" target="_blank">Tripadvisor</a> and many other websites. Feel free to check them out! It+also poses a great place to practice our jedi scraping skills since it has categories, lists and details. That's almost like our imaginary+`online-store.com` from the previous chapter.### The importance of having a plan-Sometimes scraping is really straightforward, but most of the times, it really pays out to do a little bit of research first. How is the website structured? Can I scrape it only with HTTP requests (read "with `CheerioCrawler`") or would I need a full browser solution? Are there any anti-scraping protections in place? Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API. Jakub, one of Apify's founders wrote a <a href="https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c" target="_blank">great article about all the different techniques</a> and tips and tricks so make sure to check that out!+Sometimes scraping is really straightforward, but most of the times, it really pays out to do a little bit of research first. How is the website+structured? Can I scrape it only with HTTP requests (read "with `CheerioCrawler`") or would I need a full browser solution? Are there any+anti-scraping protections in place? Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API. Jakub, one+of Apify's founders wrote a+<a href="https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c" target="_blank">great article+about all the different techniques</a> and tips and tricks so make sure to check that out!-For the purposes of this tutorial, let's just go ahead with HTTP requests and HTML parsing using `CheerioCrawler`. The number one reason being: We already know how to use it and we want to build on that knowledge to learn specific crawling and scraping techniques.+For the purposes of this tutorial, let's just go ahead with HTTP requests and HTML parsing using `CheerioCrawler`. The number one reason being: We+already know how to use it and we want to build on that knowledge to learn specific crawling and scraping techniques.#### Choosing the data we need-A good first step is always to figure out what it is we want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors (see the `Show` dropdown) in all categories (which can be found on the left side of the page) and for each actor we want to get its+A good first step is always to figure out what it is we want to scrape and where to find it. For the time being, let's just agree that we want to+scrape all actors (see the `Show` dropdown) in all categories (which can be found on the left side of the page) and for each actor we want to get its1. URL2. Owner@@ -722,31 +857,41 @@ A good first step is always to figure out what it is we want to scrape and where6. Last run date7. Number of runs-We can see that some of the information is available directly on the list page, but for details such as "Last run date" or "Number of runs" we'll need to open the actor detail pages too.+We can see that some of the information is available directly on the list page, but for details such as "Last run date" or "Number of runs" we'll need+to open the actor detail pages too.-+#### Analyzing the target-Knowing that we will use plain HTTP requests, we immediately know that we won't be able to manipulate the website in any way. We will only be able to go through the HTML it gives us and parse our data from there. This might sound like a huge limitation, but you might be surprised in how effective it might be. Let's get on it!+Knowing that we will use plain HTTP requests, we immediately know that we won't be able to manipulate the website in any way. We will only be able to+go through the HTML it gives us and parse our data from there. This might sound like a huge limitation, but you might be surprised in how effective it+might be. Let's get on it!#### The start URL(s)-This is where we start our crawl. It's convenient to start as close to our data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.+This is where we start our crawl. It's convenient to start as close to our data as possible. For example, it wouldn't make much sense to start at+`apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.-Once we look at the `apify.com/store` page more carefully though, we see that the categories themselves produce URLs that we can use to access those individual categories.+Once we look at the `apify.com/store` page more carefully though, we see that the categories themselves produce URLs that we can use to access those+individual categories.```https://apify.com/store?category=ENTERTAINMENT```-Should we write down all the category URLs down and use all of them as start URLs? It's definitely possible, but what if a new category appears on the page later? We would not learn about it unless we manually visit the page and inspect it again. So scraping the category links off the store page definitely makes sense. This way we always get an up to date list of categories.+Should we write down all the category URLs down and use all of them as start URLs? It's definitely possible, but what if a new category appears on the+page later? We would not learn about it unless we manually visit the page and inspect it again. So scraping the category links off the store page+definitely makes sense. This way we always get an up to date list of categories.-But is it really that straightforward? By digging further into the store page's HTML we find that it does not actually contain the category links. The menu on the left uses JavaScript to display the items from a given category and, as we've learned earlier, `CheerioCrawler` cannot execute JavaScript.+But is it really that straightforward? By digging further into the store page's HTML we find that it does not actually contain the category links. The+menu on the left uses JavaScript to display the items from a given category and, as we've learned earlier, `CheerioCrawler` cannot execute JavaScript.-> We've deliberately chosen this scenario to show an example of the number one weakness of `CheerioCrawler`. We will overcome this difficulty in our `PuppeteerCrawler` tutorial, but at the cost of compute resources and speed. Always remember that no tool is best for everything!+> We've deliberately chosen this scenario to show an example of the number one weakness of `CheerioCrawler`. We will overcome this difficulty in our+> `PuppeteerCrawler` tutorial, but at the cost of compute resources and speed. Always remember that no tool is best for everything!-So we're back to the pre-selected list of URLs. Since we cannot scrape the list dynamically, we have to manually collect the links and then use them in our crawler. We lose the ability to scrape new categories, but we keep the low resource consumption and speed advantages of `CheerioCrawler`.+So we're back to the pre-selected list of URLs. Since we cannot scrape the list dynamically, we have to manually collect the links and then use them+in our crawler. We lose the ability to scrape new categories, but we keep the low resource consumption and speed advantages of `CheerioCrawler`.Therefore, after careful consideration, we've determined that we should use multiple start URLs and that they should look as follows:@@ -758,87 +903,106 @@ https://apify.com/store?category=ENTERTAINMENT### The crawling strategy-Now that we know where to start, we need to figure out where to go next. Since we've eliminated one level of crawling by selecting the categories manually, we only need to crawl the actor detail pages now. The algorithm therefore follows:+Now that we know where to start, we need to figure out where to go next. Since we've eliminated one level of crawling by selecting the categories+manually, we only need to crawl the actor detail pages now. The algorithm therefore follows:1. Visit the category list page (one of our start URLs).2. Enqueue all links to actor details.3. Visit all actor details and extract data.4. Repeat 1 - 3 for all categories.-> Technically, this is a depth first crawl and the crawler will perform a breadth first crawl by default, but that's an implementation detail. We've chosen this notation since a breadth first crawl would be less readable.+> Technically, this is a depth first crawl and the crawler will perform a breadth first crawl by default, but that's an implementation detail. We've+> chosen this notation since a breadth first crawl would be less readable.-`CheerioCrawler` will make sure to visit the pages for us, if we provide the correct `Requests` and we already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are two more tricks that we'd like to show you.+`CheerioCrawler` will make sure to visit the pages for us, if we provide the correct `Requests` and we already know how to enqueue pages, so this+should be fairly easy. Nevertheless, there are two more tricks that we'd like to show you.#### Using a `RequestList`-`RequestList` is a perfect tool for scraping a pre-existing list of URLs and if you think about our start URLs, this is exactly what we have! A list of links to the different categories of the store. Let's see how we'd get them into a `RequestList`.+`RequestList` is a perfect tool for scraping a pre-existing list of URLs and if you think about our start URLs, this is exactly what we have! A list+of links to the different categories of the store. Let's see how we'd get them into a `RequestList`.```jsconst sources = [- "https://apify.com/store?category=TRAVEL",- "https://apify.com/store?category=ECOMMERCE",- "https://apify.com/store?category=ENTERTAINMENT"+ 'https://apify.com/store?category=TRAVEL',+ 'https://apify.com/store?category=ECOMMERCE',+ 'https://apify.com/store?category=ENTERTAINMENT',];-const requestList = await Apify.openRequestList("categories", sources);+const requestList = await Apify.openRequestList('categories', sources);```-As you can see, similarly to the `Apify.openRequestQueue()` function, there is an `Apify.openRequestList()` function that will create a `RequestList` instance for you. The first argument is a name of the `RequestList`. It is used to persist the crawling state of the list. This is useful when you want to continue where you left off after an error or a process restart. The second argument is the `sources` array, which is nothing more than a list of URLs you wish to crawl.+As you can see, similarly to the `Apify.openRequestQueue()` function, there is an `Apify.openRequestList()` function that will create a `RequestList`+instance for you. The first argument is a name of the `RequestList`. It is used to persist the crawling state of the list. This is useful when you+want to continue where you left off after an error or a process restart. The second argument is the `sources` array, which is nothing more than a list+of URLs you wish to crawl.-> `RequestQueue` is a persistent store by default, so no name is needed, while the `RequestList` only lives in memory and giving it a name enables it to become persistent.+> `RequestQueue` is a persistent store by default, so no name is needed, while the `RequestList` only lives in memory and giving it a name enables it+> to become persistent.You might now want to ask one of the below questions:- Can I enqueue into `RequestList` too?- How do I make `RequestList` work together with `RequestQueue` since I need the queue to enqueue new `Requests`.-The answer to the first one is a definitive no. `RequestList` is immutable and once you create it, you cannot add or remove `Requests` from it. The answer to the second one is simple. `RequestList` and `RequestQueue` are made to work together out of the box in crawlers, so all you need to do is use them both and the crawlers will do the rest.+The answer to the first one is a definitive no. `RequestList` is immutable and once you create it, you cannot add or remove `Requests` from it. The+answer to the second one is simple. `RequestList` and `RequestQueue` are made to work together out of the box in crawlers, so all you need to do is+use them both and the crawlers will do the rest.```jsconst crawler = new Apify.CheerioCrawler({requestList,requestQueue,- handlePageFunction+ handlePageFunction,});```-> For those wondering how this works, the `RequestList` `Requests` are enqueued into the `RequestQueue` right before their execution and only processed by the `RequestQueue` afterwards. You can, of course, enqueue the `Requests` to the queue manually, but that would take some boilerplate code and perhaps quite a long time, if we were talking about tens of thousands or more `Requests`. The crawlers do it while running, so the time to enqueue is spread out and you won't even notice it.+> For those wondering how this works, the `RequestList` `Requests` are enqueued into the `RequestQueue` right before their execution and only+> processed by the `RequestQueue` afterwards. You can, of course, enqueue the `Requests` to the queue manually, but that would take some boilerplate+> code and perhaps quite a long time, if we were talking about tens of thousands or more `Requests`. The crawlers do it while running, so the time to+> enqueue is spread out and you won't even notice it.-#### Sanity check-It's always useful to create some simple boilerplate code to see that we've got everything set up correctly before we start to write the scraping logic itself. We might realize that something in our previous analysis doesn't quite add up, or the website might not behave exactly the same as we expected.+#### Sanity check It's always useful to create some simple boilerplate code to see that we've got everything set up correctly before we start to write+the scraping logic itself. We might realize that something in our previous analysis doesn't quite add up, or the website might not behave exactly the+same as we expected.-Let's use the newly acquired `RequestList` knowledge and everything we know from the previous chapters to create a new crawler that'll just visit all the category URLs we selected and print the text content of all the actors in the category. Try running the code below in your selected environment. You should see, albeit very badly formatted, the text of the individual actor cards that are displayed in the selected categories.+Let's use the newly acquired `RequestList` knowledge and everything we know from the previous chapters to create a new crawler that'll just visit all+the category URLs we selected and print the text content of all the actors in the category. Try running the code below in your selected environment.+You should see, albeit very badly formatted, the text of the individual actor cards that are displayed in the selected categories.```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const sources = [- "https://apify.com/store?category=TRAVEL",- "https://apify.com/store?category=ECOMMERCE",- "https://apify.com/store?category=ENTERTAINMENT"+ 'https://apify.com/store?category=TRAVEL',+ 'https://apify.com/store?category=ECOMMERCE',+ 'https://apify.com/store?category=ENTERTAINMENT',];- const requestList = await Apify.openRequestList("categories", sources);+ const requestList = await Apify.openRequestList('categories', sources);const crawler = new Apify.CheerioCrawler({requestList,handlePageFunction: async ({ $, request }) => {// Select all the actor cards.- $(".item").each((i, el) => {+ $('.item').each((i, el) => {const text = $(el).text();console.log(`ITEM: ${text}\n`);});- }+ },});await crawler.run();});```-> If there's anything you don't understand, refer to the previous chapters on setting up your environment, building your first crawler and `CheerioCrawler`.+> If there's anything you don't understand, refer to the previous chapters on setting up your environment, building your first crawler and+> `CheerioCrawler`.-You might be wondering how we got that `.item` selector. After analyzing the category pages using a browser's DevTools, we've determined that it's a good selector to select all the currently displayed actor cards. DevTools and CSS selectors are quite a large topic, so we can't go into too much detail here, but here are a few general pointers.+You might be wondering how we got that `.item` selector. After analyzing the category pages using a browser's DevTools, we've determined that it's a+good selector to select all the currently displayed actor cards. DevTools and CSS selectors are quite a large topic, so we can't go into too much+detail here, but here are a few general pointers.#### DevTools crash course@@ -850,22 +1014,29 @@ We could pick any category, but let's just go with Travel because it includes sohttps://apify.com/store?category=TRAVEL```-and open DevTools either by right clicking anywhere in the page and selecting `Inspect`, or by pressing `F12` or by any other means relevant to your system. Once you're there, you'll see a bunch of DevToolsy stuff and a view of the category page with the individual actor cards.+and open DevTools either by right clicking anywhere in the page and selecting `Inspect`, or by pressing `F12` or by any other means relevant to your+system. Once you're there, you'll see a bunch of DevToolsy stuff and a view of the category page with the individual actor cards.-Now, find the `Select an element` tool and use it to select one of the actor cards. Make sure to select the whole card, not some of its contents, such as its title or description.+Now, find the `Select an element` tool and use it to select one of the actor cards. Make sure to select the whole card, not some of its contents, such+as its title or description.-In the resulting HTML display, it will put your cursor somewhere. Inspect the HTML around it. You'll see that there are CSS classes attached to the different HTML elements.+In the resulting HTML display, it will put your cursor somewhere. Inspect the HTML around it. You'll see that there are CSS classes attached to the+different HTML elements.-By hovering over the individual elements, you will see their placement in the page's view. It's easy to see the page's structure around the actor cards now. All the cards are displayed in a `<div>` with a classname that starts with `ItemsGrid__StyledDiv`, which holds another `<div>` with some computer generated class names and finally, inside this `<div>`, the individual cards are represented by other `<div>` elements with the class of `item`.+By hovering over the individual elements, you will see their placement in the page's view. It's easy to see the page's structure around the actor+cards now. All the cards are displayed in a `<div>` with a classname that starts with `ItemsGrid__StyledDiv`, which holds another `<div>` with some+computer generated class names and finally, inside this `<div>`, the individual cards are represented by other `<div>` elements with the class of+`item`.> Yes, there are other HTML elements and other classes too. We can safely ignore them.-It should now make sense how we got that `.item` selector. It's just a selector that finds all elements that are annotated with the `item` class and those just happen to be the actor cards only.+It should now make sense how we got that `.item` selector. It's just a selector that finds all elements that are annotated with the `item` class and+those just happen to be the actor cards only.It's always a good idea to double check that though, so go into the DevTools Console and run```js-document.querySelectorAll(".item");+document.querySelectorAll('.item');```You will see that only the actor cards will be returned, and nothing else.@@ -878,12 +1049,14 @@ In the previous chapter, we've used the `Apify.utils.enqueueLinks()` function liawait enqueueLinks({$,requestQueue,- pseudoUrls: ["http[s?]://apify.com[.*]"],- baseUrl: request.loadedUrl+ pseudoUrls: ['http[s?]://apify.com[.*]'],+ baseUrl: request.loadedUrl,});```-While very useful in that scenario, we need something different now. Instead of finding all the `<a href="..">` links that match the `pseudoUrl`, we need to find only the specific ones that will take us to the actor detail pages. Otherwise, we'd be visiting a lot of other pages that we're not interested in. Using the power of DevTools and yet another `enqueueLinks()` parameter, this becomes fairly easy.+While very useful in that scenario, we need something different now. Instead of finding all the `<a href="..">` links that match the `pseudoUrl`, we+need to find only the specific ones that will take us to the actor detail pages. Otherwise, we'd be visiting a lot of other pages that we're not+interested in. Using the power of DevTools and yet another `enqueueLinks()` parameter, this becomes fairly easy.```jsconst handlePageFunction = async ({ $, request }) => {@@ -894,47 +1067,59 @@ const handlePageFunction = async ({ $, request }) => {await Apify.utils.enqueueLinks({$,requestQueue,- selector: "div.item > a",+ selector: 'div.item > a',baseUrl: request.loadedUrl,userData: {- detailPage: true- }+ detailPage: true,+ },});}};```-The code should look pretty familiar to you. It's a very simple `handlePageFunction` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.+The code should look pretty familiar to you. It's a very simple `handlePageFunction` where we log the currently processed URL to the console and+enqueue more links. But there are also a few new, interesting additions. Let's break it down.##### The `selector` parameter of `enqueueLinks()`-When we previously used `enqueueLinks()`, we were not providing any `selector` parameter and it was fine, because we wanted to use the default setting, which is `a` - finds all `<a>` elements. But now, we need to be more specific. There are multiple `<a>` links on the given category page, but we're only interested in those that will take us to item (actor) details. Using the DevTools, we found out that we can select the links we wanted using the `div.item > a` selector, which selects all the `<a>` elements that have a `<div class="item ...">` parent. And those are exactly the ones we're interested in.+When we previously used `enqueueLinks()`, we were not providing any `selector` parameter and it was fine, because we wanted to use the default+setting, which is `a` - finds all `<a>` elements. But now, we need to be more specific. There are multiple `<a>` links on the given category page, but+we're only interested in those that will take us to item (actor) details. Using the DevTools, we found out that we can select the links we wanted+using the `div.item > a` selector, which selects all the `<a>` elements that have a `<div class="item ...">` parent. And those are exactly the ones+we're interested in.##### The missing `pseudoUrls`-Earlier we've learned that `pseudoUrls` are not required and if omitted, all links matching the given `selector` will be enqueued. This is exactly what we need, so we're skipping `pseudoUrls` this time. That does not mean that you can't use `pseudoUrls` together with a custom `selector` though, because you absolutely can!+Earlier we've learned that `pseudoUrls` are not required and if omitted, all links matching the given `selector` will be enqueued. This is exactly+what we need, so we're skipping `pseudoUrls` this time. That does not mean that you can't use `pseudoUrls` together with a custom `selector` though,+because you absolutely can!##### Finally, the `userData` of `enqueueLinks()`-You will see `userData` used often throughout Apify SDK and it's nothing more than a place to store your own data on a `Request` instance. You can access it by `request.userData` and it's a plain `Object` that can be used to store anything that needs to survive the full life-cycle of the `Request`.+You will see `userData` used often throughout Apify SDK and it's nothing more than a place to store your own data on a `Request` instance. You can+access it by `request.userData` and it's a plain `Object` that can be used to store anything that needs to survive the full life-cycle of the+`Request`.-We can use the `transformRequestFunction` option of `enqueueLinks()` to modify all the `Request` instances it creates and enqueues. In our case, we use it to set a `detailPage` property to the enqueued `Requests` to be able to easily differentiate between the category pages and the detail pages.+We can use the `transformRequestFunction` option of `enqueueLinks()` to modify all the `Request` instances it creates and enqueues. In our case, we+use it to set a `detailPage` property to the enqueued `Requests` to be able to easily differentiate between the category pages and the detail pages.#### Another sanity check-It's always good to work step by step. We have this new enqueueing logic in place and since the previous [Sanity check](#sanity-check) worked only with a `RequestList`, because we were not enqueueing anything so don't forget to add back the `RequestQueue` and `maxRequestsPerCrawl` limit. Let's test it out!+It's always good to work step by step. We have this new enqueueing logic in place and since the previous [Sanity check](#sanity-check) worked only+with a `RequestList`, because we were not enqueueing anything so don't forget to add back the `RequestQueue` and `maxRequestsPerCrawl` limit. Let's+test it out!```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const sources = [- "https://apify.com/store?category=TRAVEL",- "https://apify.com/store?category=ECOMMERCE",- "https://apify.com/store?category=ENTERTAINMENT"+ 'https://apify.com/store?category=TRAVEL',+ 'https://apify.com/store?category=ECOMMERCE',+ 'https://apify.com/store?category=ENTERTAINMENT',];- const requestList = await Apify.openRequestList("categories", sources);+ const requestList = await Apify.openRequestList('categories', sources);const requestQueue = await Apify.openRequestQueue(); // <----------------const crawler = new Apify.CheerioCrawler({@@ -949,28 +1134,32 @@ Apify.main(async () => {await Apify.utils.enqueueLinks({$,requestQueue,- selector: "div.item > a",+ selector: 'div.item > a',baseUrl: request.loadedUrl,- transformRequestFunction: (req) => {+ transformRequestFunction: req => {req.userData.detailPage = true;return req;- }+ },});}- }+ },});await crawler.run();});```-We've added the `handlePageFunction()` with the `enqueueLinks()` logic from the previous section to the code we've written earlier. As always, try running it in an environment of your choice. You should see the crawler output a number of links to the console, as it crawls the category pages first and then all the links to the actor detail pages it found.+We've added the `handlePageFunction()` with the `enqueueLinks()` logic from the previous section to the code we've written earlier. As always, try+running it in an environment of your choice. You should see the crawler output a number of links to the console, as it crawls the category pages first+and then all the links to the actor detail pages it found.-This concludes our Crawling strategy section, because we have taught the crawler to visit all the pages we need. Let's continue with scraping the tasty data.+This concludes our Crawling strategy section, because we have taught the crawler to visit all the pages we need. Let's continue with scraping the+tasty data.### Scraping data-At the beginning of this chapter, we've created a list of information we wanted to collect about the actors in the store. Let's review that and figure out ways to access it.+At the beginning of this chapter, we've created a list of information we wanted to collect about the actors in the store. Let's review that and figure+out ways to access it.1. URL2. Owner@@ -980,34 +1169,41 @@ At the beginning of this chapter, we've created a list of information we wanted6. Last run date7. Number of runs-+#### Scraping the URL, Owner and Unique identifier-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!+Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the+`request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the+`string` and be on our way then!```js// request.url = https://apify.com/apify/web-scraper-const urlArr = request.url.split("/").slice(-2); // ['apify', 'web-scraper']-const uniqueIdentifier = urlArr.join("/"); // 'apify/web-scraper'+const urlArr = request.url.split('/').slice(-2); // ['apify', 'web-scraper']+const uniqueIdentifier = urlArr.join('/'); // 'apify/web-scraper'const owner = urlArr[0]; // 'apify'```-> It's always a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should we duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's always better to make the dataset as readable as possible. Someone might want to filter by `owner` for example and keeping only the `URL` in the dataset would make this complicated without using additional tools.+> It's always a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can+> easily parse the `owner` from the `URL`, so should we duplicate the data unnecessarily? Our opinion is that unless the increased data consumption+> would be too large to bear, it's always better to make the dataset as readable as possible. Someone might want to filter by `owner` for example and+> keeping only the `URL` in the dataset would make this complicated without using additional tools.#### Scraping Title, Description, Last run date and Number of runs-Now it's time to add more data to the results. Let's open one of the actor detail pages in the Store, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu to figure out how to get the title of the actor.+Now it's time to add more data to the results. Let's open one of the actor detail pages in the Store, for example the+[`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu to figure out how to get the title of the actor.##### Title-+Let's start really easy. By using the element selector tool, we find out that the title is there under an `<h1>` tag, as titles should be.-> Remember that you can press CTRL+F (CMD+F) in the Elements tab of DevTools to open the search bar where you can quickly search for elements using their selectors.-And always make sure to use the DevTools to verify your scraping process and assumptions. It's faster than changing the crawler code all the time.+> Remember that you can press CTRL+F (CMD+F) in the Elements tab of DevTools to open the search bar where you can quickly search for elements using+> their selectors. And always make sure to use the DevTools to verify your scraping process and assumptions. It's faster than changing the crawler+> code all the time.To get the title we just need to find it using `Cheerio`:@@ -1019,12 +1215,12 @@ return {##### Description-Getting the actor's description is a little more involved, but still pretty straightforward. We can't just simply search for a `<p>` tag,-because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description-is nested within the `<header>` element, which is nested itself in the `<main>` element. Sadly, we're still left with two `<p>` tags.-To finally select only the description, we choose the `<p>` tag that has a `class` that starts with `Text__Paragraph`.+Getting the actor's description is a little more involved, but still pretty straightforward. We can't just simply search for a `<p>` tag, because+there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is nested within+the `<header>` element, which is nested itself in the `<main>` element. Sadly, we're still left with two `<p>` tags. To finally select only the+description, we choose the `<p>` tag that has a `class` that starts with `Text__Paragraph`.-+```jsreturn {@@ -1037,51 +1233,85 @@ return {The DevTools tell us that the `lastRunDate` can be found in the second of the two `<time>` elements in the page.-+```jsreturn {title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),};```-It might look a little too complex at first glance, but let me walk you through it. We find all the `<time>` elements. There are two, so we grab the second one using the `.eq(1)` call (it's zero indexed) and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.+It might look a little too complex at first glance, but let me walk you through it. We find all the `<time>` elements. There are two, so we grab the+second one using the `.eq(1)` call (it's zero indexed) and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a+`string`.-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!+But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()`+constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`.+Phew!##### Run count-And so we're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.+And so we're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a+transformation on the result.```jsreturn {title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),- runCount: Number($('ul.stats li:nth-of-type(3)').text().match(/\d+/)[0]),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),+ runCount: Number(+ $('ul.stats li:nth-of-type(3)')+ .text()+ .match(/\d+/)[0],+ ),};```-The `ul.stats > li:nth-of-type(3)` looks complicated, but it only reads that we're looking for a `<ul class="stats ...">` element and within that element we're looking for the third `<li>` element. We grab its text, but we're only interested in the number of runs. So we parse the number out using a regular expression, but its type is still a `string`, so we finally convert the result to a `number` by wrapping it with a `Number()` call.+The `ul.stats > li:nth-of-type(3)` looks complicated, but it only reads that we're looking for a `<ul class="stats ...">` element and within that+element we're looking for the third `<li>` element. We grab its text, but we're only interested in the number of runs. So we parse the number out+using a regular expression, but its type is still a `string`, so we finally convert the result to a `number` by wrapping it with a `Number()` call.-And there we have it! All the data we needed in a single object. For the sake of completeness, let's add the properties we parsed from the URL earlier and we're good to go.+And there we have it! All the data we needed in a single object. For the sake of completeness, let's add the properties we parsed from the URL earlier+and we're good to go.```js-const urlArr = request.url.split("/").slice(-2);+const urlArr = request.url.split('/').slice(-2);const results = {url: request.url,- uniqueIdentifier: urlArr.join("/"),+ uniqueIdentifier: urlArr.join('/'),owner: urlArr[0],title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),- runCount: Number($('ul.stats li:nth-of-type(3)').text().match(/\d+/)[0]),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),+ runCount: Number(+ $('ul.stats li:nth-of-type(3)')+ .text()+ .match(/\d+/)[0],+ ),};-console.log("RESULTS: ", results);+console.log('RESULTS: ', results);```#### Trying it out (sanity check #3)@@ -1089,16 +1319,16 @@ console.log("RESULTS: ", results);We have everything we need so just grab our newly created scraping logic, dump it into our original `handlePageFunction()` and see the magic!```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const sources = [- "https://apify.com/store?category=TRAVEL",- "https://apify.com/store?category=ECOMMERCE",- "https://apify.com/store?category=ENTERTAINMENT"+ 'https://apify.com/store?category=TRAVEL',+ 'https://apify.com/store?category=ECOMMERCE',+ 'https://apify.com/store?category=ENTERTAINMENT',];- const requestList = await Apify.openRequestList("categories", sources);+ const requestList = await Apify.openRequestList('categories', sources);const requestQueue = await Apify.openRequestQueue();const crawler = new Apify.CheerioCrawler({@@ -1110,18 +1340,28 @@ Apify.main(async () => {// This is our new scraping logic.if (request.userData.detailPage) {- const urlArr = request.url.split("/").slice(-2);+ const urlArr = request.url.split('/').slice(-2);const results = {url: request.url,- uniqueIdentifier: urlArr.join("/"),+ uniqueIdentifier: urlArr.join('/'),owner: urlArr[0],title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),- runCount: Number($('ul.stats li:nth-of-type(3)').text().match(/\d+/)[0]),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),+ runCount: Number(+ $('ul.stats li:nth-of-type(3)')+ .text()+ .match(/\d+/)[0],+ ),};- console.log("RESULTS", results);+ console.log('RESULTS', results);}// Only enqueue new links from the category pages.@@ -1129,28 +1369,30 @@ Apify.main(async () => {await Apify.utils.enqueueLinks({$,requestQueue,- selector: "div.item > a",+ selector: 'div.item > a',baseUrl: request.loadedUrl,- transformRequestFunction: (req) => {+ transformRequestFunction: req => {req.userData.detailPage = true;return req;- }+ },});}- }+ },});await crawler.run();});```-> Notice again that we're scraping on the detail pages `request.userData.detailPage === true`, but we're only enqueueing on the category pages `request.userData.detailPage === undefined`.+> Notice again that we're scraping on the detail pages `request.userData.detailPage === true`, but we're only enqueueing on the category pages+> `request.userData.detailPage === undefined`.When running the actor in an environment of your choice, you should see the crawled URLs and their scraped data printed to the console.### Saving the scraped data-A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this chapter so make sure to pay attention very carefully!+A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of+this chapter so make sure to pay attention very carefully!First, replace the `console.log('RESULTS', results)` call with@@ -1161,16 +1403,16 @@ await Apify.pushData(results);and that's it. Unlike in the previous paragraph, I'm serious now. That's it, we're done. The final code therefore looks exactly like this:```js-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const sources = [- "https://apify.com/store?category=TRAVEL",- "https://apify.com/store?category=ECOMMERCE",- "https://apify.com/store?category=ENTERTAINMENT"+ 'https://apify.com/store?category=TRAVEL',+ 'https://apify.com/store?category=ECOMMERCE',+ 'https://apify.com/store?category=ENTERTAINMENT',];- const requestList = await Apify.openRequestList("categories", sources);+ const requestList = await Apify.openRequestList('categories', sources);const requestQueue = await Apify.openRequestQueue();const crawler = new Apify.CheerioCrawler({@@ -1182,16 +1424,26 @@ Apify.main(async () => {// This is our new scraping logic.if (request.userData.detailPage) {- const urlArr = request.url.split("/").slice(-2);+ const urlArr = request.url.split('/').slice(-2);const results = {url: request.url,- uniqueIdentifier: urlArr.join("/"),+ uniqueIdentifier: urlArr.join('/'),owner: urlArr[0],title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),- runCount: Number($('ul.stats li:nth-of-type(3)').text().match(/\d+/)[0]),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),+ runCount: Number(+ $('ul.stats li:nth-of-type(3)')+ .text()+ .match(/\d+/)[0],+ ),};await Apify.pushData(results);}@@ -1201,15 +1453,15 @@ Apify.main(async () => {await Apify.utils.enqueueLinks({$,requestQueue,- selector: "div.item > a",+ selector: 'div.item > a',baseUrl: request.loadedUrl,- transformRequestFunction: (req) => {+ transformRequestFunction: req => {req.userData.detailPage = true;return req;- }+ },});}- }+ },});await crawler.run();@@ -1218,9 +1470,12 @@ Apify.main(async () => {#### What's `Apify.pushData()`-[`Apify.pushData()`](../api/apify#apify.pushData) is a helper function that saves data to the default [`Dataset`](../api/dataset). `Dataset` is a storage designed to hold virtually unlimited amount of data in a format similar to a table. Each time you call `Apify.pushData()` a new row in the table is created, with the property names serving as column titles.+[`Apify.pushData()`](../api/apify#apify.pushData) is a helper function that saves data to the default [`Dataset`](../api/dataset). `Dataset` is a+storage designed to hold virtually unlimited amount of data in a format similar to a table. Each time you call `Apify.pushData()` a new row in the+table is created, with the property names serving as column titles.-> Each actor run has one default `Dataset` so no need to initialize it or create an instance first. It just gets done automatically for you. You can also create named datasets at will.+> Each actor run has one default `Dataset` so no need to initialize it or create an instance first. It just gets done automatically for you. You can+> also create named datasets at will.#### Finding my saved data@@ -1228,29 +1483,40 @@ It might not be perfectly obvious where the data we saved using the previous com##### Dataset on the Apify Platform-Open any Run of your actor on the Platform and you will see a Dataset as one of the available tabs. Clicking on it will reveal basic information about the Dataset and a list of options that you can use to download your data. There are various formats such as JSON, XLSX or CSV available and there's also the possibility of downloading only Clean items, i.e. a filtered dataset with empty rows and hidden fields removed.+Open any Run of your actor on the Platform and you will see a Dataset as one of the available tabs. Clicking on it will reveal basic information about+the Dataset and a list of options that you can use to download your data. There are various formats such as JSON, XLSX or CSV available and there's+also the possibility of downloading only Clean items, i.e. a filtered dataset with empty rows and hidden fields removed.##### Local Dataset-Unless you changed the environment variables that Apify SDK uses locally, which would suggest that you knew what you were doing and you didn't need this tutorial anyway, you'll find your data in your local Apify Storage.+Unless you changed the environment variables that Apify SDK uses locally, which would suggest that you knew what you were doing and you didn't need+this tutorial anyway, you'll find your data in your local Apify Storage.```{PROJECT_FOLDER}/apify_storage/datasets/default/```-The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of `Apify.pushData()` or one table row.+The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of+`Apify.pushData()` or one table row.-> Unfortunately, the local datasets don't yet support the export in various formats functionality that the Platform Dataset page offers, so for the time being, we're stuck with JSON.+> Unfortunately, the local datasets don't yet support the export in various formats functionality that the Platform Dataset page offers, so for the+> time being, we're stuck with JSON.### Final touch-It may seem that the data are extracted and the actor is done, but honestly, this is just the beginning. For the sake of brevity, we've completely omitted error handling, proxies, debug logging, tests, documentation and other stuff that a reliable software should have. The good thing is, **error handling is mostly done by Apify SDK itself**, so no worries on that front, unless you need some custom magic.+It may seem that the data are extracted and the actor is done, but honestly, this is just the beginning. For the sake of brevity, we've completely+omitted error handling, proxies, debug logging, tests, documentation and other stuff that a reliable software should have. The good thing is, **error+handling is mostly done by Apify SDK itself**, so no worries on that front, unless you need some custom magic.-Anyway, to spark some ideas, let's look at two more things. First, passing an input to the actor, which will enable us to change the categories we want to scrape without changing the source code itself! And then some refactoring, to show you how we reckon is preferable to structure and annotate actor code.+Anyway, to spark some ideas, let's look at two more things. First, passing an input to the actor, which will enable us to change the categories we+want to scrape without changing the source code itself! And then some refactoring, to show you how we reckon is preferable to structure and annotate+actor code.#### Meet the `INPUT`-`INPUT` is just a convention on how we call the actor's input. Because there's no magic in actors, just features, the `INPUT` is actually nothing more than a key in the default [`KeyValueStore`](../api/keyvaluestore) that's, by convention, used as input on Apify Platform. Also by convention, the `INPUT` is mostly expected to be of `Content-Type: application/json`.+`INPUT` is just a convention on how we call the actor's input. Because there's no magic in actors, just features, the `INPUT` is actually nothing more+than a key in the default [`KeyValueStore`](../api/keyvaluestore) that's, by convention, used as input on Apify Platform. Also by convention, the+`INPUT` is mostly expected to be of `Content-Type: application/json`.We will not go into `KeyValueStore` details here, but for the sake of `INPUT` you need to remember that there is a function that helps you get it.@@ -1258,7 +1524,8 @@ We will not go into `KeyValueStore` details here, but for the sake of `INPUT` yoconst input = await Apify.getInput();```-On the Apify Platform, the actor's input that you can set in the Console is automatically saved to the default `KeyValueStore` under the key `INPUT` and by calling [`Apify.getInput()`](../api/apify#apify.getValue) you retrieve the value from the `KeyValueStore`.+On the Apify Platform, the actor's input that you can set in the Console is automatically saved to the default `KeyValueStore` under the key `INPUT`+and by calling [`Apify.getInput()`](../api/apify#apify.getValue) you retrieve the value from the `KeyValueStore`.Running locally, you need to place an `INPUT.json` file in your default key value store for this to work.@@ -1268,15 +1535,20 @@ Running locally, you need to place an `INPUT.json` file in your default key valu#### Use `INPUT` to seed our actor with categories-Currently we're using the full URLs of categories as sources, but it's quite obvious that we only need the final parameters, the rest of the URL is always the same. Knowing that, we can pass an array of those parameters on `INPUT` and build the URLs dynamically, which would allow us to scrape different categories without changing the source code. Let's get to it!+Currently we're using the full URLs of categories as sources, but it's quite obvious that we only need the final parameters, the rest of the URL is+always the same. Knowing that, we can pass an array of those parameters on `INPUT` and build the URLs dynamically, which would allow us to scrape+different categories without changing the source code. Let's get to it!-First, we set up our `INPUT`, either in the `INPUT` form of the actor on Apify Platform, or by creating an `INPUT.json` in our default key value store locally.+First, we set up our `INPUT`, either in the `INPUT` form of the actor on Apify Platform, or by creating an `INPUT.json` in our default key value store+locally.```json["TRAVEL", "ECOMMERCE", "ENTERTAINMENT"]```-Once we have that, we can load it in the actor and populate the crawler's sources with it. In the following example, we're using the categories in the input to construct the category URLs and we're also passing custom `userData` to the sources. This means that the `Requests` that get created will automatically contain this `userData`.+Once we have that, we can load it in the actor and populate the crawler's sources with it. In the following example, we're using the categories in the+input to construct the category URLs and we're also passing custom `userData` to the sources. This means that the `Requests` that get created will+automatically contain this `userData`.```js// ...@@ -1285,21 +1557,24 @@ const input = await Apify.getInput();const sources = input.map(category => ({url: `https://apify.com/store?category=${category}`,userData: {- label: "CATEGORY"- }+ label: 'CATEGORY',+ },}));-const requestList = await Apify.openRequestList("categories", sources);+const requestList = await Apify.openRequestList('categories', sources);// ...```-The `userData.label` is also a convention that we've been using for quite some time to label different `Requests`. We know that this is a category URL so we `label` it `CATEGORY`. This way, we can easily make decisions in the `handlePageFunction` without having to inspect the URL itself.+The `userData.label` is also a convention that we've been using for quite some time to label different `Requests`. We know that this is a category URL+so we `label` it `CATEGORY`. This way, we can easily make decisions in the `handlePageFunction` without having to inspect the URL itself.-We can then refactor the `if` clauses in the `handlePageFunction` to use the `label` for decision making. This does not make much sense for a crawler with only two different pages, because a simple `boolean` would suffice, but for pages with multiple different views, it becomes very useful.+We can then refactor the `if` clauses in the `handlePageFunction` to use the `label` for decision making. This does not make much sense for a crawler+with only two different pages, because a simple `boolean` would suffice, but for pages with multiple different views, it becomes very useful.#### Structuring the code better-But perhaps we should not stop at refactoring the `if` clauses. There are several ways we can make the actor look more elegant and - most importantly - easier to reason about and make changes to.+But perhaps we should not stop at refactoring the `if` clauses. There are several ways we can make the actor look more elegant and - most+importantly - easier to reason about and make changes to.In the following code we've made several changes.@@ -1315,22 +1590,19 @@ In our `main.js` file, we place the general structure of the crawler:```js// main.js-const Apify = require("apify");-const tools = require("./tools");+const Apify = require('apify');+const tools = require('./tools');const {- utils: { log }+ utils: { log },} = Apify;Apify.main(async () => {- log.info("Starting actor.");- const requestList = await Apify.openRequestList(- "categories",- await tools.getSources()- );+ log.info('Starting actor.');+ const requestList = await Apify.openRequestList('categories', await tools.getSources());const requestQueue = await Apify.openRequestQueue();const router = tools.createRouter({ requestQueue });- log.debug("Setting up crawler.");+ log.debug('Setting up crawler.');const crawler = new Apify.CheerioCrawler({requestList,requestQueue,@@ -1338,12 +1610,12 @@ Apify.main(async () => {const { request } = context;log.info(`Processing ${request.url}`);await router(request.userData.label, context);- }+ },});- log.info("Starting the crawl.");+ log.info('Starting the crawl.');await crawler.run();- log.info("Actor finished.");+ log.info('Actor finished.');});```@@ -1351,20 +1623,20 @@ Then in a separate `tools.js`, we add our helper functions:```js// tools.js-const Apify = require("apify");-const routes = require("./routes");+const Apify = require('apify');+const routes = require('./routes');const {- utils: { log }+ utils: { log },} = Apify;exports.getSources = async () => {- log.debug("Getting sources.");+ log.debug('Getting sources.');const input = await Apify.getInput();return input.map(category => ({url: `https://apify.com/store?category=${category}`,userData: {- label: "CATEGORY"- }+ label: 'CATEGORY',+ },}));};@@ -1382,61 +1654,82 @@ And finally our routes in a separate `routes.js` file:```js// routes.js-const Apify = require("apify");+const Apify = require('apify');const {- utils: { log }+ utils: { log },} = Apify;exports.CATEGORY = async ({ $, request }, { requestQueue }) => {return Apify.utils.enqueueLinks({$,requestQueue,- selector: "div.item > a",+ selector: 'div.item > a',baseUrl: request.loadedUrl,- transformRequestFunction: (req) => {+ transformRequestFunction: req => {req.userData.label = 'DETAIL';return req;- }+ },});};exports.DETAIL = async ({ $, request }) => {- const urlArr = request.url.split("/").slice(-2);+ const urlArr = request.url.split('/').slice(-2);- log.debug("Scraping results.");+ log.debug('Scraping results.');const results = {url: request.url,- uniqueIdentifier: urlArr.join("/"),+ uniqueIdentifier: urlArr.join('/'),owner: urlArr[0],title: $('h1').text(),description: $('main header p[class^=Text__Paragraph]').text(),- lastRunDate: new Date(Number($('time').eq(1).attr('datetime'))),- runCount: Number($('ul.stats li:nth-of-type(3)').text().match(/\d+/)[0]),+ lastRunDate: new Date(+ Number(+ $('time')+ .eq(1)+ .attr('datetime'),+ ),+ ),+ runCount: Number(+ $('ul.stats li:nth-of-type(3)')+ .text()+ .match(/\d+/)[0],+ ),};- log.debug("Pushing data to dataset.");+ log.debug('Pushing data to dataset.');await Apify.pushData(results);};```-Let us tell you a little bit more about the changes. We're hopeful that in the end, you'll agree that this structure makes the actor more readable and manageable.+Let us tell you a little bit more about the changes. We're hopeful that in the end, you'll agree that this structure makes the actor more readable and+manageable.#### Splitting your code into multiple files-It was not always the case, but now that Apify Platform has a multifile editor, there's no reason not to split your code into multiple files and keep your logic separate. Less code in a single file means less code you need to think about at any time, and that's a great thing!+It was not always the case, but now that Apify Platform has a multifile editor, there's no reason not to split your code into multiple files and keep+your logic separate. Less code in a single file means less code you need to think about at any time, and that's a great thing!#### Using `Apify.utils.log` instead of `console.log`-We wont go into great lengths here to talk about `utils.log` here, because you can read [it all in the documentation](../api/log), but there's just one thing that we need to stress: **log levels**.+We wont go into great lengths here to talk about `utils.log` here, because you can read [it all in the documentation](../api/log), but there's just+one thing that we need to stress: **log levels**.-`utils.log` enables you to use different log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `utils.log.setLevel()` function or by setting an `APIFY_LOG_LEVEL` variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple `INPUT` change, or by setting an environment variable.+`utils.log` enables you to use different log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but+it also allows selective turning off of some levels by either calling the `utils.log.setLevel()` function or by setting an `APIFY_LOG_LEVEL` variable.+This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a+simple `INPUT` change, or by setting an environment variable.The punch line? Use `Apify.utils.log` instead of `console.log` now and thank us later when something goes wrong!#### Using a router to structure your crawling-At first, it might seem more readable using just a simple `if / else` statement to select different logic based on the crawled pages, but trust me, it becomes far less impressive when working with more than 2 different pages and it definitely starts to fall apart when the logic to handle each page spans tens or hundreds of lines of code.+At first, it might seem more readable using just a simple `if / else` statement to select different logic based on the crawled pages, but trust me, it+becomes far less impressive when working with more than 2 different pages and it definitely starts to fall apart when the logic to handle each page+spans tens or hundreds of lines of code.-It's a good practice in any programming to split your logic into bite sized chunks that are easy to read and reason about. Scrolling through a thousand line long `handlePageFunction()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into a special file and with large routes, we would even suggest having one file per route.+It's a good practice in any programming to split your logic into bite sized chunks that are easy to read and reason about. Scrolling through a+thousand line long `handlePageFunction()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to+do and a pain to debug. That's why we prefer the separation of routes into a special file and with large routes, we would even suggest having one file+per route.> TO BE CONTINUED with details on `PuppeteerCrawler` and other features...diff --git a/docs/guides/getting_started_quick.md b/docs/guides/getting_started_quick.mdindex fc9862f31..10ee7d7f8 100644--- a/docs/guides/getting_started_quick.md+++ b/docs/guides/getting_started_quick.md@@ -13,17 +13,16 @@ Add Apify SDK to any Node.js project by running:npm install apify --save```-Run the following example to perform a recursive crawl of a website using Puppeteer.-For more examples showcasing various features of the Apify SDK,+Run the following example to perform a recursive crawl of a website using Puppeteer. For more examples showcasing various features of the Apify SDK,[see the Examples section of the documentation](../examples/basiccrawler).```javascript-const Apify = require("apify");+const Apify = require('apify');Apify.main(async () => {const requestQueue = await Apify.openRequestQueue();- await requestQueue.addRequest({ url: "https://www.iana.org/" });- const pseudoUrls = [new Apify.PseudoUrl("https://www.iana.org/[.*]")];+ await requestQueue.addRequest({ url: 'https://www.iana.org/' });+ const pseudoUrls = [new Apify.PseudoUrl('https://www.iana.org/[.*]')];const crawler = new Apify.PuppeteerCrawler({requestQueue,@@ -32,13 +31,13 @@ Apify.main(async () => {console.log(`Title of ${request.url}: ${title}`);await Apify.utils.puppeteer.enqueueLinks({page,- selector: "a",+ selector: 'a',pseudoUrls,- requestQueue+ requestQueue,});},maxRequestsPerCrawl: 100,- maxConcurrency: 10+ maxConcurrency: 10,});await crawler.run();@@ -49,22 +48,15 @@ When you run the example, you should see Apify SDK automating a Chrome browser.-By default, Apify SDK stores data to-`./apify_storage` in the current working directory.-You can override this behavior by setting either the-`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable.-For details, see [Environment variables](environmentvariables)-and [Data storage](datastorage).+By default, Apify SDK stores data to `./apify_storage` in the current working directory. You can override this behavior by setting either the+`APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variable. For details, see [Environment variables](environmentvariables) and+[Data storage](datastorage).## Local usage with Apify command-line interface (CLI)-To avoid the need to set the environment variables manually,-to create a boilerplate of your project,-and to enable pushing and running your code on the-<a href="https://apify.com" target="_blank">Apify Cloud</a>,-you can use the-<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a>-(CLI) tool.+To avoid the need to set the environment variables manually, to create a boilerplate of your project, and to enable pushing and running your code on+the <a href="https://apify.com" target="_blank">Apify Cloud</a>, you can use the+<a href="https://github.com/apifytech/apify-cli" target="_blank">Apify command-line interface</a> (CLI) tool.Install the CLI by running:@@ -80,18 +72,16 @@ Now create a boilerplate of your new web crawling project by running:apify create my-hello-world```-The CLI will prompt you to select a project boilerplate template - just pick "Hello world".-The tool will create a directory called `my-hello-world` with a Node.js project files.-You can run the project as follows:+The CLI will prompt you to select a project boilerplate template - just pick "Hello world". The tool will create a directory called `my-hello-world`+with a Node.js project files. You can run the project as follows:```bashcd my-hello-worldapify run```-By default, the crawling data will be stored in a local directory at `./apify_storage`.-For example, the input JSON file for the actor is expected to be in the default key-value store-in `./apify_storage/key_value_stores/default/INPUT.json`.+By default, the crawling data will be stored in a local directory at `./apify_storage`. For example, the input JSON file for the actor is expected to+be in the default key-value store in `./apify_storage/key_value_stores/default/INPUT.json`.Now you can easily deploy your code to the Apify Cloud by running:@@ -103,22 +93,14 @@ apify loginapify push```-Your script will be uploaded to the Apify Cloud and built there so that it can be run.-For more information, view the-<a href="https://apify.com/docs/cli" target="_blank">Apify CLI</a>-and-<a href="https://apify.com/docs/actor" target="_blank">Apify Actor</a>+Your script will be uploaded to the Apify Cloud and built there so that it can be run. For more information, view the+<a href="https://apify.com/docs/cli" target="_blank">Apify CLI</a> and <a href="https://apify.com/docs/actor" target="_blank">Apify Actor</a>documentation.## Usage on the Apify Cloud-You can also develop your web scraping project-in an online code editor directly on the-<a href="https://apify.com" target="_blank">Apify Cloud</a>.-You'll need to have an Apify Account.-Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>,-page in the app, click <i>Create new</i> and then go to the-<i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.+You can also develop your web scraping project in an online code editor directly on the <a href="https://apify.com" target="_blank">Apify Cloud</a>.+You'll need to have an Apify Account. Go to <a href="https://my.apify.com/actors" target="_blank">Actors</a>, page in the app, click <i>Create new</i>+and then go to the <i>Source</i> tab and start writing your code or paste one of the examples from the Examples section.-For more information, view the-<a href="https://apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.+For more information, view the <a href="https://apify.com/docs/actor#quick-start" target="_blank">Apify actors quick start guide</a>.diff --git a/docs/guides/motivation.md b/docs/guides/motivation.mdindex 1d742c779..cb695544f 100644--- a/docs/guides/motivation.md+++ b/docs/guides/motivation.md@@ -4,23 +4,19 @@ title: Motivation---Thanks to tools like <a href="https://github.com/GoogleChrome/puppeteer" target="_blank">Puppeteer</a> or-<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>,-it is easy to write Node.js code to extract data from web pages.-But eventually things will get complicated. For example, when you try to:+<a href="https://www.npmjs.com/package/cheerio" target="_blank">Cheerio</a>, it is easy to write Node.js code to extract data from web pages. But+eventually things will get complicated. For example, when you try to:- Perform a deep crawl of an entire website using a persistent queue of URLs.-- Run your scraping code on a list of 100k URLs in a CSV file,- without losing any data when your code crashes.+- Run your scraping code on a list of 100k URLs in a CSV file, without losing any data when your code crashes.- Rotate proxies to hide your browser origin.- Schedule the code to run periodically and send notification on errors.- Disable browser fingerprinting protections used by websites.-Python has <a href="https://scrapy.org/" target="_blank">Scrapy</a> for these tasks, but there was no-such library for **JavaScript, the language of the web**.-The use of JavaScript is natural,-since the same language is used to write the scripts as well as the data extraction code running in a browser.+Python has <a href="https://scrapy.org/" target="_blank">Scrapy</a> for these tasks, but there was no such library for **JavaScript, the language of+the web**. The use of JavaScript is natural, since the same language is used to write the scripts as well as the data extraction code running in a+browser.-The goal of the Apify SDK is to fill this gap and provide a toolbox-for generic web scraping, crawling and automation tasks in JavaScript.-So don't reinvent the wheel every time you need data from the web,-and focus on writing code specific to the target website, rather than developing commonalities.+The goal of the Apify SDK is to fill this gap and provide a toolbox for generic web scraping, crawling and automation tasks in JavaScript. So don't+reinvent the wheel every time you need data from the web, and focus on writing code specific to the target website, rather than developing+commonalities.diff --git a/docs/guides/puppeteer_live_view.md b/docs/guides/puppeteer_live_view.mdindex f282357a2..65cd7c8a2 100644--- a/docs/guides/puppeteer_live_view.md+++ b/docs/guides/puppeteer_live_view.md@@ -3,20 +3,18 @@ id: puppeteerliveviewtitle: Puppeteer Live View----Apify SDK enables real-time view of launched Puppeteer browser instances and their open tabs,-including screenshots of pages and snapshots of HTML.+Apify SDK enables real-time view of launched Puppeteer browser instances and their open tabs, including screenshots of pages and snapshots of HTML.This is useful for debugging your crawlers that run in headless mode.-The live view dashboard is run on a web server that is started on a port specified-by the `APIFY_CONTAINER_PORT` environment variable (typically 4321).-To enable live view, pass the `useliveView: true` option to-the `puppeteerPoolOptions` of [`PuppeteerCrawler`](../api/puppeteercrawler#new_PuppeteerCrawler_new):+The live view dashboard is run on a web server that is started on a port specified by the `APIFY_CONTAINER_PORT` environment variable (typically+4321). To enable live view, pass the `useliveView: true` option to the `puppeteerPoolOptions` of+[`PuppeteerCrawler`](../api/puppeteercrawler#new_PuppeteerCrawler_new):```jsconst crawler = new Apify.PuppeteerCrawler({puppeteerPoolOptions: {- useLiveView: true- }+ useLiveView: true,+ },// other options});```@@ -25,17 +23,15 @@ or directly to the [`PuppeteerPool`](../api/puppeteerpool) constructor, when usi```jsconst pool = new Apify.PuppeteerPool({- useLiveView: true+ useLiveView: true,// other options});```-To simplify debugging, you may also want to add the-`{ slowMo: 300 }` option to slow down all browser operations.-See <a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer documentation</a> for details.+To simplify debugging, you may also want to add the `{ slowMo: 300 }` option to slow down all browser operations. See+<a href="https://pptr.dev/#?product=Puppeteer&show=api-puppeteerlaunchoptions" target="_blank">Puppeteer documentation</a> for details.-Once live view is enabled, you can open `http://localhost:4321` and as the crawler runs,-you should see screenshots and HTML of the pages it opens displayed in your browser.+Once live view is enabled, you can open `http://localhost:4321` and as the crawler runs, you should see screenshots and HTML of the pages it opens+displayed in your browser.-To use live view on the Apify Platform (after enabling it in your actor),-simply select the Live View tab in your actor run view.+To use live view on the Apify Platform (after enabling it in your actor), simply select the Live View tab in your actor run view.diff --git a/docs/guides/what_is_an_actor.md b/docs/guides/what_is_an_actor.mdindex e874922f7..ca0ea0083 100644--- a/docs/guides/what_is_an_actor.md+++ b/docs/guides/what_is_an_actor.md@@ -3,18 +3,14 @@ id: whatisanactortitle: What is an Actor----When you deploy your script to the <a href="https://apify.com/" target="_blank">Apify Cloud</a>,-it becomes an actor.-An actor is a serverless microservice that accepts an input and produces an output.-It can run for a few seconds, hours or even infinitely.-An actor can perform anything from a simple action such as filling out a web form or sending an email,-to complex operations such as crawling an entire website and removing duplicates from a large dataset.+When you deploy your script to the <a href="https://apify.com/" target="_blank">Apify Cloud</a>, it becomes an actor. An actor is a serverless+microservice that accepts an input and produces an output. It can run for a few seconds, hours or even infinitely. An actor can perform anything from+a simple action such as filling out a web form or sending an email, to complex operations such as crawling an entire website and removing duplicates+from a large dataset.-To run an actor, you need to have an <a href="https://my.apify.com/" target="_blank">Apify Account</a>.-Actors can be shared in the <a href="https://apify.com/store" target="_blank">Apify Store</a>-so that other people can use them.-But don't worry, if you share your actor in the store-and somebody uses it, it runs under their account, not yours.+To run an actor, you need to have an <a href="https://my.apify.com/" target="_blank">Apify Account</a>. Actors can be shared in the+<a href="https://apify.com/store" target="_blank">Apify Store</a> so that other people can use them. But don't worry, if you share your actor in the+store and somebody uses it, it runs under their account, not yours.**Related links**diff --git a/website/package.json b/website/package.jsonindex e7ba51690..429d4a2f9 100644--- a/website/package.json+++ b/website/package.json@@ -7,7 +7,8 @@"write-translations": "docusaurus-write-translations","version": "docusaurus-version","rename-version": "docusaurus-rename-version",- "generate": "node ./tools/build_docs.js"+ "generate": "node ./tools/build_docs.js",+ "prettify": "prettier --write --config ./tools/prettier.config.js ../docs/guides/*.md"},"devDependencies": {"docusaurus": "^1.10.0",diff --git a/website/tools/build_docs.js b/website/tools/build_docs.jsindex b369401e5..4d3131cf1 100644--- a/website/tools/build_docs.js+++ b/website/tools/build_docs.js@@ -3,16 +3,8 @@ const readline = require('readline');const fs = require('fs');const path = require('path');const { promisify } = require('util');-const prettier = require('prettier');--const prettierConfig = {- parser: 'markdown',- arrowParens: 'avoid',- trailingComma: 'all',- singleQuote: true,- tabWidth: 4,- printWidth: 150,-};+const prettier = require('prettier'); // eslint-disable-line+const prettierConfig = require('./prettier.config');const writeFile = promisify(fs.writeFile);diff --git a/website/tools/prettier.config.js b/website/tools/prettier.config.jsnew file mode 100644index 000000000..b233e263b--- /dev/null+++ b/website/tools/prettier.config.js@@ -0,0 +1,9 @@+module.exports = {+ parser: 'markdown',+ arrowParens: 'avoid',+ trailingComma: 'all',+ singleQuote: true,+ tabWidth: 4,+ printWidth: 150,+ proseWrap: 'always',+};