Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Update README examples
Ondra Urbanfc833c83a43e
diff --git a/README-new.md b/README-new.mdindex c78c63737..6370fb709 100644--- a/README-new.md+++ b/README-new.md@@ -70,7 +70,7 @@ But eventually things will get complicated, for example when you try to:* Perform a deep crawl of an entire website using a persistent queue of URLs.* Run your scraping code on a list of 100k URLs in a CSV file,- without loosing any data when your code crashes.+ without losing any data when your code crashes.* Rotate proxies to hide your browser origin.* Schedule the code to run periodically and send notification on errors.* Disable browser fingerprinting protections used by websites.@@ -116,7 +116,7 @@ The Apify SDK package provides the following tools:- Represents a queue of URLs to crawl, which is stored either on local filesystem or in cloud.The queue is used for deep crawling of websites, where you start withseveral URLs and then recursively follow links to other pages.- The data structure supports both breath-first and depth-first crawling orders.+ The data structure supports both breadth-first and depth-first crawling orders.</li><li><a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#Dataset">Dataset</a>@@ -164,7 +164,8 @@ You can add Apify SDK to any Node.js project by running:npm install apify```-However, to make the package work at its full potential,+It works right off the bat locally. No configuration needed.+To make the package work with Apify cloud servicesyou'll need to set one or more of the following environment variablesfor your Node.js process, depending on your circumstances:@@ -176,50 +177,19 @@ for your Node.js process, depending on your circumstances:</tr></thead><tbody>- <tr>- <td><code>APIFY_LOCAL_EMULATION_DIR</code></td>- <td>Defines the path to a local directory where key-value stores, request lists and request queues store their data.- If omitted, the package will try to use cloud storage instead and will expect that the- <code>APIFY_TOKEN</code> environment variable is defined.- </td>- </tr><tr><td><code>APIFY_TOKEN</code></td><td>The API token for your Apify account. It is used to access Apify APIs, e.g. to access cloud storage.You can find your API token on the <a href="https://my.apify.com/account#intergrations" target="_blank">Apify - Account - Integrations</a> page.- If omitted, you should define <code>APIFY_LOCAL_EMULATION_DIR</code> environment variable instead.</td></tr><tr><td><code>APIFY_PROXY_PASSWORD</code></td><td>Password to <a href="https://www.apify.com/docs/proxy" target="_blank">Apify Proxy</a> for IP address rotation.If you have have an Apify account, you can find the password on the- <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.</td>- </tr>- <tr>- <td><code>APIFY_DEFAULT_KEY_VALUE_STORE_ID</code></td>- <td>ID of the default key-value store, where the- <code>Apify.getValue()</code> or <code>Apify.setValue()</code> functions store the values.- If you defined <code>APIFY_LOCAL_EMULATION_DIR</code>, then each value is stored as a file at- <code>[APIFY_LOCAL_EMULATION_DIR]/key-value-stores/[APIFY_DEFAULT_KEY_VALUE_STORE_ID]/[KEY].[EXT]</code>,- where <code>[KEY]</code> is the key nad <code>[EXT]</code> corresponds to the MIME content type of the value.- </td>- </tr>- <tr>- <td><code>APIFY_DEFAULT_DATASET_ID</code></td>- <td>ID of the default dataset, where the <code>Apify.pushData()</code> function store the data.- If you defined <code>APIFY_LOCAL_EMULATION_DIR</code>, then dataset items are stored as files at- <code>[APIFY_LOCAL_EMULATION_DIR]/datasets/[APIFY_DEFAULT_DATASET_ID]/[INDEX].json</code>,- where <code>[INDEX]</code> is a zero-based index of the item.- </td>- </tr>- <tr>- <td><code>APIFY_DEFAULT_REQUEST_QUEUE_ID</code></td>- <td>ID of the default request queue, where functions like <code>RequestQueue.addRequest()</code> store the data.- If you defined <code>APIFY_LOCAL_EMULATION_DIR</code>, then request queue records are stored as files at- <code>[APIFY_LOCAL_EMULATION_DIR]/request-queues/[APIFY_DEFAULT_REQUEST_QUEUE_ID]/[NUM].json</code>,- where <code>[NUM]</code> indicates the order of the item in the queue.+ <a href="https://my.apify.com/proxy" target="_blank">Proxy page</a> in the Apify app.+ You may freely use your own proxies, instead of the Apify pool.</td></tr></tbody>@@ -233,7 +203,7 @@ section of the Apify actor documentation.### Local usage with Apify command-line interface (CLI)-To avoid the need to set all the necessary environment variables manually,+To avoid the need to set the necessary environment variables manually,to create a boilerplate of your project,and to enable pushing and running your code on the Apify cloud,you can take advantage of the@@ -260,11 +230,9 @@ cd my-hello-worldapify run```-Note that the CLI automatically sets all necessary environment variables.-The `APIFY_LOCAL_EMULATION_DIR` variable is set to `./apify_local` directory,-where all the data will be stored.+The default local storage folder is set to `./apify_local` directory, where all the data will be stored.For example, the input JSON file for the actor is expected to be in the default key-value store-in `./apify_local/key-value-stores/default/INPUT.json`.+in `./apify_local/key_value_stores/default/INPUT.json`.With the CLI you can also easily deploy your code to Apify cloud by running:@@ -290,27 +258,226 @@ For more information, view the [Apify actors quick start guide](https://www.apif## Examples-TODO: This sections need to be finished+Because examples are often the best way to explain anything, let's look at some of the above+described features put to good use in solving various scraping challenges.All the following examples can be found in the [./examples] directory in the repository.-### Load few pages in raw HTML+Or just run the following command to see the Cheerio Crawler in action.+```+node ./examples/crawler_cheerio+```-TODO: maybe use example from https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler, but make sure it's working+### Load few pages in raw HTML+This is the most basic example of using the Apify SDK. Start with it. It explains some+essential concepts that are used throughout the SDK.+```javascript+// We require the Apify SDK and a popular client to make HTTP requests.+const Apify = require('apify');+const requestPromise = require('request-promise');++// The Apify.main() function wraps the crawler logic and is a mandatory+// part of every crawler run using Apify SDK.+Apify.main(async () => {+ // Prepare a list of URLs to crawl. For that we use an instance of the RequestList class.+ // Here we just throw some URLs into an array of sources, but the RequestList can do much more.+ const requestList = new Apify.RequestList({+ sources: [+ { url: 'http://www.google.com/' },+ { url: 'http://www.example.com/' },+ { url: 'http://www.bing.com/' },+ { url: 'http://www.wikipedia.com/' },+ ],+ });++ // Since initialization of the RequestList is asynchronous, you must always+ // call .initialize() before using it.+ await requestList.initialize();++ // To crawl the URLs, we use an instance of the BasicCrawler class which is our simplest,+ // but still powerful crawler. Its constructor takes an options object where you can+ // configure it to your liking. Here, we're keeping things simple.+ const crawler = new Apify.BasicCrawler({++ // We use the request list created earlier to feed URLs to the crawler.+ requestList,++ // We define a handleRequestFunction that describes the actions+ // we wish to perform for each URL.+ handleRequestFunction: async ({ request }) => {+ // 'request' contains an instance of the Request class which is a container+ // for request related data such as URL or Method (GET, POST ...) and is supplied by the requestList we defined.+ console.log(`Processing ${request.url}...`);++ // Here we simply fetch the HTML of the page and store it to the default Dataset.+ await Apify.pushData({+ url: request.url,+ html: await requestPromise(request.url),+ });+ },+ });++ // Once started the crawler, will automatically work through all the pages in the requestList+ // and the created promise will resolve once the crawl is completed. The collected HTML will be+ // saved in the ./apify_storage/datasets/default folder, unless configured differently.+ await crawler.run();+ console.log('Crawler finished.');+});+```### Crawl a large list of URLs with Cheerio+This example shows how to extract data (the content of title and all h1 tags) from an external+list of URLs (parsed from a CSV file) using CheerioCrawler.-Demonstrates how to create a crawler that will take-a list of URLs from a CSV file and crawls the pages using-<a href="https://www.npmjs.com/package/cheerio" target="_blank">cheerio</a>-HTML parser. The results are stored into a dataset.--TODO+It builds upon the previous BasicCrawler example, so if you missed that one, you should check it out.+```javascript+const Apify = require('apify');++// Utils is a namespace with nice to have things, such as logging control.+const { log } = Apify.utils;+// This is how you can turn internal logging off.+log.setLevel(log.LEVELS.OFF);++// This is just a list of Fortune 500 companies' websites available on GitHub.+const CSV_LINK = 'https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/2d21bb4132b77b38f2992dfaab99649397f238e9/f1000.csv';++Apify.main(async () => {+ // Using the 'requestsFromUrl' parameter instead of 'url' tells the RequestList to download+ // the document available at the given URL and parse URLs out of it.+ const requestList = new Apify.RequestList({+ sources: [{ requestsFromUrl: CSV_LINK }],+ });+ await requestList.initialize();++ // We're using the CheerioCrawler here. Its core difference from the BasicCrawler is the fact+ // that the HTTP request is already handled for you and you get a parsed HTML of the+ // page in the form of the cheerio object - $.+ const crawler = new Apify.CheerioCrawler({+ requestList,++ // We define some boundaries for concurrency. It will be automatically managed.+ // Here we say that no less than 5 and no more than 50 parallel requests should+ // be run. The actual concurrency amount is based on memory and CPU load and is+ // managed by the AutoscaledPool class.+ minConcurrency: 10,+ maxConcurrency: 50,++ // We can also set the amount of retries.+ maxRequestRetries: 1,++ // Or the timeout for each page in seconds.+ handlePageTimeoutSecs: 3,++ // In addition to the BasicCrawler, which only provides access to the request parameter,+ // CheerioCrawler further exposes the '$' parameter, which is the cheerio object containing+ // the parsed page, and the 'html' parameter, which is just the raw HTML.+ // Also, since we're not making the request ourselves, the function is named differently.+ handlePageFunction: async ({ $, html, request }) => {+ console.log(`Processing ${request.url}...`);++ // Extract data with cheerio.+ const title = $('title').text();+ const h1texts = [];+ $('h1').each((index, el) => {+ h1texts.push({+ text: $(el).text(),+ });+ });++ // Save data to default Dataset.+ await Apify.pushData({+ url: request.url,+ title,+ h1texts,+ html,+ });+ },++ // If request failed 1 + maxRequestRetries then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed twice.`);+ },+ });++ await crawler.run();+ console.log('Crawler finished.');+});+```### Recursively crawl a website using headless Chrome / Puppeteer--Demonstrates how to recursively TODO-+This example demonstrates how to use PuppeteerCrawler in connection with the RequestQueue to recursively scrape+the Hacker News site (https://news.ycombinator.com). It starts with a single URL where it finds more links,+enqueues them to the RequestQueue and continues until no more desired links are available.+```javascript+const Apify = require('apify');++Apify.main(async () => {+ // Apify.openRequestQueue() is a factory to get preconfigured RequestQueue instance.+ const requestQueue = await Apify.openRequestQueue();++ // Enqueue only the first URL.+ await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));++ // Create a PuppeteerCrawler. It's configuration is similar to the CheerioCrawler,+ // only instead of the parsed HTML, handlePageFunction gets an instance of the+ // Puppeteer.Page class. See Puppeteer docs for more information.+ const crawler = new Apify.PuppeteerCrawler({+ // Use of requestQueue is similar to RequestList.+ requestQueue,++ // Run Puppeteer headless. If you turn this off, you'll see the scraping+ // browsers showing up on screen. Non-headless mode is great for debugging.+ launchPuppeteerOptions: { headless: true },++ // For each Request in the queue, a new Page is opened in a browser.+ // This is the place to write the Puppeteer scripts you are familiar with,+ // with the exception that browsers and pages are managed for you by Apify SDK automatically.+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);++ // A function to be evaluated by Puppeteer within+ // the browser context.+ const pageFunction = ($posts) => {+ const data = [];++ // We're getting the title, rank and url of each post on Hacker News.+ $posts.forEach(($post) => {+ data.push({+ title: $post.querySelector('.title a').innerText,+ rank: $post.querySelector('.rank').innerText,+ href: $post.querySelector('.title a').href,+ });+ });++ return data;+ };+ const data = await page.$$eval('.athing', pageFunction);++ // Save data to default Dataset.+ await Apify.pushData(data);++ // To continue crawling, we need to enqueue some more pages into+ // the requestQueue. First we find the correct URLs using Puppeteer+ // and then we add the request to the queue.+ try {+ const nextHref = await page.$eval('.morelink', el => el.href);+ // You may omit the Request constructor and just use a plain object.+ await requestQueue.addRequest(new Apify.Request({ url: nextHref }));+ } catch (err) {+ console.log(`Url ${request.url} is the last page!`);+ }+ },++ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`); // Because 3 retries is the default value.+ },+ });++ // Run crawler.+ await crawler.run();+ console.log('Crawler finished.');+});+```### Save page screenshots into key-value storediff --git a/examples/basic_crawler.js b/examples/basic_crawler.jsnew file mode 100644index 000000000..0ec9ac23f--- /dev/null+++ b/examples/basic_crawler.js@@ -0,0 +1,60 @@+/**+ * This is the most basic example of using the Apify SDK. Start with it. It explains some+ * essential concepts that are used throughout the SDK.+ *+ * Example uses:+ * - Apify BasicCrawler to manage requests and autoscale the scraping job.+ * - Apify Dataset to store data.+ * - Apify RequestList to save a list of target URLs.+ */+// We require the Apify SDK and a popular client to make HTTP requests.+const Apify = require('apify');+const requestPromise = require('request-promise');++// The Apify.main() function wraps the crawler logic and is a mandatory+// part of every crawler run using Apify SDK.+Apify.main(async () => {+ // Prepare a list of URLs to crawl. For that we use an instance of the RequestList class.+ // Here we just throw some URLs into an array of sources, but the RequestList can do much more.+ const requestList = new Apify.RequestList({+ sources: [+ { url: 'http://www.google.com/' },+ { url: 'http://www.example.com/' },+ { url: 'http://www.bing.com/' },+ { url: 'http://www.wikipedia.com/' },+ ],+ });++ // Since initialization of the RequestList is asynchronous, you must always+ // call .initialize() before using it.+ await requestList.initialize();++ // To crawl the URLs, we use an instance of the BasicCrawler class which is our simplest,+ // but still powerful crawler. Its constructor takes an options object where you can+ // configure it to your liking. Here, we're keeping things simple.+ const crawler = new Apify.BasicCrawler({++ // We use the request list created earlier to feed URLs to the crawler.+ requestList,++ // We define a handleRequestFunction that describes the actions+ // we wish to perform for each URL.+ handleRequestFunction: async ({ request }) => {+ // 'request' contains an instance of the Request class which is a container+ // for request related data such as URL or Method (GET, POST ...) and is supplied by the requestList we defined.+ console.log(`Processing ${request.url}...`);++ // Here we simply fetch the HTML of the page and store it to the default Dataset.+ await Apify.pushData({+ url: request.url,+ html: await requestPromise(request.url),+ });+ },+ });++ // Once started the crawler, will automatically work through all the pages in the requestList+ // and the created promise will resolve once the crawl is completed. The collected HTML will be+ // saved in the ./apify_storage/datasets/default folder, unless configured differently.+ await crawler.run();+ console.log('Crawler finished.');+});diff --git a/examples/cheerio_crawler.js b/examples/cheerio_crawler.jsnew file mode 100644index 000000000..a639efd0e--- /dev/null+++ b/examples/cheerio_crawler.js@@ -0,0 +1,82 @@+/**+ * This example shows how to extract data (the content of title and all h1 tags) from an external+ * list of URLs (parsed from a CSV file) using CheerioCrawler.+ *+ * It builds upon the previous BasicCrawler example, so if you missed that one, you should check it out.+ *+ * Example uses:+ * - Apify CheerioCrawler to scrape pages using the cheerio NPM package.+ * - Apify Dataset to store data.+ * - Apify RequestList to download a list of URLs from a remote resource.+ */+const Apify = require('apify');++// Utils is a namespace with nice to have things, such as logging control.+const { log } = Apify.utils;+// This is how you can turn internal logging off.+log.setLevel(log.LEVELS.OFF);++// This is just a list of Fortune 500 companies' websites available on GitHub.+const CSV_LINK = 'https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/2d21bb4132b77b38f2992dfaab99649397f238e9/f1000.csv';++Apify.main(async () => {+ // Using the 'requestsFromUrl' parameter instead of 'url' tells the RequestList to download+ // the document available at the given URL and parse URLs out of it.+ const requestList = new Apify.RequestList({+ sources: [{ requestsFromUrl: CSV_LINK }],+ });+ await requestList.initialize();++ // We're using the CheerioCrawler here. Its core difference from the BasicCrawler is the fact+ // that the HTTP request is already handled for you and you get a parsed HTML of the+ // page in the form of the cheerio object - $.+ const crawler = new Apify.CheerioCrawler({+ requestList,++ // We define some boundaries for concurrency. It will be automatically managed.+ // Here we say that no less than 5 and no more than 50 parallel requests should+ // be run. The actual concurrency amount is based on memory and CPU load and is+ // managed by the AutoscaledPool class.+ minConcurrency: 10,+ maxConcurrency: 50,++ // We can also set the amount of retries.+ maxRequestRetries: 1,++ // Or the timeout for each page in seconds.+ handlePageTimeoutSecs: 3,++ // In addition to the BasicCrawler, which only provides access to the request parameter,+ // CheerioCrawler further exposes the '$' parameter, which is the cheerio object containing+ // the parsed page, and the 'html' parameter, which is just the raw HTML.+ // Also, since we're not making the request ourselves, the function is named differently.+ handlePageFunction: async ({ $, html, request }) => {+ console.log(`Processing ${request.url}...`);++ // Extract data with cheerio.+ const title = $('title').text();+ const h1texts = [];+ $('h1').each((index, el) => {+ h1texts.push({+ text: $(el).text(),+ });+ });++ // Save data to default Dataset.+ await Apify.pushData({+ url: request.url,+ title,+ h1texts,+ html,+ });+ },++ // If request failed 1 + maxRequestRetries then this function is executed.+ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed twice.`);+ },+ });++ await crawler.run();+ console.log('Crawler finished.');+});diff --git a/examples/crawler_cheerio.js b/examples/crawler_cheerio.jsdeleted file mode 100644index 7afbbfeaf..000000000--- a/examples/crawler_cheerio.js+++ /dev/null@@ -1,70 +0,0 @@-/**- * This is example how to scrape Hacker News site (https://news.ycombinator.com) using Apify SDK- * with Cheerio and Request NPM packages.- *- * Example uses:- * - Apify BasicCrawler to scrape pages in parallel- * - Apify Dataset to store data- * - Apify RequestQueue to manage dynamic queue of pending and handled requests- * - Request NPM package to request html content of website- * - Cherio NPM package to parse html and extract data- */--const Apify = require('apify');-const rp = require('request-promise');-const cheerio = require('cheerio');--Apify.main(async () => {- // Get queue and enqueue first url.- const requestQueue = await Apify.openRequestQueue();-- // Enqueue Start url.- await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));-- // Create crawler.- const crawler = new Apify.BasicCrawler({- requestQueue,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- handleRequestFunction: async ({ request }) => {- console.log(`Processing ${request.url}...`);-- // Request html of page.- const html = await rp(request.url);-- // Extract data with cheerio.- const data = [];- const $ = cheerio.load(html);- $('.athing').each((index, el) => {- data.push({- title: $(el).find('.title a').text(),- rank: $(el).find('.rank').text(),- href: $(el).find('.title a').attr('href'),- });- });-- // Save data.- await Apify.pushData(data);-- // Enqueue next page.- const $moreLink = $('.morelink');- if ($moreLink.length) {- const path = $moreLink.attr('href')- const url = `https://news.ycombinator.com/${path}`;-- await requestQueue.addRequest(new Apify.Request({ url }));- } else {- console.log(`Url ${request.url} is the last page!`);- }- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);- },- });-- // Run crawler.- await crawler.run();-});diff --git a/examples/crawler_puppeteer.js b/examples/crawler_puppeteer.jsdeleted file mode 100644index 260a6ff63..000000000--- a/examples/crawler_puppeteer.js+++ /dev/null@@ -1,66 +0,0 @@-/**- * This is example how to scrape Hacker News site (https://news.ycombinator.com) using Apify SDK and Puppeteer.- *- * Example uses:- * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel- * - Apify Dataset to store data- * - Apify RequestQueue to manage dynamic queue of pending and handled requests- * - Puppeter to controll headless Chrome browser- */--const Apify = require('apify');--Apify.main(async () => {- // Get queue and enqueue first url.- const requestQueue = await Apify.openRequestQueue();-- // Enqueue Start url.- await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));-- // Create crawler.- const crawler = new Apify.PuppeteerCrawler({- requestQueue,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- // Parameter page is Puppeteers page object with loaded page.- handlePageFunction: async ({ page, request }) => {- console.log(`Processing ${request.url}...`);-- // Extract all posts.- const pageFunction = ($posts) => {- const data = [];-- $posts.forEach(($post) => {- data.push({- title: $post.querySelector('.title a').innerText,- rank: $post.querySelector('.rank').innerText,- href: $post.querySelector('.title a').href,- });- });-- return data;- };- const data = await page.$$eval('.athing', pageFunction);-- // Save data.- await Apify.pushData(data);-- // Enqueue next page.- try {- const nextHref = await page.$eval('.morelink', el => el.href);- await requestQueue.addRequest(new Apify.Request({ url: nextHref }));- } catch (err) {- console.log(`Url ${request.url} is the last page!`);- }- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);- },- });-- // Run crawler.- await crawler.run();-});diff --git a/examples/puppeteer_crawler.js b/examples/puppeteer_crawler.jsnew file mode 100644index 000000000..617f12181--- /dev/null+++ b/examples/puppeteer_crawler.js@@ -0,0 +1,80 @@+/**+ * This example demonstrates how to use PuppeteerCrawler in connection with the RequestQueue to recursively scrape+ * the Hacker News site (https://news.ycombinator.com). It starts with a single URL where it finds more links,+ * enqueues them to the RequestQueue and continues until no more desired links are available.+ *+ * Example uses:+ * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel.+ * - Apify Dataset to store data.+ * - Apify RequestQueue to manage dynamic queue of pending and handled requests.+ * - Puppeter to control headless Chrome browser.+ */++const Apify = require('apify');++Apify.main(async () => {+ // Apify.openRequestQueue() is a factory to get preconfigured RequestQueue instance.+ const requestQueue = await Apify.openRequestQueue();++ // Enqueue only the first URL.+ await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));++ // Create a PuppeteerCrawler. It's configuration is similar to the CheerioCrawler,+ // only instead of the parsed HTML, handlePageFunction gets an instance of the+ // Puppeteer.Page class. See Puppeteer docs for more information.+ const crawler = new Apify.PuppeteerCrawler({+ // Use of requestQueue is similar to RequestList.+ requestQueue,++ // Run Puppeteer headless. If you turn this off, you'll see the scraping+ // browsers showing up on screen. Non-headless mode is great for debugging.+ launchPuppeteerOptions: { headless: true },++ // For each Request in the queue, a new Page is opened in a browser.+ // This is the place to write the Puppeteer scripts you are familiar with,+ // with the exception that browsers and pages are managed for you by Apify SDK automatically.+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);++ // A function to be evaluated by Puppeteer within+ // the browser context.+ const pageFunction = ($posts) => {+ const data = [];++ // We're getting the title, rank and url of each post on Hacker News.+ $posts.forEach(($post) => {+ data.push({+ title: $post.querySelector('.title a').innerText,+ rank: $post.querySelector('.rank').innerText,+ href: $post.querySelector('.title a').href,+ });+ });++ return data;+ };+ const data = await page.$$eval('.athing', pageFunction);++ // Save data to default Dataset.+ await Apify.pushData(data);++ // To continue crawling, we need to enqueue some more pages into+ // the requestQueue. First we find the correct URLs using Puppeteer+ // and then we add the request to the queue.+ try {+ const nextHref = await page.$eval('.morelink', el => el.href);+ // You may omit the Request constructor and just use a plain object.+ await requestQueue.addRequest(new Apify.Request({ url: nextHref }));+ } catch (err) {+ console.log(`Url ${request.url} is the last page!`);+ }+ },++ handleFailedRequestFunction: async ({ request }) => {+ console.log(`Request ${request.url} failed 4 times`); // Because 3 retries is the default value.+ },+ });++ // Run crawler.+ await crawler.run();+ console.log('Crawler finished.');+});diff --git a/examples/url_list_cheerio.js b/examples/url_list_cheerio.jsdeleted file mode 100644index cb5ca107a..000000000--- a/examples/url_list_cheerio.js+++ /dev/null@@ -1,67 +0,0 @@-/**- * This example shows how to extract data (title and "see also" links) form a list of Wikipedia articles- * using Cheerio and Request NPM packages.- *- * Example uses:- * - Apify BasicCrawler to scrape pages using Puppeteer in parallel- * - Apify Dataset to store data- * - Apify RequestList to manage a list of urls to be processed- * - Request NPM package to request html content of website- * - Cherio NPM package to parse html and extract data- */--const Apify = require('apify');-const rp = require('request-promise');-const cheerio = require('cheerio');--Apify.main(async () => {- const sources = [- { url: 'https://en.wikipedia.org/wiki/Amazon_Web_Services' },- { url: 'https://en.wikipedia.org/wiki/Google_Cloud_Platform' },- { url: 'https://en.wikipedia.org/wiki/Microsoft_Azure' },- { url: 'https://en.wikipedia.org/wiki/Rackspace_Cloud' },- ];-- // Create a request list.- const requestList = new Apify.RequestList({ sources });- await requestList.initialize();-- const crawler = new Apify.BasicCrawler({- requestList,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- handleRequestFunction: async ({ request }) => {- console.log(`Processing ${request.url}...`);-- // Request html of page.- const html = await rp(request.url);-- // Extract data with cheerio.- const $ = cheerio.load(html);- const $seeAlsoElement = $('#See_also').parent().next();- const seeAlsoLinks = [];- $seeAlsoElement.find('a').each((index, el) => {- seeAlsoLinks.push({- url: $(el).attr('href'),- text: $(el).text(),- });- });-- // Save data.- await Apify.pushData({- url: request.url,- title: $('h1').text(),- seeAlsoLinks,- });- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);- },- });-- // Run crawler for request list.- await crawler.run();-});