Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
Finalize example revamp
Ondra Urbanab0c06a8389b
diff --git a/README-new.md b/README-new.mdindex 6370fb709..d751780ed 100644--- a/README-new.md+++ b/README-new.md@@ -268,7 +268,7 @@ Or just run the following command to see the Cheerio Crawler in action.node ./examples/crawler_cheerio```-### Load few pages in raw HTML+### 1 - Load few pages in raw HTMLThis is the most basic example of using the Apify SDK. Start with it. It explains someessential concepts that are used throughout the SDK.```javascript@@ -325,7 +325,7 @@ Apify.main(async () => {});```-### Crawl a large list of URLs with Cheerio+### 2 - Crawl a large list of URLs with CheerioThis example shows how to extract data (the content of title and all h1 tags) from an externallist of URLs (parsed from a CSV file) using CheerioCrawler.@@ -404,7 +404,7 @@ Apify.main(async () => {});```-### Recursively crawl a website using headless Chrome / Puppeteer+### 3 - Recursively crawl a website using headless Chrome / PuppeteerThis example demonstrates how to use PuppeteerCrawler in connection with the RequestQueue to recursively scrapethe Hacker News site (https://news.ycombinator.com). It starts with a single URL where it finds more links,enqueues them to the RequestQueue and continues until no more desired links are available.@@ -479,25 +479,178 @@ Apify.main(async () => {});```-### Save page screenshots into key-value store+### 4 - Save page screenshots into KeyValueStore+This example shows how to work with KeyValueStore. It crawls a list of URLs using Puppeteer,+capture a screenshot of each page and saves it to the KeyValueStore. The list of URLs is+provided as INPUT, which is a standard way of passing initial configuration to Apify actors.+Locally, INPUT needs to be placed in the KeyValueStore. On the platform, it can either be set+using the applications UI or passed as the body of the Run Actor API call.-TODO+For more information on RequestList, see example 1. For PuppeteerCrawler, see example 3.+```javascript+const Apify = require('apify');-### Run Puppeteer with Apify Proxy+Apify.main(async () => {+ // Apify.getValue() is a shorthand to read the value of the provided key (INPUT) from the default KeyValueStore.+ // To read the INPUT on your local machine, you first need to create it.+ // Place an INPUT.json file with the desired input into the+ // ./apify_storage/key_value_stores/default directory (unless configured otherwise).+ // Example input: { "sources": [{ "url": "https://www.google.com" }, { "url": "https://www.duckduckgo.com" }] }+ const { sources } = await Apify.getValue('INPUT');++ const requestList = new Apify.RequestList({ sources });+ await requestList.initialize();-TODO+ const crawler = new Apify.PuppeteerCrawler({+ requestList,+ launchPuppeteerOptions: { headless: true },+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);+ // This is a Puppeteer function that takes a screenshot of the Page and returns its buffer.+ const screenshotBuffer = await page.screenshot();-### Invoke another actor+ // uniqueKey is a normalized URL of the request,+ // but KeyValueStore keys may only include [a-zA-Z0-9!-_.'()] characters.+ const key = request.uniqueKey.replace(/[:/]/g, '_');-This example demonstrates how to call another actor on Apify cloud - in this case `apify/send-mail`-to send an email.+ // Here we save the screenshot. Choosing the right content type will automatically+ // assign the local file the right extension. In this case: .png+ await Apify.setValue(key, screenshotBuffer, { contentType: 'image/png' });+ console.log('Screenshot saved.');+ },+ });-TODO+ // Run crawler.+ await crawler.run();+ console.log('Crawler finished.');+});+```+### 5 - Run Puppeteer with Apify Proxy+This example demonstrates the use of Apify features with Puppeteer.+We'll show you how to use Apify Proxy without using our Crawlers+and instead using only Puppeteer itself.+```javascript+const Apify = require('apify');+Apify.main(async () => {+ // Apify enhances not only our Crawlers, but also plain Puppeteer+ // with useful tools such as automatic proxy use (from our own pool).+ // To use the Proxy, you need to either log in using the Apify CLI, or set+ // the APIFY_PROXY_PASSWORD environment variable. You will find the password+ // under your account in the Apify Platform.+ // Other options such as LiveView may also be enabled for Puppeteer.+ const options = {+ useApifyProxy: true,+ headless: true,+ };++ // Apify.launchPuppeteer() is a shortcut to get a preconfigured Puppeteer.Browser+ // instance with extra features provided by Apify. All original Puppeteer options+ // are passed directly to Puppeteer.+ const browser = await Apify.launchPuppeteer(options);++ console.log('Running Puppeteer...');+ // Proceed with a plain Puppeteer script.+ const page = await browser.newPage();+ const url = 'https://en.wikipedia.org/wiki/Main_Page';+ await page.goto(url);+ const html = await page.content();++ // Use any Apify feature.+ await Apify.pushData({ url, html });++ // Cleaning up resources is a good practice.+ await browser.close();+ console.log('Puppeteer closed.');+});+```+### 6 - Invoke another actor+This example shows how to call another actor - in this case apify/send-mail to send+an email.++For this demonstration, we've chosen to scrape BTC prices. If you don't want to miss the chance of+of your life then you can use this code to get current BTC prices from Kraken.com+and mail them to your mailbox.++If you deploy this actor to Apify platform then you can setup a scheduler for early+morning.+```javascript+const Apify = require('apify');++const YOUR_MAIL = '[email protected]';++Apify.main(async () => {+ // Start browser.+ const browser = await Apify.launchPuppeteer({ headless: true });++ // Load Kraken and get last traded price of BTC.+ const page = await browser.newPage();+ await page.goto('https://www.kraken.com/charts');+ const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);++ console.log('Calling another actor. This may take a few seconds...');+ // Send prices to your email. For that, you can use an actor we already+ // have available on the platform under the name: apify/send-mail.+ // The second parameter to the Apify.call() invocation is the actor's+ // desired input. You can find the required input parameters by checking+ // the actor's documentation page: https://www.apify.com/apify/send-mail+ await Apify.call('apify/send-mail', {+ to: YOUR_MAIL,+ subject: 'Kraken.com BTC',+ html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,+ });++ console.log('Actor successfully called. Go check your email.');+});+```++### 7 - Run actor as an API+This example shows shows an actor that has short runtime - just few seconds. It opens a webpage+http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate+bridge, takes a screenshot and saves it as output. This makes actor executable on Apify platform+synchronously with a single request that also returns its output.++Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam+so you can easily run it with request to+https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]+```javascript+const Apify = require('apify');++Apify.main(async () => {+ // Start browser.+ const browser = await Apify.launchPuppeteer({ headless: true });++ // Load http://goldengatebridge75.org/news/webcam.html and get an iframe+ // containing webcam stream.+ console.log('Opening page.');+ const page = await browser.newPage();+ await page.goto('http://goldengatebridge75.org/news/webcam.html');+ const iframe = (await page.frames()).pop();++ // Get webcam image element handle.+ const imageElementHandle = await iframe.$('.VideoColm img');++ // Give the webcam image some time to load.+ console.log('Waiting for some time...');+ await Apify.utils.sleep(3000);++ // Get a screenshot of that image.+ const imageBuffer = await imageElementHandle.screenshot();+ console.log('Screenshot captured.');++ // Save it as an OUTPUT. Just as INPUT, OUTPUT has a special meaning.+ // Anything you save as an OUTPUT to KeyValueStore will be sent to you+ // as an API response once the actor finishes its run, if you use the+ // run-sync API. This way, you can really Apify any website.+ await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });+ console.log('Actor finished.');+});+```+## Puppeteer live viewApify SDK enables real-time view of launched Puppeteer browser instances and their open tabs,diff --git a/examples/basic_crawler.js b/examples/1_basic_crawler.jssimilarity index 92%rename from examples/basic_crawler.jsrename to examples/1_basic_crawler.jsindex 0ec9ac23f..4484065eb 100644--- a/examples/basic_crawler.js+++ b/examples/1_basic_crawler.js@@ -18,10 +18,10 @@ Apify.main(async () => {// Here we just throw some URLs into an array of sources, but the RequestList can do much more.const requestList = new Apify.RequestList({sources: [- { url: 'http://www.google.com/' },- { url: 'http://www.example.com/' },- { url: 'http://www.bing.com/' },- { url: 'http://www.wikipedia.com/' },+ { url: 'https://www.google.com/' },+ { url: 'https://www.example.com/' },+ { url: 'https://www.bing.com/' },+ { url: 'https://www.wikipedia.org/' },],});diff --git a/examples/cheerio_crawler.js b/examples/2_cheerio_crawler.jssimilarity index 100%rename from examples/cheerio_crawler.jsrename to examples/2_cheerio_crawler.jsdiff --git a/examples/puppeteer_crawler.js b/examples/3_puppeteer_crawler.jssimilarity index 98%rename from examples/puppeteer_crawler.jsrename to examples/3_puppeteer_crawler.jsindex 617f12181..8cdfe8d69 100644--- a/examples/puppeteer_crawler.js+++ b/examples/3_puppeteer_crawler.js@@ -7,7 +7,7 @@* - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel.* - Apify Dataset to store data.* - Apify RequestQueue to manage dynamic queue of pending and handled requests.- * - Puppeter to control headless Chrome browser.+ * - Puppeteer to control headless Chrome browser.*/const Apify = require('apify');diff --git a/examples/4_save_screenshots.js b/examples/4_save_screenshots.jsnew file mode 100644index 000000000..85ec29472--- /dev/null+++ b/examples/4_save_screenshots.js@@ -0,0 +1,53 @@+/**+ * This example shows how to work with KeyValueStore. It crawls a list of URLs using Puppeteer,+ * capture a screenshot of each page and saves it to the KeyValueStore. The list of URLs is+ * provided as INPUT, which is a standard way of passing initial configuration to Apify actors.+ * Locally, INPUT needs to be placed in the KeyValueStore. On the platform, it can either be set+ * using the applications UI or passed as the body of the Run Actor API call.+ *+ * For more information on RequestList, see example 1. For PuppeteerCrawler, see example 3.+ *+ * Example uses:+ * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel.+ * - Apify KeyValueStore to read INPUT and store screenshots.+ * - Apify RequestList to save a list of target URLs.+ * - Puppeteer to control headless Chrome browser.+ */++const Apify = require('apify');++Apify.main(async () => {+ // Apify.getValue() is a shorthand to read the value of the provided key (INPUT) from the default KeyValueStore.+ // To read the INPUT on your local machine, you first need to create it.+ // Place an INPUT.json file with the desired input into the+ // ./apify_storage/key_value_stores/default directory (unless configured otherwise).+ // Example input: { "sources": [{ "url": "https://www.google.com" }, { "url": "https://www.duckduckgo.com" }] }+ const { sources } = await Apify.getValue('INPUT');++ const requestList = new Apify.RequestList({ sources });+ await requestList.initialize();++ const crawler = new Apify.PuppeteerCrawler({+ requestList,+ launchPuppeteerOptions: { headless: true },+ handlePageFunction: async ({ page, request }) => {+ console.log(`Processing ${request.url}...`);++ // This is a Puppeteer function that takes a screenshot of the Page and returns its buffer.+ const screenshotBuffer = await page.screenshot();++ // uniqueKey is a normalized URL of the request,+ // but KeyValueStore keys may only include [a-zA-Z0-9!-_.'()] characters.+ const key = request.uniqueKey.replace(/[:/]/g, '_');++ // Here we save the screenshot. Choosing the right content type will automatically+ // assign the local file the right extension. In this case: .png+ await Apify.setValue(key, screenshotBuffer, { contentType: 'image/png' });+ console.log('Screenshot saved.');+ },+ });++ // Run crawler.+ await crawler.run();+ console.log('Crawler finished.');+});diff --git a/examples/5_puppeteer_proxy.js b/examples/5_puppeteer_proxy.jsnew file mode 100644index 000000000..5c2c1fa19--- /dev/null+++ b/examples/5_puppeteer_proxy.js@@ -0,0 +1,44 @@+/**+ * This example demonstrates the use of Apify features with Puppeteer.+ * We'll show you how to use Apify Proxy without using our Crawlers+ * and instead using only Puppeteer itself.+ *+ * Example uses:+ * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.+ * - Puppeteer to control headless Chrome browser.+ * - Apify Dataset to store data.+ */++const Apify = require('apify');++Apify.main(async () => {+ // Apify enhances not only our Crawlers, but also plain Puppeteer+ // with useful tools such as automatic proxy use (from our own pool).+ // To use the Proxy, you need to either log in using the Apify CLI, or set+ // the APIFY_PROXY_PASSWORD environment variable. You will find the password+ // under your account in the Apify Platform.+ // Other options such as LiveView may also be enabled for Puppeteer.+ const options = {+ useApifyProxy: true,+ headless: true,+ };++ // Apify.launchPuppeteer() is a shortcut to get a preconfigured Puppeteer.Browser+ // instance with extra features provided by Apify. All original Puppeteer options+ // are passed directly to Puppeteer.+ const browser = await Apify.launchPuppeteer(options);++ console.log('Running Puppeteer...');+ // Proceed with a plain Puppeteer script.+ const page = await browser.newPage();+ const url = 'https://en.wikipedia.org/wiki/Main_Page';+ await page.goto(url);+ const html = await page.content();++ // Use any Apify feature.+ await Apify.pushData({ url, html });++ // Cleaning up resources is a good practice.+ await browser.close();+ console.log('Puppeteer closed.');+});diff --git a/examples/6_call_another_act.js b/examples/6_call_another_act.jsnew file mode 100644index 000000000..98be38685--- /dev/null+++ b/examples/6_call_another_act.js@@ -0,0 +1,44 @@+/**+ * This example shows how to call another actor - in this case apify/send-mail to send+ * an email.+ *+ * For this demonstration, we've chosen to scrape BTC prices. If you don't want to miss the chance of+ * of your life then you can use this code to get current BTC prices from Kraken.com+ * and mail them to your mailbox.+ *+ * If you deploy this actor to Apify platform then you can setup a scheduler for early+ * morning.+ *+ * Example uses:+ * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.+ * - Puppeteer to control headless Chrome browser.+ * - Apify.call() to invoke another actor with a provided input.+ */++const Apify = require('apify');++const YOUR_MAIL = '[email protected]';++Apify.main(async () => {+ // Start browser.+ const browser = await Apify.launchPuppeteer({ headless: true });++ // Load Kraken and get last traded price of BTC.+ const page = await browser.newPage();+ await page.goto('https://www.kraken.com/charts');+ const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);++ console.log('Calling another actor. This may take a few seconds...');+ // Send prices to your email. For that, you can use an actor we already+ // have available on the platform under the name: apify/send-mail.+ // The second parameter to the Apify.call() invocation is the actor's+ // desired input. You can find the required input parameters by checking+ // the actor's documentation page: https://www.apify.com/apify/send-mail+ await Apify.call('apify/send-mail', {+ to: YOUR_MAIL,+ subject: 'Kraken.com BTC',+ html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,+ });++ console.log('Actor successfully called. Go check your email.');+});diff --git a/examples/synchronous_run.js b/examples/7_synchronous_run.jssimilarity index 60%rename from examples/synchronous_run.jsrename to examples/7_synchronous_run.jsindex 2eeeba548..9adce457a 100644--- a/examples/synchronous_run.js+++ b/examples/7_synchronous_run.js@@ -7,16 +7,22 @@* Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam* so you can easily run it with request to* https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]+ *+ * Example uses:+ * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.+ * - Puppeteer to control headless Chrome browser.+ * - Apify KeyValueStore to store data and provide an API response.*/const Apify = require('apify');Apify.main(async () => {// Start browser.- const browser = await Apify.launchPuppeteer();+ const browser = await Apify.launchPuppeteer({ headless: true });// Load http://goldengatebridge75.org/news/webcam.html and get an iframe// containing webcam stream.+ console.log('Opening page.');const page = await browser.newPage();await page.goto('http://goldengatebridge75.org/news/webcam.html');const iframe = (await page.frames()).pop();@@ -24,9 +30,18 @@ Apify.main(async () => {// Get webcam image element handle.const imageElementHandle = await iframe.$('.VideoColm img');+ // Give the webcam image some time to load.+ console.log('Waiting for some time...');+ await Apify.utils.sleep(3000);+// Get a screenshot of that image.const imageBuffer = await imageElementHandle.screenshot();+ console.log('Screenshot captured.');- // Save it as an output.+ // Save it as an OUTPUT. Just as INPUT, OUTPUT has a special meaning.+ // Anything you save as an OUTPUT to KeyValueStore will be sent to you+ // as an API response once the actor finishes its run, if you use the+ // run-sync API. This way, you can really Apify any website.await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });+ console.log('Actor finished.');});diff --git a/examples/autoscaled_pool.js b/examples/autoscaled_pool.jsdeleted file mode 100644index 421ee69a4..000000000--- a/examples/autoscaled_pool.js+++ /dev/null@@ -1,32 +0,0 @@-const Apify = require('apify');--Apify.main(async () => {- let counter = 0;-- // Worker function gets executed again and again in the pool.- // Size of the pool is auto-scaled based on memory and CPU.- // If this function returns null then all tasks are considered- // to be done.- const runTaskFunction = () => {- const current = counter++;-- if (current >= 1000) return null;-- const randomWaitMillis = Math.round(Math.random() * 100);-- return new Promise((resolve) => {- setTimeout(() => {- console.log(`${counter} is done!`);- resolve();- }, randomWaitMillis);- });- };-- const pool = new Apify.AutoscaledPool({- minConcurrency: 3,- maxConcurrency: 50,- runTaskFunction,- });-- await pool.run();-});diff --git a/examples/call_another_act.js b/examples/call_another_act.jsdeleted file mode 100644index e8218a61c..000000000--- a/examples/call_another_act.js+++ /dev/null@@ -1,32 +0,0 @@-/**- * This example shows how to call another actor - in this case apify/send-main to send- * email.- *- * BTC is falling down for few months already. If you don't want to miss the chance of- * of your life then you can use this code to get current BTC prices from Kraken.com- * and mail them to your mailbox.- *- * If you deploy this actor to Apify platform then you can setup a scheduler for early- * morning.- */--const Apify = require('apify');--const YOUR_MAIL = '[email protected]';--Apify.main(async () => {- // Start browser.- const browser = await Apify.launchPuppeteer();-- // Load Kraken and get last traded price of BTC.- const page = await browser.newPage();- await page.goto('https://www.kraken.com/charts');- const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);-- // Send prices to your email.- await Apify.call('apify/send-mail', {- to: YOUR_MAIL,- subject: 'Kraken.com BTC',- html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,- });-});diff --git a/examples/url_list_puppeteer.js b/examples/url_list_puppeteer.jsdeleted file mode 100644index b52b71650..000000000--- a/examples/url_list_puppeteer.js+++ /dev/null@@ -1,68 +0,0 @@-/**- * This example shows how to extract data (title and "see also" links) form a list of Wikipedia articles using Puppeteer.- *- * Example uses:- * - Apify BasicCrawler to scrape pages in parallel- * - Apify Dataset to store data- * - Apify RequestList to manage a list of urls to be processed- * - Puppeter to controll headless Chrome browser- */--const Apify = require('apify');-const rp = require('request-promise');-const cheerio = require('cheerio');--Apify.main(async () => {- const sources = [- { url: 'https://en.wikipedia.org/wiki/Amazon_Web_Services' },- { url: 'https://en.wikipedia.org/wiki/Google_Cloud_Platform' },- { url: 'https://en.wikipedia.org/wiki/Microsoft_Azure' },- { url: 'https://en.wikipedia.org/wiki/Rackspace_Cloud' },- ];-- // Create a request list.- const requestList = new Apify.RequestList({ sources });- await requestList.initialize();-- const crawler = new Apify.PuppeteerCrawler({- requestList,-- // This page is executed for each request.- // If request failes then it's retried 3 times.- // Parameter page is Puppeteers page object with loaded page.- handlePageFunction: async ({ request, page }) => {- console.log(`Processing ${request.url}...`);-- // Extract data with Puppeteer.- const seeAlsoLinks = await page.$eval('#See_also', (seeAlsoTitleEl) => {- const seeAlsoListEl = seeAlsoTitleEl.parentNode.nextSibling.nextSibling;- const seeAlsoLinksEls = seeAlsoListEl.querySelectorAll('a');- const data = [];-- seeAlsoLinksEls.forEach((linkEl) => {- data.push({- title: linkEl.innerText,- url: linkEl.getAttribute('href'),- });- });-- return data;- });-- // Save data.- await Apify.pushData({- url: request.url,- title: await page.$eval('h1', el => el.innerText),- seeAlsoLinks,- });- },-- // If request failed 4 times then this function is executed.- handleFailedRequestFunction: async ({ request }) => {- console.log(`Request ${request.url} failed 4 times`);- },- });-- // Run crawler for request list.- await crawler.run();-});