Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Updated README and /examples

Jan Curn4b6b6d0938bd
diff --git a/README.md b/README.md
index c27d48920..c8e28b895 100644
--- a/README.md
+++ b/README.md
@@ -305,7 +305,7 @@ To run the examples, just copy them into the directory where you installed Apify
`npm install apify` and then run them by calling:
```
-node APIFY_LOCAL_STORAGE_DIR=./apify_storage 1_basic_crawler.js
+node APIFY_LOCAL_STORAGE_DIR=./apify_storage basic_crawler.js
```
Note that it is necessary to set either the `APIFY_LOCAL_STORAGE_DIR` or `APIFY_TOKEN` environment variables in order
@@ -319,7 +319,7 @@ file created by the CLI. Then go to the project directory and run the example us
apify run
```
-### Load a few pages in raw HTML
+### Crawl several pages in raw HTML
This is the most basic example of Apify SDK, which demonstrates some of its
elementary tools such as the
@@ -595,7 +595,6 @@ Apify.main(async () => {
const crawler = new Apify.PuppeteerCrawler({
requestList,
- launchPuppeteerOptions: { headless: true },
handlePageFunction: async ({ page, request }) => {
console.log(`Processing ${request.url}...`);
@@ -639,7 +638,6 @@ Apify.main(async () => {
// Moreover, it accepts several additional options, such as useApifyProxy.
const options = {
useApifyProxy: true,
- headless: true,
};
const browser = await Apify.launchPuppeteer(options);
@@ -679,7 +677,7 @@ const Apify = require('apify');
Apify.main(async () => {
// Launch the web browser.
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer();
console.log('Obtaining email address...');
const user = await Apify.client.users.getUser();
@@ -725,7 +723,7 @@ const Apify = require('apify');
Apify.main(async () => {
// Launch web browser.
- const browser = await Apify.launchPuppeteer({ headless: true });
+ const browser = await Apify.launchPuppeteer();
// Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam stream
console.log('Opening web page...');
@@ -822,7 +820,7 @@ await store.delete('some-key');
```
To see a real-world example of how to get the input from the key-value store, see the
-[4_save_screenshots.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/4_save_screenshots.js) example.
+[screenshots.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/screenshots.js) example.
### Dataset
@@ -872,7 +870,7 @@ await dataset.pushData([
```
To see how to use the dataset to store crawler results, see the
-[2_cheerio_crawler.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/2_cheerio_crawler.js) example.
+[cheerio_crawler.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/cheerio_crawler.js) example.
### Request queue
@@ -927,7 +925,7 @@ await queue.reclaimRequest(request2);
```
To see how to use the request queue with a crawler, see the
-[3_puppeteer_crawler.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/3_puppeteer_crawler.js) example.
+[puppeteer_crawler.js](https://github.com/apifytech/apify-js/blob/feature/better-readme/examples/puppeteer_crawler.js) example.
## Puppeteer live view
diff --git a/examples/1_basic_crawler.js b/examples/1_basic_crawler.js
deleted file mode 100644
index 4484065eb..000000000
--- a/examples/1_basic_crawler.js
+++ /dev/null
@@ -1,60 +0,0 @@
-/**
- * This is the most basic example of using the Apify SDK. Start with it. It explains some
- * essential concepts that are used throughout the SDK.
- *
- * Example uses:
- * - Apify BasicCrawler to manage requests and autoscale the scraping job.
- * - Apify Dataset to store data.
- * - Apify RequestList to save a list of target URLs.
- */
-// We require the Apify SDK and a popular client to make HTTP requests.
-const Apify = require('apify');
-const requestPromise = require('request-promise');
-
-// The Apify.main() function wraps the crawler logic and is a mandatory
-// part of every crawler run using Apify SDK.
-Apify.main(async () => {
- // Prepare a list of URLs to crawl. For that we use an instance of the RequestList class.
- // Here we just throw some URLs into an array of sources, but the RequestList can do much more.
- const requestList = new Apify.RequestList({
- sources: [
- { url: 'https://www.google.com/' },
- { url: 'https://www.example.com/' },
- { url: 'https://www.bing.com/' },
- { url: 'https://www.wikipedia.org/' },
- ],
- });
-
- // Since initialization of the RequestList is asynchronous, you must always
- // call .initialize() before using it.
- await requestList.initialize();
-
- // To crawl the URLs, we use an instance of the BasicCrawler class which is our simplest,
- // but still powerful crawler. Its constructor takes an options object where you can
- // configure it to your liking. Here, we're keeping things simple.
- const crawler = new Apify.BasicCrawler({
-
- // We use the request list created earlier to feed URLs to the crawler.
- requestList,
-
- // We define a handleRequestFunction that describes the actions
- // we wish to perform for each URL.
- handleRequestFunction: async ({ request }) => {
- // 'request' contains an instance of the Request class which is a container
- // for request related data such as URL or Method (GET, POST ...) and is supplied by the requestList we defined.
- console.log(`Processing ${request.url}...`);
-
- // Here we simply fetch the HTML of the page and store it to the default Dataset.
- await Apify.pushData({
- url: request.url,
- html: await requestPromise(request.url),
- });
- },
- });
-
- // Once started the crawler, will automatically work through all the pages in the requestList
- // and the created promise will resolve once the crawl is completed. The collected HTML will be
- // saved in the ./apify_storage/datasets/default folder, unless configured differently.
- await crawler.run();
- console.log('Crawler finished.');
-});
diff --git a/examples/2_cheerio_crawler.js b/examples/2_cheerio_crawler.js
deleted file mode 100644
index a639efd0e..000000000
--- a/examples/2_cheerio_crawler.js
+++ /dev/null
@@ -1,82 +0,0 @@
-/**
- * This example shows how to extract data (the content of title and all h1 tags) from an external
- * list of URLs (parsed from a CSV file) using CheerioCrawler.
- *
- * It builds upon the previous BasicCrawler example, so if you missed that one, you should check it out.
- *
- * Example uses:
- * - Apify CheerioCrawler to scrape pages using the cheerio NPM package.
- * - Apify Dataset to store data.
- * - Apify RequestList to download a list of URLs from a remote resource.
- */
-const Apify = require('apify');
-
-// Utils is a namespace with nice to have things, such as logging control.
-const { log } = Apify.utils;
-// This is how you can turn internal logging off.
-log.setLevel(log.LEVELS.OFF);
-
-// This is just a list of Fortune 500 companies' websites available on GitHub.
-const CSV_LINK = 'https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/2d21bb4132b77b38f2992dfaab99649397f238e9/f1000.csv';
-
-Apify.main(async () => {
- // Using the 'requestsFromUrl' parameter instead of 'url' tells the RequestList to download
- // the document available at the given URL and parse URLs out of it.
- const requestList = new Apify.RequestList({
- sources: [{ requestsFromUrl: CSV_LINK }],
- });
- await requestList.initialize();
-
- // We're using the CheerioCrawler here. Its core difference from the BasicCrawler is the fact
- // that the HTTP request is already handled for you and you get a parsed HTML of the
- // page in the form of the cheerio object - $.
- const crawler = new Apify.CheerioCrawler({
- requestList,
-
- // We define some boundaries for concurrency. It will be automatically managed.
- // Here we say that no less than 5 and no more than 50 parallel requests should
- // be run. The actual concurrency amount is based on memory and CPU load and is
- // managed by the AutoscaledPool class.
- minConcurrency: 10,
- maxConcurrency: 50,
-
- // We can also set the amount of retries.
- maxRequestRetries: 1,
-
- // Or the timeout for each page in seconds.
- handlePageTimeoutSecs: 3,
-
- // In addition to the BasicCrawler, which only provides access to the request parameter,
- // CheerioCrawler further exposes the '$' parameter, which is the cheerio object containing
- // the parsed page, and the 'html' parameter, which is just the raw HTML.
- // Also, since we're not making the request ourselves, the function is named differently.
- handlePageFunction: async ({ $, html, request }) => {
- console.log(`Processing ${request.url}...`);
-
- // Extract data with cheerio.
- const title = $('title').text();
- const h1texts = [];
- $('h1').each((index, el) => {
- h1texts.push({
- text: $(el).text(),
- });
- });
-
- // Save data to default Dataset.
- await Apify.pushData({
- url: request.url,
- title,
- h1texts,
- html,
- });
- },
-
- // If request failed 1 + maxRequestRetries then this function is executed.
- handleFailedRequestFunction: async ({ request }) => {
- console.log(`Request ${request.url} failed twice.`);
- },
- });
-
- await crawler.run();
- console.log('Crawler finished.');
-});
diff --git a/examples/3_puppeteer_crawler.js b/examples/3_puppeteer_crawler.js
deleted file mode 100644
index 8cdfe8d69..000000000
--- a/examples/3_puppeteer_crawler.js
+++ /dev/null
@@ -1,80 +0,0 @@
-/**
- * This example demonstrates how to use PuppeteerCrawler in connection with the RequestQueue to recursively scrape
- * the Hacker News site (https://news.ycombinator.com). It starts with a single URL where it finds more links,
- * enqueues them to the RequestQueue and continues until no more desired links are available.
- *
- * Example uses:
- * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel.
- * - Apify Dataset to store data.
- * - Apify RequestQueue to manage dynamic queue of pending and handled requests.
- * - Puppeteer to control headless Chrome browser.
- */
-
-const Apify = require('apify');
-
-Apify.main(async () => {
- // Apify.openRequestQueue() is a factory to get preconfigured RequestQueue instance.
- const requestQueue = await Apify.openRequestQueue();
-
- // Enqueue only the first URL.
- await requestQueue.addRequest(new Apify.Request({ url: 'https://news.ycombinator.com/' }));
-
- // Create a PuppeteerCrawler. It's configuration is similar to the CheerioCrawler,
- // only instead of the parsed HTML, handlePageFunction gets an instance of the
- // Puppeteer.Page class. See Puppeteer docs for more information.
- const crawler = new Apify.PuppeteerCrawler({
- // Use of requestQueue is similar to RequestList.
- requestQueue,
-
- // Run Puppeteer headless. If you turn this off, you'll see the scraping
- // browsers showing up on screen. Non-headless mode is great for debugging.
- launchPuppeteerOptions: { headless: true },
-
- // For each Request in the queue, a new Page is opened in a browser.
- // This is the place to write the Puppeteer scripts you are familiar with,
- // with the exception that browsers and pages are managed for you by Apify SDK automatically.
- handlePageFunction: async ({ page, request }) => {
- console.log(`Processing ${request.url}...`);
-
- // A function to be evaluated by Puppeteer within
- // the browser context.
- const pageFunction = ($posts) => {
- const data = [];
-
- // We're getting the title, rank and url of each post on Hacker News.
- $posts.forEach(($post) => {
- data.push({
- title: $post.querySelector('.title a').innerText,
- rank: $post.querySelector('.rank').innerText,
- href: $post.querySelector('.title a').href,
- });
- });
-
- return data;
- };
- const data = await page.$$eval('.athing', pageFunction);
-
- // Save data to default Dataset.
- await Apify.pushData(data);
-
- // To continue crawling, we need to enqueue some more pages into
- // the requestQueue. First we find the correct URLs using Puppeteer
- // and then we add the request to the queue.
- try {
- const nextHref = await page.$eval('.morelink', el => el.href);
- // You may omit the Request constructor and just use a plain object.
- await requestQueue.addRequest(new Apify.Request({ url: nextHref }));
- } catch (err) {
- console.log(`Url ${request.url} is the last page!`);
- }
- },
-
- handleFailedRequestFunction: async ({ request }) => {
- console.log(`Request ${request.url} failed 4 times`); // Because 3 retries is the default value.
- },
- });
-
- // Run crawler.
- await crawler.run();
- console.log('Crawler finished.');
-});
diff --git a/examples/4_save_screenshots.js b/examples/4_save_screenshots.js
deleted file mode 100644
index 85ec29472..000000000
--- a/examples/4_save_screenshots.js
+++ /dev/null
@@ -1,53 +0,0 @@
-/**
- * This example shows how to work with KeyValueStore. It crawls a list of URLs using Puppeteer,
- * capture a screenshot of each page and saves it to the KeyValueStore. The list of URLs is
- * provided as INPUT, which is a standard way of passing initial configuration to Apify actors.
- * Locally, INPUT needs to be placed in the KeyValueStore. On the platform, it can either be set
- * using the applications UI or passed as the body of the Run Actor API call.
- *
- * For more information on RequestList, see example 1. For PuppeteerCrawler, see example 3.
- *
- * Example uses:
- * - Apify PuppeteerCrawler to scrape pages using Puppeteer in parallel.
- * - Apify KeyValueStore to read INPUT and store screenshots.
- * - Apify RequestList to save a list of target URLs.
- * - Puppeteer to control headless Chrome browser.
- */
-
-const Apify = require('apify');
-
-Apify.main(async () => {
- // Apify.getValue() is a shorthand to read the value of the provided key (INPUT) from the default KeyValueStore.
- // To read the INPUT on your local machine, you first need to create it.
- // Place an INPUT.json file with the desired input into the
- // ./apify_storage/key_value_stores/default directory (unless configured otherwise).
- // Example input: { "sources": [{ "url": "https://www.google.com" }, { "url": "https://www.duckduckgo.com" }] }
- const { sources } = await Apify.getValue('INPUT');
-
- const requestList = new Apify.RequestList({ sources });
- await requestList.initialize();
-
- const crawler = new Apify.PuppeteerCrawler({
- requestList,
- launchPuppeteerOptions: { headless: true },
- handlePageFunction: async ({ page, request }) => {
- console.log(`Processing ${request.url}...`);
-
- // This is a Puppeteer function that takes a screenshot of the Page and returns its buffer.
- const screenshotBuffer = await page.screenshot();
-
- // uniqueKey is a normalized URL of the request,
- // but KeyValueStore keys may only include [a-zA-Z0-9!-_.'()] characters.
- const key = request.uniqueKey.replace(/[:/]/g, '_');
-
- // Here we save the screenshot. Choosing the right content type will automatically
- // assign the local file the right extension. In this case: .png
- await Apify.setValue(key, screenshotBuffer, { contentType: 'image/png' });
- console.log('Screenshot saved.');
- },
- });
-
- // Run crawler.
- await crawler.run();
- console.log('Crawler finished.');
-});
diff --git a/examples/5_puppeteer_proxy.js b/examples/5_puppeteer_proxy.js
deleted file mode 100644
index 5c2c1fa19..000000000
--- a/examples/5_puppeteer_proxy.js
+++ /dev/null
@@ -1,44 +0,0 @@
-/**
- * This example demonstrates the use of Apify features with Puppeteer.
- * We'll show you how to use Apify Proxy without using our Crawlers
- * and instead using only Puppeteer itself.
- *
- * Example uses:
- * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.
- * - Puppeteer to control headless Chrome browser.
- * - Apify Dataset to store data.
- */
-
-const Apify = require('apify');
-
-Apify.main(async () => {
- // Apify enhances not only our Crawlers, but also plain Puppeteer
- // with useful tools such as automatic proxy use (from our own pool).
- // To use the Proxy, you need to either log in using the Apify CLI, or set
- // the APIFY_PROXY_PASSWORD environment variable. You will find the password
- // under your account in the Apify Platform.
- // Other options such as LiveView may also be enabled for Puppeteer.
- const options = {
- useApifyProxy: true,
- headless: true,
- };
-
- // Apify.launchPuppeteer() is a shortcut to get a preconfigured Puppeteer.Browser
- // instance with extra features provided by Apify. All original Puppeteer options
- // are passed directly to Puppeteer.
- const browser = await Apify.launchPuppeteer(options);
-
- console.log('Running Puppeteer...');
- // Proceed with a plain Puppeteer script.
- const page = await browser.newPage();
- const url = 'https://en.wikipedia.org/wiki/Main_Page';
- await page.goto(url);
- const html = await page.content();
-
- // Use any Apify feature.
- await Apify.pushData({ url, html });
-
- // Cleaning up resources is a good practice.
- await browser.close();
- console.log('Puppeteer closed.');
-});
diff --git a/examples/6_call_another_act.js b/examples/6_call_another_act.js
deleted file mode 100644
index 98be38685..000000000
--- a/examples/6_call_another_act.js
+++ /dev/null
@@ -1,44 +0,0 @@
-/**
- * This example shows how to call another actor - in this case apify/send-mail to send
- * an email.
- *
- * For this demonstration, we've chosen to scrape BTC prices. If you don't want to miss the chance of
- * of your life then you can use this code to get current BTC prices from Kraken.com
- * and mail them to your mailbox.
- *
- * If you deploy this actor to Apify platform then you can setup a scheduler for early
- * morning.
- *
- * Example uses:
- * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.
- * - Puppeteer to control headless Chrome browser.
- * - Apify.call() to invoke another actor with a provided input.
- */
-
-const Apify = require('apify');
-
-const YOUR_MAIL = '[email protected]';
-
-Apify.main(async () => {
- // Start browser.
- const browser = await Apify.launchPuppeteer({ headless: true });
-
- // Load Kraken and get last traded price of BTC.
- const page = await browser.newPage();
- await page.goto('https://www.kraken.com/charts');
- const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);
-
- console.log('Calling another actor. This may take a few seconds...');
- // Send prices to your email. For that, you can use an actor we already
- // have available on the platform under the name: apify/send-mail.
- // The second parameter to the Apify.call() invocation is the actor's
- // desired input. You can find the required input parameters by checking
- // the actor's documentation page: https://www.apify.com/apify/send-mail
- await Apify.call('apify/send-mail', {
- to: YOUR_MAIL,
- subject: 'Kraken.com BTC',
- html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,
- });
-
- console.log('Actor successfully called. Go check your email.');
-});
diff --git a/examples/7_synchronous_run.js b/examples/7_synchronous_run.js
deleted file mode 100644
index 9adce457a..000000000
--- a/examples/7_synchronous_run.js
+++ /dev/null
@@ -1,47 +0,0 @@
-/**
- * This example shows shows an actor that has short runtime - just few seconds. It opens a webpage
- * http://goldengatebridge75.org/news/webcam.html that contains webcam stream from Golden Gate
- * bridge, takes a screenshot and saves it as output. This makes actor executable on Apify platform
- * synchronously with a single request that also returns its output.
- *
- * Example is shared in library under https://www.apify.com/apify/example-golden-gate-webcam
- * so you can easily run it with request to
- * https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]
- *
- * Example uses:
- * - Apify.launchPuppeteer() function to get a supercharged instance of Puppeteer.
- * - Puppeteer to control headless Chrome browser.
- * - Apify KeyValueStore to store data and provide an API response.
- */
-
-const Apify = require('apify');
-
-Apify.main(async () => {
- // Start browser.
- const browser = await Apify.launchPuppeteer({ headless: true });
-
- // Load http://goldengatebridge75.org/news/webcam.html and get an iframe
- // containing webcam stream.
- console.log('Opening page.');
- const page = await browser.newPage();
- await page.goto('http://goldengatebridge75.org/news/webcam.html');
- const iframe = (await page.frames()).pop();
-
- // Get webcam image element handle.
- const imageElementHandle = await iframe.$('.VideoColm img');
-
- // Give the webcam image some time to load.
- console.log('Waiting for some time...');
- await Apify.utils.sleep(3000);
-
- // Get a screenshot of that image.
- const imageBuffer = await imageElementHandle.screenshot();
- console.log('Screenshot captured.');
-
- // Save it as an OUTPUT. Just as INPUT, OUTPUT has a special meaning.
- // Anything you save as an OUTPUT to KeyValueStore will be sent to you
- // as an API response once the actor finishes its run, if you use the
- // run-sync API. This way, you can really Apify any website.
- await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });
- console.log('Actor finished.');
-});
diff --git a/examples/basic_crawler.js b/examples/basic_crawler.js
new file mode 100644
index 000000000..500b88c1c
--- /dev/null
+++ b/examples/basic_crawler.js
@@ -0,0 +1,57 @@
+/**
+ * This is the most basic example of Apify SDK, which demonstrates some of its
+ * elementary tools such as the
+ * [BasicCrawler](https://www.apify.com/docs/sdk/apify-runtime-js/latest#BasicCrawler)
+ * and [RequestList](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList) classes.
+ * The script just downloads several web pages with plain HTTP requests (using the
+ * [request-promise](https://www.npmjs.com/package/request-promise) library)
+ * and stores their raw HTML and URL to the default dataset.
+ * In local configuration, the data will be stored as JSON files in `./apify_storage/datasets/default`.
+ */
+
+const Apify = require('apify');
+const requestPromise = require('request-promise');
+
+// Apify.main() function wraps the crawler logic (it is optional).
+Apify.main(async () => {
+ // Create and initialize an instance of the RequestList class that contains
+ // a list of URLs to crawl. Here we use just a few hard-coded URLs.
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://www.google.com/' },
+ { url: 'http://www.example.com/' },
+ { url: 'http://www.bing.com/' },
+ { url: 'http://www.wikipedia.com/' },
+ ],
+ });
+ await requestList.initialize();
+
+ // Create a BasicCrawler - the simplest crawler that enables
+ // users to implement the crawling logic themselves.
+ const crawler = new Apify.BasicCrawler({
+
+ // Let the crawler fetch URLs from our list.
+ requestList,
+
+ // This function will be called for each URL to crawl.
+ // The 'request' option is an instance of the Request class, which contains
+ // information such as URL and HTTP method, as supplied by the RequestList.
+ handleRequestFunction: async ({ request }) => {
+ console.log(`Processing ${request.url}...`);
+
+ // Fetch the page HTML
+ const html = await requestPromise(request.url);
+
+ // Store the HTML and URL to the default dataset
+ await Apify.pushData({
+ url: request.url,
+ html,
+ });
+ },
+ });
+
+ // Run the crawler and wait for its finish.
+ await crawler.run();
+
+ console.log('Crawler finished.');
+});
diff --git a/examples/call_actor.js b/examples/call_actor.js
new file mode 100644
index 000000000..7c3ede600
--- /dev/null
+++ b/examples/call_actor.js
@@ -0,0 +1,44 @@
+/**
+ * This example shows how to start an Apify actor using
+ * <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-call"><code>Apify.call()</code></a>
+ * and how to call Apify API using
+ * <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-client"><code>Apify.client</code></a>.
+ * The script extracts the current Bitcoin prices from Kraken.com
+ * and sends them to your email using the [apify/send-mail](https://www.apify.com/apify/send-mail) actor.
+ *
+ * To make the example work, you'll need an [Apify Account](https://my.apify.com/).
+ * Go to [Account - Integrations](https://my.apify.com/account#/integrations) page to obtain your API token
+ * and set it to the `APIFY_TOKEN` environment variable, or run the script using the CLI.
+ * If you deploy this actor to Apify platform then you can setup a scheduler for early
+ * morning. Don't miss the chance of your life to get rich!
+ */
+
+const Apify = require('apify');
+
+Apify.main(async () => {
+ // Launch the web browser.
+ const browser = await Apify.launchPuppeteer();
+
+ console.log('Obtaining email address...');
+ const user = await Apify.client.users.getUser();
+
+ // Load Kraken.com charts and get last traded price of BTC
+ console.log('Extracting data from kraken.com...');
+ const page = await browser.newPage();
+ await page.goto('https://www.kraken.com/charts');
+ const tradedPricesHtml = await page.$eval('#ticker-top ul', el => el.outerHTML);
+
+ // Send prices to your email. For that, you can use an actor we already
+ // have available on the platform under the name: apify/send-mail.
+ // The second parameter to the Apify.call() invocation is the actor's
+ // desired input. You can find the required input parameters by checking
+ // the actor's documentation page: https://www.apify.com/apify/send-mail
+ console.log(`Sending email to ${user.email}...`);
+ await Apify.call('apify/send-mail', {
+ to: user.email,
+ subject: 'Kraken.com BTC',
+ html: `<h1>Kraken.com BTC</h1>${tradedPricesHtml}`,
+ });
+
+ console.log('Email was sent. Good luck!');
+});
diff --git a/examples/cheerio_crawler.js b/examples/cheerio_crawler.js
new file mode 100644
index 000000000..9b7cfdd28
--- /dev/null
+++ b/examples/cheerio_crawler.js
@@ -0,0 +1,82 @@
+/**
+ * This example demonstrates how to use [CheerioCrawler](https://www.apify.com/docs/sdk/apify-runtime-js/latest#CheerioCrawler)
+ * to crawl a list of URLs from an external file,
+ * load each URL using a plain HTTP request, parse the HTML using [cheerio](https://www.npmjs.com/package/cheerio)
+ * and extract some data from it: the page title and all H1 tags.
+ */
+
+const Apify = require('apify');
+
+// Apify.utils contains various utilities, e.g. for logging.
+// Here we turn off logging of unimportant messages.
+const { log } = Apify.utils;
+log.setLevel(log.LEVELS.WARNING);
+
+// A link to a list of Fortune 500 companies' websites available on GitHub.
+const CSV_LINK = 'https://gist.githubusercontent.com/hrbrmstr/ae574201af3de035c684/raw/f1000.csv';
+
+// Apify.main() function wraps the crawler logic (it is optional).
+Apify.main(async () => {
+ // Create an instance of the RequestList class that contains a list of URLs to crawl.
+ // Here we download and parse the list of URLs from an external file.
+ const requestList = new Apify.RequestList({
+ sources: [{ requestsFromUrl: CSV_LINK }],
+ });
+ await requestList.initialize();
+
+ // Create an instance of the CheerioCrawler class - a crawler
+ // that automatically loads the URLs and parses their HTML using cheerio library.
+ const crawler = new Apify.CheerioCrawler({
+ // Let the crawler fetch URLs from our list.
+ requestList,
+
+ // The crawler downloads and processes the web pages in parallel, with a concurrency
+ // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // Here we define some hard limits for the concurrency.
+ minConcurrency: 10,
+ maxConcurrency: 50,
+
+ // On error, retry each page at most once.
+ maxRequestRetries: 1,
+
+ // Increase the timeout for processing of each page.
+ handlePageTimeoutSecs: 60,
+
+ // This function will be called for each URL to crawl.
+ // It accepts a single parameter, which is an object with the following fields:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - html: contains raw HTML of the page
+ // - $: the cheerio object containing parsed HTML
+ handlePageFunction: async ({ request, html, $ }) => {
+ console.log(`Processing ${request.url}...`);
+
+ // Extract data from the page using cheerio.
+ const title = $('title').text();
+ const h1texts = [];
+ $('h1').each((index, el) => {
+ h1texts.push({
+ text: $(el).text(),
+ });
+ });
+
+ // Store the results to the default dataset. In local configuration,
+ // the data will be stored as JSON files in ./apify_storage/datasets/default
+ await Apify.pushData({
+ url: request.url,
+ title,
+ h1texts,
+ html,
+ });
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ handleFailedRequestFunction: async ({ request }) => {
+ console.log(`Request ${request.url} failed twice.`);
+ },
+ });
+
+ // Run the crawler and wait for its finish.
+ await crawler.run();
+
+ console.log('Crawler finished.');
+});
diff --git a/examples/puppeteer_crawler.js b/examples/puppeteer_crawler.js
new file mode 100644
index 000000000..155bca9fd
--- /dev/null
+++ b/examples/puppeteer_crawler.js
@@ -0,0 +1,89 @@
+/**
+ * This example demonstrates how to use [PuppeteerCrawler](https://www.apify.com/docs/sdk/apify-runtime-js/latest#PuppeteerCrawler)
+ * in combination with [RequestList](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestList)
+ * and [RequestQueue](https://www.apify.com/docs/sdk/apify-runtime-js/latest#RequestQueue) to recursively scrape
+ * Hacker News website (https://news.ycombinator.com) using headless Chrome / Puppeteer.
+ * The crawlers starts with a single URL, finds links to next pages,
+ * enqueues them and continues until no more desired links are available.
+ * The results are stored to the default dataset. In local configuration, the results are stored as JSON files in `./apify_storage/datasets/default`
+ */
+
+const Apify = require('apify');
+
+Apify.main(async () => {
+ // Create and initialize an instance of the RequestList class that contains the start URL
+ const requestList = new Apify.RequestList({
+ sources: [
+ { url: 'http://www.ycombinator.com/' },
+ ],
+ });
+ await requestList.initialize();
+
+ // Apify.openRequestQueue() is a factory to get a preconfigured RequestQueue instance.
+ const requestQueue = await Apify.openRequestQueue();
+
+ // Create an instance of the PuppeteerCrawler class - a crawler
+ // that automatically loads the URLs in headless Chrome / Puppeteer.
+ const crawler = new Apify.PuppeteerCrawler({
+ // The crawler will first fetch start URLs from the RequestList
+ // and then the newly discovered URLs from the RequestQueue
+ requestList,
+ requestQueue,
+
+ // Run Puppeteer in headless mode. If you headless to false, you'll see the scraping
+ // browsers showing up on your screen. This is great for debugging.
+ launchPuppeteerOptions: { headless: true },
+
+ // This function will be called for each URL to crawl.
+ // Here you can write the Puppeteer scripts you are familiar with,
+ // with the exception that browsers and pages are automatically managed by Apify SDK.
+ // The function accepts a single parameter, which is an object with the following fields:
+ // - request: an instance of the Request class with information such as URL and HTTP method
+ // - page: Puppeteer's Page object (see https://pptr.dev/#show=api-class-page)
+ handlePageFunction: async ({ request, page }) => {
+ console.log(`Processing ${request.url}...`);
+
+ // A function to be evaluated by Puppeteer within the browser context.
+ const pageFunction = ($posts) => {
+ const data = [];
+
+ // We're getting the title, rank and URL of each post on Hacker News.
+ $posts.forEach(($post) => {
+ data.push({
+ title: $post.querySelector('.title a').innerText,
+ rank: $post.querySelector('.rank').innerText,
+ href: $post.querySelector('.title a').href,
+ });
+ });
+
+ return data;
+ };
+ const data = await page.$$eval('.athing', pageFunction);
+
+ // Store the results to the default dataset.
+ await Apify.pushData(data);
+
+ // Find the link to the next page using Puppeteer functions
+ let nextHref;
+ try {
+ nextHref = await page.$eval('.morelink', el => el.href);
+ } catch (err) {
+ console.log(`${request.url} is the last page!`);
+ return;
+ }
+
+ // Enqueue the link to the RequestQueue
+ await requestQueue.addRequest(new Apify.Request({ url: nextHref }));
+ },
+
+ // This function is called if the page processing failed more than maxRequestRetries+1 times.
+ handleFailedRequestFunction: async ({ request }) => {
+ console.log(`Request ${request.url} failed too many times`);
+ },
+ });
+
+ // Run the crawler and wait for its finish.
+ await crawler.run();
+
+ console.log('Crawler finished.');
+});
diff --git a/examples/puppeteer_proxy.js b/examples/puppeteer_proxy.js
new file mode 100644
index 000000000..9e66535fd
--- /dev/null
+++ b/examples/puppeteer_proxy.js
@@ -0,0 +1,35 @@
+/**
+ * This example demonstrates how to load pages in headless Chrome / Puppeteer
+ * over [Apify Proxy](https://www.apify.com/docs/proxy).
+ * To make it work, you'll need an Apify account
+ * that has access to the proxy.
+ * The proxy password is available on the [Proxy](https://my.apify.com/proxy) page in the app.
+ * Just set it to the `APIFY_PROXY_PASSWORD` environment variable
+ * or run the script using the CLI.
+ */
+
+const Apify = require('apify');
+
+Apify.main(async () => {
+ // Apify.launchPuppeteer() is similar to Puppeteer's launch() function.
+ // It accepts the same parameters and returns a preconfigured Puppeteer.Browser instance.
+ // Moreover, it accepts several additional options, such as useApifyProxy.
+ const options = {
+ useApifyProxy: true,
+ };
+ const browser = await Apify.launchPuppeteer(options);
+
+ console.log('Running Puppeteer script...');
+
+ // Proceed with a plain Puppeteer script.
+ const page = await browser.newPage();
+ const url = 'https://en.wikipedia.org/wiki/Main_Page';
+ await page.goto(url);
+ const title = await page.title();
+
+ console.log(`Page title: ${title}`);
+
+ // Cleaning up after yourself is always good.
+ await browser.close();
+ console.log('Puppeteer closed.');
+});
diff --git a/examples/screenshots.js b/examples/screenshots.js
new file mode 100644
index 000000000..982dadeef
--- /dev/null
+++ b/examples/screenshots.js
@@ -0,0 +1,61 @@
+/**
+ * This example shows how to read and write
+ * data to the default key-value store using
+ * <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-getValue"><code>Apify.getValue()</code></a>
+ * and
+ * <a href="https://www.apify.com/docs/sdk/apify-runtime-js/latest#module-Apify-setValue"><code>Apify.setValue()</code></a>.
+ * The script crawls a list of URLs using Puppeteer,
+ * captures a screenshot of each page and saves it to the store. The list of URLs is
+ * provided as actor input that is also read from the store.
+ *
+ * In local configuration, the input is stored in the default key-value store's directory as JSON file at
+ * `./apify_storage/key_value_stores/default/INPUT.json`. You need to create the file and set it the following content:
+ *
+ * ```json
+ * { "sources": [{ "url": "https://www.google.com" }, { "url": "https://www.duckduckgo.com" }] }
+ * ```
+ *
+ * On the Apify cloud platform, the input can be either set manually
+ * in the UI app or passed as the POST payload to the
+ * [Run actor API call](https://www.apify.com/docs/api/v2#/reference/actors/run-collection/run-actor).
+ * For more details, see [Input and output](https://www.apify.com/docs/actor#input-output)
+ * in the Apify Actor documentation.
+ */
+
+const Apify = require('apify');
+
+Apify.main(async () => {
+ // Read the actor input configuration containing URLs to take the screenshot off.
+ // By convention, the input is present in actor's default key-value store under the "INPUT" key.
+ const input = await Apify.getValue('INPUT');
+ if (!input) throw new Error('Have you passed the correct INPUT ?');
+
+ const { sources } = input;
+
+ const requestList = new Apify.RequestList({ sources });
+ await requestList.initialize();
+
+ const crawler = new Apify.PuppeteerCrawler({
+ requestList,
+ handlePageFunction: async ({ page, request }) => {
+ console.log(`Processing ${request.url}...`);
+
+ // This is a Puppeteer function that takes a screenshot of the page and returns its buffer.
+ const screenshotBuffer = await page.screenshot();
+
+ // The record key may only include the following characters: a-zA-Z0-9!-_.'()
+ const key = request.url.replace(/[:/]/g, '_');
+
+ // Save the screenshot. Choosing the right content type will automatically
+ // assign the local file the right extension, in this case .png.
+ // The screenshots will be stored in ./apify_storage/key_value_stores/default/
+ await Apify.setValue(key, screenshotBuffer, { contentType: 'image/png' });
+ console.log(`Screenshot of ${request.url} saved.`);
+ },
+ });
+
+ // Run crawler.
+ await crawler.run();
+
+ console.log('Crawler finished.');
+});
diff --git a/examples/synchronous_run.js b/examples/synchronous_run.js
new file mode 100644
index 000000000..8a5bef383
--- /dev/null
+++ b/examples/synchronous_run.js
@@ -0,0 +1,42 @@
+/**
+ * This example shows shows a quick actor that has a run time of just a few seconds.
+ * It opens a [web page](http://goldengatebridge75.org/news/webcam.html) that contains webcam stream from the Golden Gate
+ * bridge, takes a screenshot of the page and saves it as output.
+ *
+ * This actor
+ * can be invoked synchronously using a single HTTP request to directly obtain its output as a reponse,
+ * using the [Run actor synchronously](https://www.apify.com/docs/api/v2#/reference/actors/run-actor-synchronously/without-input)
+ * API endpoint.
+ * The example is also shared as the [apify/example-golden-gate-webcam](https://www.apify.com/apify/example-golden-gate-webcam)
+ * actor in the Apify library, so you can test it directly there simply by sending a POST request to
+ * https://api.apify.com/v2/acts/apify~example-golden-gate-webcam/run-sync?token=[YOUR_API_TOKEN]
+ */
+
+const Apify = require('apify');
+
+Apify.main(async () => {
+ // Launch web browser.
+ const browser = await Apify.launchPuppeteer();
+
+ // Load http://goldengatebridge75.org/news/webcam.html and get an IFRAME with the webcam stream
+ console.log('Opening web page...');
+ const page = await browser.newPage();
+ await page.goto('http://goldengatebridge75.org/news/webcam.html');
+ const iframe = (await page.frames()).pop();
+
+ // Get webcam image element handle.
+ const imageElementHandle = await iframe.$('.VideoColm img');
+
+ // Give the webcam image some time to load.
+ console.log('Waiting for page to load...');
+ await Apify.utils.sleep(3000);
+
+ // Get a screenshot of that image.
+ const imageBuffer = await imageElementHandle.screenshot();
+ console.log('Screenshot captured.');
+
+ // Save the screenshot as actor's output. By convention, similarly to "INPUT",
+ // the actor's output is stored in the default key-value store under the "OUTPUT" key.
+ await Apify.setValue('OUTPUT', imageBuffer, { contentType: 'image/jpeg' });
+ console.log('Actor finished.');
+});