Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: update getting started and some other guides (#1429)

Ondra Urban333faa49c027
diff --git a/docs/guides/cheerio_crawler.mdx b/docs/guides/cheerio_crawler.mdx
new file mode 100644
index 000000000..6097f77b4
--- /dev/null
+++ b/docs/guides/cheerio_crawler.mdx
@@ -0,0 +1,87 @@
+---
+id: cheerio-crawler-guide
+title: "CheerioCrawler guide"
+sidebar_label: "CheerioCrawler"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+&#8203;<ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> is our simplest and fastest crawler. If you're familiar with [jQuery](https://jquery.com/), you'll understand `CheerioCrawler` in minutes.
+
+## What is Cheerio
+
+[Cheerio](https://www.npmjs.com/package/cheerio) is essentially [jQuery](https://jquery.com/) for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would use jQuery for manipulating the DOM of an HTML page. In crawling, you'll mostly use it to select the needed elements and extract their values - the data you're interested in. But jQuery runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> comes in.
+
+## How the crawler works
+
+&#8203;<ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client. The URLs are fed to the crawler using <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are usually HTML pages. The same pages you would get in your browser when you first load a URL. But it can handle any content types with the help of the <ApiLink to="cheerio-crawler/interface/CheerioCrawlerOptions#additionalMimeTypes">`additionalMimeTypes`</ApiLink> option.
+
+:::info
+
+Modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards, and together they create the final page. To crawl those, see <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>.
+
+:::
+
+Once the page's HTML is retrieved, the crawler will pass it to [Cheerio](https://www.npmjs.com/package/cheerio) for parsing. The result is the typical `$` function, which should be familiar to jQuery users. You can use the `$` function to do all sorts of lookups and manipulation of the page's HTML, but in scraping, you will mostly use it to find specific HTML elements and extract their data.
+
+Example use of Cheerio and its `$` function in comparison to browser JavaScript:
+
+```ts
+// Return the text content of the <title> element.
+document.querySelector('title').textContent; // plain JS
+$('title').text(); // Cheerio
+
+// Return an array of all 'href' links on the page.
+Array.from(document.querySelectorAll('[href]')).map(el => el.href); // plain JS
+$('[href]')
+ .map((i, el) => $(el).attr('href'))
+ .get(); // Cheerio
+```
+
+:::note
+
+This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides. Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is your best bet to do the parsing in Node.
+
+:::
+
+## When to use `CheerioCrawler`
+
+`CheerioCrawler` really shines when you need to cope with extremely high workloads. With just 4 GBs of memory and a single CPU core, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_. To scrape this fast with a full browser scraper, such as the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, you'd need significantly more computing power.
+
+**Advantages:**
+
+- Extremely fast and cheap to run
+- Easy to set up
+- Familiar for jQuery users
+- Automatically avoids some anti-scraping bans
+
+**Disadvantages:**
+
+- Does not work for websites that require JavaScript rendering
+- May easily overload the target website with requests
+- Does not enable any manipulation of the website before scraping
+
+## Example use of Cheerio
+
+### Get text content of an element
+
+Finds the first `<h2>` element and returns its text content.
+
+```js
+$('h2').text()
+```
+
+### Find all links on a page
+
+This snippet finds all `<a>` elements which have the `href` attribute and extracts the hrefs into an array.
+
+```js
+$('a[href]')
+ .map((i, el) => $(el).attr('href'))
+ .get();
+```
+
+### Other examples
+
+Visit the [Examples](../examples) section to browse examples of `CheerioCrawler` usage. Almost all examples show `CheerioCrawler` code in their code tabs.
diff --git a/docs/guides/javascript-rendering-playwright-no-wait.ts b/docs/guides/javascript-rendering-playwright-no-wait.ts
new file mode 100644
index 000000000..77371b51b
--- /dev/null
+++ b/docs/guides/javascript-rendering-playwright-no-wait.ts
@@ -0,0 +1,14 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ page }) {
+ // Here we don't wait for the selector and immediately
+ // extract the text content from the page.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/guides/javascript-rendering-playwright.ts b/docs/guides/javascript-rendering-playwright.ts
new file mode 100644
index 000000000..6a1ca9b93
--- /dev/null
+++ b/docs/guides/javascript-rendering-playwright.ts
@@ -0,0 +1,15 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ async requestHandler({ page }) {
+ // page.locator points to an element in the DOM
+ // using a CSS selector, but it does not access it yet.
+ const actorCard = page.locator('.ActorStoreItem').first();
+ // Upon calling one of the locator methods Playwright
+ // waits for the element to render and then accesses it.
+ const actorText = await actorCard.textContent();
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/guides/javascript-rendering-puppeteer-no-wait.ts b/docs/guides/javascript-rendering-puppeteer-no-wait.ts
new file mode 100644
index 000000000..c06a4c1b7
--- /dev/null
+++ b/docs/guides/javascript-rendering-puppeteer-no-wait.ts
@@ -0,0 +1,14 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ page }) {
+ // Here we don't wait for the selector and immediately
+ // extract the text content from the page.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/guides/javascript-rendering-puppeteer.ts b/docs/guides/javascript-rendering-puppeteer.ts
new file mode 100644
index 000000000..d4ad7fedf
--- /dev/null
+++ b/docs/guides/javascript-rendering-puppeteer.ts
@@ -0,0 +1,17 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ async requestHandler({ page }) {
+ // Puppeteer does not have the automatic waiting functionality
+ // of Playwright, so we have to explicitly wait for the element.
+ await page.waitForSelector('.ActorStoreItem');
+ // Puppeteer does not have helper methods like locator.textContent,
+ // so we have to manually extract the value using in-page JavaScript.
+ const actorText = await page.$eval('.ActorStoreItem', (el) => {
+ return el.textContent;
+ });
+ console.log(`ACTOR: ${actorText}`);
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/guides/javascript-rendering.mdx b/docs/guides/javascript-rendering.mdx
new file mode 100644
index 000000000..1fd1b95f7
--- /dev/null
+++ b/docs/guides/javascript-rendering.mdx
@@ -0,0 +1,94 @@
+---
+id: javascript-rendering
+title: "JavaScript rendering"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import PlaywrightSource from '!!raw-loader!./javascript-rendering-playwright.ts';
+import PlaywrightNoWaitSource from '!!raw-loader!./javascript-rendering-playwright-no-wait.ts';
+import PuppeteerSource from '!!raw-loader!./javascript-rendering-puppeteer.ts';
+import PuppeteerNoWaitSource from '!!raw-loader!./javascript-rendering-puppeteer-no-wait.ts';
+
+JavaScript rendering is the process of executing JavaScript on a page to make changes in the page's structure or content. It's also called client-side rendering, the opposite of server-side rendering. Some modern websites render on the client, some on the server and many cutting edge websites render some things on the server and other things on the client.
+
+The Crawlee website does not use JavaScript rendering to display its content, so we have to look for an example elsewhere. [Apify Store](https://apify.com/store) is a library of scrapers and automations called **actors** that anyone can grab and use for free. It uses JavaScript rendering to display the list of actors, so let's use it to demonstrate how it works.
+
+```ts title="src/main.mjs"
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ $, request }) {
+ // Extract text content of an actor card
+ const actorText = $('.ActorStoreItem').text();
+ console.log(`ACTOR: ${actorText}`);
+ }
+})
+
+await crawler.run(['https://apify.com/store']);
+```
+
+Run the code, and you'll see that the crawler won't print the content of the actor card.
+
+```log
+ACTOR:
+```
+
+That's because Apify Store uses client-side JavaScript to render its content and `CheerioCrawler` can't execute it, so the text never appears in the page's HTML.
+
+You can confirm this using Chrome DevTools. If you go to https://apify.com/store, right-click anywhere in the page, select **View Page Source** and search for **ActorStoreItem** you won't find any results. Then, if you right-click again, select **Inspect** and search for the same **ActorStoreItem**, you will find many of them.
+
+How's this possible? Because **View Page Source** shows the original HTML, before any JavaScript executions. That's what `CheerioCrawler` gets. Whereas with **Inspect** you see the current HTML - after JavaScript execution. When you understand this, it's not a huge surprise that `CheerioCrawler` can't find the data. For that we need a headless browser.
+
+## Headless browsers
+
+To get the contents of `.ActorStoreItem`, you will have to use a headless browser. You can choose from two libraries to control your browser: [Puppeteer](https://github.com/puppeteer/puppeteer) or [Playwright](https://github.com/microsoft/playwright). The choice is simple. If you know one of them, choose the one you know. If you know both, or none, choose Playwright, because it's better in most cases.
+
+## Waiting for elements to render
+
+No matter which library you pick, here's example code for both. Playwright is a little more pleasant to use, but both libraries will get the job done. The big difference between them is that Playwright will automatically wait for elements to appear, whereas in Puppeteer, you have to explicitly wait for them.
+
+<Tabs groupId="javascript-rendering">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PlaywrightSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PuppeteerSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+When you run the code, you'll see the _badly formatted_ content of the first actor card printed to console:
+
+```log
+ACTOR: Web Scraperapify/web-scraperCrawls arbitrary websites using [...]
+```
+
+### We're not kidding
+
+If you don't believe us that the elements need to be waited for, run the following code which skips the waiting.
+
+<Tabs groupId="javascript-rendering">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PlaywrightNoWaitSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PuppeteerNoWaitSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+In both cases, the request will be retried a few times and eventually fail with an error like this:
+
+```log
+ERROR [...] Error: failed to find element matching selector ".ActorStoreItem"
+```
+
+That's because when you try to access the element in the browser, it's not been rendered in the DOM yet.
+
+:::tip
+
+This guide only touches the concept of JavaScript rendering and use of headless browsers. To learn more, continue with the [Puppeteer & Playwright course](https://developers.apify.com/academy/puppeteer-playwright) in the Apify Academy. **It's free and open-source** ❤️.
+
+:::
diff --git a/docs/guides/motivation.mdx b/docs/guides/motivation.mdx
index bbfff3686..e69de29bb 100644
--- a/docs/guides/motivation.mdx
+++ b/docs/guides/motivation.mdx
@@ -1,18 +0,0 @@
----
-id: motivation
-title: Motivation
-description: "Spoiler: we built it so you can focus on awesome scrapers"
----
-
-Thanks to the tools like [Playwright](https://github.com/microsoft/playwright), [Puppeteer](https://github.com/puppeteer/puppeteer) or [Cheerio](https://www.npmjs.com/package/cheerio), it is easy to write Node.js code to extract data from web pages. But eventually things will get complicated. For example, when we try to:
-
-- Perform a deep crawl of an entire website using a persistent queue of URLs.
-- Run our scraping code on a list of 100k URLs in a CSV file, without losing any data when our code crashes.
-- Rotate proxies to hide our browser origin and keep user-like sessions.
-- Disable browser fingerprinting protections used by websites.
-
-Python has [Scrapy](https://scrapy.org/) for these tasks, but there was no such library for **JavaScript, the language of
-the web**. The use of JavaScript is natural, since the same language is used to write the scripts as well as the data extraction code running in a browser.
-
-The goal of Crawlee is to fill this gap and provide a toolbox for generic web scraping, crawling and automation tasks in JavaScript and TypeScript, so that we could focus on writing code specific to the target website, rather than developing
-commonalities by reinventing the wheel every time we need data from the web.
diff --git a/docs/introduction/01-setting-up.mdx b/docs/introduction/01-setting-up.mdx
index a2aec19b2..7cc45467d 100644
--- a/docs/introduction/01-setting-up.mdx
+++ b/docs/introduction/01-setting-up.mdx
@@ -8,11 +8,10 @@ import ApiLink from '@site/src/components/ApiLink';
To run Crawlee on your own computer, you need to meet the following pre-requisites first:
-1. Have Node.js version 16.0 or higher installed.
- - Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
-2. Have NPM installed.
- - NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
- > Or use any other package manager of your choice.
+1. Have **Node.js version 16.0** or higher installed.
+ - Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
+2. Have **NPM** installed, or use other package manager of your choice.
+ - NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
If not certain, confirm the prerequisites by running:
@@ -24,48 +23,41 @@ node -v
npm -v
```
-### Creating a new project
+## Creating a new project
-The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). This command line tool allows you to create and run Crawlee projects with ease. You can use the `npx` utility to download and run the CLI - it is also embedded to the `crawlee` meta-package:
+The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is also embedded in the `crawlee` package:
```bash
npx crawlee create my-new-project
```
-A prompt will be shown, asking you to choose a template. Let's choose the first one called `Crawlee playwright template [TypeScript]`. The command will now create a new directory in your current working directory, called `my-new-project`, create a `package.json` in this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
+A prompt will be shown, asking you to choose a template. Crawlee is written in TypeScript so if you're familiar with it, choosing a TypeScript template will give you better code completion and static type checking, but feel free to use JavaScript as well. Functionally they're identical.
+
+Let's choose the first template called **Crawlee playwright template**. The command will create a new directory in your current working directory, called **my-new-project**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
Let's try that!
```bash
cd my-new-project
+npm start
```
-```bash
-npx crawlee run
-```
-
-We used the `crawlee run` CLI command, but all what it actually does is to call `npm run start`. You can use the `--script` or `-s` option to specify what script you want to run instead of the default `start`:
-
-```bash
-npx crawlee run --script=start:prod
-```
-
-:::info Purging of storages
+You will see log messages in the terminal as Crawlee boots up and after a second a Chromium browser window will open. In the window, you'll see quickly changing pages and back in the terminal, you will see the printed titles (contents of the `<title>` HTML tags) of the pages.
-Another option of the `crawlee run` command is the control over purging: `--purge` and more importantly `--no-purge`, as purging is enabled by default. To understand what those are about, you first need to know about the concept of storages in Crawlee. There are 3 storage types you can use: <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink> and <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>. Crawlee is using the first two for storing the runtime data about current crawl, and the third one is for storing the results.
+![Chrome Scrape](/img/chrome_scrape.gif)
-When Crawlee stores some state, it uses a storage client - by default a <ApiLink to="memory-storage/class/MemoryStorage">`MemoryStorage`</ApiLink>, which - contrary to its name - will also store the state in JSON files, so you can observe it easily. This state is being purged automatically when you run the crawler, and with the `--no-purge` flag you can disable this behaviour and reuse the state you already have.
+:::info
-More about storages and purging can be found in [Request Storage guide](../guides/request-storage).
+We picked the Playwright template, which uses Chromium to open pages. If you pick the Cheerio template instead, there won't be any browser window, as the requests to the target site will be done via a specialized HTTP client: [`got-scraping`](https://github.com/apify/got-scraping), instead of a browser.
:::
-You should start seeing log messages in the terminal as the system boots up and after a second, a Chromium browser window should pop up. In the window, you'll see quickly changing pages and back in the terminal, you should see the titles (contents of the `<title>` HTML tags) of the pages printed.
-
-> We picked the playwright template, which will use the Chromium browser to open pages. If you picked the cheerio template instead, there won't be any browser window, as the requests to the target site will be done via [`got-scraping`](https://github.com/apify/got-scraping) instead of real browser.
-
You can always terminate the crawl with a keypress in the terminal:
```bash
CTRL+C
```
+
+## Next lesson
+
+The next lesson will teach you how to create a very simple crawler and explain Crawlee components while building it.
diff --git a/docs/introduction/02-first-crawler.mdx b/docs/introduction/02-first-crawler.mdx
index e68e95e2c..68ecc5b92 100644
--- a/docs/introduction/02-first-crawler.mdx
+++ b/docs/introduction/02-first-crawler.mdx
@@ -6,118 +6,116 @@ description: Your first steps into the world of scraping with Crawlee
import ApiLink from '@site/src/components/ApiLink';
-Now it's time to start writing some actual source code. But before you do, let's just briefly introduce all Crawlee classes necessary to make it happen.
+In this lesson, you will build your first crawler. But before you do, let's briefly introduce the Crawlee classes involved in the process.
-### The general idea
+## How Crawlee works
-There are 4 crawler classes available for use in Crawlee. <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink>, <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. We'll talk about their differences later. Now, let's talk about what they have in common.
+There are 3 main crawler classes available for use in Crawlee. <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. We'll talk about their differences later. Now, let's talk about what they have in common.
-The general idea of each crawler is to go to a web page, open it, do some stuff there, save some results and continue to the next page, until it's done its job. So the crawler always needs to find answers to two questions: **Where should I go?** and **What should I do there?** Answering those two questions is the only setup mandatory for running the crawlers.
+The general idea of each crawler is to go to a web page, open it, do some stuff there, save some results, continue to the next page, and repeat this process until the crawler's done its job. So the crawler always needs to find answers to two questions: **Where should I go?** and **What should I do there?** Answering those two questions is the only required setup. The crawlers have reasonable defaults for everything else.
-### The Where - `Request`, `RequestList` and `RequestQueue`
+### The Where - `Request` and `RequestQueue`
-All crawlers use instances of the <ApiLink to="core/class/Request">`Request`</ApiLink> class to determine where they need to go. Each request may hold a lot of information, but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. You need to either have a pre-existing list of your own URLs that you wish to visit, perhaps a thousand, or a million, or you need to build this list dynamically as you crawl, adding more and more URLs to the list as you progress. Or both at the same time.
+All crawlers use instances of the <ApiLink to="core/class/Request">`Request`</ApiLink> class to determine where they need to go. Each request may hold a lot of information, but at the very least, it must hold a URL - a web page to open. But having only one URL would not make sense for crawling. Sometimes you have a pre-existing list of your own URLs that you wish to visit, perhaps a thousand. Other times you need to build this list dynamically as you crawl, adding more and more URLs to the list as you progress. Most of the time, you will use both options.
-A representation of the pre-existing list is an instance of the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> class. It is a static, immutable list of URLs and other metadata (see the <ApiLink to="core/class/Request">`Request`</ApiLink> object) that the crawler will visit, one by one, retrying whenever an error occurs, until there are no more requests to process.
+The requests are stored in a <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, a dynamic queue of `Request` instances. You can seed it with start URLs and also add more requests while the crawler is running. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain, add them to the queue (called _enqueuing_) and repeat this process to build a queue of virtually unlimited number of URLs.
-&#8203;<ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> on the other hand, represents a dynamic queue of requests. One that can be updated at runtime by adding more pages - <ApiLink to="core/class/Request">`Request`</ApiLink> objects - to process. This allows the crawler to open one page, extract interesting URLs, such as links to other pages on the same domain, add them to the queue (called _enqueuing_) and repeat this process to build a queue of tens of thousands or more URLs while knowing only a single one at the beginning.
+### The What - `requestHandler`
-`RequestList` and `RequestQueue` are essential for the crawler's operation. There is no other way to supply requests = "pages to crawl" to the crawlers. At least one of them always needs to be provided while setting up. You can also use both at the same time, if you wish.
+In the `requestHandler` you tell the crawler what to do at each and every page it visits. You can use it to handle extraction of data from the page, processing the data, saving it, calling APIs, doing calculations and so on.
-### The What - `requestHandler`
+The `requestHandler` is a user-defined function, invoked automatically by the crawler for each `Request` from the `RequestQueue`. It always receives a single argument - a <ApiLink to="core/interface/CrawlingContext">`CrawlingContext`</ApiLink>. Its properties change depending on the crawler class used, but it always includes the `request` property, which represents the currently crawled URL and related metadata.
-The `requestHandler` is the brain of the crawler. It tells it what to do at each and every page it visits. Generally it handles extraction of data from the page, processing the data, saving it, calling APIs, doing calculations and whatever else you need it to do.
+## Building a crawler
-The `requestHandler` is a user-defined function, invoked automatically by the crawler for each `Request` object, either from the `RequestList` or `RequestQueue`. It always receives a single argument - a plain object containing the crawling context. Its properties change depending on the crawler class used, but it always includes at least the `request` property, which represents the currently crawled `Request` instance (i.e. the URL the crawler is visiting and related metadata) and the `autoscaledPool` property, which is an instance of the <ApiLink to="core/class/AutoscaledPool">`AutoscaledPool`</ApiLink> class and we'll talk about it in detail later.
+Let's put the theory into practice and start with something easy. Visit a page and get its HTML title. In this tutorial, you'll scrape the Crawlee website [https://crawlee.dev](https://crawlee.dev), but the same code will work for any website.
-```ts
-// The object received as a single argument by the requestHandler
-{
- request: Request,
- autoscaledPool: AutoscaledPool,
-}
-```
+:::info
+
+We are using a JavaScript feature called [Top level await](https://blog.saeloun.com/2021/11/25/ecmascript-top-level-await.html) in our examples. To be able to use that, you might need some extra setup. Namely, it requires the use of [ECMAScript Modules](https://nodejs.org/api/esm.html) - this means you either need to add `"type": "module"` to your `package.json` file, or use `*.mjs` extension for your files. Additionally, if you are in a TypeScript project, you need to set the `module` and `target` compiler options to `ES2022` or above.
-### Putting it all together
+:::
-Enough theory! Let's put some of those hard-learned facts into practice. We learned above that we need at least one `Request` and a `requestHandler` to setup a crawler.
+### Adding requests to the crawling queue
-Let's start with something super easy. Visit a page, get its title and close. For the purposes of this tutorial, we'll be scraping our own webpage [https://apify.com](https://apify.com). Now, to get there, we need a `Request` with the page's URL in one of our sources, `RequestList` or `RequestQueue`. Let's go with `RequestQueue` for now.
+Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts
+```ts title="src/main.mjs"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
const requestQueue = await RequestQueue.open();
-// And then you add a request to it.
-await requestQueue.addRequest({ url: 'https://apify.com' });
+// And then you add one or more requests to it.
+await requestQueue.addRequest({ url: 'https://crawlee.dev' });
```
-> If you're not familiar with the `async` and `await` keywords used in the example, you should know that these are native syntax in modern JavaScript. You can [learn more about them here](https://nikgrozev.com/2017/10/01/async-await/).
+The <ApiLink to="core/class/RequestQueue#addRequest">`requestQueue.addRequest()`</ApiLink> function automatically converts the object with URL string to a <ApiLink to="core/class/Request">`Request`</ApiLink> instance. So now you have a `requestQueue` that holds one request which points to `https://crawlee.dev`.
-The <ApiLink to="core/class/RequestQueue#addRequest">`requestQueue.addRequest()`</ApiLink> function automatically converts the plain object you passed to it to a `Request` instance, so now you have a `requestQueue` that holds one `request` which points to `https://apify.com`. In case you want to add more requests, you can also use the <ApiLink to="core/class/RequestQueue#addRequests">`requestQueue.addRequests()`</ApiLink> method, that accepts an array of request-like objects (or directly an array of strings - the URLs). Even better, there is a `crawler.addRequests()` method we will describe later in the tutorial - for now just keep in mind that the `crawler.addRequests()` is the smartest method of those three, because it supports adding large numbers of requests without blocking.
+:::tip
-Now you need the `requestHandler`.
+The code above is for illustration of the request queue concept. Soon you'll learn about the `crawler.addRequests()` method which allows you to skip this initialization code, and it also supports adding a large number of requests without blocking.
-```ts
-// Type import, applies only to TypeScript projects.
-import type { CheerioCrawlingContext } from 'crawlee';
+:::
-// We'll define the function separately, so it's more obvious.
-const requestHandler: CheerioCrawlingContext = async ({ request, $ }) => {
- // This should look familiar if you ever worked with jQuery.
- // We're just getting the text content of the <title> HTML element.
- const title = $('title').text();
+### Building a CheerioCrawler
- console.log(`The title of "${request.url}" is: ${title}.`);
-};
-```
+Crawlee comes with three main crawler classes: <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. You can read their short descriptions in the [Quick start](../quick-start) lesson.
-Wait, where did the `$` come from? Remember what you learned about the `requestHandler` earlier. It accepts a plain `Object` as an argument that will always have a `request` property, but it will also have other properties, depending on the chosen crawler class. Well, `$` is a property provided by the `CheerioCrawler` class, which we'll set up right now.
+Unless you have a good reason to start with a different one, you should try building a `CheerioCrawler` first. It is an HTTP crawler with HTTP2 support, anti-blocking features and integrated HTML parser - [Cheerio](https://www.npmjs.com/package/cheerio). It's fast, simple, cheap to run and does not require complicated dependencies. The only downside is that it won't work out of the box for websites which require JavaScript rendering. But you might not need JavaScript rendering at all, because many modern websites use server-side rendering.
-```ts
-import { CheerioCrawler, RequestQueue, CheerioCrawlingContext } from 'crawlee';
+Let's continue with the earlier `RequestQueue` example.
-const requestQueue = await RequestQueue.open();
-await requestQueue.addRequests(['https://apify.com']);
+```ts title="src/main.mjs"
+// Add import of CheerioCrawler
+import { RequestQueue, CheerioCrawler } from 'crawlee';
-const requestHandler: CheerioCrawlingContext = async ({ request, $ }) => {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
-};
+const requestQueue = await RequestQueue.open();
+await requestQueue.addRequest({ url: 'https://crawlee.dev' });
-// Set up the crawler, passing a single options object as an argument.
+// Create the crawler and add the queue with our URL
+// and a request handler to process the page.
const crawler = new CheerioCrawler({
requestQueue,
- requestHandler,
-});
+ // The `$` argument is the Cheerio object
+ // which contains parsed HTML of the website.
+ async requestHandler({ $, request }) {
+ // Extract <title> text with Cheerio.
+ // See Cheerio documentation for API docs.
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ }
+})
+// Start the crawler and wait for it to finish
await crawler.run();
```
-That starts to look almost good. Almost? Yes, you can do even better! Every crawler has an implicit `RequestQueue` instance, so you can save yourself from all the `RequestQueue` related code and just use `crawler.addRequests()` method. In fact, you can even go further and just use the first parameter of `crawler.run()`! You can also move the `requestHandler` method definition directly to the crawler options, this way you get type inference for free:
+When you run the example, you will see the title of https://crawlee.dev printed to the log. What really happens is that CheerioCrawler first makes an HTTP request to `https://crawlee.dev`, then parses the received HTML with Cheerio and makes it available as the `$` argument of the `requestHandler`.
+
+```log
+The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee.
+```
+
+### Add requests faster
+
+Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts
+```ts title="src/main.mjs"
+// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
-// Set up the crawler, passing a single options object as an argument.
const crawler = new CheerioCrawler({
- async requestHandler({ request, $ }) {
+ async requestHandler({ $, request }) {
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
+ }
+})
-// Enqueue the initial request and run the crawler
-await crawler.run(['https://apify.com']);
+// Start the crawler with the provided URLs
+await crawler.run(['https://crawlee.dev']);
```
-And you're done! You just created your first crawler from scratch. It will download the HTML of `https://apify.com`, find the `<title>` element, get its text content and print it to console. Good job!
+When you run this code, you'll see exactly the same output as with the earlier, longer example. The `RequestQueue` is still there, it's just managed by the crawler automatically.
-You should see the message `The title of "https://apify.com" is: Web Scraping, Data Extraction and Automation · Apify.` printed to the screen. If you do, congratulations and let's move onto some bigger challenges! And if you feel like you don't really know what just happened there, no worries, it will all become clear when you learn more about `CheerioCrawler`.
-
-:::info
-
-You are using a feature called [Top level await](https://blog.saeloun.com/2021/11/25/ecmascript-top-level-await.html) in your examples. To be able to use that, you might need some extra setup. Namely, it requires the use of [ECMAScript Modules](https://nodejs.org/api/esm.html) - this means you either need to add `"type": "module"` to your `package.json` file, or use `*.mjs` extension for your files. Additionally, if you are in a TypeScript project, you need to set the `module` and `target` compiler options to `ES2022` or above.
-
-:::
+## Next lesson
+In the next lesson you'll learn about crawling links. That means finding new URLs on the pages you crawl and adding them to the `RequestQueue` for the crawler to visit.
diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdx
new file mode 100644
index 000000000..52f2a99f0
--- /dev/null
+++ b/docs/introduction/03-adding-urls.mdx
@@ -0,0 +1,167 @@
+---
+id: adding-urls
+title: "Adding more URLs"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import FindWithELSource from '!!raw-loader!./03-find-el.ts';
+import FilterWithELSource from '!!raw-loader!./03-filter-el.ts';
+import FindWithoutELSource from '!!raw-loader!./03-find-without-el.ts';
+import FilterWithoutELSource from '!!raw-loader!./03-filter-without-el.ts';
+
+In the previous lesson you built a very simple crawler that downloads HTML of a single page, reads its title and prints
+it to the console. This is the original source code:
+
+```ts
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ $, request }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ }
+})
+
+await crawler.run(['https://crawlee.dev']);
+```
+
+In this lesson you'll use the example from the previous section and improve on it. You'll add more URLs to the queue and thanks to that the crawler will keep going, finding new links, enqueuing them into the `RequestQueue` and then scraping them.
+
+## How crawling works
+
+The process is simple:
+
+1. Find new links on the page.
+2. Filter only those pointing to the same domain, in this case `crawlee.dev`.
+3. Enqueue (add) them to the `RequestQueue`.
+4. Visit the newly enqueued links.
+5. Repeat the process.
+
+In the following paragraphs you will learn about the <ApiLink to="core/function/enqueueLinks">`enqueueLinks`</ApiLink> function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
+
+:::tip
+
+The `enqueueLinks` function is context aware. It means that it will read the information about the currently crawled page from the context, and you don't need to explicitly provide any arguments. It will find the links using the Cheerio function `$` and automatically add the links to the running crawler's `RequestQueue`.
+
+:::
+
+## Limit your crawls with `maxRequestsPerCrawl`
+
+ When you're just testing your code or when your crawler could potentially find millions of links, it's very useful to set a maximum limit of crawled pages. The option is called `maxRequestsPerCrawl`, is available in all crawlers, and you can set it like this:
+
+```ts
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ // ...
+});
+```
+
+This means that no new requests will be started after the 20th request is finished. The actual number of processed requests might be a little higher thanks to parallelization, because the running requests won't be forcefully aborted. It's not even possible in most cases.
+
+
+## Finding new links
+
+There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute because that's what you need in most cases. For example:
+
+```html
+<a href="https://crawlee.dev/docs/introduction">This is a link to Crawlee introduction</a>
+```
+
+Since this is the most common case, it is also the `enqueueLinks` default.
+
+<Tabs groupId="crawling-links">
+<TabItem value="enqueue-links" label="with enqueueLinks" default>
+<CodeBlock language="js" title="src/main.mjs">{FindWithELSource}</CodeBlock>
+</TabItem>
+<TabItem value="plain-js" label="without enqueueLinks">
+<CodeBlock language="js" title="src/main.mjs">{FindWithoutELSource}</CodeBlock>
+</TabItem>
+</Tabs>
+
+If you need to override the default selection of elements in `enqueueLinks`, you can use the `selector` argument.
+
+```ts
+await enqueueLinks({
+ selector: 'div.has-link'
+});
+```
+
+## Filtering links to same domain
+
+Websites typically contain a lot of links that lead away from the original page. This is normal, but when crawling a website, we usually want to crawl that one site and not let our crawler wander away to Google, Facebook and Twitter. Therefore, we need to filter out the off-domain links and only keep the ones that lead to the same domain.
+
+<Tabs groupId="crawling-links">
+ <TabItem value="enqueue-links" label="with enqueueLinks" default>
+ <CodeBlock language="js" title="src/main.mjs">{FilterWithELSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="plain-js" label="without enqueueLinks">
+ <CodeBlock language="js" title="src/main.mjs">{FilterWithoutELSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+The default behavior of `enqueueLinks` is to stay on the same hostname. This **does not include subdomains**. To include subdomains in your crawl, use the `strategy` argument.
+
+```ts
+await enqueueLinks({
+ strategy: 'same-domain'
+});
+```
+
+When you run the code, you will see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the first enqueued page and so on and so on.
+
+## Skipping duplicate URLs
+
+Skipping of duplicate URLs is critical, because visiting the same page multiple times would lead to duplicate results. This is automatically handled by the `RequestQueue` which deduplicates requests using their `uniqueKey`. This `uniqueKey` is automatically generated from the request's URL by lowercasing the URL, lexically ordering query parameters, removing fragments and a few other tweaks that ensure the queue only includes unique URLs.
+
+## Advanced filtering arguments
+
+While the defaults for `enqueueLinks` can be often exactly what you need, it also gives you fine-grained control over which URLs should be enqueued. One way we already mentioned above is by using the <ApiLink to="core/enum/EnqueueStrategy">`EnqueueStrategy`</ApiLink>. You can use the <ApiLink to="core/enum/EnqueueStrategy#All">`All`</ApiLink> strategy if you want to follow every single link, regardless of its domain, or you can enqueue links that target the same domain name with the <ApiLink to="core/enum/EnqueueStrategy#SameDomain">`SameDomain`</ApiLink> strategy.
+
+```ts
+await enqueueLinks({
+ strategy: 'all', // wander the internet
+});
+```
+
+### Filter URLs with patterns
+
+For even more control, you can use `globs`, `regexps` and `pseudoUrls` to filter the URLs. Each of those arguments is always an `Array`, but the contents can take on many forms. <ApiLink to="core/interface/EnqueueLinksOptions">See the reference</ApiLink> for more information about them as well as other options.
+
+:::caution
+
+If you provide one of those options, the default `same-hostname` strategy will **not** be applied unless explicitly set in the options.
+
+:::
+
+```ts
+await enqueueLinks({
+ globs: ['http?(s)://apify.com/*/*'],
+});
+```
+
+### Transform requests
+
+To have absolute control, we have the <ApiLink to="core/interface/EnqueueLinksOptions/#transformRequestFunction">`transformRequestFunction`</ApiLink>. Just before a new <ApiLink to="core/class/Request">`Request`</ApiLink> is constructed and enqueued to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, this function can be used to skip it or modify its contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+
+```ts
+await enqueueLinks({
+ globs: ['http?(s)://apify.com/*/*'],
+ transformRequestFunction(req) {
+ // ignore all links ending with `.pdf`
+ if (req.url.endsWith('.pdf')) return false;
+ return req;
+ },
+});
+```
+
+And that's it! `enqueueLinks()` is just one example of Crawlee's powerful helper functions. They're all designed to make your life easier, so you can focus on getting your data, while leaving the mundane crawling management to the tools.
+
+## Next lesson
+
+In the next lesson you will start your project of scraping a production website and learn some more Crawlee tricks in the process.
diff --git a/docs/introduction/03-cheerio-crawler.mdx b/docs/introduction/03-cheerio-crawler.mdx
deleted file mode 100644
index da4fae978..000000000
--- a/docs/introduction/03-cheerio-crawler.mdx
+++ /dev/null
@@ -1,227 +0,0 @@
----
-id: cheerio-crawler
-title: "CheerioCrawler aka jQuery crawler"
-sidebar_label: "CheerioCrawler"
-description: Your first steps into the world of scraping with Crawlee
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-This is the crawler that we used in our earlier example. Our simplest and also the fastest crawling solution. If you're familiar with `jQuery`, you'll understand <ApiLink to="cheerio-crawler/class/CheerioCrawler">CheerioCrawler</ApiLink> in minutes. [`Cheerio`](https://www.npmjs.com/package/cheerio) is essentially `jQuery` for Node.js. It offers the same API, including the familiar `$` object. You can use it, as you would use `jQuery`, for manipulating the DOM of an HTML page. In crawling, you'll mostly use it to select the right elements and extract their text values - the data you're interested in. But `jQuery` runs in a browser and attaches directly to the browser's DOM. Where does `cheerio` get its HTML? This is where the `Crawler` part of <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> comes in.
-
-### Overview
-
-&#8203;<ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> crawls by making plain HTTP requests to the provided URLs. As you remember from the previous section, the URLs are fed to the crawler using either the <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> or the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>. The HTTP responses it gets back are HTML pages, the same pages you would get in your browser when you first load a URL.
-
-> Note, however, that modern web pages often do not serve all of their content in the first HTML response, but rather the first HTML contains links to other resources such as CSS and JavaScript that get downloaded afterwards, and together they create the final page. See <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> to crawl those.
-
-Once the page's HTML is retrieved, the crawler will pass it to [Cheerio](https://www.npmjs.com/package/cheerio) for parsing. The result is the typical `$` function, which should be familiar to `jQuery` users. You can use this `$` to do all sorts of lookups and manipulation of the page's HTML, but in scraping, you will mostly use it to find specific HTML elements and extract their data.
-
-Example use of Cheerio and its `$` function in comparison to browser JavaScript:
-
-```ts
-// Return the text content of the <title> element.
-document.querySelector('title').textContent; // plain JS
-$('title').text(); // Cheerio
-
-// Return an array of all 'href' links on the page.
-Array.from(document.querySelectorAll('[href]')).map(el => el.href); // plain JS
-$('[href]')
- .map((i, el) => $(el).attr('href'))
- .get(); // Cheerio
-```
-
-> This is not to show that Cheerio is better than plain browser JavaScript. Some might actually prefer the more expressive way plain JS provides. Unfortunately, the browser JavaScript methods are not available in Node.js, so Cheerio is your best bet to do the parsing.
-
-### When to use `CheerioCrawler`
-
-Even though using `CheerioCrawler` is extremely easy, it probably will not be your first choice for most kinds of crawling or scraping in production environments. Since most websites nowadays use modern JavaScript to create rich, responsive and data-driven user experiences, the plain HTTP requests the crawler uses may just fall short of your needs.
-
-But <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink> is far from useless! It really shines when you need to cope with extremely high workloads. With just 4 GBs of memory and a single CPU core, you can scrape 500 or more pages a minute! _(assuming each page contains approximately 400KB of HTML)_. To scrape this fast with a full browser scraper, such as the <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, you'd need significantly more computing power.
-
-**Advantages:**
-
-- Extremely fast
-- Easy to set up
-- Familiar for jQuery users
-- Super cheap to run
-- Each request can go through a different proxy
-
-**Disadvantages:**
-
-- Does not work for all websites
-- May easily overload the target website with requests
-- Does not enable any manipulation of the website before scraping
-
-### Basic use of `CheerioCrawler`
-
-Now that you have an idea of the crawler's inner workings, let's build one. We'll use the example from the previous section and improve on it by letting it truly crawl the page, finding new links as it goes, enqueuing them into the `RequestQueue` and then scraping them.
-
-#### Refresher
-
-Just to refresh your memory, in the previous section you built a very simple crawler that downloads the HTML of a single page, reads its title and prints
-it to the console. This is the original source code:
-
-```ts
-import { CheerioCrawler } from 'crawlee';
-
-// Set up the crawler, passing a single options object as an argument.
-const crawler = new CheerioCrawler({
- async requestHandler({ request, $ }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
-
-// Enqueue the initial request and run the crawler
-await crawler.run(['https://apify.com']);
-```
-
-Earlier we said that we would let the crawler:
-
-1. Find new links on the page
-2. Filter only those pointing to the same hostname, in this case `apify.com`
-3. Enqueue them to the `RequestQueue`
-4. Scrape the newly enqueued links
-
-So let's get to it!
-
-#### Finding new links
-
-There are numerous approaches to finding links to follow when crawling the web. For our purposes, we will be looking for `<a>` elements that contain the `href` attribute. For example `<a href="https://apify.com/store">This is a link to Apify Store</a>`. To do this, we need to update our Cheerio function.
-
-```ts
-const links = $('a[href]')
- .map((i, el) => $(el).attr('href'))
- .get();
-```
-
-Our new function finds all the `<a>` elements that contain the `href` attribute and extracts the attributes into an array of strings. But there's a problem. There could be relative links in the list and those can't be used on their own. We need to resolve them using our domain as base URL and we will use one of Node.js' standard libraries to do this.
-
-Crawlee exposes two URL properties: <ApiLink to="core/class/Request#url">`request.url`</ApiLink> and <ApiLink to="core/class/Request#loadedUrl">`request.loadedUrl`</ApiLink>. What's the difference? Well, `request.url` is the URL of the first request. In case of redirects, the URL changes. The final one is stored as `request.loadedUrl`.
-
-```ts
-// At the top of the file:
-import { URL } from 'url';
-
-// ...
-
-const { hostname } = new URL(request.loadedUrl);
-const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-```
-
-#### Filtering links to same domain
-
-Websites typically contain a lot of links that lead away from the original page. This is normal, but when crawling a website, we usually want to crawl that one site and not let our crawler wander away to Google, Facebook and Twitter. Therefore, we need to filter out the off-domain links and only keep the ones that lead to the same domain.
-
-> Don't worry, we'll soon learn about much easier way to handle this.
-
-```ts
-// At the top of the file:
-import { URL } from 'url';
-
-// ...
-
-const links = $('a[href]')
- .map((i, el) => $(el).attr('href'))
- .get();
-
-const { hostname } = new URL(request.loadedUrl);
-const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-
-const sameDomainLinks = absoluteUrls.filter(url => url.hostname.endsWith(hostname));
-
-// ...
-```
-
-This includes subdomains. In order to filter the same origin, simply compare the `url.origin` property instead:
-
-```ts
-const { origin } = new URL(request.loadedUrl);
-const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-
-const sameDomainLinks = absoluteUrls.filter(url => url.origin === origin);
-```
-
-> The `URL` class contains many other useful properties. You can read more about `url.origin` [here](https://developer.mozilla.org/en-US/docs/Web/API/URL/origin).
-
-#### Enqueueing links to `RequestQueue`
-
-This should be easy, because we already did that [earlier](#putting-it-all-together), remember? Just call `requestQueue.addRequests()` for all the new links. This will add them to the end of the queue for processing.
-
-```ts
-// At the top of the file:
-import { URL } from 'url';
-
-// ...
-
-const links = $('a[href]')
- .map((i, el) => $(el).attr('href'))
- .get();
-
-const { origin } = new URL(request.loadedUrl);
-const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-
-const sameDomainLinks = absoluteUrls
- .filter(url => url.origin === origin)
- .map(url => ({ url: url.href }));
-
-// Add the requests in series. There's of course room for speed
-// improvement by parallelization. Try to implement it, if you wish.
-console.log(`Enqueueing ${sameDomainLinks.length} URLs.`);
-await requestQueue.addRequests(sameDomainLinks);
-
-// ...
-```
-
-#### Scrape the newly enqueued links
-
-And we're approaching the finishing line. All we need to do now is to integrate the new code into our original crawler. It will be easy, because almost everything needs to go into the `requestHandler`. But just before we do that, let's introduce the first crawler configuration option that is not a `requestHandler` or `requestQueue`. It's called `maxRequestsPerCrawl`.
-
-##### The `maxRequestsPerCrawl` limit
-
-This configuration option is available in all crawler classes, and you can use it to limit the number of requests the crawler should process. It's very useful when you're just testing your code or when your crawler could potentially crawl millions of pages, and you want to save resources. You can add it to the crawler options like this:
-
-```ts
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20,
- requestHandler,
-});
-```
-
-This limits the number of successfully handled requests to 20. Bear in mind that the actual number of processed requests might be a little higher and that's because usually there are multiple requests processed at the same time and once the 20th `Request` finishes, the other running requests will be allowed to finish too.
-
-#### Putting it all together
-
-```ts
-import { URL } from 'url'; // <------ This is new.
-import { CheerioCrawler } from 'crawlee';
-
-// Set up the crawler, passing a single options object as an argument.
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20, // <------ This is new too.
- async requestHandler({ request, $ }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
-
- // Here starts the new part of requestHandler.
- const links = $('a[href]')
- .map((i, el) => $(el).attr('href'))
- .get();
-
- const { origin } = new URL(request.loadedUrl);
- const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-
- const sameDomainLinks = absoluteUrls
- .filter(url => url.origin === origin)
- .map(url => ({ url: url.href} ));
-
- console.log(`Enqueueing ${sameDomainLinks.length} URLs.`);
- await crawler.addRequests(sameDomainLinks);
- },
-});
-
-await crawler.run(['https://apify.com']);
-```
-
-No matter if you followed along with your coding or just copy-pasted the resulting source, try running it now, perhaps even in both environments. You should see the crawler log the **title** of the first page, then the **enqueueing** message showing number of URLs, followed by the **title** of the first enqueued page and so on and so on.
diff --git a/docs/introduction/03-filter-el.ts b/docs/introduction/03-filter-el.ts
new file mode 100644
index 000000000..00c9b0f9a
--- /dev/null
+++ b/docs/introduction/03-filter-el.ts
@@ -0,0 +1,12 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ $, request, enqueueLinks }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/introduction/03-filter-without-el.ts b/docs/introduction/03-filter-without-el.ts
new file mode 100644
index 000000000..b945dd64b
--- /dev/null
+++ b/docs/introduction/03-filter-without-el.ts
@@ -0,0 +1,26 @@
+import { CheerioCrawler } from 'crawlee';
+import { URL } from 'node:url';
+
+const crawler = new CheerioCrawler({
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ request, $ }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+
+ // Without enqueueLinks, we first have to extract all
+ // the URLs from the page with Cheerio.
+ const links = $('a[href]')
+ .map((_, el) => $(el).attr('href'))
+ .get();
+
+ // Then we need to resolve relative URLs,
+ // otherwise they would be unusable for crawling.
+ const absoluteUrls = links
+ .map((link) => new URL(link, request.loadedUrl).href);
+
+ // Finally, we have to add the URLs to the queue
+ await crawler.addRequests(absoluteUrls);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/introduction/03-find-el.ts b/docs/introduction/03-find-el.ts
new file mode 100644
index 000000000..50983a24f
--- /dev/null
+++ b/docs/introduction/03-find-el.ts
@@ -0,0 +1,17 @@
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // Let's limit our crawls to make our
+ // tests shorter and safer.
+ maxRequestsPerCrawl: 20,
+ // enqueueLinks is an argument of the requestHandler
+ async requestHandler({ $, request, enqueueLinks }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+ // The enqueueLinks function is context aware,
+ // so it does not require any parameters.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/introduction/03-find-without-el.ts b/docs/introduction/03-find-without-el.ts
new file mode 100644
index 000000000..9b3aa6543
--- /dev/null
+++ b/docs/introduction/03-find-without-el.ts
@@ -0,0 +1,33 @@
+import { CheerioCrawler } from 'crawlee';
+import { URL } from 'node:url';
+
+const crawler = new CheerioCrawler({
+ // Let's limit our crawls to make our
+ // tests shorter and safer.
+ maxRequestsPerCrawl: 20,
+ async requestHandler({ request, $ }) {
+ const title = $('title').text();
+ console.log(`The title of "${request.url}" is: ${title}.`);
+
+ const links = $('a[href]')
+ .map((_, el) => $(el).attr('href'))
+ .get();
+
+ // Besides resolving the URLs, we now also need to
+ // grab their hostname for filtering.
+ const { hostname } = new URL(request.loadedUrl);
+ const absoluteUrls = links
+ .map((link) => new URL(link, request.loadedUrl));
+
+ // We use the hostname to filter links that point
+ // to a different domain, even subdomain.
+ const sameHostnameLinks = absoluteUrls
+ .filter((url) => url.hostname === hostname)
+ .map((url) => ({ url: url.href }));
+
+ // Finally, we have to add the URLs to the queue
+ await crawler.addRequests(sameHostnameLinks);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/introduction/04-enqueue-links.mdx b/docs/introduction/04-enqueue-links.mdx
deleted file mode 100644
index d5ea122e8..000000000
--- a/docs/introduction/04-enqueue-links.mdx
+++ /dev/null
@@ -1,145 +0,0 @@
----
-id: enqueue-links
-title: "Using Crawlee to enqueue links like a boss"
-sidebar_label: "Enqueuing links"
-description: Your first steps into the world of scraping with Crawlee
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-If you were paying attention carefully in the previous chapter, we said there is much easier way to enqueue new <ApiLink to="core/class/Request">`Request`</ApiLink> objects with a single function call. You might be wondering why you had to go through the whole process of getting the individual links, filtering the same domain ones and then manually enqueuing them into the `RequestQueue`, when there is a simpler way.
-
-Well, the obvious reason is practice. This is a tutorial after all. The other reason is to make you think about all the bits and pieces that come together, so that in the end, a new page, not previously entered in by you, can be scraped. We think that by seeing the bigger picture, you will be able to get the most out of Crawlee.
-
-### Introduction to `enqueueLinks()`
-
-Since enqueuing new links to crawl is such an integral part of web crawling, we created a function that attempts to simplify this process as much as possible. With a single function call, it allows you to find all the links on a page that match specified criteria and add them to a `RequestQueue`. It also allows you to modify the resulting requests to match your crawling needs.
-
-`enqueueLinks` is quite a powerful function so, like crawlers, it gets its arguments from the `options` object. This is useful, because you don't have to remember their order! But also because we can easily extend its API and add new features. You can <ApiLink to="core/function/enqueueLinks">find the full reference here</ApiLink>.
-
-```ts
-import { enqueueLinks } from 'crawlee';
-
-// Now you can use enqueueLinks like this:
-await enqueueLinks({
- /* options */
-});
-```
-
-### Basic usage of `enqueueLinks()` with `CheerioCrawler`
-
-We already implemented logic that takes care of enqueueing new links to a `RequestQueue` in the previous chapter on `CheerioCrawler`. Let's look at that logic and implement the same functionality using `enqueueLinks()`.
-
-We found that the crawler needed to do these 4 things to crawl `apify.com` website:
-
-1. Find new links on the page
-2. Filter only those pointing to `apify.com`
-3. Enqueue them to the `RequestQueue`
-4. Scrape the newly enqueued links
-
-Using `enqueueLinks()` we can squash the first 3 into a single function call. We will use the context bound `enqueueLinks` variant:
-
-```ts
-import { CheerioCrawler } from 'crawlee';
-
-const crawler = new CheerioCrawler({
- async requestHandler({ enqueueLinks }) {
- await enqueueLinks();
- },
-});
-
-await crawler.run(['https://apify.com']);
-```
-
-Wait, what? Yes! That's all we need to find and enqueue all the links on currently handled page. We are using the context bound variant (the one from the `requestHandler`'s first context parameter), so it already knows what the current request body looks like, how to extract the links from it, how to handle relative links, it uses the crawler's `RequestQueue` (and creates a default one if you don't provide your own). By default, it will enqueue only links that target the same hostname (via the so-called <ApiLink to="core/enum/EnqueueStrategy#SameHostname">`SameHostname`</ApiLink> <ApiLink to="core/enum/EnqueueStrategy">EnqueueStrategy</ApiLink>).
-
-#### Using `enqueueLinks()` to filter links
-
-While the defaults for `enqueueLinks` can be often exactly what you need, it also gives you fine-grained control over _what_ links should be enqueued. One way we already mentioned above is by using the <ApiLink to="core/enum/EnqueueStrategy">`EnqueueStrategy`</ApiLink>. You can use the <ApiLink to="core/enum/EnqueueStrategy#All">`All`</ApiLink> strategy if you want to follow every single link, regardless of its domain, or you can enqueue links that target the same domain name with the <ApiLink to="core/enum/EnqueueStrategy#SameDomain">`SameDomain`</ApiLink> strategy.
-
-```ts
-await enqueueLinks({
- strategy: EnqueueStrategy.SameDomain, // or 'same-domain'
-});
-```
-
-For even more control, you can use `globs`, `regexps` and `pseudoUrls` to filter the URLs. Each of those arguments is always an `Array`, but the contents can take on many forms. <ApiLink to="core/interface/EnqueueLinksOptions">See the reference</ApiLink> for more information about them as well as other options.
-
-> If you provide one of those options, the default `SameHostname` strategy will **not** be applied unless explicitly set in the options.
-
-```ts
-await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
-});
-```
-
-To have absolute control, we have the <ApiLink to="core/interface/EnqueueLinksOptions/#transformRequestFunction">`transformRequestFunction`</ApiLink>. Just before a new <ApiLink to="core/class/Request">`Request`</ApiLink> is constructed and enqueued to the <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink>, this function can be used to skip it or modify its contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
-
-```ts
-await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
- transformRequestFunction(req) {
- // ignore all links ending with `.html`
- if (req.url.endsWith('.html')) return false;
- return req;
- },
-});
-```
-
-#### Integrating `enqueueLinks()` into our crawler
-
-Now let's get back to our crawler. Let's take a look at the original crawler code, where we enqueued all the links manually.
-
-```ts
-import { URL } from 'url';
-import { CheerioCrawler } from 'crawlee';
-
-// Set up the crawler, passing a single options object as an argument.
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20,
- async requestHandler({ request, $ }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
-
- // Here starts the new part of requestHandler.
- const links = $('a[href]')
- .map((i, el) => $(el).attr('href'))
- .get();
-
- const { origin } = new URL(request.loadedUrl);
- const absoluteUrls = links.map(link => new URL(link, request.loadedUrl));
-
- const sameDomainLinks = absoluteUrls
- .filter(url => url.origin === origin)
- .map(url => ({ url: url.href} ));
-
- console.log(`Enqueueing ${sameDomainLinks.length} URLs.`);
- await crawler.addRequests(sameDomainLinks);
- },
-});
-
-await crawler.run(['https://apify.com']);
-```
-
-Since we've already learned that the context bound `enqueueLinks()` function does exactly what we need, we can just replace all the above enqueuing logic with a single function call, as promised.
-
-```ts
-import { CheerioCrawler } from 'crawlee';
-
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20,
- async requestHandler({ request, $, enqueueLinks }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
-
- const enqueued = await enqueueLinks();
- console.log(`Enqueued ${enqueued.length} URLs.`);
- },
-});
-
-await crawler.run(['https://apify.com']);
-```
-
-And that's it! No more parsing the links from HTML using Cheerio, filtering them and enqueueing them one by one. It all gets done automatically! `enqueueLinks()` is just one example of Crawlee's powerful helper functions. They're all designed to make your life easier, so you can focus on getting your data, while leaving the mundane crawling management to the tools.
-
-`enqueueLinks()` has a lot more tricks up its sleeve. Make sure to check out the <ApiLink to="core/interface/EnqueueLinksOptions">options reference</ApiLink> to see what else it can do for you. Namely the feature to prepopulate the `Request` instances it creates with `userData` of your choice is extremely useful!
diff --git a/docs/introduction/04-pw-w-cheerio.ts b/docs/introduction/04-pw-w-cheerio.ts
new file mode 100644
index 000000000..2873909e8
--- /dev/null
+++ b/docs/introduction/04-pw-w-cheerio.ts
@@ -0,0 +1,21 @@
+// Instead of CheerioCrawler let's use Playwright
+// to be able to render JavaScript.
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, parseWithCheerio }) => {
+ // Wait for the actor cards to render.
+ await page.waitForSelector('.ActorStoreItem');
+ // Extract the page's HTML from browser
+ // and parse it with Cheerio.
+ const $ = await parseWithCheerio();
+ // Use familiar Cheerio syntax to
+ // select all the actor cards.
+ $('.ActorStoreItem').each((i, el) => {
+ const text = $(el).text();
+ console.log(`ACTOR_${i + 1}: ${text}\n`);
+ });
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/introduction/04-pw.ts b/docs/introduction/04-pw.ts
new file mode 100644
index 000000000..8260549a3
--- /dev/null
+++ b/docs/introduction/04-pw.ts
@@ -0,0 +1,21 @@
+// Instead of CheerioCrawler let's use Playwright
+// to be able to render JavaScript.
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page }) => {
+ // Wait for the actor cards to render.
+ await page.waitForSelector('.ActorStoreItem');
+ // Execute a function in the browser which targets
+ // the actor card elements and allows their manipulation.
+ const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {
+ // Extract text content from the actor cards
+ return els.map((el) => el.textContent);
+ });
+ actorTexts.forEach((text, i) => {
+ console.log(`ACTOR_${i + 1}: ${text}\n`);
+ });
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
diff --git a/docs/introduction/04-real-world-project.mdx b/docs/introduction/04-real-world-project.mdx
new file mode 100644
index 000000000..a54e29bea
--- /dev/null
+++ b/docs/introduction/04-real-world-project.mdx
@@ -0,0 +1,176 @@
+---
+id: real-world-project
+title: "Getting some real-world data"
+sidebar_label: "Real-world project"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import PwSource from '!!raw-loader!./04-pw.ts';
+import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
+
+> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+
+We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
+
+## Making a production-grade crawler
+
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.
+
+The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
+
+:::tip
+
+If you're not interested in crawling theory, feel free to [skip to the next lesson](./crawling) and get right back to coding.
+
+:::
+
+## Drawing a plan
+
+Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
+
+## Choosing the data you need
+
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:
+
+- URL
+- Owner
+- Unique identifier (such as `apify/web-scraper`)
+- Title
+- Description
+- Last modification date
+- Number of runs
+
+You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
+
+![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
+
+### The start URL(s)
+
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+
+## Exploring the page
+
+Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+
+### Categories and sorting
+
+When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+
+:::caution
+
+Be careful, because on some websites, like [amazon.com](https://amazon.com), this is not true and the sum of products in categories is actually larger than what's available without filters. Learn more in our [tutorial on scraping websites with limited pagination](https://docs.apify.com/tutorials/scrape-paginated-sites).
+
+:::
+
+### Pagination
+
+The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+
+```
+https://apify.com/store?page=2
+```
+
+Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+
+:::caution
+
+Similarly to the issue with filters explained above, the existence of pagination does not guarantee that you can simply paginate through all the results. Always test your assumptions about pagination. Otherwise, you might miss a chunk of results, and not even know about it.
+
+:::
+
+At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
+
+```
+https://apify.com/store?page=44
+```
+
+It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.
+
+At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+
+If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+
+## The crawling strategy
+
+Now that you know where to start and how to find all the actor details, let's look at the crawling process.
+
+1. Visit the store page containing the list of actors (our start URL).
+2. Enqueue all links to actor details.
+3. Enqueue links to next pages of results.
+4. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's an actor detail page, scrape the data.
+5. Repeat until all results pages and all actor details have been processed.
+
+`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
+
+## Sanity check
+
+Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
+
+The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.
+
+<Tabs groupId="sanity-check">
+ <TabItem value="playwright" label="Playwright" default>
+ <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ </TabItem>
+ <TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
+ <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ </TabItem>
+</Tabs>
+
+If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.
+
+## DevTools - the scrapers toolbox
+
+:::info
+
+We'll use Chrome DevTools here, since it's the most common browser, but feel free to use any other, they're all very similar.
+
+:::
+
+Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+
+## Selecting elements
+
+In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.
+
+![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
+
+You'll see that you can select different elements inside the card. Instead, select the whole card, not just some of its contents, such as its title or description.
+
+![selected element](/img/getting-started/selected-element.png 'Selecting an element by hovering over it.')
+
+Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
+
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.
+
+It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
+
+```ts
+document.querySelectorAll('.ActorStoreItem');
+```
+
+You will see that only the 24 actor cards will be returned, and nothing else.
+
+:::tip
+
+CSS selectors and DevTools are quite a big topic. If you want to learn more, visit the [Web scraping for beginners course](https://developers.apify.com/academy/web-scraping-for-beginners) in the Apify Academy. **It's free and open-source** ❤️.
+
+:::
+
+## Next lesson
+
+In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.
diff --git a/docs/introduction/05-crawling.mdx b/docs/introduction/05-crawling.mdx
new file mode 100644
index 000000000..979779f7e
--- /dev/null
+++ b/docs/introduction/05-crawling.mdx
@@ -0,0 +1,92 @@
+---
+id: crawling
+title: "Crawling the Store"
+sidebar_label: "Crawling"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.
+
+## Crawling the listing pages
+
+In previous lessons, you used the `enqueueLinks()` function like this:
+
+```js
+await enqueueLinks();
+```
+
+While useful in that scenario, you need something different now. Instead of finding all the `<a href="..">` elements with links to the same hostname, you need to find only the specific ones that will take your crawler to the next page of results. Otherwise, the crawler will visit a lot of other pages that you're not interested in. Using the power of DevTools and yet another `enqueueLinks()` parameter, this becomes fairly easy.
+
+```ts
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`)
+ // Wait for the actor cards to render,
+ // otherwise enqueueLinks wouldn't enqueue anything.
+ await page.waitForSelector('.ActorStorePagination-pages a');
+
+ // Add links to the queue, but only from
+ // elements matching the provided selector.
+ await enqueueLinks({
+ selector: '.ActorStorePagination-pages > a',
+ label: 'LIST',
+ })
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
+```
+
+The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
+
+### The `selector` parameter of `enqueueLinks()`
+
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.
+
+### The `label` of `enqueueLinks()`
+
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+
+## Crawling the detail pages
+
+In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+
+```ts
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`)
+ if (request.label === 'DETAIL') {
+ // We're not doing anything with the details yet.
+ } else {
+ // This means we're either on the start page, with no label,
+ // or on a list page, with LIST label.
+
+ await page.waitForSelector('.ActorStorePagination-pages a');
+ await enqueueLinks({
+ selector: '.ActorStorePagination-pages > a',
+ label: 'LIST',
+ })
+
+ // In addition to adding the listing URLs, we now also
+ // add the detail URLs from all the listing pages.
+ await page.waitForSelector('.ActorStoreItem');
+ await enqueueLinks({
+ selector: '.ActorStoreItem',
+ label: 'DETAIL', // <= note the different label
+ })
+ }
+ },
+});
+
+await crawler.run(['https://apify.com/store']);
+```
+
+The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
+
+## Next lesson
+
+This concludes the Crawling lesson, because you have taught the crawler to visit all the pages it needs. Let's continue with scraping data.
diff --git a/docs/introduction/05-realworld-example.mdx b/docs/introduction/05-realworld-example.mdx
deleted file mode 100644
index e9e941d17..000000000
--- a/docs/introduction/05-realworld-example.mdx
+++ /dev/null
@@ -1,717 +0,0 @@
----
-id: realworld-example
-title: "Getting some real-world data"
-sidebar_label: "Real-world example"
-description: Your first steps into the world of scraping with Crawlee
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why we started reading this tutorial in the first place!*
-
-We hear you, young padawan! First, learn how to crawl, we must. Only then, walk through data, we can!
-
-### Making a store crawler
-
-Fortunately, you don't have to travel to a galaxy far far away to find a good candidate for learning how to scrape structured data. The [Apify Store](https://apify.com/store) is a library of public actors that anyone can grab and use. You can find ready-made solutions for crawling [Google Places](https://apify.com/drobnikj/crawler-google-places), [Amazon](https://apify.com/vaclavrut/amazon-crawler), [Google Search](https://apify.com/apify/google-search-scraper), [Booking](https://apify.com/dtrungtin/booking-scraper), [Instagram](https://apify.com/jaroslavhejlek/instagram-scraper), [Tripadvisor](https://apify.com/maxcopell/tripadvisor) and many other websites. Feel free to check them out! It's also a great place to practice your Jedi scraping skills since it has categories, lists and details.
-
-### The importance of having a plan
-
-Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Would I need a full browser solution?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
-
-Jakub, one of Apify's founders, wrote a [great article](https://blog.apify.com/web-scraping-in-2018-forget-html-use-xhrs-metadata-or-javascript-variables-8167f252439c) about all the different techniques, tips and tricks related to web scraping, so make sure to check that out!
-
-For the purposes of this tutorial, let's just go ahead with HTTP requests and HTML parsing using `CheerioCrawler`. The number one reason being: you already know how to use it, and we want to build on that knowledge to learn specific crawling and scraping techniques.
-
-#### Choosing the data you need
-
-A good first step is always to figure out what is it you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) in all categories (which can be found on the left side of the page) and for each actor we want to get its:
-
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
-
-You can notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
-
-![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-
-#### Analyzing the target
-
-Knowing that we will use plain HTTP requests, we immediately know that we won't be able to manipulate the website in any way. We will only be able to go through the HTML it gives us and parse our data from there. This might sound like a huge limitation, but you might be surprised in how effective it can be. Let's get to it!
-
-#### The start URL(s)
-
-This is where you start your crawl. It's convenient to start as close to our data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
-
-Once we look at the `apify.com/store` page more carefully, we see that the categories themselves produce URLs that we can use to access those individual categories.
-
-```
-https://apify.com/store?category=AUTOMATION
-```
-
-Should we write down all the category URLs down and use all of them as start URLs? It's definitely possible, but what if a new category appears on the page later? You would not know about it unless you manually visit the page and inspect it again. So scraping the category links off the store page certainly makes sense. This way you always get an up-to-date list of categories.
-
-But is it really that straightforward? By digging further into the store page's HTML we can eventually find out that it does not actually contain the category links. The menu on the left uses JavaScript to display the items from a given category and, as we've learned earlier, `CheerioCrawler` cannot execute JavaScript. Moreover - the page loads the actors when opened in the browser, but it does not "show" us any when processed with a plain HTTP request.
-
-> We've deliberately chosen this scenario to show an example of the number one weakness of `CheerioCrawler`. We will overcome this difficulty in our `PlaywrightCrawler` tutorial, but at the cost of compute resources and speed. Always remember that no tool is best for everything!
-
-So firstly, let's agree that within the `CheerioCrawler` tutorial we would scrape all public actors made by Apify from the [Apify team account](https://apify.com/apify) page. Secondly, it means that we're back to the pre-selected list of URLs. Since we cannot scrape the list dynamically, we have to manually collect the links and then use them in our crawler. We lose the ability to scrape all the categories/actor details and instead scrape public actors of a specific user, but we keep the low resource consumption and speed advantages of `CheerioCrawler`.
-
-Therefore, we've determined that we should use Apify team account page as a start URL and that it should look as follows:
-
-```
-https://apify.com/apify
-```
-
-### The crawling strategy
-
-Now that we know where to start, we need to figure out where to go next. Since we've eliminated one level of crawling by selecting a specific user page manually, we now only need to crawl the actor detail pages. The algorithm therefore follows:
-
-1. Visit the user page containing the list of their public actors (our start URL).
-2. Enqueue all links to actor details.
-3. Visit all actor details and extract data.
-4. Repeat 1 - 3 for all start URLs (we have only one in this tutorial, but feel free to try it for more user pages).
-
-> Technically, this is a depth first crawl and the crawler will perform a breadth first crawl by default, but that's an implementation detail. We've chosen this notation since a breadth first crawl would be less readable.
-
-`CheerioCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are two more tricks that we'd like to showcase.
-
-#### Using a `RequestList`
-
-`RequestList` is a perfect tool for scraping a pre-existing list of URLs and if we think about our start URLs, this is exactly what we have! A list of links to the different user pages. Let's see how we'd get them into a `RequestList`.
-
-```js
-const sources = [
- 'https://apify.com/apify',
-];
-
-const requestList = await RequestList.open('users', sources);
-```
-
-As you can see, similarly to the `RequestQueue.open()` function, there is a `RequestList.open()` function that will create a `RequestList` instance for you. The first argument is the name of the `RequestList` (or an options object). It is used to persist the crawling state of the list. This is useful when you want to continue where you left off after an error or a process restart. The second argument is the `sources` array, which is nothing more than a list of URLs you wish to crawl.
-
-> `RequestQueue` is a persistent store by default, so no name is needed, while the `RequestList` only lives in memory and giving it a name enables it to become persistent.
-
-You might now want to ask one of these questions:
-- Can I enqueue into `RequestList` too?
-- How do I make `RequestList` work together with `RequestQueue` since I need the queue to enqueue new requests.
-
-The answer to the first one is a definitive no. `RequestList` is immutable and once you create it, you cannot add or remove requests from it. The answer to the second one is simple. `RequestList` and `RequestQueue` are made to work together out-of-the-box in crawlers, so all you need to do is use them both and the crawlers will do the rest.
-
-```js
-const crawler = new CheerioCrawler({
- requestList,
- requestQueue,
- requestHandler,
-});
-```
-
-> For those wondering how this works, the `RequestList` requests are enqueued into the `RequestQueue` right before their execution and only processed by the `RequestQueue` afterwards.
-
-:::tip Prefer `crawler.addRequests()`
-
-While `RequestList` might be a good fit for some use cases like handling very large input lists, or when you don't need any dynamic enqueuing at all, using `crawler.addRequests()` should be generally the preferred way to handle enqueuing of initial requests. It will enqueue the initial batch of 1000 requests and resolve right after that, so it won't block the start of crawling. Instead, it will continue adding more requests to the queue in batches in the background while the crawler runs.
-
-:::
-
-Following section is here mainly to show how to work with `RequestList`.
-
-#### Sanity check
-
-It's always useful to create some simple boilerplate code to see that you've got everything set up correctly before you start to write the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-
-Let's use your newly acquired `RequestList` knowledge and everything you know from the previous chapters to create a new crawler that'll just visit all the user page URLs we selected and print the text content of all the public actors of the user. Try running the code below in your selected environment. You should see, albeit very badly formatted, the text of the individual public actor cards of the selected user.
-
-```ts
-import { CheerioCrawler, RequestList } from 'crawlee';
-
-const sources = [
- 'https://apify.com/apify',
-];
-
-const requestList = await RequestList.open('users', sources);
-
-const crawler = new CheerioCrawler({
- requestList,
- requestHandler: async ({ $ }) => {
- // Select all the actor cards.
- $('.ActorStoreItem').each((i, el) => {
- const text = $(el).text();
- console.log(`ITEM: ${text}\n`);
- });
- },
-});
-
-await crawler.run();
-```
-
-You might be wondering how we got that `.ActorStoreItem` selector. After analyzing the user page using a browser's DevTools, we've determined that it's a good selector to select all the currently displayed actor cards. DevTools and CSS selectors are quite a large topic, so we can't go into too much detail now, but here are a few general pointers.
-
-#### DevTools crash course
-
-> We'll use Chrome DevTools here, since it's the most common browser, but feel free to use any other, it's all very similar.
-
-You could pick any user page, but let's just go with Apify team account. Let's open `https://apify.com/apify` in Chrome and open DevTools either by right-clicking anywhere in the page and selecting `Inspect`, or by pressing `F12` or by any other means relevant to our system. Once we're there, we'll see a bunch of DevToolsy stuff and a view of the user's page with the individual public actor cards.
-
-Now, let's find the `Select an element` tool and use it to select one of the actor cards. You need to make sure to select the whole card, not just some of its contents, such as its title or description.
-
-In the resulting HTML display, it will put your cursor somewhere. Inspecting the HTML around it, you'll see that there are some CSS classes attached to the different HTML elements.
-
-By hovering over the individual elements, you will see their placement in the page's view. It's easy to see the page's structure around the actor cards now. All the cards are displayed in a `<div>` with a `data-test` attribute with `actorCard` value, which holds an `<a>` with some computer-generated class names and also with the class of `ActorStoreItem`.
-
-> Yes, there are other HTML elements and other classes too. You can safely ignore them.
-
-It should now make sense how we got that `.ActorStoreItem` selector. It's just a selector that finds all elements that are annotated with the `ActorStoreItem` class and those just happen to be the actor cards only.
-
-It's always a good idea to double-check that though, so go into the DevTools Console and run
-
-```ts
-document.querySelectorAll('.ActorStoreItem');
-```
-
-You will see that only the actor cards will be returned, and nothing else.
-
-#### Enqueueing the detail links using a custom selector
-
-In the previous chapter, we used the `enqueueLinks()` function like this:
-
-```js
-await enqueueLinks();
-```
-
-While very useful in that scenario, we need something different now. Instead of finding all the `<a href="..">` targeting the same hostname, we need to find only the specific ones that will take us to the actor detail pages. Otherwise, we'd be visiting a lot of other pages that we're not interested in. Using the power of DevTools and yet another `enqueueLinks()` parameter, this becomes fairly easy.
-
-```ts
-import { CheerioCrawlingContext } from 'crawlee';
-
-export async function requestHandler({ request, enqueueLinks }: CheerioCrawlingContext) {
- console.log(`Processing ${request.url}`);
-
- // Only enqueue new links from the user pages.
- if (!request.userData.detailPage) {
- await enqueueLinks({
- selector: 'div[data-test=actorCard] > a.ActorStoreItem',
- userData: { detailPage: true },
- });
- }
-}
-```
-
-The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
-
-##### The `selector` parameter of `enqueueLinks()`
-
-When we previously used `enqueueLinks()`, we were not providing any `selector` parameter, and it was fine, because we wanted to use the default value, which is `a` - finds all `<a>` elements. But now, we need to be more specific. There are multiple `<a>` links on the given user page, and we're only interested in those that will take us to item (actor) details. Using the DevTools, we found out that we can select the links we wanted using the `div[data-test=actorCard] > a.ActorStoreItem` selector, which selects all the `<a class="ActorStoreItem ...">` elements. And those are exactly the ones we're interested in.
-
-##### Finally, the `userData` of `enqueueLinks()`
-
-You will see `userData` used often throughout Crawlee, and it's nothing more than a place to store our own data on a `Request` instance. You can access it with `request.userData` and it's a plain `Object` that can be used to store anything that needs to survive the full life-cycle of the `Request`. You can also use `Request.label` shortcut, that under the hood controls the `userData.label`.
-
-To modify all the `Request` instances before enqueueing, you can use the `transformRequestFunction` option of `enqueueLinks()`. In our case, we use it to set a `userData.detailPage` property to the enqueued requests to be able to easily differentiate between the user pages and the detail pages.
-
-#### Another sanity check
-
-It's always good to work step by step. We have this new enqueueing logic in place and since the previous [Sanity check](#sanity-check) worked only with a `RequestList`, because we were not enqueueing anything, don't forget to add back the `RequestQueue` and `maxRequestsPerCrawl` limit. Let's test it out!
-
-```ts
-import { CheerioCrawler, RequestList, RequestQueue } from 'crawlee';
-
-const sources = [
- 'https://apify.com/apify',
-];
-
-const requestList = await RequestList.open('users', sources);
-const requestQueue = await RequestQueue.open(); // <---------------------
-
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20, // <----------------------------------------
- requestList,
- requestQueue, // <---------------------------------------------------
- async requestHandler({ request, enqueueLinks }) {
- console.log(`Processing ${request.url}`);
-
- // Only enqueue new links from the user pages.
- if (!request.userData.detailPage) {
- await enqueueLinks({
- selector: 'div[data-test=actorCard] > a.ActorStoreItem',
- userData: { detailPage: true },
- });
- }
- },
-});
-
-await crawler.run();
-```
-
-We've added the `requestHandler()` with the `enqueueLinks()` logic from the previous section to the code we wrote earlier. As always, try running it in the environment of your choice. You should see the crawler logging a number of links to the console, as it crawls the user page first and then all the links to the actor detail pages it found.
-
-This concludes our Crawling strategy section, because we have taught the crawler to visit all the pages we need. Let's continue with scraping the tasty data.
-
-### Scraping data
-
-At the beginning of this chapter, we created a list of the information we wanted to collect about the actors in the store. Let's review that and figure out ways to access it.
-
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
-
-![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-
-#### Scraping the URL, Owner and Unique identifier
-
-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
-
-> You could also use the `request.loadedUrl` instead. Remember the difference: `request.url` is what you enqueue, `request.loadedUrl` is what gets processed (after possible redirects).
-
-```js
-// request.url = https://apify.com/apify/web-scraper
-
-const urlArr = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlArr.join('/'); // 'apify/web-scraper'
-const owner = urlArr[0]; // 'apify'
-```
-
-> It's always a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should we duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's always better to make the dataset as readable as possible. Someone might want to filter for example by `owner` and keeping only the `URL` in the dataset would make this complicated without using additional tools.
-
-#### Scraping Title, Description, Last modification date and Number of runs
-
-Now it's time to add more data to the results. Let's open one of the public actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu to figure out how to get the title of the actor.
-
-##### Title
-
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
-
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. However, in some cases there could actually be more `<h1>` tags on the same page. This should get you thinking. Is there any parent element that includes your `<h1>` tag, but not the other ones, i.e. narrows down the matching elements? Yes, there is! There is a `<header>` element that we can use to select only the heading we're interested in. And although in this case it's not really necessary, it's still a good idea to use more specific selectors to make sure the crawler would work as expected even if there will be some changes in the page HTML after some time.
-
-> Remember that you can press CTRL+F (or CMD+F on Mac) in the Elements tab of DevTools to open the search bar where you can quickly search for elements using their selectors. And always make sure to use the DevTools to verify your scraping process and assumptions. It's faster than changing the crawler code all the time.
-
-To get the title we just need to find it using `Cheerio` and a `header h1` selector, which selects all `<h1>` elements that have a `<header>` ancestor. And as we already know, there's only one.
-
-```js
-return {
- title: $('header h1').text(),
-};
-```
-
-##### Description
-
-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find out that the actor description is nested within the `<header>` element too, same as the title. Moreover, the actual description is nested inside a `<span>` tag with a class `actor-description`.
-
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
-
-```js
-return {
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
-};
-```
-
-##### Last modification date
-
-The DevTools tell us that the `modifiedDate` can be found in the `<time>` element inside `<ul class="ActorHeader-stats">`.
-
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
-
-```js
-return {
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- // or you can do just: new Date(+$('ul.ActorHeader-stats time').attr('datetime'))
- ),
-};
-```
-
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
-
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
-
-##### Run count
-
-And so we're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
-
-```js
-return {
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- ),
- runCount: Number(
- $('ul.ActorHeader-stats > li:nth-of-type(3)')
- .text()
- .match(/[\d,]+/)[0]
- .replaceAll(',', ''),
- ),
-};
-```
-
-The `ul.ActorHeader-stats > li:nth-of-type(3)` looks complicated, but it only reads that we're looking for a `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the third `<li>` element. We grab its text, but we're only interested in the number of runs. So we parse the number out using a regular expression, but its type is still a `string`, so we finally convert the result to a `number` by wrapping it with a `Number()` call.
-
-> The numbers are formatted with commas as thousands separators (e.g. `'1,234,567'`), so to extract it, we first use regular expression `/[\d,]+/` - it will search for consecutive number or comma characters. Then we extract the match via `.match(/[\d,]+/)[0]` and finally remove the commas by calling `.replaceAll(',', '')`. This will give us a string (e.g. `'1234567'`) that can be converted via `Number` function.
-
-And there we have it! All the data we needed in a single object. For the sake of completeness, let's add the properties we parsed from the URL earlier, and we're good to go.
-
-```js
-const urlArr = request.url.split('/').slice(-2);
-
-const results = {
- url: request.url,
- uniqueIdentifier: urlArr.join('/'),
- owner: urlArr[0],
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- ),
- runCount: Number(
- $('ul.ActorHeader-stats > li:nth-of-type(3)')
- .text()
- .match(/[\d,]+/)[0]
- .replaceAll(',', ''),
- ),
-};
-
-console.log('RESULTS: ', results);
-```
-
-#### Trying it out (sanity check #3)
-
-We have everything we need so just grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-
-```js
-import { CheerioCrawler, RequestList, RequestQueue } from 'crawlee';
-
-const sources = [
- 'https://apify.com/apify',
-];
-
-const requestList = await RequestList.open('users', sources);
-const requestQueue = await RequestQueue.open();
-
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20,
- requestList,
- requestQueue,
- async requestHandler({ $, request, enqueueLinks }) {
- console.log(`Processing ${request.url}`);
-
- // This is our new scraping logic.
- if (request.userData.detailPage) {
- const urlArr = request.url.split('/').slice(-2);
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlArr.join('/'),
- owner: urlArr[0],
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- ),
- runCount: Number(
- $('ul.ActorHeader-stats > li:nth-of-type(3)')
- .text()
- .match(/[\d,]+/)[0]
- .replaceAll(',', ''),
- ),
- };
- console.log('RESULTS', results);
- }
-
- // Only enqueue new links from the user pages.
- if (!request.userData.detailPage) {
- await enqueueLinks({
- selector: 'div[data-test=actorCard] > a.ActorStoreItem',
- userData: { detailPage: true },
- });
- }
- },
-});
-
-await crawler.run();
-```
-
-> Notice again that we're scraping on the detail pages `request.userData.detailPage === true`, but we're only enqueueing on the user pages `request.userData.detailPage === undefined`.
-
-When running the actor in the environment of your choice, you should see the crawled URLs and their scraped data printed to the console.
-
-### Saving the scraped data
-
-A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this chapter so make sure to pay attention very carefully!
-
-First, replace the `console.log('RESULTS', results)` call with
-
-```js
-import { Dataset } from 'crawlee';
-
-await Dataset.pushData(results);
-```
-
-and that's it. Unlike in the previous paragraph, we are being serious now. That's it, we're done. The final code therefore looks exactly like this:
-
-```js
-import { CheerioCrawler, RequestList, RequestQueue, Dataset } from 'crawlee';
-
-const sources = [
- 'https://apify.com/apify',
-];
-
-const requestList = await RequestList.open('users', sources);
-const requestQueue = await RequestQueue.open();
-
-const crawler = new CheerioCrawler({
- maxRequestsPerCrawl: 20,
- requestList,
- requestQueue,
- async requestHandler({ $, request, enqueueLinks }) {
- console.log(`Processing ${request.url}`);
-
- // This is our new scraping logic.
- if (request.userData.detailPage) {
- const urlArr = request.url.split('/').slice(-2);
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlArr.join('/'),
- owner: urlArr[0],
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- ),
- runCount: Number(
- $('ul.ActorHeader-stats > li:nth-of-type(3)')
- .text()
- .match(/[\d,]+/)[0]
- .replaceAll(',', ''),
- ),
- };
- await Dataset.pushData(results);
- }
-
- // Only enqueue new links from the user pages.
- if (!request.userData.detailPage) {
- await enqueueLinks({
- selector: 'div[data-test=actorCard] > a.ActorStoreItem',
- userData: { detailPage: true },
- });
- }
- },
-});
-
-await crawler.run();
-```
-
-#### What's `Dataset.pushData()`
-
-&#8203;<ApiLink to="core/class/Dataset#pushData">`Dataset.pushData()`</ApiLink> is a helper function that saves data to the default <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>. `Dataset` is a storage designed to hold virtually unlimited amount of data in a format similar to a table. Each time you call `Dataset.pushData()` a new row in the table is created, with the property names serving as column titles.
-
-> Each crawler run has one default `Dataset` so no need to initialize it or create an instance first. It just gets done automatically for you. You can also create named datasets at will.
-
-#### Finding your saved data
-
-It might not be perfectly obvious where the data you saved using the previous command went. Unless you changed the environment variables that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the `storage` directory:
-
-```
-{PROJECT_FOLDER}/storage/datasets/default/
-```
-
-The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of `Dataset.pushData()` or one table row.
-
-### Final touch
-
-It may seem that the data are extracted and the crawler is done, but honestly, this is just the beginning. For the sake of brevity, we've completely omitted error handling, proxies, debug logging, tests, documentation and other stuff that a reliable software should have. The good thing is, **error handling is mostly done by Crawlee itself**, so no worries on that front, unless you need some custom magic.
-
-Anyway, to spark some ideas, let's look at two more things. First, passing an input to the crawler, which will enable you to change the user page you want to scrape without changing the source code itself! And then some refactoring, to show how we reckon is preferable to structure and annotate crawler code.
-
-#### Meet the `INPUT`
-
-`INPUT` is just a convention on how we call the crawler's input. Because there's no magic in crawlers, just features, the `INPUT` is actually nothing more than a key in the default <ApiLink to="core/class/KeyValueStore">`KeyValueStore`</ApiLink>. Also by convention, the `INPUT` is mostly expected to be of `Content-Type: application/json`.
-
-We will not go into `KeyValueStore` details here, but for the sake of `INPUT` you need to remember that there is a function that helps you get it.
-
-```js
-const input = await KeyValueStore.getInput();
-```
-
-You need to place an `INPUT.json` file in your default key-value store for this to work.
-
-```
-{PROJECT_FOLDER}/storage/key_value_stores/default/INPUT.json
-```
-
-#### Use `INPUT` to seed your crawler with users
-
-Currently, we're using the full URLs of user pages as sources, but it's quite obvious that we only need the final parameter, the rest of the URL is always the same. Knowing that, we can pass an array of those parameters on `INPUT` and build the URLs dynamically, which would allow us to scrape public actors created by different users without changing the source code. Let's get to it!
-
-For simplicity, let's set the default input in the code directly, as a fallback in case the input is not found in the default `KeyValueStore`:
-
-> We are using the [nullish coalescing operator (`??`)](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_operator), it will use the value on the right hand side only if the left hand side resolves to `null` or `undefined`.
-
-```js
-const input = await KeyValueStore.getInput() ?? ['apify'];
-```
-
-Next, we will be using the usernames in the input to construct the user URLs, and we'll also pass custom `userData` to the sources. This means that the requests that get created will automatically contain this `userData`.
-
-```js
-// ...
-const input = await KeyValueStore.getInput() ?? ['apify'];
-
-const sources = input.map((username) => ({
- url: `https://apify.com/${username}`,
- userData: {
- label: 'USER_PAGE',
- },
-}));
-
-const requestList = await RequestList.open('users', sources);
-// ...
-```
-
-The `userData.label` is also a convention that we've been using for quite some time to label different requests. We know that this is a user page URL so we `label` it `USER_PAGE`. This way, we can easily make decisions in the `requestHandler` without having to inspect the URL itself.
-
-We can then refactor the `if` clauses in the `requestHandler` to use the `label` for decision-making. This does not make much sense for a crawler with only two different pages, because a simple `boolean` would suffice, but for pages with multiple different views, it becomes very useful.
-
-#### Structuring the code better
-
-But perhaps we should not stop at refactoring the `if` clauses. There are several ways you can make the crawler look more elegant and - most importantly - easier to reason about and make changes to.
-
-In the following code we've made several changes:
-- Split the code into multiple files;
-- Replaced `console.log` with the Crawlee logger;
-- Added a `getSources()` function to encapsulate `INPUT` consumption;
-- Added a `createRouter()` function to make our routing cleaner, without nested `if` clauses;
-- Removed the `maxRequestsPerCrawl` limit.
-
-In our `main.js` file, we place the general structure of the crawler:
-
-```js title="main.js"
-import { CheerioCrawler, log } from 'crawlee';
-import { router } from './routes.js';
-import { getSources } from './tools.js';
-
-log.info('Starting crawler.');
-
-log.debug('Setting up crawler.');
-const crawler = new CheerioCrawler({
- requestHandler: router,
-});
-
-await crawler.addRequests(await getSources());
-
-log.info('Starting the crawl.');
-await crawler.run();
-log.info('Crawl finished.');
-```
-
-Then in a separate `tools.js`, we add our helper functions:
-
-```ts title="tools.js"
-import { KeyValueStore, log } from 'crawlee';
-
-export const DEFAULT_USERNAMES = ['apify'];
-
-export async function getSources() {
- log.debug('Getting sources.');
- const input = await KeyValueStore.getInput() ?? DEFAULT_USERNAMES;
- return input.map((username) => ({
- url: `https://apify.com/${username}`,
- label: 'USER_PAGE',
- }));
-}
-```
-
-And finally our routes in a separate `routes.js` file:
-
-```js title="routes.js"
-import { createCheerioRouter, Dataset } from 'crawlee';
-
-export const router = createCheerioRouter();
-
-router.addHandler('USER_PAGE', async ({ enqueueLinks }) => {
- await enqueueLinks({
- selector: 'div[data-test=actorCard] > a.ActorStoreItem',
- label: 'DETAIL',
- });
-});
-
-router.addHandler('DETAIL', async ({ $, log, request }) => {
- const urlArr = request.url.split('/').slice(-2);
-
- log.debug('Scraping results.');
- const results = {
- url: request.url,
- uniqueIdentifier: urlArr.join('/'),
- owner: urlArr[0],
- title: $('header h1').text(),
- description: $('header span.actor-description').text(),
- modifiedDate: new Date(
- Number(
- $('ul.ActorHeader-stats time').attr('datetime'),
- ),
- ),
- runCount: Number(
- $('ul.ActorHeader-stats > li:nth-of-type(3)')
- .text()
- .match(/[\d,]+/)[0]
- .replaceAll(',', ''),
- ),
- };
-
- log.debug('Pushing data to dataset.');
- await Dataset.pushData(results);
-});
-```
-
-Let's describe the changes a bit more in detail. We hope that in the end, you can agree that this structure makes the actor more readable and manageable.
-
-#### Splitting your code into multiple files
-
-There's no reason not to split your code into multiple files and keep your logic separate.
-Less code in a single file means less code you need to think about at any time, and that's a great thing!
-
-#### Using Crawlee `log` instead of `console.log`
-
-We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
-
-`utils.log` enables the usage of different log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple `INPUT` change, or by setting an environment variable.
-
-The punch line? Use `log` exported from `crawlee` instead of `console.log` now and thank us later when something goes wrong!
-
-#### Using a router to structure your crawling
-
-At first, it might seem more readable using just a simple `if / else` statement to select different logic based on the crawled pages, but trust us, it becomes far less impressive when working with more than two different pages, and it definitely starts to fall apart when the logic to handle each page spans tens or hundreds of lines of code.
-
-It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into a special file and with large routes, we would even suggest having one file per route.
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. You should already have docker image for the crawler ready, as it was generated by the CLI. To read more about how to run this docker image in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-[//]: # (> TO BE CONTINUED with details on `PlaywrightCrawler` and other features...)
diff --git a/docs/introduction/06-scraping.mdx b/docs/introduction/06-scraping.mdx
new file mode 100644
index 000000000..38b2289c2
--- /dev/null
+++ b/docs/introduction/06-scraping.mdx
@@ -0,0 +1,180 @@
+---
+id: scraping
+title: "Scraping the Store"
+sidebar_label: "Scraping"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.
+
+- URL
+- Owner
+- Unique identifier (such as `apify/web-scraper`)
+- Title
+- Description
+- Last modification date
+- Number of runs
+
+![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
+
+### Scraping the URL, Owner and Unique identifier
+
+Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+
+:::info
+
+You can use `request.loadedUrl` as well. Remember the difference: `request.url` is what you enqueue, `request.loadedUrl` is what gets processed (after possible redirects).
+
+:::
+
+```js
+// request.url = https://apify.com/apify/web-scraper
+
+const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
+const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
+const owner = urlParts[0]; // 'apify'
+```
+
+:::tip
+
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+
+:::
+
+Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.
+
+### Title
+
+![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.
+
+:::caution
+
+In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.
+
+:::
+
+:::tip
+
+Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of DevTools to open the search bar where you can quickly search for elements using their selectors. Always verify your scraping process and assumptions using the DevTools. It's faster than changing the crawler code all the time.
+
+:::
+
+To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.
+
+```js
+const title = await page.locator('h1').textContent();
+```
+
+### Description
+
+Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.
+
+![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+
+```js
+const description = await page.locator('span.actor-description').textContent();
+```
+
+### Last modification date
+
+DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+
+![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+
+```js
+const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
+const modifiedDate = new Date(Number(modifiedTimestamp));
+```
+
+It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+
+But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+
+### Run count
+
+We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+
+```js
+const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
+const runCountString = await runsRow.locator('span').last().textContent();
+const runCount = Number(runCountString.replaceAll(',', ''));
+```
+
+The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.
+
+And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
+
+```js
+const urlParts = request.url.split('/').slice(-2);
+const uniqueIdentifier = urlParts.join('/');
+const owner = urlParts[0];
+
+const title = await page.locator('h1').textContent();
+
+const description = await page.locator('span.actor-description').textContent();
+
+const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
+const modifiedDate = new Date(Number(modifiedTimestamp));
+
+const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
+const runCountString = await runsRow.locator('span').last().textContent();
+const runCount = Number(runCountString.replaceAll(',', ''));
+```
+
+## Trying it out
+
+We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
+
+```js
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`)
+ if (request.label === 'DETAIL') {
+ const urlParts = request.url.split('/').slice(-2);
+ const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
+ const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
+ const runCountString = await runsRow.locator('span').last().textContent();
+
+ const results = {
+ url: request.url,
+ uniqueIdentifier: urlParts.join('/'),
+ owner: urlParts[0],
+ title: await page.locator('h1').textContent(),
+ description: await page.locator('span.actor-description').textContent(),
+ modifiedDate: new Date(Number(modifiedTimestamp)),
+ runCount: Number(runCountString.replaceAll(',', '')),
+ }
+
+ console.log(results)
+ } else {
+ await page.waitForSelector('.ActorStorePagination-pages a');
+ await enqueueLinks({
+ selector: '.ActorStorePagination-pages > a',
+ label: 'LIST',
+ })
+ await page.waitForSelector('.ActorStoreItem');
+ await enqueueLinks({
+ selector: '.ActorStoreItem',
+ label: 'DETAIL', // <= note the different label
+ })
+ }
+ }
+});
+
+await crawler.run(['https://apify.com/store']);
+```
+
+
+
+When you run the actor, you should see the crawled URLs and their scraped data printed to the console.
+
+## Next lesson
+
+In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
+
diff --git a/docs/introduction/07-saving-data.mdx b/docs/introduction/07-saving-data.mdx
new file mode 100644
index 000000000..717b9f342
--- /dev/null
+++ b/docs/introduction/07-saving-data.mdx
@@ -0,0 +1,94 @@
+---
+id: saving-data
+title: "Saving data"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
+
+First, add a new import to the top of the file:
+
+```ts
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+```
+
+Then, replace the `console.log(results)` call with:
+
+```js
+await Dataset.pushData(results);
+```
+
+and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
+
+```js
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`)
+ if (request.label === 'DETAIL') {
+ const urlParts = request.url.split('/').slice(-2);
+ const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
+ const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
+ const runCountString = await runsRow.locator('span').last().textContent();
+
+ const results = {
+ url: request.url,
+ uniqueIdentifier: urlParts.join('/'),
+ owner: urlParts[0],
+ title: await page.locator('h1').textContent(),
+ description: await page.locator('span.actor-description').textContent(),
+ modifiedDate: new Date(Number(modifiedTimestamp)),
+ runCount: Number(runCountString.replaceAll(',', '')),
+ }
+
+ await Dataset.pushData(results);
+ } else {
+ await page.waitForSelector('.ActorStorePagination-pages a');
+ await enqueueLinks({
+ selector: '.ActorStorePagination-pages > a',
+ label: 'LIST',
+ })
+ await page.waitForSelector('.ActorStoreItem');
+ await enqueueLinks({
+ selector: '.ActorStoreItem',
+ label: 'DETAIL', // <= note the different label
+ })
+ }
+ }
+});
+
+await crawler.run(['https://apify.com/store']);
+```
+
+## What's `Dataset.pushData()`
+
+&#8203;<ApiLink to="core/class/Dataset#pushData">`Dataset.pushData()`</ApiLink> is a function that saves data to the default <ApiLink to="core/class/Dataset">`Dataset`</ApiLink>. `Dataset` is a storage designed to hold data in a format similar to a table. Each time you call `Dataset.pushData()` a new row in the table is created, with the property names serving as column titles. In the default configuration, the rows are represented as JSON files saved on your disk, but other storage systems can be plugged into Crawlee as well.
+
+:::info
+
+Each time you start Crawlee a default `Dataset` is automatically created, so there's no need to initialize it or create an instance first. You can create as many datasets as you want and even give them names. For more details see the [Result storage guide](../guides/result-storage#dataset) and the <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> function.
+
+:::
+
+## Finding saved data
+
+Unless you changed the configuration that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the `storage` directory that Crawlee creates in the working directory of the running script:
+
+```
+{PROJECT_FOLDER}/storage/datasets/default/
+```
+
+The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of `Dataset.pushData()` or one table row.
+
+:::tip
+
+If you would like to store your data in a single big file, instead of many small ones, see the [Result storage guide](../guides/result-storage#key-value-store) for Key-value stores.
+
+:::
+
+## Next lesson
+
+In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdx
new file mode 100644
index 000000000..0a23142d5
--- /dev/null
+++ b/docs/introduction/08-refactoring.mdx
@@ -0,0 +1,137 @@
+---
+id: refactoring
+title: "Refactoring"
+description: Your first steps into the world of scraping with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Admonition from '@theme/Admonition';
+
+It may seem that the data is extracted and the crawler is done, but honestly, this is just the beginning. For the sake of brevity, we've completely omitted error handling, proxies, logging, architecture, tests, documentation and other stuff that a reliable software should have. The good thing is, **error handling is mostly done by Crawlee itself**, so no worries on that front, unless you need some custom magic.
+
+:::info
+
+If you've got to this point and are wondering where all the **anti-blocking, bot-protection avoiding stealthy features** are, you're right, we haven't shown you any. But that's the point! They are **automatically used** with the default configuration.
+
+That does not mean the default configuration can handle everything, but it should get you far enough. If you want to learn more, browse the [Avoid getting blocked](../guides/avoid-blocking), [Proxy management](../guides/proxy-management) and [Session management](../guides/session-management) guides.
+
+:::
+
+Anyway, to promote good coding practices, let's look at how you can use a <ApiLink to="core/class/Router">`Router`</ApiLink> to better structure your crawler code.
+
+## Routing
+
+In the following code we've made several changes:
+
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
+
+In our `main.mjs` file, we place the general structure of the crawler:
+
+```js title="src/main.mjs"
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.js';
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+log.debug('Adding requests to the queue.');
+await crawler.addRequests(['https://apify.com/store']);
+
+// crawler.run has its own logs 🙂
+await crawler.run();
+```
+
+Then in a separate `routes.mjs` file:
+
+```js title="src/routes.mjs"
+import { createPlaywrightRouter, Dataset } from 'crawlee';
+
+// createPlaywrightRouter() is only a helper to get better
+// intellisense and typings. You can use Router.create() too.
+export const router = createPlaywrightRouter();
+
+// This replaces the request.label === DETAIL branch of the if clause.
+router.addHandler('DETAIL', async ({ request, page, log }) => {
+ log.debug(`Extracting data: ${request.url}`)
+ const urlParts = request.url.split('/').slice(-2);
+ const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
+ const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
+ const runCountString = await runsRow.locator('span').last().textContent();
+
+ const results = {
+ url: request.url,
+ uniqueIdentifier: urlParts.join('/'),
+ owner: urlParts[0],
+ title: await page.locator('h1').textContent(),
+ description: await page.locator('span.actor-description').textContent(),
+ modifiedDate: new Date(Number(modifiedTimestamp)),
+ runCount: Number(runCountString.replaceAll(',', '')),
+ }
+
+ log.debug(`Saving data: ${request.url}`)
+ await Dataset.pushData(results);
+});
+
+// This is a fallback route which will handle the start URL
+// as well as the LIST labelled URLs.
+router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
+ log.debug(`Enqueueing pagination: ${request.url}`)
+ await page.waitForSelector('.ActorStorePagination-pages a');
+ await enqueueLinks({
+ selector: '.ActorStorePagination-pages > a',
+ label: 'LIST',
+ })
+ log.debug(`Enqueueing actor details: ${request.url}`)
+ await page.waitForSelector('.ActorStoreItem');
+ await enqueueLinks({
+ selector: '.ActorStoreItem',
+ label: 'DETAIL', // <= note the different label
+ })
+});
+```
+
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.
+
+## Splitting your code into multiple files
+
+There's no reason not to split your code into multiple files and keep your logic separate. Less code in a single file means less code you need to think about at any time, and that's good. We would most likely go even further and split even the routes into separate files.
+
+## Using Crawlee `log` instead of `console.log`
+
+We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
+
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.
+
+The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!
+
+## Using a router to structure your crawling
+
+At first, it might seem more readable using just a simple `if / else` statement to select different logic based on the crawled pages, but trust us, it becomes far less impressive when working with more than two different pages, and it definitely starts to fall apart when the logic to handle each page spans tens or hundreds of lines of code.
+
+It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Running your crawler in the Cloud
+
+Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/docs/introduction/index.mdx b/docs/introduction/index.mdx
index 6ce8be867..a6895d500 100644
--- a/docs/introduction/index.mdx
+++ b/docs/introduction/index.mdx
@@ -1,12 +1,49 @@
---
+id: introduction
title: Introduction
description: Your first steps into the world of scraping with Crawlee
---
import ApiLink from '@site/src/components/ApiLink';
-Without the right tools, crawling and scraping the web can be difficult. At the very least, you need an HTTP client to make the necessary requests, but that only gets you raw HTML and sometimes not even that. Then you have to read this HTML and extract the data you're interested in. Once extracted, it must be stored in a machine-readable format and easily accessible for further processing, because it is the processed data that holds value.
+Crawlee covers your crawling and scraping end-to-end and helps you **build reliable scrapers. Fast.**
-Crawlee covers the process end-to-end. From crawling the web for links and scraping the raw data to storing it in various machine-readable formats, ready for processing. With this guide in hand, you should have your own data extraction solutions up and running in a few hours.
+Your crawlers will appear human-like and fly under the radar of modern bot protections even with the default configuration. Crawlee gives you the tools to crawl the web for links, scrape data and persistently store it in machine-readable formats, without having to worry about the technical details. And thanks to rich configuration options, you can tweak almost any aspect of Crawlee to suit your project's needs if the default settings don't cut it.
-The goal of this guide is to provide a step-by-step introduction to all the features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to complex systems that crawl pages, interact with them as if a real user were sitting in front of a real browser and output structured data.
+## What you will learn
+
+The goal of the introduction is to provide a step-by-step guide to all the features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
+
+## 🛠 Features
+
+- Single interface for **HTTP and headless browser** crawling
+- Persistent **queue** for URLs to crawl (breadth & depth first)
+- Pluggable **storage** of both tabular data and files
+- Automatic **scaling** with available system resources
+- Integrated **proxy rotation** and session management
+- Lifecycles customizable with **hooks**
+- **CLI** to bootstrap your projects
+- Configurable **routing**, **error handling** and **retries**
+- **Dockerfiles** ready to deploy
+- Written in **TypeScript** with generics
+
+### 👾 HTTP crawling
+
+- Zero config **HTTP2 support**, even for proxies
+- Automatic generation of **browser-like headers**
+- Replication of browser **TLS fingerprints**
+- Integrated fast **HTML parser**
+- Yes, you can scrape **JSON APIs** as well
+
+### 💻 Real browser crawling
+
+- JavaScript **rendering** and **screenshots**
+- **Headless** and **headful** support
+- Zero-config generation of **human-like fingerprints**
+- Automatic **browser management**
+- Use **Playwright** and **Puppeteer** with the same interface
+- **Chrome**, **Firefox**, **Webkit** and many others
+
+## Next lesson
+
+In the next lesson you will install Crawlee and learn how to bootstrap projects with the Crawlee CLI.
diff --git a/docs/quick-start/headful_playwright.ts b/docs/quick-start/headful_playwright.ts
new file mode 100644
index 000000000..149482390
--- /dev/null
+++ b/docs/quick-start/headful_playwright.ts
@@ -0,0 +1,16 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const { url } = request;
+ const title = await page.title();
+ log.info(`Title of ${url}: ${title}`);
+ await enqueueLinks({ strategy: 'same-domain' });
+ await Dataset.pushData({ url, title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev/']);
diff --git a/docs/quick-start/headful_puppeteer.ts b/docs/quick-start/headful_puppeteer.ts
new file mode 100644
index 000000000..c8e44fcbd
--- /dev/null
+++ b/docs/quick-start/headful_puppeteer.ts
@@ -0,0 +1,16 @@
+import { PuppeteerCrawler, Dataset } from 'crawlee';
+
+const crawler = new PuppeteerCrawler({
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const { url } = request;
+ const title = await page.title();
+ log.info(`Title of ${url}: ${title}`);
+ await enqueueLinks({ strategy: 'same-domain' });
+ await Dataset.pushData({ url, title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev/']);
diff --git a/docs/quick-start/index.mdx b/docs/quick-start/index.mdx
index 4f4153084..68e1bfd52 100644
--- a/docs/quick-start/index.mdx
+++ b/docs/quick-start/index.mdx
@@ -12,7 +12,9 @@ import CodeBlock from '@theme/CodeBlock';
import CheerioSource from '!!raw-loader!./quick_start_cheerio.ts';
import PlaywrightSource from '!!raw-loader!./quick_start_playwright.ts';
+import PlaywrightHeadful from '!!raw-loader!./headful_playwright.ts';
import PuppeteerSource from '!!raw-loader!./quick_start_puppeteer.ts';
+import PuppeteerHeadful from '!!raw-loader!./headful_puppeteer.ts';
import CheerioLog from '!!raw-loader!./quick_start_cheerio.txt';
@@ -20,7 +22,7 @@ With this short tutorial you can start scraping with Crawlee in a minute or two.
## Choose your crawler
-Crawlee comes with three different crawler classes: <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. All classes share the same interface for maximum flexibility when switching between them.
+Crawlee comes with three main crawler classes: <ApiLink to="cheerio-crawler/class/CheerioCrawler">`CheerioCrawler`</ApiLink>, <ApiLink to="puppeteer-crawler/class/PuppeteerCrawler">`PuppeteerCrawler`</ApiLink> and <ApiLink to="playwright-crawler/class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>. All classes share the same interface for maximum flexibility when switching between them.
### CheerioCrawler
This is a plain HTTP crawler. It parses HTML using the [Cheerio](https://github.com/cheeriojs/cheerio) library and crawls the web using the specialized [got-scraping](https://github.com/apify/got-scraping) HTTP client which masks as a browser. It's very fast and efficient, but can't handle JavaScript rendering.
@@ -41,30 +43,30 @@ It can be added to any Node.js project by running:
<CodeBlock language="bash">npm install crawlee</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
-<CodeBlock language="bash">npm install crawlee playwright</CodeBlock>
:::caution
-`playwright` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM.
+`playwright` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM. 👇
:::
+<CodeBlock language="bash">npm install crawlee playwright</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
-<CodeBlock language="bash">npm install crawlee puppeteer</CodeBlock>
:::caution
-`puppeteer` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM.
+`puppeteer` is not bundled with Crawlee to reduce install size and allow greater flexibility. You need to explicitly install it with NPM. 👇
:::
+<CodeBlock language="bash">npm install crawlee puppeteer</CodeBlock>
</TabItem>
</Tabs>
## Crawling
-Run the following example to perform a recursive crawl of a website using:
+Run the following example to perform a recursive crawl of the Crawlee website using:
<Tabs groupId="quick_start">
<TabItem value="cheerio" label="CheerioCrawler" default>
@@ -78,34 +80,47 @@ Run the following example to perform a recursive crawl of a website using:
</TabItem>
</Tabs>
-When you run the example, you will see Crawlee automating the data extraction process.
+When you run the example, you will see Crawlee automating the data extraction process in your terminal.
-<Tabs groupId="quick_start">
-<TabItem value="cheerio" label="CheerioCrawler" default>
-Logs in your terminal will show the progress of your scraping:
<CodeBlock language="log">{CheerioLog}</CodeBlock>
-</TabItem>
-<TabItem value="playwright" label="PlaywrightCrawler">
-Logs in your terminal will show the progress of your scraping and you will also see Crawlee automating a browser:
-![Chrome Scrape](/img/chrome_scrape.gif)
-</TabItem>
-<TabItem value="puppeteer" label="PuppeteerCrawler">
-Logs in your terminal will show the progress of your scraping and you will also see Crawlee automating a browser:
+### Running headful browsers
-![Chrome Scrape](/img/chrome_scrape.gif)
+Browsers controlled by Puppeteer and Playwright run headless (without a visible window). You can switch to headful by adding the `headless: false` option to the crawlers' constructor. This is useful in the development phase when you want to see what's going on in the browser.
-</TabItem>
+<Tabs groupId="quick_start">
+ <TabItem value="playwright" label="PlaywrightCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PlaywrightHeadful}</CodeBlock>
+ </TabItem>
+ <TabItem value="puppeteer" label="PuppeteerCrawler">
+ <CodeBlock language="js" title="src/main.mjs">{PuppeteerHeadful}</CodeBlock>
+ </TabItem>
</Tabs>
+When you run the example code, you'll see an automated browser blaze through the Crawlee website.
+
+![Chrome Scrape](/img/chrome_scrape.gif)
+
+## Results
+
+Crawlee stores data to the `./storage` directory in your current working directory. The results of your crawl will be available under `./storage/datasets/default/*.json` as JSON files.
+
+```json title="./storage/datasets/default/000000001.json"
+{
+ "url": "https://crawlee.dev/",
+ "title": "Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee"
+}
+```
+
:::tip
-Crawlee stores data to `./storage` in the current working directory. The results of your crawl will be available under `./storage/datasets/default/*`. You can override this behavior by setting the `CRAWLEE_STORAGE_DIR` environment variable.
+You can override the storage directory by setting the `CRAWLEE_STORAGE_DIR` environment variable.
:::
-More examples showcasing various features of Crawlee could be found in [Examples](./examples) section of the documentation.
+## Examples and further reading
+You can find more examples showcasing various features of Crawlee in the [Examples](./examples) section of the documentation. To better understand Crawlee and its components you should read the [Introduction](./introduction) step-by-step guide.
**Related links**
diff --git a/docs/quick-start/quick_start_cheerio.ts b/docs/quick-start/quick_start_cheerio.ts
index de12102dd..038aa3b03 100644
--- a/docs/quick-start/quick_start_cheerio.ts
+++ b/docs/quick-start/quick_start_cheerio.ts
@@ -8,15 +8,14 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
log.info(`Title of ${url}: ${title}`);
- // Add URLs that match the provided pattern.
- await enqueueLinks({
- globs: ['https://www.iana.org/*'],
- });
+ // Add links from the page that point
+ // to the same domain as the original request.
+ await enqueueLinks({ strategy: 'same-domain' });
- // Save extracted data to dataset.
+ // Save extracted data to storage.
await Dataset.pushData({ url, title });
},
});
-// Enqueue the initial request and run the crawler.
-await crawler.run(['https://www.iana.org/']);
+// Add a start URL to the queue and run the crawler.
+await crawler.run(['https://crawlee.dev/']);
diff --git a/docs/quick-start/quick_start_cheerio.txt b/docs/quick-start/quick_start_cheerio.txt
index 8a433b4f8..fb7c410e3 100644
--- a/docs/quick-start/quick_start_cheerio.txt
+++ b/docs/quick-start/quick_start_cheerio.txt
@@ -1,18 +1,6 @@
INFO CheerioCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.7,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
-INFO CheerioCrawler: Title of https://www.iana.org/: Internet Assigned Numbers Authority
-INFO CheerioCrawler: Title of https://www.iana.org/domains: Domain Name Services
-INFO CheerioCrawler: Title of https://www.iana.org/numbers: Number Resources
-INFO CheerioCrawler: Title of https://www.iana.org/about/: About us
-INFO CheerioCrawler: Title of https://www.iana.org/abuse: Abuse Issues
-INFO CheerioCrawler: Title of https://www.iana.org/protocols: Protocol Registries
-INFO CheerioCrawler: Title of https://www.iana.org/time-zones: Time Zone Database
-INFO CheerioCrawler: Title of https://www.iana.org/performance: Performance
-INFO CheerioCrawler: Title of https://www.iana.org/reports: Reports
-INFO CheerioCrawler: Title of https://www.iana.org/reviews: Reviews
-INFO CheerioCrawler: Title of https://www.iana.org/contact: Contact Us
-INFO CheerioCrawler: Title of https://www.iana.org/dnssec: DNSSEC Information
-INFO CheerioCrawler: Title of https://www.iana.org/procedures: Procedures
-INFO CheerioCrawler: Title of https://www.iana.org/glossary: Glossary
-INFO CheerioCrawler: Title of https://www.iana.org/audits: Audit Programs
-INFO CheerioCrawler: All the requests from request list and/or request queue have been processed, the crawler will shut down.
-INFO CheerioCrawler: Final request statistics: {"requestsFinished":15,"requestsFailed":0,"retryHistogram":[15],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":482,"requestsFinishedPerMinute":209,"requestsFailedPerMinute":0,"requestTotalDurationMillis":7233,"requestsTotal":15,"crawlerRuntimeMillis":4316}
+INFO CheerioCrawler: Title of https://crawlee.dev/: Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples: Examples | Crawlee
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start: Quick Start | Crawlee
+INFO CheerioCrawler: Title of https://crawlee.dev/api/core: @crawlee/core | API | Crawlee
+INFO CheerioCrawler: Title of https://crawlee.dev/api/core/changelog: Changelog | API | Crawlee
diff --git a/docs/quick-start/quick_start_playwright.ts b/docs/quick-start/quick_start_playwright.ts
index 3fa70d89b..d7d12c27b 100644
--- a/docs/quick-start/quick_start_playwright.ts
+++ b/docs/quick-start/quick_start_playwright.ts
@@ -8,15 +8,14 @@ const crawler = new PlaywrightCrawler({
const title = await page.title();
log.info(`Title of ${url}: ${title}`);
- // Add URLs that match the provided pattern.
- await enqueueLinks({
- globs: ['https://www.iana.org/*'],
- });
+ // Add links from the page that point
+ // to the same domain as the original request.
+ await enqueueLinks({ strategy: 'same-domain' });
- // Save extracted data to dataset.
+ // Save extracted data to storage.
await Dataset.pushData({ url, title });
},
});
-// Enqueue the initial request and run the crawler.
-await crawler.run(['https://www.iana.org/']);
+// Add a start URL to the queue and run the crawler.
+await crawler.run(['https://crawlee.dev/']);
diff --git a/docs/quick-start/quick_start_puppeteer.ts b/docs/quick-start/quick_start_puppeteer.ts
index bbcd30114..001254010 100644
--- a/docs/quick-start/quick_start_puppeteer.ts
+++ b/docs/quick-start/quick_start_puppeteer.ts
@@ -8,15 +8,14 @@ const crawler = new PuppeteerCrawler({
const title = await page.title();
log.info(`Title of ${url}: ${title}`);
- // Add URLs that match the provided pattern.
- await enqueueLinks({
- globs: ['https://www.iana.org/*'],
- });
+ // Add links from the page that point
+ // to the same domain as the original request.
+ await enqueueLinks({ strategy: 'same-domain' });
- // Save extracted data to dataset.
+ // Save extracted data to storage.
await Dataset.pushData({ url, title });
},
});
-// Enqueue the initial request and run the crawler.
-await crawler.run(['https://www.iana.org/']);
+// Add a start URL to the queue and run the crawler.
+await crawler.run(['https://crawlee.dev/']);
diff --git a/packages/core/src/storages/request_queue.ts b/packages/core/src/storages/request_queue.ts
index cccdc487c..830e4ed67 100644
--- a/packages/core/src/storages/request_queue.ts
+++ b/packages/core/src/storages/request_queue.ts
@@ -282,7 +282,6 @@ export class RequestQueue {
* it will not be updated. You can find out whether this happened by finding the request in the resulting
* {@link BatchAddRequestsResult} object.
*
- * @experimental
* @param requestsLike {@link Request} objects or vanilla objects with request data.
* Note that the function sets the `uniqueKey` and `id` fields to the passed requests if missing.
* @param [options] Request queue operation options.
diff --git a/website/sidebars.js b/website/sidebars.js
index edfe2e25b..593586d3c 100644
--- a/website/sidebars.js
+++ b/website/sidebars.js
@@ -7,14 +7,17 @@ module.exports = {
collapsed: false,
link: {
type: 'doc',
- id: 'introduction/index',
+ id: 'introduction/introduction',
},
items: [
'introduction/setting-up',
'introduction/first-crawler',
- 'introduction/cheerio-crawler',
- 'introduction/enqueue-links',
- 'introduction/realworld-example',
+ 'introduction/adding-urls',
+ 'introduction/real-world-project',
+ 'introduction/crawling',
+ 'introduction/scraping',
+ 'introduction/saving-data',
+ 'introduction/refactoring',
],
},
{
@@ -27,10 +30,11 @@ module.exports = {
keywords: ['guides'],
},
items: [
- 'guides/motivation',
'guides/request-storage',
'guides/result-storage',
'guides/environment-variables',
+ 'guides/cheerio-crawler-guide',
+ 'guides/javascript-rendering',
'guides/proxy-management',
'guides/session-management',
'guides/scaling-crawlers',
diff --git a/website/src/pages/index.js b/website/src/pages/index.js
index a9cb41b6b..9030b27fb 100644
--- a/website/src/pages/index.js
+++ b/website/src/pages/index.js
@@ -14,11 +14,11 @@ function Hero() {
<div className="row">
<div className="col col--7">
<div className="row">
- <p className={styles.tagline}>
- The scalable <span>web crawling</span>,<br />
- <span>scraping</span> and <span>automation library</span><br />
+ <h1 className={styles.tagline}>
+ The scalable <span>web crawling</span>, <br />
+ <span>scraping</span> and <span>automation library</span> <br />
for JavaScript/Node.js
- </p>
+ </h1>
</div>
<div className={styles.heroButtons}>
<Link to="#try" className={styles.getStarted}>Try it out</Link>
diff --git a/website/static/img/chrome_scrape.gif b/website/static/img/chrome_scrape.gif
index 08a455e70..83de50c8f 100644
Binary files a/website/static/img/chrome_scrape.gif and b/website/static/img/chrome_scrape.gif differ
diff --git a/website/static/img/chrome_scrape_old.gif b/website/static/img/chrome_scrape_old.gif
new file mode 100644
index 000000000..08a455e70
Binary files /dev/null and b/website/static/img/chrome_scrape_old.gif differ
diff --git a/website/static/img/getting-started/select-an-element.png b/website/static/img/getting-started/select-an-element.png
new file mode 100644
index 000000000..722f07286
Binary files /dev/null and b/website/static/img/getting-started/select-an-element.png differ
diff --git a/website/static/img/getting-started/selected-element.png b/website/static/img/getting-started/selected-element.png
new file mode 100644
index 000000000..d272f1492
Binary files /dev/null and b/website/static/img/getting-started/selected-element.png differ