Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: rewrite introduction to scraping guide (#2143)

Vlad Frangu52b06faae7a9
diff --git a/docs/introduction/03-filter-without-el.ts b/docs/introduction/03-filter-without-el.ts
index de7e6e21c..e50b05d8f 100644
--- a/docs/introduction/03-filter-without-el.ts
+++ b/docs/introduction/03-filter-without-el.ts
@@ -15,7 +15,7 @@ const crawler = new CheerioCrawler({
// grab their hostname for filtering.
const { hostname } = new URL(request.loadedUrl);
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl)
+ (link) => new URL(link, request.loadedUrl),
);
// We use the hostname to filter links that point
diff --git a/docs/introduction/03-find-without-el.ts b/docs/introduction/03-find-without-el.ts
index 0414c504b..33561d019 100644
--- a/docs/introduction/03-find-without-el.ts
+++ b/docs/introduction/03-find-without-el.ts
@@ -18,7 +18,7 @@ const crawler = new CheerioCrawler({
// Then we need to resolve relative URLs,
// otherwise they would be unusable for crawling.
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl).href
+ (link) => new URL(link, request.loadedUrl).href,
);
// Finally, we have to add the URLs to the queue
diff --git a/docs/introduction/04-pw-w-cheerio.ts b/docs/introduction/04-pw-w-cheerio.ts
index b698f9fc4..448d8fd2a 100644
--- a/docs/introduction/04-pw-w-cheerio.ts
+++ b/docs/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('div[data-test="actorCard"]').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/docs/introduction/04-pw.ts b/docs/introduction/04-pw.ts
index 37becf161..076094467 100644
--- a/docs/introduction/04-pw.ts
+++ b/docs/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('div[data-test="actorCard"]', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/docs/introduction/04-real-world-project.mdx b/docs/introduction/04-real-world-project.mdx
index 775b34fd6..d0c0c893b 100644
--- a/docs/introduction/04-real-world-project.mdx
+++ b/docs/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **Actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all Actors from [Apify Store](https://apify.com/store) and for each Actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the Actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the Actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - Actors. Quick count of Actors on one page of results makes 24. 8 rows times 3 Actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain Actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of Actors (our start URL).
-2. Enqueue all links to Actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an Actor detail page, scrape the data.
-5. Repeat until all results pages and all Actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,18 +113,22 @@ Now that you know where to start and how to find all the Actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the Actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual Actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `div[data-test="actorCard"]` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
## DevTools - the scraper's toolbox
@@ -141,7 +138,7 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the Actor cards. You'll see that all the card's data is displayed in an `<a>` element with a `data-test` attribute **actorCard**. It should now make sense how we got that `div[data-test="actorCard"]` selector. It's just a way to find all elements that are annotated with the `actorCard`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('div[data-test="actorCard"]');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 Actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson, we will crawl the whole store, including all the listing pages and all the Actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/docs/introduction/05-crawling.mdx b/docs/introduction/05-crawling.mdx
index 848007e66..a646699e5 100644
--- a/docs/introduction/05-crawling.mdx
+++ b/docs/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with Actors - listing pages and also all the Actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the Actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-buttons a` selector, which selects all the `<a>` elements that are children of an element with `class=ActorStorePagination-buttons`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the Actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('div[data-test="actorCard"] a');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/docs/introduction/06-example.ts b/docs/introduction/06-example.ts
index 85b4e36f4..cd1eceeb0 100644
--- a/docs/introduction/06-example.ts
+++ b/docs/introduction/06-example.ts
@@ -4,33 +4,68 @@ const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString?.replace('Runs ', ''),
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
};
console.log(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
});
- await page.waitForSelector('div[data-test="actorCard"] a');
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- // highlight-next-line
- label: 'DETAIL', // <= note the different label
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
});
}
},
@@ -39,4 +74,4 @@ const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 50,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/docs/introduction/06-scraping.mdx b/docs/introduction/06-scraping.mdx
index 5220038fd..bf8b2d64f 100644
--- a/docs/introduction/06-scraping.mdx
+++ b/docs/introduction/06-scraping.mdx
@@ -1,7 +1,7 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
@@ -9,21 +9,20 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the Actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the Actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -32,26 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the Actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the Actor.
+:::caution
+
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
+
+:::
+
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
### Title
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. Web pages usually have only a single `<h1>` tag, but this one has two. Because of that, we have to extend the selector with more information. The `<h1>` tag is enclosed in a `<div>` with class `ActorHeader-identificator`. We can leverage this to create a combined selector `.ActorHeader-identificator h1`. It selects any `<h1>` element that is a child of a different element with the class `ActorHeader-identificator`.
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -59,66 +63,90 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title, we need to find it using `Playwright` and a `.ActorHeader-identificator h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Using the DevTools, we find that the Actor description is inside a `<p>` tag with a class `ActorHeader-description`. And since there's no other `<p>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('p.ActorHeader-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-userMedallion ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the count from the last (second) `<li>` element. But it contains the "Runs " string as well, so we remove the "Runs " by replacing the word with empty string.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
-
-const description = await page.locator('p.ActorHeader-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
@@ -133,17 +161,15 @@ When you run the crawler, you will see the crawled URLs and their scraped data p
```json
{
- "url": "https://apify.com/clockworks/tiktok-scraper",
- "uniqueIdentifier": "clockworks/tiktok-scraper",
- "owner": "clockworks",
- "title": "TikTok Scraper",
- "description": "Powerful TikTok scraper to extract data from TikTok videos, hashtags, and users. Use it to scrape TikTok profiles, hashtags, posts, URLs, numbers of shares, followers, hearts, names, video, and music-related data. Download TikTok data as a HTML, JSON, CSV, Excel, or XML doc.",
- "modifiedDate": "2023-04-26T12:13:19.387Z",
- "runCount": "1.6M"
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
}
```
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/docs/introduction/07-example.ts b/docs/introduction/07-example.ts
index a72159427..43937be28 100644
--- a/docs/introduction/07-example.ts
+++ b/docs/introduction/07-example.ts
@@ -4,34 +4,69 @@ const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString?.replace('Runs ', ''),
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
};
// highlight-next-line
await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
});
- await page.waitForSelector('div[data-test="actorCard"] a');
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- // highlight-next-line
- label: 'DETAIL', // <= note the different label
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
});
}
},
@@ -40,4 +75,4 @@ const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 50,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdx
index d8f8aa1b3..8c526cdcb 100644
--- a/docs/introduction/08-refactoring.mdx
+++ b/docs/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,9 +23,9 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
@@ -44,8 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -59,41 +58,81 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString.replace('Runs ', ''),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-buttons a');
- await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('div[data-test="actorCard"] a');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
diff --git a/docs/introduction/09-deployment.mdx b/docs/introduction/09-deployment.mdx
index 255056898..5273181be 100644
--- a/docs/introduction/09-deployment.mdx
+++ b/docs/introduction/09-deployment.mdx
@@ -1,7 +1,7 @@
---
id: deployment
-title: "Running your crawler in the Cloud"
-sidebar_label: "Running in the Cloud"
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
description: Deploying Crawlee projects to the Apify Platform
---
@@ -23,12 +23,13 @@ We started this guide by using the Crawlee CLI to bootstrap the project - it off
The first step will be installing two new dependencies:
-- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
```bash
npm install apify
```
-- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
```bash
npm install -g apify-cli
```
@@ -67,7 +68,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
// highlight-next-line
await Actor.exit();
diff --git a/test/e2e/playwright-introduction-guide/actor/main.js b/test/e2e/playwright-introduction-guide/actor/main.js
index 9b154b035..66c5ce498 100644
--- a/test/e2e/playwright-introduction-guide/actor/main.js
+++ b/test/e2e/playwright-introduction-guide/actor/main.js
@@ -7,66 +7,91 @@ await Actor.init({ storage: process.env.STORAGE_IMPLEMENTATION === 'LOCAL' ? new
// intellisense and typings. You can use Router.create() too.
export const router = createPlaywrightRouter();
-// This replaces the request.label === DETAIL branch of the if clause.
-router.addHandler('DETAIL', async ({
- request,
- page,
- log,
-}) => {
+router.addHandler('DETAIL', async ({ request, page, log }) => {
log.debug(`Extracting data: ${request.url}`);
- const urlParts = request.url.split('/')
- .slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]')
- .getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li')
- .filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1')
- .textContent(),
- description: await page.locator('p.ActorHeader-description')
- .textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString.replace('Runs ', ''),
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
};
- log.info(`Saving data: ${request.url}`);
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
-// This is a fallback route which will handle the start URL
-// as well as the LIST labeled URLs.
-router.addDefaultHandler(async ({
- request,
- page,
- enqueueLinks,
- log,
-}) => {
- log.debug(`Enqueueing pagination: ${request.url}`);
- await page.waitForSelector('.ActorStorePagination-buttons a');
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
});
- log.debug(`Enqueueing actor details: ${request.url}`);
- await page.waitForSelector('div[data-test="actorCard"] a');
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
+// This is a fallback route which will handle the start URL
+// as well as the LIST labeled URLs.
+router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
});
});
const crawler = new PlaywrightCrawler({
- maxRequestsPerCrawl: 10, // so the test runs faster
+ maxRequestsPerCrawl: 15, // so the test runs faster
// Instead of the long requestHandler with
// if clauses we provide a router instance.
requestHandler: router,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
await Actor.exit({ exit: Actor.isAtHome() });
diff --git a/test/e2e/puppeteer-store-pagination-jquery/actor/main.js b/test/e2e/puppeteer-store-pagination-jquery/actor/main.js
index 2f342e713..288579abe 100644
--- a/test/e2e/puppeteer-store-pagination-jquery/actor/main.js
+++ b/test/e2e/puppeteer-store-pagination-jquery/actor/main.js
@@ -9,8 +9,10 @@ const mainOptions = {
await Actor.main(async () => {
const crawler = new PuppeteerCrawler({
maxRequestsPerCrawl: 10,
- preNavigationHooks: [({ session, request }, goToOptions) => {
- session?.setCookies([{ name: 'OptanonAlertBoxClosed', value: new Date().toISOString() }], request.url);
+ preNavigationHooks: [async ({ page }, goToOptions) => {
+ await page.evaluateOnNewDocument(() => {
+ localStorage.setItem('themeExitPopup', 'true');
+ });
goToOptions.waitUntil = ['networkidle2'];
}],
async requestHandler({ page, request, log, enqueueLinks, injectJQuery }) {
@@ -18,15 +20,16 @@ await Actor.main(async () => {
if (label === 'START') {
log.info('Store opened');
- const nextButtonSelector = '[data-test="pagination-button-next"]:not([disabled])';
- // enqueue actor details from the first three pages of the store
- for (let pageNo = 1; pageNo <= 3; pageNo++) {
+ const nextButtonSelector = '.pagination__next';
+ // enqueue product details from the first three pages of the store
+ for (let pageNo = 1; pageNo < 3; pageNo++) {
// Wait for network events to finish
await page.waitForNetworkIdle();
// Enqueue all loaded links
await enqueueLinks({
- selector: 'div.ActorStore-main div > a',
- globs: [{ glob: 'https://apify.com/*/*', userData: { label: 'DETAIL' } }],
+ selector: 'a.product-item__image-wrapper',
+ label: 'DETAIL',
+ globs: ['https://warehouse-theme-metal.myshopify.com/*/*'],
});
log.info(`Enqueued actors for page ${pageNo}`);
log.info('Loading the next page');
@@ -35,18 +38,38 @@ await Actor.main(async () => {
} else if (label === 'DETAIL') {
log.info(`Scraping ${url}`);
await injectJQuery();
- const uniqueIdentifier = url.split('/').slice(-2).join('/');
- const results = await page.evaluate(() => ({
- title: $('header h1').text(), // eslint-disable-line
- description: $('div.Section-body > div > p').text(), // eslint-disable-line
- modifiedDate: $('div:nth-of-type(2) > ul > li:nth-of-type(3)').text(), // eslint-disable-line
- runCount: $('ul.ActorHeader-userMedallion li:nth-of-type(4)').text(), // eslint-disable-line
- }));
-
- await Dataset.pushData({ url, uniqueIdentifier, ...results });
+ const urlPart = url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ /* eslint-disable no-undef */
+ const results = await page.evaluate(() => {
+ const rawPrice = $('span.price')
+ .filter((_, el) => $(el).text().includes('$'))
+ .first()
+ .text()
+ .split('$')[1];
+
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStock = $('span.product-form__inventory')
+ .first()
+ .filter((_, el) => $(el).text().includes('In stock'))
+ .length !== 0;
+
+ return {
+ title: $('.product-meta h1').text(),
+ sku: $('span.product-meta__sku-number').text(),
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+ });
+
+ /* eslint-enable no-undef */
+
+ await Dataset.pushData({ url, manufacturer, ...results });
}
},
});
- await crawler.run([{ url: 'https://apify.com/store', userData: { label: 'START' } }]);
+ await crawler.run([{ url: 'https://warehouse-theme-metal.myshopify.com/collections/all-tvs', userData: { label: 'START' } }]);
}, mainOptions);
diff --git a/test/e2e/puppeteer-store-pagination-jquery/test.mjs b/test/e2e/puppeteer-store-pagination-jquery/test.mjs
index 308a11c07..8f87841e7 100644
--- a/test/e2e/puppeteer-store-pagination-jquery/test.mjs
+++ b/test/e2e/puppeteer-store-pagination-jquery/test.mjs
@@ -8,17 +8,6 @@ const { stats, datasetItems } = await runActor(testActorDirname, 16384);
await expect(stats.requestsFinished >= 10, 'All requests finished');
await expect(datasetItems.length > 5 && datasetItems.length < 15, 'Number of dataset items');
await expect(
- validateDataset(
- datasetItems,
- [
- 'url',
- 'title',
- 'uniqueIdentifier',
- 'description',
- // Skip modifiedAt and runCount since they changed
- // 'modifiedDate',
- // 'runCount',
- ],
- ),
+ validateDataset(datasetItems, ['url', 'manufacturer', 'title', 'sku', 'currentPrice', 'availableInStock']),
'Dataset items validation',
);
diff --git a/test/e2e/puppeteer-store-pagination/actor/main.js b/test/e2e/puppeteer-store-pagination/actor/main.js
index 97daf15c9..2aaaf7236 100644
--- a/test/e2e/puppeteer-store-pagination/actor/main.js
+++ b/test/e2e/puppeteer-store-pagination/actor/main.js
@@ -5,24 +5,26 @@ await Actor.init({ storage: process.env.STORAGE_IMPLEMENTATION === 'LOCAL' ? new
const crawler = new PuppeteerCrawler({
maxRequestsPerCrawl: 10,
- preNavigationHooks: [({ session, request }, goToOptions) => {
- session?.setCookies([{ name: 'OptanonAlertBoxClosed', value: new Date().toISOString() }], request.url);
+ preNavigationHooks: [async ({ page }, goToOptions) => {
+ await page.evaluateOnNewDocument(() => {
+ localStorage.setItem('themeExitPopup', 'true');
+ });
goToOptions.waitUntil = ['networkidle2'];
}],
});
crawler.router.addHandler('START', async ({ log, enqueueLinks, page }) => {
log.info('Store opened');
- const nextButtonSelector = '[data-test="pagination-button-next"]:not([disabled])';
- // enqueue actor details from the first three pages of the store
- for (let pageNo = 1; pageNo <= 3; pageNo++) {
+ const nextButtonSelector = '.pagination__next';
+ // enqueue product details from the first three pages of the store
+ for (let pageNo = 1; pageNo < 3; pageNo++) {
// Wait for network events to finish
await page.waitForNetworkIdle();
// Enqueue all loaded links
await enqueueLinks({
- selector: 'div.ActorStore-main div > a',
+ selector: 'a.product-item__image-wrapper',
label: 'DETAIL',
- globs: ['https://apify.com/*/*'],
+ globs: ['https://warehouse-theme-metal.myshopify.com/*/*'],
});
log.info(`Enqueued actors for page ${pageNo}`);
log.info('Loading the next page');
@@ -33,27 +35,39 @@ crawler.router.addHandler('START', async ({ log, enqueueLinks, page }) => {
crawler.router.addHandler('DETAIL', async ({ log, page, request: { url } }) => {
log.info(`Scraping ${url}`);
- const uniqueIdentifier = url.split('/').slice(-2).join('/');
- const titleP = page.$eval('header h1', ((el) => el.textContent));
- const descriptionP = page.$eval('div.Section-body > div > p', ((el) => el.textContent));
- const modifiedTimestampP = page.$eval('div:nth-of-type(2) > ul > li:nth-of-type(3)', (el) => el.textContent);
- const runCountTextP = page.$eval('div:nth-of-type(2) > ul > li:nth-of-type(2)', ((el) => el.textContent));
+ const urlPart = url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').map((el) => el.textContent).wait();
+ const sku = await page.locator('span.product-meta__sku-number').map((el) => el.textContent).wait();
+
+ const rawPriceString = await page
+ .locator('span.price')
+ .filter((el) => el.textContent.includes('$'))
+ .map((el) => el.textContent)
+ .wait();
- const [
+ const rawPrice = rawPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStock = await page
+ .locator('span.product-form__inventory')
+ .filter((el) => el.textContent.includes('In stock'))
+ .map((el) => (!!el))
+ .wait();
+
+ const results = {
+ url,
+ manufacturer,
title,
- description,
- modifiedTimestamp,
- runCountText,
- ] = await Promise.all([
- titleP,
- descriptionP,
- modifiedTimestampP,
- runCountTextP,
- ]);
-
- await Dataset.pushData({ url, uniqueIdentifier, title, description, modifiedDate: modifiedTimestamp, runCount: runCountText });
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ await Dataset.pushData(results);
});
-await crawler.run([{ url: 'https://apify.com/store?page=1', userData: { label: 'START' } }]);
+await crawler.run([{ url: 'https://warehouse-theme-metal.myshopify.com/collections/all-tvs', userData: { label: 'START' } }]);
await Actor.exit({ exit: Actor.isAtHome() });
diff --git a/test/e2e/puppeteer-store-pagination/test.mjs b/test/e2e/puppeteer-store-pagination/test.mjs
index 308a11c07..8f87841e7 100644
--- a/test/e2e/puppeteer-store-pagination/test.mjs
+++ b/test/e2e/puppeteer-store-pagination/test.mjs
@@ -8,17 +8,6 @@ const { stats, datasetItems } = await runActor(testActorDirname, 16384);
await expect(stats.requestsFinished >= 10, 'All requests finished');
await expect(datasetItems.length > 5 && datasetItems.length < 15, 'Number of dataset items');
await expect(
- validateDataset(
- datasetItems,
- [
- 'url',
- 'title',
- 'uniqueIdentifier',
- 'description',
- // Skip modifiedAt and runCount since they changed
- // 'modifiedDate',
- // 'runCount',
- ],
- ),
+ validateDataset(datasetItems, ['url', 'manufacturer', 'title', 'sku', 'currentPrice', 'availableInStock']),
'Dataset items validation',
);
diff --git a/test/e2e/tools.mjs b/test/e2e/tools.mjs
index 56c17340c..ae18eda80 100644
--- a/test/e2e/tools.mjs
+++ b/test/e2e/tools.mjs
@@ -438,7 +438,7 @@ function checkDatasetItem(item, propName) {
case 'runCount':
return Number.isInteger(item.runCount);
default:
- return typeof item[propName] === 'string';
+ return ['string', 'number', 'boolean'].includes(typeof item[propName]);
}
}
diff --git a/website/static/img/getting-started/current-price.jpg b/website/static/img/getting-started/current-price.jpg
new file mode 100644
index 000000000..157229bb5
Binary files /dev/null and b/website/static/img/getting-started/current-price.jpg differ
diff --git a/website/static/img/getting-started/description.jpg b/website/static/img/getting-started/description.jpg
deleted file mode 100644
index a6ce595be..000000000
Binary files a/website/static/img/getting-started/description.jpg and /dev/null differ
diff --git a/website/static/img/getting-started/modified-date.jpg b/website/static/img/getting-started/modified-date.jpg
deleted file mode 100644
index 631efda5b..000000000
Binary files a/website/static/img/getting-started/modified-date.jpg and /dev/null differ
diff --git a/website/static/img/getting-started/scraping-practice.jpg b/website/static/img/getting-started/scraping-practice.jpg
index 3b358015a..f6c32ed9b 100644
Binary files a/website/static/img/getting-started/scraping-practice.jpg and b/website/static/img/getting-started/scraping-practice.jpg differ
diff --git a/website/static/img/getting-started/select-an-element.png b/website/static/img/getting-started/select-an-element.png
index 722f07286..f72f48ddf 100644
Binary files a/website/static/img/getting-started/select-an-element.png and b/website/static/img/getting-started/select-an-element.png differ
diff --git a/website/static/img/getting-started/selected-element.png b/website/static/img/getting-started/selected-element.png
index 64de73a7a..11a48635f 100644
Binary files a/website/static/img/getting-started/selected-element.png and b/website/static/img/getting-started/selected-element.png differ
diff --git a/website/static/img/getting-started/sku.jpg b/website/static/img/getting-started/sku.jpg
new file mode 100644
index 000000000..6d0cbef45
Binary files /dev/null and b/website/static/img/getting-started/sku.jpg differ
diff --git a/website/static/img/getting-started/title.jpg b/website/static/img/getting-started/title.jpg
index 9c6d09d31..5c78c5ab8 100644
Binary files a/website/static/img/getting-started/title.jpg and b/website/static/img/getting-started/title.jpg differ
diff --git a/website/versioned_docs/version-3.0/deployment/apify_platform.mdx b/website/versioned_docs/version-3.0/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.0/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.0/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.0/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.0/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.0/deployment/aws-browsers.md b/website/versioned_docs/version-3.0/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.0/deployment/aws-cheerio.md b/website/versioned_docs/version-3.0/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.0/deployment/gcp-browsers.md b/website/versioned_docs/version-3.0/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.0/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.0/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.0/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
index 9257a4ce9..a69852298 100644
--- a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
+++ b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
@@ -11,9 +11,9 @@ import ApiLink from '@site/src/components/ApiLink';
To run Crawlee on your own computer, you need to meet the following pre-requisites first:
1. Have **Node.js version 16.0** or higher installed.
-- Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
+ - Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
2. Have **NPM** installed, or use other package manager of your choice.
-- NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
+ - NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
If not certain, confirm the prerequisites by running:
diff --git a/website/versioned_docs/version-3.0/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.0/introduction/02-first-crawler.mdx
index 68ecc5b92..fd36ebedd 100644
--- a/website/versioned_docs/version-3.0/introduction/02-first-crawler.mdx
+++ b/website/versioned_docs/version-3.0/introduction/02-first-crawler.mdx
@@ -40,7 +40,7 @@ We are using a JavaScript feature called [Top level await](https://blog.saeloun.
Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
@@ -65,7 +65,7 @@ Unless you have a good reason to start with a different one, you should try buil
Let's continue with the earlier `RequestQueue` example.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';
@@ -100,7 +100,7 @@ The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scr
Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
diff --git a/website/versioned_docs/version-3.0/introduction/03-filter-el.ts b/website/versioned_docs/version-3.0/introduction/03-filter-el.ts
index 00c9b0f9a..9f9c09f73 100644
--- a/website/versioned_docs/version-3.0/introduction/03-filter-el.ts
+++ b/website/versioned_docs/version-3.0/introduction/03-filter-el.ts
@@ -5,6 +5,9 @@ const crawler = new CheerioCrawler({
async requestHandler({ $, request, enqueueLinks }) {
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // The default behavior of enqueueLinks is to stay on the same hostname,
+ // so it does not require any parameters.
+ // This will ensure the subdomain stays the same.
await enqueueLinks();
},
});
diff --git a/website/versioned_docs/version-3.0/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.0/introduction/03-filter-without-el.ts
index b945dd64b..e50b05d8f 100644
--- a/website/versioned_docs/version-3.0/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.0/introduction/03-filter-without-el.ts
@@ -7,19 +7,25 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
- // Without enqueueLinks, we first have to extract all
- // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Then we need to resolve relative URLs,
- // otherwise they would be unusable for crawling.
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl).href);
+ // Besides resolving the URLs, we now also need to
+ // grab their hostname for filtering.
+ const { hostname } = new URL(request.loadedUrl);
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl),
+ );
+
+ // We use the hostname to filter links that point
+ // to a different domain, even subdomain.
+ const sameHostnameLinks = absoluteUrls
+ .filter((url) => url.hostname === hostname)
+ .map((url) => ({ url: url.href }));
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(absoluteUrls);
+ await crawler.addRequests(sameHostnameLinks);
},
});
diff --git a/website/versioned_docs/version-3.0/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.0/introduction/03-find-without-el.ts
index 9b3aa6543..33561d019 100644
--- a/website/versioned_docs/version-3.0/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.0/introduction/03-find-without-el.ts
@@ -9,24 +9,20 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // Without enqueueLinks, we first have to extract all
+ // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Besides resolving the URLs, we now also need to
- // grab their hostname for filtering.
- const { hostname } = new URL(request.loadedUrl);
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl));
-
- // We use the hostname to filter links that point
- // to a different domain, even subdomain.
- const sameHostnameLinks = absoluteUrls
- .filter((url) => url.hostname === hostname)
- .map((url) => ({ url: url.href }));
+ // Then we need to resolve relative URLs,
+ // otherwise they would be unusable for crawling.
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl).href,
+ );
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(sameHostnameLinks);
+ await crawler.addRequests(absoluteUrls);
},
});
diff --git a/website/versioned_docs/version-3.0/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.0/introduction/04-pw-w-cheerio.ts
index 2873909e8..448d8fd2a 100644
--- a/website/versioned_docs/version-3.0/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.0/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('.ActorStoreItem').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.0/introduction/04-pw.ts b/website/versioned_docs/version-3.0/introduction/04-pw.ts
index 8260549a3..076094467 100644
--- a/website/versioned_docs/version-3.0/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.0/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.0/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.0/introduction/04-real-world-project.mdx
index a54e29bea..d0c0c893b 100644
--- a/website/versioned_docs/version-3.0/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.0/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
-Now that you know where to start and how to find all the actor details, let's look at the crawling process.
+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of actors (our start URL).
-2. Enqueue all links to actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an actor detail page, scrape the data.
-5. Repeat until all results pages and all actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,20 +113,24 @@ Now that you know where to start and how to find all the actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
-## DevTools - the scrapers toolbox
+## DevTools - the scraper's toolbox
:::info
@@ -141,11 +138,11 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
-In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.
+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.
![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('.ActorStoreItem');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.0/introduction/05-crawling.mdx b/website/versioned_docs/version-3.0/introduction/05-crawling.mdx
index 979779f7e..a646699e5 100644
--- a/website/versioned_docs/version-3.0/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.0/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.0/introduction/06-example.ts b/website/versioned_docs/version-3.0/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.0/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.0/introduction/06-scraping.mdx b/website/versioned_docs/version-3.0/introduction/06-scraping.mdx
index 38b2289c2..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.0/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.0/introduction/06-scraping.mdx
@@ -1,27 +1,28 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -30,32 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.
+:::caution
-### Title
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+:::
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
-:::caution
+### Title
-In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-:::
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -63,118 +63,113 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('span.actor-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('h1').textContent();
-
-const description = await page.locator('span.actor-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-```js
-import { PlaywrightCrawler } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- console.log(results)
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
+
+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
+
+```json
+{
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
+}
```
-
-
-When you run the actor, you should see the crawled URLs and their scraped data printed to the console.
-
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.0/introduction/07-example.ts b/website/versioned_docs/version-3.0/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.0/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.0/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.0/introduction/07-saving-data.mdx
index 717b9f342..8f059d74c 100644
--- a/website/versioned_docs/version-3.0/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.0/introduction/07-saving-data.mdx
@@ -4,7 +4,9 @@ title: "Saving data"
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -22,46 +24,9 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
-```js
-import { PlaywrightCrawler, Dataset } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
## What's `Dataset.pushData()`
@@ -91,4 +56,4 @@ If you would like to store your data in a single big file, instead of many small
## Next lesson
-In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.0/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.0/introduction/08-refactoring.mdx
index 0a23142d5..8c526cdcb 100644
--- a/website/versioned_docs/version-3.0/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.0/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,15 +23,15 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
```js title="src/main.mjs"
import { PlaywrightCrawler, log } from 'crawlee';
-import { router } from './routes.js';
+import { router } from './routes.mjs';
// This is better set with CRAWLEE_LOG_LEVEL env var
// or a configuration option. This is just for show 😈
@@ -44,11 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-log.debug('Adding requests to the queue.');
-await crawler.addRequests(['https://apify.com/store']);
-
-// crawler.run has its own logs 🙂
-await crawler.run();
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -62,45 +58,85 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
-// as well as the LIST labelled URLs.
+// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('.ActorStoreItem');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
-Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.
## Splitting your code into multiple files
@@ -110,9 +146,7 @@ There's no reason not to split your code into multiple files and keep your logic
We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
-Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.
-
-The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.
## Using a router to structure your crawling
@@ -120,18 +154,6 @@ At first, it might seem more readable using just a simple `if / else` statement
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
-## Learning more about web scraping
-
-:::tip
-
-If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
-
-:::
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-## Thank you! 🎉
+## Next lesson
-That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.0/introduction/09-deployment.mdx b/website/versioned_docs/version-3.0/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.0/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.1/deployment/apify_platform.mdx b/website/versioned_docs/version-3.1/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.1/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.1/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.1/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.1/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.1/deployment/aws-browsers.md b/website/versioned_docs/version-3.1/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.1/deployment/aws-cheerio.md b/website/versioned_docs/version-3.1/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.1/deployment/gcp-browsers.md b/website/versioned_docs/version-3.1/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.1/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.1/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.1/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.1/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.1/introduction/02-first-crawler.mdx
index 68ecc5b92..fd36ebedd 100644
--- a/website/versioned_docs/version-3.1/introduction/02-first-crawler.mdx
+++ b/website/versioned_docs/version-3.1/introduction/02-first-crawler.mdx
@@ -40,7 +40,7 @@ We are using a JavaScript feature called [Top level await](https://blog.saeloun.
Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
@@ -65,7 +65,7 @@ Unless you have a good reason to start with a different one, you should try buil
Let's continue with the earlier `RequestQueue` example.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';
@@ -100,7 +100,7 @@ The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scr
Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
diff --git a/website/versioned_docs/version-3.1/introduction/03-filter-el.ts b/website/versioned_docs/version-3.1/introduction/03-filter-el.ts
index 00c9b0f9a..9f9c09f73 100644
--- a/website/versioned_docs/version-3.1/introduction/03-filter-el.ts
+++ b/website/versioned_docs/version-3.1/introduction/03-filter-el.ts
@@ -5,6 +5,9 @@ const crawler = new CheerioCrawler({
async requestHandler({ $, request, enqueueLinks }) {
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // The default behavior of enqueueLinks is to stay on the same hostname,
+ // so it does not require any parameters.
+ // This will ensure the subdomain stays the same.
await enqueueLinks();
},
});
diff --git a/website/versioned_docs/version-3.1/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.1/introduction/03-filter-without-el.ts
index b945dd64b..e50b05d8f 100644
--- a/website/versioned_docs/version-3.1/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.1/introduction/03-filter-without-el.ts
@@ -7,19 +7,25 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
- // Without enqueueLinks, we first have to extract all
- // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Then we need to resolve relative URLs,
- // otherwise they would be unusable for crawling.
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl).href);
+ // Besides resolving the URLs, we now also need to
+ // grab their hostname for filtering.
+ const { hostname } = new URL(request.loadedUrl);
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl),
+ );
+
+ // We use the hostname to filter links that point
+ // to a different domain, even subdomain.
+ const sameHostnameLinks = absoluteUrls
+ .filter((url) => url.hostname === hostname)
+ .map((url) => ({ url: url.href }));
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(absoluteUrls);
+ await crawler.addRequests(sameHostnameLinks);
},
});
diff --git a/website/versioned_docs/version-3.1/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.1/introduction/03-find-without-el.ts
index 9b3aa6543..33561d019 100644
--- a/website/versioned_docs/version-3.1/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.1/introduction/03-find-without-el.ts
@@ -9,24 +9,20 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // Without enqueueLinks, we first have to extract all
+ // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Besides resolving the URLs, we now also need to
- // grab their hostname for filtering.
- const { hostname } = new URL(request.loadedUrl);
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl));
-
- // We use the hostname to filter links that point
- // to a different domain, even subdomain.
- const sameHostnameLinks = absoluteUrls
- .filter((url) => url.hostname === hostname)
- .map((url) => ({ url: url.href }));
+ // Then we need to resolve relative URLs,
+ // otherwise they would be unusable for crawling.
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl).href,
+ );
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(sameHostnameLinks);
+ await crawler.addRequests(absoluteUrls);
},
});
diff --git a/website/versioned_docs/version-3.1/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.1/introduction/04-pw-w-cheerio.ts
index 2873909e8..448d8fd2a 100644
--- a/website/versioned_docs/version-3.1/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.1/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('.ActorStoreItem').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.1/introduction/04-pw.ts b/website/versioned_docs/version-3.1/introduction/04-pw.ts
index 8260549a3..076094467 100644
--- a/website/versioned_docs/version-3.1/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.1/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.1/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.1/introduction/04-real-world-project.mdx
index a54e29bea..d0c0c893b 100644
--- a/website/versioned_docs/version-3.1/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.1/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
-Now that you know where to start and how to find all the actor details, let's look at the crawling process.
+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of actors (our start URL).
-2. Enqueue all links to actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an actor detail page, scrape the data.
-5. Repeat until all results pages and all actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,20 +113,24 @@ Now that you know where to start and how to find all the actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
-## DevTools - the scrapers toolbox
+## DevTools - the scraper's toolbox
:::info
@@ -141,11 +138,11 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
-In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.
+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.
![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('.ActorStoreItem');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.1/introduction/05-crawling.mdx b/website/versioned_docs/version-3.1/introduction/05-crawling.mdx
index 979779f7e..a646699e5 100644
--- a/website/versioned_docs/version-3.1/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.1/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.1/introduction/06-example.ts b/website/versioned_docs/version-3.1/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.1/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.1/introduction/06-scraping.mdx b/website/versioned_docs/version-3.1/introduction/06-scraping.mdx
index 38b2289c2..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.1/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.1/introduction/06-scraping.mdx
@@ -1,27 +1,28 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -30,32 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.
+:::caution
-### Title
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+:::
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
-:::caution
+### Title
-In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-:::
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -63,118 +63,113 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('span.actor-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('h1').textContent();
-
-const description = await page.locator('span.actor-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-```js
-import { PlaywrightCrawler } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- console.log(results)
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
+
+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
+
+```json
+{
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
+}
```
-
-
-When you run the actor, you should see the crawled URLs and their scraped data printed to the console.
-
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.1/introduction/07-example.ts b/website/versioned_docs/version-3.1/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.1/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.1/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.1/introduction/07-saving-data.mdx
index 717b9f342..8f059d74c 100644
--- a/website/versioned_docs/version-3.1/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.1/introduction/07-saving-data.mdx
@@ -4,7 +4,9 @@ title: "Saving data"
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -22,46 +24,9 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
-```js
-import { PlaywrightCrawler, Dataset } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
## What's `Dataset.pushData()`
@@ -91,4 +56,4 @@ If you would like to store your data in a single big file, instead of many small
## Next lesson
-In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.1/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.1/introduction/08-refactoring.mdx
index 0a23142d5..8c526cdcb 100644
--- a/website/versioned_docs/version-3.1/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.1/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,15 +23,15 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
```js title="src/main.mjs"
import { PlaywrightCrawler, log } from 'crawlee';
-import { router } from './routes.js';
+import { router } from './routes.mjs';
// This is better set with CRAWLEE_LOG_LEVEL env var
// or a configuration option. This is just for show 😈
@@ -44,11 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-log.debug('Adding requests to the queue.');
-await crawler.addRequests(['https://apify.com/store']);
-
-// crawler.run has its own logs 🙂
-await crawler.run();
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -62,45 +58,85 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
-// as well as the LIST labelled URLs.
+// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('.ActorStoreItem');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
-Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.
## Splitting your code into multiple files
@@ -110,9 +146,7 @@ There's no reason not to split your code into multiple files and keep your logic
We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
-Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.
-
-The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.
## Using a router to structure your crawling
@@ -120,18 +154,6 @@ At first, it might seem more readable using just a simple `if / else` statement
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
-## Learning more about web scraping
-
-:::tip
-
-If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
-
-:::
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-## Thank you! 🎉
+## Next lesson
-That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.1/introduction/09-deployment.mdx b/website/versioned_docs/version-3.1/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.1/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.2/deployment/apify_platform.mdx b/website/versioned_docs/version-3.2/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.2/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.2/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.2/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.2/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.2/deployment/aws-browsers.md b/website/versioned_docs/version-3.2/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.2/deployment/aws-cheerio.md b/website/versioned_docs/version-3.2/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.2/deployment/gcp-browsers.md b/website/versioned_docs/version-3.2/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.2/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.2/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.2/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.2/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.2/introduction/02-first-crawler.mdx
index 68ecc5b92..fd36ebedd 100644
--- a/website/versioned_docs/version-3.2/introduction/02-first-crawler.mdx
+++ b/website/versioned_docs/version-3.2/introduction/02-first-crawler.mdx
@@ -40,7 +40,7 @@ We are using a JavaScript feature called [Top level await](https://blog.saeloun.
Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
@@ -65,7 +65,7 @@ Unless you have a good reason to start with a different one, you should try buil
Let's continue with the earlier `RequestQueue` example.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';
@@ -100,7 +100,7 @@ The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scr
Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
diff --git a/website/versioned_docs/version-3.2/introduction/03-filter-el.ts b/website/versioned_docs/version-3.2/introduction/03-filter-el.ts
index 00c9b0f9a..9f9c09f73 100644
--- a/website/versioned_docs/version-3.2/introduction/03-filter-el.ts
+++ b/website/versioned_docs/version-3.2/introduction/03-filter-el.ts
@@ -5,6 +5,9 @@ const crawler = new CheerioCrawler({
async requestHandler({ $, request, enqueueLinks }) {
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // The default behavior of enqueueLinks is to stay on the same hostname,
+ // so it does not require any parameters.
+ // This will ensure the subdomain stays the same.
await enqueueLinks();
},
});
diff --git a/website/versioned_docs/version-3.2/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.2/introduction/03-filter-without-el.ts
index b945dd64b..e50b05d8f 100644
--- a/website/versioned_docs/version-3.2/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.2/introduction/03-filter-without-el.ts
@@ -7,19 +7,25 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
- // Without enqueueLinks, we first have to extract all
- // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Then we need to resolve relative URLs,
- // otherwise they would be unusable for crawling.
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl).href);
+ // Besides resolving the URLs, we now also need to
+ // grab their hostname for filtering.
+ const { hostname } = new URL(request.loadedUrl);
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl),
+ );
+
+ // We use the hostname to filter links that point
+ // to a different domain, even subdomain.
+ const sameHostnameLinks = absoluteUrls
+ .filter((url) => url.hostname === hostname)
+ .map((url) => ({ url: url.href }));
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(absoluteUrls);
+ await crawler.addRequests(sameHostnameLinks);
},
});
diff --git a/website/versioned_docs/version-3.2/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.2/introduction/03-find-without-el.ts
index 9b3aa6543..33561d019 100644
--- a/website/versioned_docs/version-3.2/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.2/introduction/03-find-without-el.ts
@@ -9,24 +9,20 @@ const crawler = new CheerioCrawler({
const title = $('title').text();
console.log(`The title of "${request.url}" is: ${title}.`);
+ // Without enqueueLinks, we first have to extract all
+ // the URLs from the page with Cheerio.
const links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get();
- // Besides resolving the URLs, we now also need to
- // grab their hostname for filtering.
- const { hostname } = new URL(request.loadedUrl);
- const absoluteUrls = links
- .map((link) => new URL(link, request.loadedUrl));
-
- // We use the hostname to filter links that point
- // to a different domain, even subdomain.
- const sameHostnameLinks = absoluteUrls
- .filter((url) => url.hostname === hostname)
- .map((url) => ({ url: url.href }));
+ // Then we need to resolve relative URLs,
+ // otherwise they would be unusable for crawling.
+ const absoluteUrls = links.map(
+ (link) => new URL(link, request.loadedUrl).href,
+ );
// Finally, we have to add the URLs to the queue
- await crawler.addRequests(sameHostnameLinks);
+ await crawler.addRequests(absoluteUrls);
},
});
diff --git a/website/versioned_docs/version-3.2/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.2/introduction/04-pw-w-cheerio.ts
index 2873909e8..448d8fd2a 100644
--- a/website/versioned_docs/version-3.2/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.2/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('.ActorStoreItem').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.2/introduction/04-pw.ts b/website/versioned_docs/version-3.2/introduction/04-pw.ts
index 8260549a3..076094467 100644
--- a/website/versioned_docs/version-3.2/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.2/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.2/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.2/introduction/04-real-world-project.mdx
index a54e29bea..d0c0c893b 100644
--- a/website/versioned_docs/version-3.2/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.2/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
-Now that you know where to start and how to find all the actor details, let's look at the crawling process.
+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of actors (our start URL).
-2. Enqueue all links to actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an actor detail page, scrape the data.
-5. Repeat until all results pages and all actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,20 +113,24 @@ Now that you know where to start and how to find all the actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
-## DevTools - the scrapers toolbox
+## DevTools - the scraper's toolbox
:::info
@@ -141,11 +138,11 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
-In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.
+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.
![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('.ActorStoreItem');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.2/introduction/05-crawling.mdx b/website/versioned_docs/version-3.2/introduction/05-crawling.mdx
index 979779f7e..a646699e5 100644
--- a/website/versioned_docs/version-3.2/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.2/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.2/introduction/06-example.ts b/website/versioned_docs/version-3.2/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.2/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.2/introduction/06-scraping.mdx b/website/versioned_docs/version-3.2/introduction/06-scraping.mdx
index 38b2289c2..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.2/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.2/introduction/06-scraping.mdx
@@ -1,27 +1,28 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -30,32 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.
+:::caution
-### Title
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+:::
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
-:::caution
+### Title
-In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-:::
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -63,118 +63,113 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('span.actor-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('h1').textContent();
-
-const description = await page.locator('span.actor-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-```js
-import { PlaywrightCrawler } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- console.log(results)
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
+
+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
+
+```json
+{
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
+}
```
-
-
-When you run the actor, you should see the crawled URLs and their scraped data printed to the console.
-
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.2/introduction/07-example.ts b/website/versioned_docs/version-3.2/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.2/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.2/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.2/introduction/07-saving-data.mdx
index 717b9f342..8f059d74c 100644
--- a/website/versioned_docs/version-3.2/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.2/introduction/07-saving-data.mdx
@@ -4,7 +4,9 @@ title: "Saving data"
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -22,46 +24,9 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
-```js
-import { PlaywrightCrawler, Dataset } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
## What's `Dataset.pushData()`
@@ -91,4 +56,4 @@ If you would like to store your data in a single big file, instead of many small
## Next lesson
-In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.2/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.2/introduction/08-refactoring.mdx
index 0a23142d5..8c526cdcb 100644
--- a/website/versioned_docs/version-3.2/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.2/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,15 +23,15 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
```js title="src/main.mjs"
import { PlaywrightCrawler, log } from 'crawlee';
-import { router } from './routes.js';
+import { router } from './routes.mjs';
// This is better set with CRAWLEE_LOG_LEVEL env var
// or a configuration option. This is just for show 😈
@@ -44,11 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-log.debug('Adding requests to the queue.');
-await crawler.addRequests(['https://apify.com/store']);
-
-// crawler.run has its own logs 🙂
-await crawler.run();
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -62,45 +58,85 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
-// as well as the LIST labelled URLs.
+// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('.ActorStoreItem');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
-Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.
## Splitting your code into multiple files
@@ -110,9 +146,7 @@ There's no reason not to split your code into multiple files and keep your logic
We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
-Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.
-
-The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.
## Using a router to structure your crawling
@@ -120,18 +154,6 @@ At first, it might seem more readable using just a simple `if / else` statement
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
-## Learning more about web scraping
-
-:::tip
-
-If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
-
-:::
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-## Thank you! 🎉
+## Next lesson
-That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.2/introduction/09-deployment.mdx b/website/versioned_docs/version-3.2/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.2/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.3/deployment/apify_platform.mdx b/website/versioned_docs/version-3.3/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.3/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.3/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.3/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.3/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.3/deployment/aws-browsers.md b/website/versioned_docs/version-3.3/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.3/deployment/aws-cheerio.md b/website/versioned_docs/version-3.3/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.3/deployment/gcp-browsers.md b/website/versioned_docs/version-3.3/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.3/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.3/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.3/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.3/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.3/introduction/02-first-crawler.mdx
index 68ecc5b92..fd36ebedd 100644
--- a/website/versioned_docs/version-3.3/introduction/02-first-crawler.mdx
+++ b/website/versioned_docs/version-3.3/introduction/02-first-crawler.mdx
@@ -40,7 +40,7 @@ We are using a JavaScript feature called [Top level await](https://blog.saeloun.
Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
@@ -65,7 +65,7 @@ Unless you have a good reason to start with a different one, you should try buil
Let's continue with the earlier `RequestQueue` example.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';
@@ -100,7 +100,7 @@ The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scr
Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
diff --git a/website/versioned_docs/version-3.3/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.3/introduction/03-filter-without-el.ts
index de7e6e21c..e50b05d8f 100644
--- a/website/versioned_docs/version-3.3/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.3/introduction/03-filter-without-el.ts
@@ -15,7 +15,7 @@ const crawler = new CheerioCrawler({
// grab their hostname for filtering.
const { hostname } = new URL(request.loadedUrl);
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl)
+ (link) => new URL(link, request.loadedUrl),
);
// We use the hostname to filter links that point
diff --git a/website/versioned_docs/version-3.3/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.3/introduction/03-find-without-el.ts
index 0414c504b..33561d019 100644
--- a/website/versioned_docs/version-3.3/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.3/introduction/03-find-without-el.ts
@@ -18,7 +18,7 @@ const crawler = new CheerioCrawler({
// Then we need to resolve relative URLs,
// otherwise they would be unusable for crawling.
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl).href
+ (link) => new URL(link, request.loadedUrl).href,
);
// Finally, we have to add the URLs to the queue
diff --git a/website/versioned_docs/version-3.3/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.3/introduction/04-pw-w-cheerio.ts
index 2873909e8..448d8fd2a 100644
--- a/website/versioned_docs/version-3.3/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.3/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('.ActorStoreItem').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.3/introduction/04-pw.ts b/website/versioned_docs/version-3.3/introduction/04-pw.ts
index 8260549a3..076094467 100644
--- a/website/versioned_docs/version-3.3/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.3/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.3/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.3/introduction/04-real-world-project.mdx
index a54e29bea..d0c0c893b 100644
--- a/website/versioned_docs/version-3.3/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.3/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
-Now that you know where to start and how to find all the actor details, let's look at the crawling process.
+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of actors (our start URL).
-2. Enqueue all links to actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an actor detail page, scrape the data.
-5. Repeat until all results pages and all actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,20 +113,24 @@ Now that you know where to start and how to find all the actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
-## DevTools - the scrapers toolbox
+## DevTools - the scraper's toolbox
:::info
@@ -141,11 +138,11 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
-In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.
+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.
![select an element](/img/getting-started/select-an-element.png 'Finding the select an element tool.')
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('.ActorStoreItem');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.3/introduction/05-crawling.mdx b/website/versioned_docs/version-3.3/introduction/05-crawling.mdx
index 979779f7e..a646699e5 100644
--- a/website/versioned_docs/version-3.3/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.3/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-pages a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('.ActorStoreItem');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.3/introduction/06-example.ts b/website/versioned_docs/version-3.3/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.3/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.3/introduction/06-scraping.mdx b/website/versioned_docs/version-3.3/introduction/06-scraping.mdx
index 38b2289c2..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.3/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.3/introduction/06-scraping.mdx
@@ -1,27 +1,28 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -30,32 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.
+:::caution
-### Title
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+:::
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
-:::caution
+### Title
-In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-:::
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -63,118 +63,113 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('span.actor-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('h1').textContent();
-
-const description = await page.locator('span.actor-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.locator('span').last().textContent();
-const runCount = Number(runCountString.replaceAll(',', ''));
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-```js
-import { PlaywrightCrawler } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- console.log(results)
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
+
+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
+
+```json
+{
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
+}
```
-
-
-When you run the actor, you should see the crawled URLs and their scraped data printed to the console.
-
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.3/introduction/07-example.ts b/website/versioned_docs/version-3.3/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.3/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.3/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.3/introduction/07-saving-data.mdx
index 717b9f342..8f059d74c 100644
--- a/website/versioned_docs/version-3.3/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.3/introduction/07-saving-data.mdx
@@ -4,7 +4,9 @@ title: "Saving data"
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -22,46 +24,9 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
-```js
-import { PlaywrightCrawler, Dataset } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- await page.waitForSelector('.ActorStoreItem');
- await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
## What's `Dataset.pushData()`
@@ -91,4 +56,4 @@ If you would like to store your data in a single big file, instead of many small
## Next lesson
-In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.3/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.3/introduction/08-refactoring.mdx
index 0a23142d5..8c526cdcb 100644
--- a/website/versioned_docs/version-3.3/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.3/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,15 +23,15 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
```js title="src/main.mjs"
import { PlaywrightCrawler, log } from 'crawlee';
-import { router } from './routes.js';
+import { router } from './routes.mjs';
// This is better set with CRAWLEE_LOG_LEVEL env var
// or a configuration option. This is just for show 😈
@@ -44,11 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-log.debug('Adding requests to the queue.');
-await crawler.addRequests(['https://apify.com/store']);
-
-// crawler.run has its own logs 🙂
-await crawler.run();
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -62,45 +58,85 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.locator('span').last().textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('h1').textContent(),
- description: await page.locator('span.actor-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: Number(runCountString.replaceAll(',', '')),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
-// as well as the LIST labelled URLs.
+// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-pages a');
- await enqueueLinks({
- selector: '.ActorStorePagination-pages > a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('.ActorStoreItem');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: '.ActorStoreItem',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
-Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.
+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.
## Splitting your code into multiple files
@@ -110,9 +146,7 @@ There's no reason not to split your code into multiple files and keep your logic
We won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.
-Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.
-
-The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!
+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.
## Using a router to structure your crawling
@@ -120,18 +154,6 @@ At first, it might seem more readable using just a simple `if / else` statement
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
-## Learning more about web scraping
-
-:::tip
-
-If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
-
-:::
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-## Thank you! 🎉
+## Next lesson
-That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.3/introduction/09-deployment.mdx b/website/versioned_docs/version-3.3/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.3/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.4/deployment/apify_platform.mdx b/website/versioned_docs/version-3.4/deployment/apify_platform.mdx
new file mode 100644
index 000000000..e5cb39e6a
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/apify_platform.mdx
@@ -0,0 +1,305 @@
+---
+id: apify-platform
+title: Apify Platform
+description: Apify platform - large-scale and high-performance web scraping
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import MainSource from '!!raw-loader!./apify_platform_main.ts';
+import InitExitSource from '!!raw-loader!./apify_platform_init_exit.ts';
+
+Apify is a [platform](https://apify.com) built to serve large-scale and high-performance web scraping
+and automation needs. It provides easy access to [compute instances (Actors)](#what-is-an-actor),
+convenient [request](../guides/request-storage) and [result](../guides/result-storage) storages, [proxies](../guides/proxy-management),
+[scheduling](https://docs.apify.com/scheduler), [webhooks](https://docs.apify.com/webhooks)
+and [more](https://docs.apify.com/), accessible through a [web interface](https://console.apify.com)
+or an [API](https://docs.apify.com/api).
+
+While we think that the Apify platform is super cool, and it's definitely worth signing up for a
+[free account](https://console.apify.com/sign-up), **Crawlee is and will always be open source**,
+runnable locally or on any cloud infrastructure.
+
+:::note
+
+We do not test Crawlee in other cloud environments such as Lambda or on specific
+architectures such as Raspberry PI. We strive to make it work, but there are no guarantees.
+
+:::
+
+## Logging into Apify platform from Crawlee
+
+To access your [Apify account](https://console.apify.com/sign-up) from Crawlee, you must provide
+credentials - your [API token](https://console.apify.com/account?tab=integrations). You can do that
+either by utilizing [Apify CLI](https://github.com/apify/apify-cli) or with environment
+variables.
+
+Once you provide credentials to your scraper, you will be able to use all the Apify platform
+features, such as calling actors, saving to cloud storages, using Apify proxies,
+setting up webhooks and so on.
+
+### Log in with CLI
+
+Apify CLI allows you to log in to your Apify account on your computer. If you then run your
+scraper using the CLI, your credentials will automatically be added.
+
+```bash
+npm install -g apify-cli
+apify login -t YOUR_API_TOKEN
+```
+
+### Log in with environment variables
+
+Alternatively, you can always provide credentials to your scraper
+by setting the [`APIFY_TOKEN`](#apify_token) environment
+variable to your API token.
+
+> There's also the [`APIFY_PROXY_PASSWORD`](#apify_proxy_password)
+> environment variable. Actor automatically infers that from your token, but it can be useful
+> when you need to access proxies from a different account than your token represents.
+
+### Log in with Configuration
+
+Another option is to use the [`Configuration`](https://apify.github.io/apify-sdk-js/api/apify/class/Configuration) instance and set your api token there.
+
+```javascript
+import { Actor } from 'apify';
+
+const sdk = new Actor({ token: 'your_api_token' });
+```
+
+## What is an actor
+
+When you deploy your script to the Apify platform, it becomes an [actor](https://apify.com/actors).
+An actor is a serverless microservice that accepts an input and produces an output. It can run for
+a few seconds, hours or even infinitely. An actor can perform anything from a simple action such
+as filling out a web form or sending an email, to complex operations such as crawling an entire website
+and removing duplicates from a large dataset.
+
+Actors can be shared in the [Apify Store](https://apify.com/store) so that other people can use them.
+But don't worry, if you share your actor in the store and somebody uses it, it runs under their account,
+not yours.
+
+**Related links**
+
+- [Store of existing actors](https://apify.com/store)
+- [Documentation](https://docs.apify.com/actors)
+- [View actors in Apify Console](https://console.apify.com/actors)
+- [API reference](https://apify.com/docs/api/v2#/reference/actors)
+
+## Running an actor locally
+
+First let's create a boilerplate of the new actor. You could use Apify CLI and just run:
+
+```bash
+apify create my-hello-world
+```
+
+The CLI will prompt you to select a project boilerplate template - let's pick "Hello world". The tool will create a directory called `my-hello-world` with a Node.js project files. You can run the actor as follows:
+
+```bash
+cd my-hello-world
+apify run
+```
+
+## Running Crawlee code as an actor
+
+For running Crawlee code as an actor on [Apify platform](https://apify.com/actors) you should either:
+- use a combination of [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) functions;
+- or wrap it into [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) function.
+
+:::info NOTE
+- Adding [`Actor.init()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#init) and [`Actor.exit()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#exit) to your code are the only two important things needed to run it on Apify platform as an actor. `Actor.init()` is needed to initialize your actor (e.g. to set the correct storage implementation), while without `Actor.exit()` the process will simply never stop.
+- [`Actor.main()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#main) is an alternative to `Actor.init()` and `Actor.exit()` as it calls both behind the scenes.
+:::
+
+Let's look at the `CheerioCrawler` example from the [Quick Start](../quick-start) guide:
+
+<Tabs>
+ <TabItem value="main" label="Using Actor.main()" default>
+ <CodeBlock language="js">
+ {MainSource}
+ </CodeBlock>
+ </TabItem>
+ <TabItem value="init_exit" label="Using Actor.init() and Actor.exit()">
+ <CodeBlock language="js">
+ {InitExitSource}
+ </CodeBlock>
+ </TabItem>
+</Tabs>
+
+Note that you could also run your actor (that is using Crawlee) locally with Apify CLI. You could start it via the following command in your project folder:
+
+```bash
+apify run
+```
+
+## Deploying an actor to Apify platform
+
+Now (assuming you are already logged in to your Apify account) you can easily deploy your code to the Apify platform by running:
+
+```bash
+apify push
+```
+
+Your script will be uploaded to and built on the Apify platform so that it can be run there. For more information, view the
+[Apify Actor](https://docs.apify.com/cli) documentation.
+
+## Usage on Apify platform
+
+You can also develop your actor in an online code editor directly on the platform (you'll need an Apify Account). Let's go to the [Actors](https://console.apify.com/actors) page in the app, click *Create new* and then go to the *Source* tab and start writing the code or paste one of the examples from the [Examples](../examples) section.
+
+## Storages
+
+There are several things worth mentioning here.
+
+### Helper functions for default Key-Value Store and Dataset
+
+To simplify access to the _default_ storages, instead of using the helper functions of respective storage classes, you could use:
+- [`Actor.setValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#setValue), [`Actor.getValue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getValue), [`Actor.getInput()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#getInput) for `Key-Value Store`
+- [`Actor.pushData()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#pushData) for `Dataset`
+
+### Using platform storage in a local actor
+
+When you plan to use the platform storage while developing and running your actor locally, you should use [`Actor.openKeyValueStore()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openKeyValueStore), [`Actor.openDataset()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openDataset) and [`Actor.openRequestQueue()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#openRequestQueue) to open the respective storage.
+
+Using each of these methods allows to pass the [`OpenStorageOptions`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions) as a second argument, which has only one optional property: [`forceCloud`](https://apify.github.io/apify-sdk-js/api/apify/interface/OpenStorageOptions#forceCloud). If set to `true` - cloud storage will be used instead of the folder on the local disk.
+
+:::note
+If you don't plan to force usage of the platform storages when running the actor locally, there is no need to use the [`Actor`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor) class for it. The Crawlee variants <ApiLink to="core/class/KeyValueStore#open">`KeyValueStore.open()`</ApiLink>, <ApiLink to="core/class/Dataset#open">`Dataset.open()`</ApiLink> and <ApiLink to="core/class/RequestQueue#open">`RequestQueue.open()`</ApiLink> will work the same.
+:::
+
+### Getting public url of an item in the platform storage
+
+If you need to share a link to some file stored in a Key-Value Store on Apify Platform, you can use [`getPublicUrl()`](https://apify.github.io/apify-sdk-js/api/apify/class/KeyValueStore#getPublicUrl) method. It accepts only one parameter: `key` - the key of the item you want to share.
+
+```js
+import { KeyValueStore } from 'apify';
+
+const store = await KeyValueStore.open();
+await store.setValue('your-file', { foo: 'bar' });
+const url = store.getPublicUrl('your-file');
+// https://api.apify.com/v2/key-value-stores/<your-store-id>/records/your-file
+```
+
+### Exporting dataset data
+
+When the <ApiLink to="core/class/Dataset">`Dataset`</ApiLink> is stored on the [Apify platform](https://apify.com/actors), you can export its data to the following formats: HTML, JSON, CSV, Excel, XML and RSS. The datasets are displayed on the actor run details page and in the [Storage](https://console.apify.com/storage) section in the Apify Console. The actual data is exported using the [Get dataset items](https://apify.com/docs/api/v2#/reference/datasets/item-collection/get-items) Apify API endpoint. This way you can easily share the crawling results.
+
+**Related links**
+
+- [Apify platform storage documentation](https://docs.apify.com/storage)
+- [View storage in Apify Console](https://console.apify.com/storage)
+- [Key-value stores API reference](https://apify.com/docs/api/v2#/reference/key-value-stores)
+- [Datasets API reference](https://docs.apify.com/api/v2#/reference/datasets)
+- [Request queues API reference](https://docs.apify.com/api/v2#/reference/request-queues)
+
+## Environment variables
+
+The following are some additional environment variables specific to Apify platform. More Crawlee specific environment variables could be found in the [Environment Variables](../guides/configuration#environment-variables) guide.
+
+:::note
+
+It's important to notice that `CRAWLEE_` environment variables don't need to be replaced with equivalent `APIFY_` ones. Likewise, Crawlee understands `APIFY_` environment variables after calling `Actor.init()` or when using `Actor.main()`.
+
+:::
+
+### `APIFY_TOKEN`
+
+The API token for your Apify account. It is used to access the Apify API, e.g. to access cloud storage
+or to run an actor on the Apify platform. You can find your API token on the
+[Account Settings / Integrations](https://console.apify.com/account?tab=integrations) page.
+
+### Combinations of `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`
+
+> `CRAWLEE_STORAGE_DIR` env variable description could be found in [Environment Variables](../guides/configuration#crawlee_storage_dir) guide.
+
+By combining the env vars in various ways, you can greatly influence the actor's behavior.
+
+| Env Vars | API | Storages |
+| --------------------------------------- | --- | ---------------- |
+| none OR `CRAWLEE_STORAGE_DIR` | no | local |
+| `APIFY_TOKEN` | yes | Apify platform |
+| `APIFY_TOKEN` AND `CRAWLEE_STORAGE_DIR` | yes | local + platform |
+
+When using both `APIFY_TOKEN` and `CRAWLEE_STORAGE_DIR`, you can use all the Apify platform
+features and your data will be stored locally by default. If you want to access platform storages,
+you can use the `{ forceCloud: true }` option in their respective functions.
+
+```js
+import { Actor } from 'apify';
+import { Dataset } from 'crawlee';
+
+// or Dataset.open('my-local-data')
+const localDataset = await Actor.openDataset('my-local-data');
+// but here we need the `Actor` class
+const remoteDataset = await Actor.openDataset('my-dataset', { forceCloud: true });
+```
+
+### `APIFY_PROXY_PASSWORD`
+
+Optional password to [Apify Proxy](https://docs.apify.com/proxy) for IP address rotation.
+Assuming Apify Account was already created, you can find the password on the [Proxy page](https://console.apify.com/proxy)
+in the Apify Console. The password is automatically inferred using the `APIFY_TOKEN` env var,
+so in most cases, you don't need to touch it. You should use it when, for some reason,
+you need access to Apify Proxy, but not access to Apify API, or when you need access to
+proxy from a different account than your token represents.
+
+## Proxy management
+
+In addition to your own proxy servers and proxy servers acquired from
+third-party providers used together with Crawlee, you can also rely on [Apify Proxy](https://apify.com/proxy)
+for your scraping needs.
+
+### Apify Proxy
+
+If you are already subscribed to Apify Proxy, you can start using them immediately in only a few lines of code (for local usage you first should be [logged in](#logging-into-apify-platform-from-crawlee) to your Apify account.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration();
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Note that unlike using your own proxies in Crawlee, you shouldn't use the constructor to create <ApiLink to="core/class/ProxyConfiguration">`ProxyConfiguration`</ApiLink> instance. For using Apify Proxy you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function instead.
+
+### Apify Proxy Configuration
+
+With Apify Proxy, you can select specific proxy groups to use, or countries to connect from.
+This allows you to get better proxy performance after some initial research.
+
+```javascript
+import { Actor } from 'apify';
+
+const proxyConfiguration = await Actor.createProxyConfiguration({
+ groups: ['RESIDENTIAL'],
+ countryCode: 'US',
+});
+const proxyUrl = await proxyConfiguration.newUrl();
+```
+
+Now your crawlers will use only Residential proxies from the US. Note that you must first get access
+to a proxy group before you are able to use it. You can check proxy groups available to you
+in the [proxy dashboard](https://console.apify.com/proxy).
+
+### Apify Proxy vs. Own proxies
+
+The `ProxyConfiguration` class covers both Apify Proxy and custom proxy URLs so that
+you can easily switch between proxy providers. However, some features of the class
+are available only to Apify Proxy users, mainly because Apify Proxy is what
+one would call a super-proxy. It's not a single proxy server, but an API endpoint
+that allows connection through millions of different IP addresses. So the class
+essentially has two modes: Apify Proxy or Own (third party) proxy.
+
+The difference is easy to remember.
+- If you're using your own proxies - you should create an instance with the ProxyConfiguration <ApiLink to="core/class/ProxyConfiguration#constructor">`constructor`</ApiLink> function based on the provided <ApiLink to="core/interface/ProxyConfigurationOptions">`ProxyConfigurationOptions`</ApiLink>.
+- If you are planning to use Apify Proxy - you should create an instance using the [`Actor.createProxyConfiguration()`](https://apify.github.io/apify-sdk-js/api/apify/class/Actor#createProxyConfiguration) function. <ApiLink to="core/interface/ProxyConfigurationOptions#proxyUrls">`ProxyConfigurationOptions.proxyUrls`</ApiLink> and <ApiLink to="core/interface/ProxyConfigurationOptions#newUrlFunction">`ProxyConfigurationOptions.newUrlFunction`</ApiLink> enable use of your custom proxy URLs, whereas all the other options are there to configure Apify Proxy.
+
+**Related links**
+
+- [Apify Proxy docs](https://docs.apify.com/proxy)
diff --git a/website/versioned_docs/version-3.4/deployment/apify_platform_init_exit.ts b/website/versioned_docs/version-3.4/deployment/apify_platform_init_exit.ts
new file mode 100644
index 000000000..49a10f100
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/apify_platform_init_exit.ts
@@ -0,0 +1,27 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.init();
+
+const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+});
+
+// Enqueue the initial request and run the crawler
+await crawler.run(['https://www.iana.org/']);
+
+await Actor.exit();
diff --git a/website/versioned_docs/version-3.4/deployment/apify_platform_main.ts b/website/versioned_docs/version-3.4/deployment/apify_platform_main.ts
new file mode 100644
index 000000000..a338047e8
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/apify_platform_main.ts
@@ -0,0 +1,25 @@
+import { Actor } from 'apify';
+import { CheerioCrawler } from 'crawlee';
+
+await Actor.main(async () => {
+ const crawler = new CheerioCrawler({
+ async requestHandler({ request, $, enqueueLinks }) {
+ const { url } = request;
+
+ // Extract HTML title of the page.
+ const title = $('title').text();
+ console.log(`Title of ${url}: ${title}`);
+
+ // Add URLs that match the provided pattern.
+ await enqueueLinks({
+ globs: ['https://www.iana.org/*'],
+ });
+
+ // Save extracted data to dataset.
+ await Actor.pushData({ url, title });
+ },
+ });
+
+ // Enqueue the initial request and run the crawler
+ await crawler.run(['https://www.iana.org/']);
+});
diff --git a/website/versioned_docs/version-3.4/deployment/aws-browsers.md b/website/versioned_docs/version-3.4/deployment/aws-browsers.md
new file mode 100644
index 000000000..e2160fc1f
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/aws-browsers.md
@@ -0,0 +1,124 @@
+---
+id: aws-browsers
+title: Browsers on AWS Lambda
+---
+
+Running browser-enabled Crawlee crawlers in AWS Lambda is a bit complicated - but not too much. The main problem is that we have to upload not only our code and the dependencies, but also the **browser binaries**.
+
+## Managing browser binaries
+
+Fortunately, there are already some NPM packages that can help us with managing the browser binaries installation:
+
+- [@sparticuz/chromium](https://www.npmjs.com/package/@sparticuz/chromium) is an NPM package containing brotli-compressed chromium binaries. When run in the Lambda environment, the package unzips the binaries under the `/tmp/` path and returns the path to the executable.
+
+We just add this package to the project dependencies and zip the `node_modules` folder.
+
+```bash
+# Install the package
+npm i -S @sparticuz/chromium
+
+# Zip the dependencies
+zip -r dependencies.zip ./node_modules
+```
+
+We will now upload the `dependencies.zip` as a Lambda Layer to AWS. Unfortunately, we cannot do this directly - there is a 50MB limit on direct uploads (and the compressed Chromium build is around that size itself). Instead, we'll upload it as an object into an S3 storage and provide the link to that object during the layer creation.
+
+## Updating the code
+
+We also have to slightly update the Crawlee code:
+
+- First, we pass a new `Configuration` instance to the Crawler. This way, every crawler instance we create will have its own storage and won’t interfere with other crawler instances running in your Lambda environment.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Now, we actually have to supply the code with the Chromium path from the `@sparticuz/chromium` package. AWS Lambda execution also lacks some hardware support for GPU acceleration etc. - you can tell Chrome about this by passing the `aws_chromium.args` to the `args` parameter.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-next-line
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ // highlight-start
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ // highlight-end
+}, new Configuration({
+ persistStorage: false,
+}));
+
+```
+
+- Last but not least, we have to wrap the code in the exported `handler` function - this will become the Lambda AWS will be executing.
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+import aws_chromium from '@sparticuz/chromium';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ launchContext: {
+ launchOptions: {
+ executablePath: await aws_chromium.executablePath(),
+ args: aws_chromium.args,
+ headless: true
+ }
+ }
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ };
+}
+// highlight-end
+
+```
+
+## Deploying the code
+
+Now we can simply pack the code into a zip archive (minus the `node_modules` folder, we have put that in the Lambda Layer, remember?). We upload the code archive to AWS as the Lambda body, set up the Lambda so it uses the dependencies Layer, and test our newly created Lambda.
+
+:::tip Memory settings
+
+Since we’re using full-size browsers here, we have to update the Lambda configurations a bit. Most importantly, make sure to set the memory setting to **1024 MB or more** and update the **Lambda timeout**.
+
+The target timeout value depends on how long your crawler will be running. Try measuring the execution time when running your crawler locally and set the timeout accordingly.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.4/deployment/aws-cheerio.md b/website/versioned_docs/version-3.4/deployment/aws-cheerio.md
new file mode 100644
index 000000000..0a047ead7
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/aws-cheerio.md
@@ -0,0 +1,126 @@
+---
+id: aws-cheerio
+title: Cheerio on AWS Lambda
+---
+
+Locally, we can conveniently create a Crawlee project with `npx crawlee create`. In order to run this project on AWS Lambda, however, we need to do a few tweaks.
+
+## Updating the code
+
+Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.
+
+Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration, ProxyConfiguration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+Now, we wrap all the logic in a `handler` function. This is the actual “Lambda” that AWS will be executing later on.
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+// highlight-next-line
+};
+```
+
+:::tip **Important**
+
+Make sure to always instantiate a **new crawler instance for every Lambda**. AWS always keeps the environment running for some time after the first Lambda execution (in order to reduce cold-start times) - so any subsequent Lambda calls will access the already-used crawler instance.
+
+**TLDR: Keep your Lambda stateless.**
+
+:::
+
+
+Last things last, we also want to return the scraped data from the Lambda when the crawler run ends.
+
+In the end, your `main.js` script should look something like this:
+
+```javascript title="src/main.js"
+// For more information, see https://crawlee.dev/
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+export const handler = async (event, context) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-start
+ return {
+ statusCode: 200,
+ body: await crawler.getData(),
+ }
+ // highlight-end
+};
+```
+
+## Deploying the project
+
+Now it’s time to deploy our script on AWS!
+
+Let’s create a zip archive from our project (including the `node_modules` folder) by running `zip -r package.zip .` in the project folder.
+
+:::note Large `node_modules` folder?
+
+AWS has a limit of 50MB for direct file upload. Usually, our Crawlee projects won’t be anywhere near this limit, but we can easily exceed this with large dependency trees.
+
+A better way to install your project dependencies is by using Lambda Layers. With Layers, we can also share files between multiple Lambdas - and keep the actual “code” part of the Lambdas as slim as possible.
+
+**To create a Lambda Layer, we need to:**
+
+- Pack the `node_modules` folder into a separate zip file (the archive should contain one folder named `node_modules`).
+- Create a new Lambda layer from this archive. We’ll probably need to upload this file to AWS S3 storage and create the Lambda Layer like this.
+- After creating it, we simply tell our new Lambda function to use this layer.
+
+:::
+
+To deploy our actual code, we upload the `package.zip` archive as our code source.
+
+In Lambda Runtime Settings, we point the `handler` to the main function that runs the crawler. You can use slashes to describe directory structure and `.` to denote a named export. Our handler function is called `handler` and is exported from the `src/main.js` file, so we’ll use `src/main.handler` as the handler name.
+
+Now we’re all set! By clicking the **Test** button, we can send an example testing event to our new Lambda. The actual contents of the event don’t really matter for now - if you want, further parameterize your crawler run by analyzing the `event` object AWS passes as the first argument to the handler.
+
+:::tip
+
+In the Configuration tab in the AWS Lambda dashboard, you can configure the amount of memory the Lambda is running with or the size of the ephemeral storage.
+
+The memory size can greatly affect the execution speed of your Lambda.
+
+See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.4/deployment/gcp-browsers.md b/website/versioned_docs/version-3.4/deployment/gcp-browsers.md
new file mode 100644
index 000000000..0c6162e34
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/gcp-browsers.md
@@ -0,0 +1,91 @@
+---
+id: gcp-browsers
+title: Browsers in GCP Cloud Run
+---
+
+Running full-size browsers on GCP Cloud Functions is actually a bit different from doing so on AWS Lambda - [apparently](https://pptr.dev/troubleshooting#running-puppeteer-on-google-cloud-functions), the latest runtime versions miss dependencies required to run Chromium.
+
+If we want to run browser-enabled Crawlee crawlers on GCP, we’ll need to turn towards **Cloud Run.** Cloud Run is GCP’s platform for running Docker containers - other than that, (almost) everything is the same as with Cloud Functions / AWS Lambdas.
+
+GCP can spin up your containers on demand, so you’re only billed for the time it takes your container to return an HTTP response to the requesting client. In a way, it also provides a slightly better developer experience (than regular FaaS), as you can debug your Docker containers locally and be sure you’re getting the same setup in the cloud.
+
+## Preparing the project
+
+As always, we first pass a new `Configuration` instance to the crawler constructor:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+All we now need to do is wrap our crawler with an Express HTTP server handler, so it can communicate with the client via HTTP. Because the Cloud Run platform sees only an opaque Docker container, we have to take care of this bit ourselves.
+
+:::info
+
+GCP passes you an environment variable called `PORT` - your HTTP server is expected to be listening on this port (GCP exposes this one to the outer world).
+
+:::
+
+The `main.js` script should be looking like this in the end:
+
+```javascript title="src/main.js"
+import { Configuration, PlaywrightCrawler } from 'crawlee';
+import { router } from './routes.js';
+// highlight-start
+import express from 'express';
+const app = express();
+// highlight-end
+
+const startUrls = ['https://crawlee.dev'];
+
+
+// highlight-next-line
+app.get('/', async (req, res) => {
+ const crawler = new PlaywrightCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData());
+// highlight-next-line
+});
+
+// highlight-next-line
+app.listen(parseInt(process.env.PORT) || 3000);
+```
+
+:::tip
+
+Always make sure to keep all the logic in the request handler - as with other FaaS services, your request handlers have to be **stateless.**
+
+:::
+
+## Deploying to GCP
+
+Now, we’re ready to deploy! If you have initialized your project using `npx crawlee create`, the initialization script has prepared a Dockerfile for you.
+
+All you have to do now is run `gcloud run deploy` in your project folder (the one with your Dockerfile in it). The gcloud CLI application will ask you a few questions, such as what region you want to deploy your application in, or whether you want to make your application public or private.
+
+After answering those questions, you should be able to see your application in the GCP dashboard and run it using the link you find there.
+
+:::tip
+
+In case your first execution of your newly created Cloud Run fails, try editing the Run configuration - mainly setting the available memory to 1GiB or more and updating the request timeout according to the size of the website you are scraping.
+
+:::
\ No newline at end of file
diff --git a/website/versioned_docs/version-3.4/deployment/gcp-cheerio.md b/website/versioned_docs/version-3.4/deployment/gcp-cheerio.md
new file mode 100644
index 000000000..918ff29de
--- /dev/null
+++ b/website/versioned_docs/version-3.4/deployment/gcp-cheerio.md
@@ -0,0 +1,81 @@
+---
+id: gcp-cheerio
+title: Cheerio on GCP Cloud Functions
+---
+
+Running CheerioCrawler-based project in GCP functions is actually quite easy - you just have to make a few changes to the project code.
+
+## Updating the project
+
+Let’s first create the Crawlee project locally with `npx crawlee create`. Set the `"main"` field in the `package.json` file to `"src/main.js"`.
+
+```json title="package.json"
+{
+ "name": "my-crawlee-project",
+ "version": "1.0.0",
+ // highlight-next-line
+ "main": "src/main.js",
+ ...
+}
+```
+
+Now, let’s update the `main.js` file, namely:
+
+- Pass a separate `Configuration` instance (with the `persistStorage` option set to `false`) to the crawler constructor.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+const crawler = new CheerioCrawler({
+ requestHandler: router,
+// highlight-start
+}, new Configuration({
+ persistStorage: false,
+}));
+// highlight-end
+
+await crawler.run(startUrls);
+```
+
+- Wrap the crawler call in a separate handler function. This function:
+ - Can be asynchronous
+ - Takes two positional arguments - `req` (containing details about the user-made request to your cloud function) and `res` (response object you can modify).
+ - Call `res.send(data)` to return any data from the cloud function.
+- Export this function from the `src/main.js` module as a named export.
+
+```javascript title="src/main.js"
+import { CheerioCrawler, Configuration } from 'crawlee';
+import { router } from './routes.js';
+
+const startUrls = ['https://crawlee.dev'];
+
+// highlight-next-line
+export const handler = async (req, res) => {
+ const crawler = new CheerioCrawler({
+ requestHandler: router,
+ }, new Configuration({
+ persistStorage: false,
+ }));
+
+ await crawler.run(startUrls);
+
+ // highlight-next-line
+ return res.send(await crawler.getData())
+// highlight-next-line
+}
+```
+
+## Deploying to Google Cloud Platform
+
+In the Google Cloud dashboard, create a new function, allocate memory and CPUs to it, set region and function timeout.
+
+When deploying, pick **ZIP Upload**. You have to create a new GCP storage bucket to store the zip packages in.
+
+Now, for the package - you should zip all the contents of your project folder **excluding the `node_modules` folder** - GCP doesn’t have Layers like AWS Lambda does, but takes care of the project setup for us based on the `package.json` file).
+
+Also, make sure to set the **Entry point** to the name of the function you’ve exported from the `src/main.js` file. GCP takes the file from the `package.json`'s `main` field.
+
+After the Function deploys, you can test it by clicking the “Testing” tab. This tab contains a `curl` script that calls your new Cloud Function. To avoid having to install the `gcloud` CLI application locally, you can also run this script in the Cloud Shell by clicking the link above the code block.
diff --git a/website/versioned_docs/version-3.4/introduction/02-first-crawler.mdx b/website/versioned_docs/version-3.4/introduction/02-first-crawler.mdx
index 68ecc5b92..fd36ebedd 100644
--- a/website/versioned_docs/version-3.4/introduction/02-first-crawler.mdx
+++ b/website/versioned_docs/version-3.4/introduction/02-first-crawler.mdx
@@ -40,7 +40,7 @@ We are using a JavaScript feature called [Top level await](https://blog.saeloun.
Earlier you learned that the crawler uses a queue of requests as its source of URLs to crawl. Let's create it and add the first request.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
import { RequestQueue } from 'crawlee';
// First you create the request queue instance.
@@ -65,7 +65,7 @@ Unless you have a good reason to start with a different one, you should try buil
Let's continue with the earlier `RequestQueue` example.
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';
@@ -100,7 +100,7 @@ The title of "https://crawlee.dev" is: Crawlee · The scalable web crawling, scr
Earlier we mentioned that you'll learn how to use the `crawler.addRequests()` method to skip the request queue initialization. It's simple. Every crawler has an implicit `RequestQueue` instance, and you can add requests to it with the `crawler.addRequests()` method. In fact, you can go even further and just use the first parameter of `crawler.run()`!
-```ts title="src/main.mjs"
+```ts title="src/main.js"
// You don't need to import RequestQueue anymore
import { CheerioCrawler } from 'crawlee';
diff --git a/website/versioned_docs/version-3.4/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.4/introduction/03-filter-without-el.ts
index de7e6e21c..e50b05d8f 100644
--- a/website/versioned_docs/version-3.4/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.4/introduction/03-filter-without-el.ts
@@ -15,7 +15,7 @@ const crawler = new CheerioCrawler({
// grab their hostname for filtering.
const { hostname } = new URL(request.loadedUrl);
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl)
+ (link) => new URL(link, request.loadedUrl),
);
// We use the hostname to filter links that point
diff --git a/website/versioned_docs/version-3.4/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.4/introduction/03-find-without-el.ts
index 0414c504b..33561d019 100644
--- a/website/versioned_docs/version-3.4/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.4/introduction/03-find-without-el.ts
@@ -18,7 +18,7 @@ const crawler = new CheerioCrawler({
// Then we need to resolve relative URLs,
// otherwise they would be unusable for crawling.
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl).href
+ (link) => new URL(link, request.loadedUrl).href,
);
// Finally, we have to add the URLs to the queue
diff --git a/website/versioned_docs/version-3.4/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.4/introduction/04-pw-w-cheerio.ts
index b698f9fc4..448d8fd2a 100644
--- a/website/versioned_docs/version-3.4/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.4/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('div[data-test="actorCard"]').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.4/introduction/04-pw.ts b/website/versioned_docs/version-3.4/introduction/04-pw.ts
index 37becf161..076094467 100644
--- a/website/versioned_docs/version-3.4/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.4/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('div[data-test="actorCard"]', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.4/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.4/introduction/04-real-world-project.mdx
index 775b34fd6..d0c0c893b 100644
--- a/website/versioned_docs/version-3.4/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.4/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **Actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all Actors from [Apify Store](https://apify.com/store) and for each Actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the Actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the Actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - Actors. Quick count of Actors on one page of results makes 24. 8 rows times 3 Actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain Actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of Actors (our start URL).
-2. Enqueue all links to Actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an Actor detail page, scrape the data.
-5. Repeat until all results pages and all Actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,18 +113,22 @@ Now that you know where to start and how to find all the Actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the Actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual Actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `div[data-test="actorCard"]` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
## DevTools - the scraper's toolbox
@@ -141,7 +138,7 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the Actor cards. You'll see that all the card's data is displayed in an `<a>` element with a `data-test` attribute **actorCard**. It should now make sense how we got that `div[data-test="actorCard"]` selector. It's just a way to find all elements that are annotated with the `actorCard`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('div[data-test="actorCard"]');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 Actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson, we will crawl the whole store, including all the listing pages and all the Actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.4/introduction/05-crawling.mdx b/website/versioned_docs/version-3.4/introduction/05-crawling.mdx
index 848007e66..a646699e5 100644
--- a/website/versioned_docs/version-3.4/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.4/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with Actors - listing pages and also all the Actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the Actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-buttons a` selector, which selects all the `<a>` elements that are children of an element with `class=ActorStorePagination-buttons`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the Actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('div[data-test="actorCard"] a');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.4/introduction/06-example.ts b/website/versioned_docs/version-3.4/introduction/06-example.ts
new file mode 100644
index 000000000..cd1eceeb0
--- /dev/null
+++ b/website/versioned_docs/version-3.4/introduction/06-example.ts
@@ -0,0 +1,77 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ console.log(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.4/introduction/06-scraping.mdx b/website/versioned_docs/version-3.4/introduction/06-scraping.mdx
index 4a81741b9..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.4/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.4/introduction/06-scraping.mdx
@@ -1,27 +1,28 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the Actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the Actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -30,26 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the Actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the Actor.
+:::caution
+
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
+
+:::
+
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
### Title
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. Web pages usually have only a single `<h1>` tag, but this one has two. Because of that, we have to extend the selector with more information. The `<h1>` tag is enclosed in a `<div>` with class `ActorHeader-identificator`. We can leverage this to create a combined selector `.ActorHeader-identificator h1`. It selects any `<h1>` element that is a child of a different element with the class `ActorHeader-identificator`.
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -57,129 +63,113 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title, we need to find it using `Playwright` and a `.ActorHeader-identificator h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Using the DevTools, we find that the Actor description is inside a `<p>` tag with a class `ActorHeader-description`. And since there's no other `<p>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('p.ActorHeader-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-userMedallion ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the count from the last (second) `<li>` element. But it contains the "Runs " string as well, so we remove the "Runs " by replacing the word with empty string.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
-
-const description = await page.locator('p.ActorHeader-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
We have everything we need so grab our newly created scraping logic, dump it into our original `requestHandler()` and see the magic happen!
-```js
-import { PlaywrightCrawler } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString.replace('Runs ', ''),
- }
-
- console.log(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
- await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
- await page.waitForSelector('div[data-test="actorCard"] a');
- await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
-
-
+<RunnableCodeBlock className="language-js" type="playwright">
+ {ExampleCode}
+</RunnableCodeBlock>
When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:
```json
{
- "url": "https://apify.com/clockworks/tiktok-scraper",
- "uniqueIdentifier": "clockworks/tiktok-scraper",
- "owner": "clockworks",
- "title": "TikTok Scraper",
- "description": "Powerful TikTok scraper to extract data from TikTok videos, hashtags, and users. Use it to scrape TikTok profiles, hashtags, posts, URLs, numbers of shares, followers, hearts, names, video, and music-related data. Download TikTok data as a HTML, JSON, CSV, Excel, or XML doc.",
- "modifiedDate": "2023-04-26T12:13:19.387Z",
- "runCount": "1.6M"
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
}
```
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.4/introduction/07-example.ts b/website/versioned_docs/version-3.4/introduction/07-example.ts
new file mode 100644
index 000000000..43937be28
--- /dev/null
+++ b/website/versioned_docs/version-3.4/introduction/07-example.ts
@@ -0,0 +1,78 @@
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+const crawler = new PlaywrightCrawler({
+ requestHandler: async ({ page, request, enqueueLinks }) => {
+ console.log(`Processing: ${request.url}`);
+ if (request.label === 'DETAIL') {
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
+
+ const results = {
+ url: request.url,
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ // highlight-next-line
+ await Dataset.pushData(results);
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
+ await enqueueLinks({
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
+ }
+ },
+
+ // Let's limit our crawls to make our tests shorter and safer.
+ maxRequestsPerCrawl: 50,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.4/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.4/introduction/07-saving-data.mdx
index 3d7ea9202..8f059d74c 100644
--- a/website/versioned_docs/version-3.4/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.4/introduction/07-saving-data.mdx
@@ -4,7 +4,9 @@ title: "Saving data"
description: Your first steps into the world of scraping with Crawlee
---
+import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -22,46 +24,9 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
-```js
-import { PlaywrightCrawler, Dataset } from 'crawlee';
-
-const crawler = new PlaywrightCrawler({
- requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
-
- const results = {
- url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: (runCountString.replace('Runs ', '')),
- }
-
- await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
- await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
- await page.waitForSelector('div[data-test="actorCard"] a');
- await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
- }
- }
-});
-
-await crawler.run(['https://apify.com/store']);
-```
+<RunnableCodeBlock className="language-js" type="playwright">
+ {Example}
+</RunnableCodeBlock>
## What's `Dataset.pushData()`
@@ -91,4 +56,4 @@ If you would like to store your data in a single big file, instead of many small
## Next lesson
-In the next and final lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
+In the next lesson, we will show you some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
diff --git a/website/versioned_docs/version-3.4/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.4/introduction/08-refactoring.mdx
index 99a55f424..8c526cdcb 100644
--- a/website/versioned_docs/version-3.4/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.4/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,9 +23,9 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
@@ -44,8 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -59,41 +58,81 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString.replace('Runs ', ''),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-buttons a');
- await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('div[data-test="actorCard"] a');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
@@ -115,18 +154,6 @@ At first, it might seem more readable using just a simple `if / else` statement
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
-## Learning more about web scraping
-
-:::tip
-
-If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
-
-:::
-
-## Running your crawler in the Cloud
-
-Now that you have your crawler ready, it's the right time to think about where you want to run it. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready. To read more about how to run this Dockerfile in the cloud, check out the [Apify Platform guide](../guides/apify-platform).
-
-## Thank you! 🎉
+## Next lesson
-That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
+In the next and final lesson, we will show you how you can deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/website/versioned_docs/version-3.4/introduction/09-deployment.mdx b/website/versioned_docs/version-3.4/introduction/09-deployment.mdx
new file mode 100644
index 000000000..5273181be
--- /dev/null
+++ b/website/versioned_docs/version-3.4/introduction/09-deployment.mdx
@@ -0,0 +1,115 @@
+---
+id: deployment
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
+description: Deploying Crawlee projects to the Apify Platform
+---
+
+## Apify Platform
+
+Crawlee is developed by [**Apify**](https://apify.com), the web scraping and automation platform. You could say it is the **home of Crawlee projects**. In this section we will show you how to deploy the crawler there with just a few simple steps. You can deploy a **Crawlee** project wherever you want, but using the [**Apify Platform**](https://console.apify.com) will give you the best experience.
+
+In case you want to deploy your Crawlee project to other platforms, check out the [**Deployment**](../deployment) section of the documentation instead.
+
+With a few simple steps, you can convert your Crawlee project into a so-called **Actor**. Actors are serverless micro-apps that are easy to develop, run, share, and integrate. The infra, proxies, and storages are ready to go. [Learn more about Actors](https://apify.com/actors).
+
+:::info
+
+We started this guide by using the Crawlee CLI to bootstrap the project - it offers the basic Crawlee templates, including a ready-made `Dockerfile`. If you know you will be deploying your project to the Apify Platform, you might want to start with the Apify CLI instead. It also offers several project templates, and those are all set up to be used on the Apify Platform right ahead.
+
+:::
+
+## Dependencies
+
+The first step will be installing two new dependencies:
+
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
+ ```bash
+ npm install apify
+ ```
+
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+ ```bash
+ npm install -g apify-cli
+ ```
+
+## Logging in to the Apify Platform
+
+The next step will be [creating your Apify account](https://console.apify.com/sign-up). Don't worry, we have a **free tier**, so you can try things out before you buy in! Once you have that, it's time to log in with the just-installed [Apify CLI](https://docs.apify.com/cli/). You will need your personal access token, which you can find at https://console.apify.com/account#/integrations.
+
+```bash
+apify login
+```
+
+## Adjusting the code
+
+Now that you have your account set up, you will need to adjust the code a tiny bit. We will use the [Apify SDK](https://docs.apify.com/sdk/js/), which will help us to wire the Crawlee storages (like the `RequestQueue`) to their Apify Platform counterparts - otherwise Crawlee would keep things only in memory.
+
+Open your `src/main.js` file (or `src/main.ts` if you used a TypeScript template), and add `Actor.init()` to the beginning of your main script and `Actor.exit()` to the end of it. Don't forget to `await` those calls, as both functions are async. Your code should look like this:
+
+```ts title="src/main.js"
+// highlight-next-line
+import { Actor } from 'apify';
+import { PlaywrightCrawler, log } from 'crawlee';
+import { router } from './routes.mjs';
+
+// highlight-next-line
+await Actor.init();
+
+// This is better set with CRAWLEE_LOG_LEVEL env var
+// or a configuration option. This is just for show 😈
+log.setLevel(log.LEVELS.DEBUG);
+
+log.debug('Setting up crawler.');
+const crawler = new PlaywrightCrawler({
+ // Instead of the long requestHandler with
+ // if clauses we provide a router instance.
+ requestHandler: router,
+});
+
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
+
+// highlight-next-line
+await Actor.exit();
+```
+
+The `Actor.init()` call will configure Crawlee to use the Apify API instead of its default memory storage interface. It also sets up few other things, like listening to the platform events via websockets. The `Actor.exit()` call then handles graceful shutdown - it will close the open handles created by the `Actor.init()` call, as without that, the Node.js process would be stuck.
+
+:::info
+
+The `Actor.init()` call works conditionally based on the environment variables, namely based on the `APIFY_IS_AT_HOME` env var, which is set to `true` on the Apify Platform. This means that your project will remain working the same locally, but will use the Apify API when deployed to the Apify Platform.
+
+:::
+
+## Initializing the project
+
+We will also need to initialize the project for Apify, to do that, let's use the Apify CLI again:
+
+```bash
+apify init
+```
+
+This will create a folder called `.actor`, and an `actor.json` file inside it - this file contains the configuration relevant to the Apify Platform, namely the Actor name, version, build tag, and few other things. Check out the [relevant documentation](https://docs.apify.com/platform/actors/development/actor-definition/actor-json) to see all the different things you can set there up.
+
+## Ship it!
+
+And that's all, our project is now ready to be published on the Apify Platform. We will use the Apify CLI once more to do that:
+
+```bash
+apify push
+```
+
+This command will create an archive from your project, upload it to the Apify Platform and initiate a Docker build. Once finished, you will get a link to your new Actor on the platform.
+
+## Learning more about web scraping
+
+:::tip
+
+If you want to learn more about web scraping and browser automation, check out the [Apify Academy](https://developers.apify.com/academy). It's full of courses and tutorials on the topic. From beginner to advanced. And the best thing: **It's free and open source** ❤️
+
+:::
+
+## Thank you! 🎉
+
+That's it! Thanks for reading the whole introduction and if there's anything wrong, please 🙏 let us know on [GitHub](https://github.com/apify/crawlee) or in our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping! 👋
diff --git a/website/versioned_docs/version-3.5/introduction/03-filter-without-el.ts b/website/versioned_docs/version-3.5/introduction/03-filter-without-el.ts
index de7e6e21c..e50b05d8f 100644
--- a/website/versioned_docs/version-3.5/introduction/03-filter-without-el.ts
+++ b/website/versioned_docs/version-3.5/introduction/03-filter-without-el.ts
@@ -15,7 +15,7 @@ const crawler = new CheerioCrawler({
// grab their hostname for filtering.
const { hostname } = new URL(request.loadedUrl);
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl)
+ (link) => new URL(link, request.loadedUrl),
);
// We use the hostname to filter links that point
diff --git a/website/versioned_docs/version-3.5/introduction/03-find-without-el.ts b/website/versioned_docs/version-3.5/introduction/03-find-without-el.ts
index 0414c504b..33561d019 100644
--- a/website/versioned_docs/version-3.5/introduction/03-find-without-el.ts
+++ b/website/versioned_docs/version-3.5/introduction/03-find-without-el.ts
@@ -18,7 +18,7 @@ const crawler = new CheerioCrawler({
// Then we need to resolve relative URLs,
// otherwise they would be unusable for crawling.
const absoluteUrls = links.map(
- (link) => new URL(link, request.loadedUrl).href
+ (link) => new URL(link, request.loadedUrl).href,
);
// Finally, we have to add the URLs to the queue
diff --git a/website/versioned_docs/version-3.5/introduction/04-pw-w-cheerio.ts b/website/versioned_docs/version-3.5/introduction/04-pw-w-cheerio.ts
index b698f9fc4..448d8fd2a 100644
--- a/website/versioned_docs/version-3.5/introduction/04-pw-w-cheerio.ts
+++ b/website/versioned_docs/version-3.5/introduction/04-pw-w-cheerio.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Extract the page's HTML from browser
// and parse it with Cheerio.
const $ = await parseWithCheerio();
// Use familiar Cheerio syntax to
// select all the actor cards.
- $('div[data-test="actorCard"]').each((i, el) => {
+ $('.collection-block-item').each((i, el) => {
const text = $(el).text();
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.5/introduction/04-pw.ts b/website/versioned_docs/version-3.5/introduction/04-pw.ts
index 37becf161..076094467 100644
--- a/website/versioned_docs/version-3.5/introduction/04-pw.ts
+++ b/website/versioned_docs/version-3.5/introduction/04-pw.ts
@@ -5,17 +5,17 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page }) => {
// Wait for the actor cards to render.
- await page.waitForSelector('div[data-test="actorCard"]');
+ await page.waitForSelector('.collection-block-item');
// Execute a function in the browser which targets
// the actor card elements and allows their manipulation.
- const actorTexts = await page.$$eval('div[data-test="actorCard"]', (els) => {
+ const categoryTexts = await page.$$eval('.collection-block-item', (els) => {
// Extract text content from the actor cards
return els.map((el) => el.textContent);
});
- actorTexts.forEach((text, i) => {
- console.log(`ACTOR_${i + 1}: ${text}\n`);
+ categoryTexts.forEach((text, i) => {
+ console.log(`CATEGORY_${i + 1}: ${text}\n`);
});
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.5/introduction/04-real-world-project.mdx b/website/versioned_docs/version-3.5/introduction/04-real-world-project.mdx
index 775b34fd6..d0c0c893b 100644
--- a/website/versioned_docs/version-3.5/introduction/04-real-world-project.mdx
+++ b/website/versioned_docs/version-3.5/introduction/04-real-world-project.mdx
@@ -1,7 +1,7 @@
---
id: real-world-project
-title: "Getting some real-world data"
-sidebar_label: "Real-world project"
+title: 'Getting some real-world data'
+sidebar_label: 'Real-world project'
description: Your first steps into the world of scraping with Crawlee
---
@@ -13,13 +13,13 @@ import CodeBlock from '@theme/CodeBlock';
import PwSource from '!!raw-loader!./04-pw.ts';
import PwWCheerioSource from '!!raw-loader!./04-pw-w-cheerio.ts';
-> *Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!*
+> _Hey, guys, you know, it's cool that we can scrape the `<title>` elements of web pages, but that's not very useful. Can we finally scrape some real data and save it somewhere in a machine-readable format? Because that's why I started reading this tutorial in the first place!_
We hear you, young padawan! First, learn how to crawl, you must. Only then, walk through data, you can!
## Making a production-grade crawler
-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **Actors** that anyone can grab and use for free.
+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape an [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) instead of the Crawlee website. It contains a list of products of different categories, and each product has its own detail page.
The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.
@@ -32,41 +32,41 @@ If you're not interested in crawling theory, feel free to [skip to the next less
## Drawing a plan
Sometimes scraping is really straightforward, but most of the time, it really pays off to do a bit of research first and try to answer some of these questions:
-- How is the website structured?
-- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
-- Do I need a headless browser for something?
-- Are there any anti-scraping protections in place?
-- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
+
+- How is the website structured?
+- Can I scrape it only with HTTP requests (read "with `CheerioCrawler`")?
+- Do I need a headless browser for something?
+- Are there any anti-scraping protections in place?
+- Do I need to parse the HTML or can I get the data otherwise, such as directly from the website's API?
For the purposes of this tutorial, let's assume that the website cannot be scraped with `CheerioCrawler`. It actually can, but we would have to dive a bit deeper than this introductory guide allows. So for now we will make things easier for you, scrape it with `PlaywrightCrawler`, and you'll learn about headless browsers in the process.
## Choosing the data you need
-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all Actors from [Apify Store](https://apify.com/store) and for each Actor we want to get its:
+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all products from all categories available on the [All collections page of the store](https://warehouse-theme-metal.myshopify.com/collections) and for each product we want to get its:
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the Actor detail pages.
+You will notice that some information is available directly on the list page, but for details such as "SKU" we'll also need to open the product's detail page.
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
### The start URL(s)
-This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `apify.com` and look for a `store` link there, when we already know that everything we want to extract can be found at the `apify.com/store` page.
+This is where you start your crawl. It's convenient to start as close to the data as possible. For example, it wouldn't make much sense to start at `https://warehouse-theme-metal.myshopify.com/` and look for a `collections` link there, when we already know that everything we want to extract can be found at the `https://warehouse-theme-metal.myshopify.com/collections` page.
## Exploring the page
-Let's take a look at the `apify.com/store` page more carefully. There are some **categories** on the left, **sorting** on the right, and at the very bottom, there are links to the next pages of results. This is usually called **the pagination**.
+Let's take a look at the `https://warehouse-theme-metal.myshopify.com/collections` page more carefully. There are some **categories** on the page, and each category has a list of **items**. On some category pages, at the bottom you will notice there are links to the next pages of results. This is usually called **the pagination**.
### Categories and sorting
-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the Actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.
+When you click the categories, you'll see that they load a page of products filtered by that category. By going through a few categories and observing the behavior, we can also observe that we can sort by different conditions (such as `Best selling`, or `Price, low to high`), but for this example, we will not be looking into those.
:::caution
@@ -76,13 +76,13 @@ Be careful, because on some websites, like [amazon.com](https://amazon.com), thi
### Pagination
-The pagination of Apify Store is simple enough. The only setback is that there's no button that would allow you to go directly to the last page, so you can't confirm how many results there are. But you can trick it a bit. When you click on the page 2 button, you'll see that the URL changes to:
+The pagination of the demo Warehouse Store is simple enough. When switching between pages, you will see that the URL changes to:
```
-https://apify.com/store?page=2
+https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2
```
-Try clicking on the link to page 6. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
+Try clicking on the link to page 4. You'll see that the pagination links update and show more pages. But can you trust that this will include all pages and won't stop at some point?
:::caution
@@ -90,29 +90,22 @@ Similarly to the issue with filters explained above, the existence of pagination
:::
-At the time of writing the Store results counter showed 1047 results - Actors. Quick count of Actors on one page of results makes 24. 8 rows times 3 Actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).
-
-```
-https://apify.com/store?page=44
-```
-
-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain Actors were hidden for some reason, but the count does not reflect it.
-
-At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍
+At the time of writing the `Headphones` collection results counter showed 75 results - products. Quick count of products on one page of results makes 24. 6 rows times 4 products. This means that there are 4 pages of results.
-If you're not convinced, you can visit a page somewhere in the middle, like `https://apify.com/store?page=20` and see how the pagination looks there.
+If you're not convinced, you can visit a page somewhere in the middle, like `https://warehouse-theme-metal.myshopify.com/collections/headphones?page=2` and see how the pagination looks there.
## The crawling strategy
Now that you know where to start and how to find all the Actor details, let's look at the crawling process.
-1. Visit the store page containing the list of Actors (our start URL).
-2. Enqueue all links to Actor details.
-3. Enqueue links to next pages of results.
-4. Open the next page in queue.
- - When it's a results list page, go to 2.
- - When it's an Actor detail page, scrape the data.
-5. Repeat until all results pages and all Actor details have been processed.
+1. Visit the store page containing the list of categories (our start URL).
+2. Enqueue all links to all categories.
+3. Enqueue all product pages from the current page.
+4. Enqueue links to next pages of results.
+5. Open the next page in queue.
+ - When it's a results list page, go to 2.
+ - When it's a product page, scrape the data.
+6. Repeat until all results pages and all products have been processed.
`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.
@@ -120,18 +113,22 @@ Now that you know where to start and how to find all the Actor details, let's lo
Let's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.
-The example below creates a new crawler that visits the start URL and prints the text content of all the Actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual Actor cards.
+The example below creates a new crawler that visits the start URL and prints the text content of all the categories on that page. When you run the code, you will see the _very badly formatted_ content of the individual category card.
<Tabs groupId="sanity-check">
<TabItem value="playwright" label="Playwright" default>
- <CodeBlock language="js" title="src/main.mjs">{PwSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwSource}
+ </CodeBlock>
</TabItem>
<TabItem value="playwright-with-cheerio" label="Playwright with Cheerio">
- <CodeBlock language="js" title="src/main.mjs">{PwWCheerioSource}</CodeBlock>
+ <CodeBlock language="js" title="src/main.mjs">
+ {PwWCheerioSource}
+ </CodeBlock>
</TabItem>
</Tabs>
-If you're wondering how to get that `div[data-test="actorCard"]` selector. We'll explain it in the next chapter on DevTools.
+If you're wondering how to get that `.collection-block-item` selector. We'll explain it in the next chapter on DevTools.
## DevTools - the scraper's toolbox
@@ -141,7 +138,7 @@ We'll use Chrome DevTools here, since it's the most common browser, but feel fre
:::
-Let's open DevTools by going to https://apify.com/store in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentaion](https://developer.chrome.com/docs/devtools/).
+Let's open DevTools by going to https://warehouse-theme-metal.myshopify.com/collections in Chrome and then right-clicking anywhere in the page and selecting **Inspect**, or by pressing **F12** or whatever your system prefers. With DevTools, you can inspect or manipulate any aspect of the currently open web page. You can learn more about DevTools in their [official documentation](https://developer.chrome.com/docs/devtools/).
## Selecting elements
@@ -155,15 +152,15 @@ You'll see that you can select different elements inside the card. Instead, sele
Selecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.
-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the Actor cards. You'll see that all the card's data is displayed in an `<a>` element with a `data-test` attribute **actorCard**. It should now make sense how we got that `div[data-test="actorCard"]` selector. It's just a way to find all elements that are annotated with the `actorCard`.
+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the collection card. You'll see that all the card's data is displayed in an `<a>` element with a `class` attribute that includes **collection-block-item**. It should now make sense how we got that `.collection-block-item` selector. It's just a way to find all elements that are annotated with the `collection-block-item`.
It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:
```ts
-document.querySelectorAll('div[data-test="actorCard"]');
+document.querySelectorAll('.collection-block-item');
```
-You will see that only the 24 Actor cards will be returned, and nothing else.
+You will see that only the 31 collection cards will be returned, and nothing else.
:::tip
@@ -173,4 +170,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis
## Next lesson
-In the next lesson, we will crawl the whole store, including all the listing pages and all the Actor detail pages.
+In the next lesson, we will crawl the whole store, including all the listing pages and all the product detail pages.
diff --git a/website/versioned_docs/version-3.5/introduction/05-crawling.mdx b/website/versioned_docs/version-3.5/introduction/05-crawling.mdx
index 848007e66..a646699e5 100644
--- a/website/versioned_docs/version-3.5/introduction/05-crawling.mdx
+++ b/website/versioned_docs/version-3.5/introduction/05-crawling.mdx
@@ -1,11 +1,11 @@
---
id: crawling
-title: "Crawling the Store"
-sidebar_label: "Crawling"
+title: 'Crawling the Store'
+sidebar_label: 'Crawling'
description: Your first steps into the world of scraping with Crawlee
---
-To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with Actors - listing pages and also all the Actor detail pages.
+To crawl the whole [example Warehouse Store](https://warehouse-theme-metal.myshopify.com/collections) and find all the data, you first need to visit all the pages with products - going through all categories available and also all the product detail pages.
## Crawling the listing pages
@@ -22,67 +22,77 @@ import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
- // Wait for the Actor cards to render,
+ console.log(`Processing: ${request.url}`);
+ // Wait for the category cards to render,
// otherwise enqueueLinks wouldn't enqueue anything.
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.collection-block-item');
// Add links to the queue, but only from
// elements matching the provided selector.
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The code should look pretty familiar to you. It's a very simple `requestHandler` where we log the currently processed URL to the console and enqueue more links. But there are also a few new, interesting additions. Let's break it down.
### The `selector` parameter of `enqueueLinks()`
-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-buttons a` selector, which selects all the `<a>` elements that are children of an element with `class=ActorStorePagination-buttons`.
+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the `Categories` page, and you're only interested in those that will take your crawler to the available list of results. Using the DevTools, you'll find that you can select the links you need using the `.collection-block-item` selector, which selects all the elements that have the `class=collection-block-item` attribute.
### The `label` of `enqueueLinks()`
-You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `LIST` to note that we're enqueueing pages with lists of results. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
+You will see `label` used often throughout Crawlee, as it's a convenient way of labelling a `Request` instance for quick identification later. You can access it with `request.label` and it's a `string`. You can name your requests any way you want. Here, we used the label `CATEGORY` to note that we're enqueueing pages that represent a category of products. The `enqueueLinks()` function will add this label to all requests before enqueueing them to the `RequestQueue`. Why this is useful will become obvious in a minute.
## Crawling the detail pages
-In a similar fashion, you need to collect all the URLs to the Actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
+In a similar fashion, you need to collect all the URLs to the product detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.
```ts
import { PlaywrightCrawler } from 'crawlee';
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
- console.log(`Processing: ${request.url}`)
+ console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
// We're not doing anything with the details yet.
- } else {
- // This means we're either on the start page, with no label,
- // or on a list page, with LIST label.
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
- // In addition to adding the listing URLs, we now also
- // add the detail URLs from all the listing pages.
- await page.waitForSelector('div[data-test="actorCard"] a');
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
}
},
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
The crawling code is now complete. When you run the code, you'll see the crawler visit all the listing URLs and all the detail URLs.
diff --git a/website/versioned_docs/version-3.5/introduction/06-example.ts b/website/versioned_docs/version-3.5/introduction/06-example.ts
index 85b4e36f4..cd1eceeb0 100644
--- a/website/versioned_docs/version-3.5/introduction/06-example.ts
+++ b/website/versioned_docs/version-3.5/introduction/06-example.ts
@@ -4,33 +4,68 @@ const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString?.replace('Runs ', ''),
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
};
console.log(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
});
- await page.waitForSelector('div[data-test="actorCard"] a');
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- // highlight-next-line
- label: 'DETAIL', // <= note the different label
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
});
}
},
@@ -39,4 +74,4 @@ const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 50,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.5/introduction/06-scraping.mdx b/website/versioned_docs/version-3.5/introduction/06-scraping.mdx
index 5220038fd..bf8b2d64f 100644
--- a/website/versioned_docs/version-3.5/introduction/06-scraping.mdx
+++ b/website/versioned_docs/version-3.5/introduction/06-scraping.mdx
@@ -1,7 +1,7 @@
---
id: scraping
-title: "Scraping the Store"
-sidebar_label: "Scraping"
+title: 'Scraping the Store'
+sidebar_label: 'Scraping'
description: Your first steps into the world of scraping with Crawlee
---
@@ -9,21 +9,20 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import ExampleCode from '!!raw-loader!roa-loader!./06-example.ts';
-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the Actors in the Apify Store. Let's review that and figure out ways to access the data.
+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the products in the example Warehouse store. Let's review that and figure out ways to access the data.
-- URL
-- Owner
-- Unique identifier (such as `apify/web-scraper`)
-- Title
-- Description
-- Last modification date
-- Number of runs
+- URL
+- Manufacturer
+- SKU
+- Title
+- Current price
+- Stock available
![data to scrape](/img/getting-started/scraping-practice.jpg 'Overview of data to be scraped.')
-### Scraping the URL, Owner and Unique identifier
+### Scraping the URL, Manufacturer and SKU
-Some information is lying right there in front of us without even having to touch the Actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!
+Some information is lying right there in front of us without even having to touch the product detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that we can also extract the manufacturer from the URL (as all product urls start with `/products/<manufacturer>`). We can just split the `string` and be on our way then!
:::info
@@ -32,26 +31,31 @@ You can use `request.loadedUrl` as well. Remember the difference: `request.url`
:::
```js
-// request.url = https://apify.com/apify/web-scraper
+// request.url = https://warehouse-theme-metal.myshopify.com/products/sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440
-const urlParts = request.url.split('/').slice(-2); // ['apify', 'web-scraper']
-const uniqueIdentifier = urlParts.join('/'); // 'apify/web-scraper'
-const owner = urlParts[0]; // 'apify'
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
```
:::tip
-It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `owner` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `owner`.
+It's a matter of preference, whether to store this information separately in the resulting dataset, or not. Whoever uses the dataset can easily parse the `manufacturer` from the `URL`, so should you duplicate the data unnecessarily? Our opinion is that unless the increased data consumption would be too large to bear, it's better to make the dataset as rich as possible. For example, someone might want to filter by `manufacturer`.
:::
-Now it's time to add more data to the results. Let's open one of the Actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the Actor.
+:::caution
+
+One thing you may notice is that the `manufacturer` might have a `-` in its name. If that's the case, your best bet is extracting it from the details page instead, but it's not mandatory. At the end of the day, you should always adjust and pick the best solution for your use case, and website you are crawling.
+
+:::
+
+Now it's time to add more data to the results. Let's open one of the product detail pages, for example the [`Sony XBR-950G`](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) page and use our DevTools-Fu 🥋 to figure out how to get the title of the product.
### Title
-![actor title](/img/getting-started/title.jpg 'Finding actor title in DevTools.')
+![product title](/img/getting-started/title.jpg 'Finding product title in DevTools.')
-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. Web pages usually have only a single `<h1>` tag, but this one has two. Because of that, we have to extend the selector with more information. The `<h1>` tag is enclosed in a `<div>` with class `ActorHeader-identificator`. We can leverage this to create a combined selector `.ActorHeader-identificator h1`. It selects any `<h1>` element that is a child of a different element with the class `ActorHeader-identificator`.
+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. The `<h1>` tag is enclosed in a `<div>` with class `product-meta`. We can leverage this to create a combined selector `.product-meta h1`. It selects any `<h1>` element that is a child of a different element with the class `product-meta`.
:::tip
@@ -59,66 +63,90 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of
:::
-To get the title, we need to find it using `Playwright` and a `.ActorHeader-identificator h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
+To get the title, we need to find it using `Playwright` and a `.product-meta h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.
```js
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
+const title = await page.locator('.product-meta h1').textContent();
```
-### Description
+### SKU
-Using the DevTools, we find that the Actor description is inside a `<p>` tag with a class `ActorHeader-description`. And since there's no other `<p>` with that class on the page, we can safely use it.
+Using the DevTools, we find that the product SKU is inside a `<span>` tag with a class `product-meta__sku-number`. And since there's no other `<span>` with that class on the page, we can safely use it.
-![actor description selector](/img/getting-started/description.jpg 'Finding actor description in DevTools.')
+![product sku selector](/img/getting-started/sku.jpg 'Finding product SKU in DevTools.')
```js
-const description = await page.locator('p.ActorHeader-description').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
```
-### Last modification date
+### Current price
-DevTools tell us that the `modifiedDate` can be found in a `<time>` element with a `datetime` property. You can use the [attribute CSS selector](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) to get it.
+DevTools tell us that the `currentPrice` can be found in a `<span>` element tagged with the `price` class. But it also shows us that it is nested as raw text alongside another `<span>` element with the `visually-hidden` class. We don't want that, so we need to filter it out, and we can use the `hasText` helper for that.
-![actor last modification date selector](/img/getting-started/modified-date.jpg 'Finding actor last modification date in DevTools.')
+![product current price selector](/img/getting-started/current-price.jpg 'Finding product current price in DevTools.')
```js
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
```
-It might look a little too complex at first glance, but let's walk through it. We find the right `<time>` element, and then we read its `datetime` attribute, because that's where a unix timestamp is stored as a `string`.
+It might look a little too complex at first glance, but let's walk through what we did. First off, we find the right part of the `price` span (specifically the actual price) by filtering the element that has the `$` sign in it. When we do that, we will get a string similar to `Sale price$1,398.00`. This, by itself, is not that useful, so we extract the actual numeric part by splitting by the `$` sign.
-But we would much rather see a readable date in our results, not a unix timestamp, so we need to convert it. Unfortunately the `new Date()` constructor will not accept a `string`, so we cast the `string` to a `number` using the `Number()` function before actually calling `new Date()`. Phew!
+Once we do that, we receive a string that represents our price, but we will be converting it to a number. We do that by replacing all the commas with nothingness (so we can parse it into a number), then we parse it into a number using `Number()`.
-### Run count
+### Stock available
-We're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.
+We're finishing up with the `availableInStock`. We can see there is a span with the `product-form__inventory` class, and it contains the text `In stock`. We can use the `hasText` helper again to filter out the right element.
```js
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-userMedallion ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the count from the last (second) `<li>` element. But it contains the "Runs " string as well, so we remove the "Runs " by replacing the word with empty string.
+For this, all we care about is whether the element exists or not, so we can use the `count()` method to check if there are any elements that match our selector. If there are, we know that the product is in stock.
And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.
```js
-const urlParts = request.url.split('/').slice(-2);
-const uniqueIdentifier = urlParts.join('/');
-const owner = urlParts[0];
-
-const title = await page.locator('.ActorHeader-identificator h1').textContent();
-
-const description = await page.locator('p.ActorHeader-description').textContent();
-
-const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
-const modifiedDate = new Date(Number(modifiedTimestamp));
-
-const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
-const runCountString = await runsRow.textContent();
-const runCount = runCountString.replace('Runs ', '');
+const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+const manufacturer = urlPart.split('-')[0]; // 'sennheiser'
+
+const title = await page.locator('.product-meta h1').textContent();
+const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+const currentPriceString = await priceElement.textContent();
+const rawPrice = currentPriceString.split('$')[1];
+const price = Number(rawPrice.replaceAll(',', ''));
+
+const inStockElement = await page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+const inStock = (await inStockElement.count()) > 0;
```
## Trying it out
@@ -133,17 +161,15 @@ When you run the crawler, you will see the crawled URLs and their scraped data p
```json
{
- "url": "https://apify.com/clockworks/tiktok-scraper",
- "uniqueIdentifier": "clockworks/tiktok-scraper",
- "owner": "clockworks",
- "title": "TikTok Scraper",
- "description": "Powerful TikTok scraper to extract data from TikTok videos, hashtags, and users. Use it to scrape TikTok profiles, hashtags, posts, URLs, numbers of shares, followers, hearts, names, video, and music-related data. Download TikTok data as a HTML, JSON, CSV, Excel, or XML doc.",
- "modifiedDate": "2023-04-26T12:13:19.387Z",
- "runCount": "1.6M"
+ "url": "https://warehouse-theme-metal.myshopify.com/products/sony-str-za810es-7-2-channel-hi-res-wi-fi-network-av-receiver",
+ "manufacturer": "sony",
+ "title": "Sony STR-ZA810ES 7.2-Ch Hi-Res Wi-Fi Network A/V Receiver",
+ "sku": "SON-692802-STR-DE",
+ "currentPrice": 698,
+ "availableInStock": true
}
```
## Next lesson
In the next lesson, we will show you how to save the data you scraped to the disk for further processing.
-
diff --git a/website/versioned_docs/version-3.5/introduction/07-example.ts b/website/versioned_docs/version-3.5/introduction/07-example.ts
index a72159427..43937be28 100644
--- a/website/versioned_docs/version-3.5/introduction/07-example.ts
+++ b/website/versioned_docs/version-3.5/introduction/07-example.ts
@@ -4,34 +4,69 @@ const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, request, enqueueLinks }) => {
console.log(`Processing: ${request.url}`);
if (request.label === 'DETAIL') {
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page.locator('span.product-meta__sku-number').textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString?.replace('Runs ', ''),
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
};
// highlight-next-line
await Dataset.pushData(results);
- } else {
- await page.waitForSelector('.ActorStorePagination-buttons a');
+ } else if (request.label === 'CATEGORY') {
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
});
- await page.waitForSelector('div[data-test="actorCard"] a');
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+ } else {
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- // highlight-next-line
- label: 'DETAIL', // <= note the different label
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
});
}
},
@@ -40,4 +75,4 @@ const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 50,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
diff --git a/website/versioned_docs/version-3.5/introduction/07-saving-data.mdx b/website/versioned_docs/version-3.5/introduction/07-saving-data.mdx
index fe6cae26e..8f059d74c 100644
--- a/website/versioned_docs/version-3.5/introduction/07-saving-data.mdx
+++ b/website/versioned_docs/version-3.5/introduction/07-saving-data.mdx
@@ -6,7 +6,7 @@ description: Your first steps into the world of scraping with Crawlee
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
-import ExampleCode from '!!raw-loader!roa-loader!./07-example.ts';
+import Example from '!!raw-loader!roa-loader!./07-example.ts';
A data extraction job would not be complete without saving the data for later use and processing. We've come to the final and most difficult part of this tutorial so make sure to pay attention very carefully!
@@ -25,7 +25,7 @@ await Dataset.pushData(results);
and that's it. Unlike earlier, we are being serious now. That's it, we're done. The final code looks like this:
<RunnableCodeBlock className="language-js" type="playwright">
- {ExampleCode}
+ {Example}
</RunnableCodeBlock>
## What's `Dataset.pushData()`
diff --git a/website/versioned_docs/version-3.5/introduction/08-refactoring.mdx b/website/versioned_docs/version-3.5/introduction/08-refactoring.mdx
index d8f8aa1b3..8c526cdcb 100644
--- a/website/versioned_docs/version-3.5/introduction/08-refactoring.mdx
+++ b/website/versioned_docs/version-3.5/introduction/08-refactoring.mdx
@@ -1,6 +1,6 @@
---
id: refactoring
-title: "Refactoring"
+title: 'Refactoring'
description: Your first steps into the world of scraping with Crawlee
---
@@ -23,9 +23,9 @@ Anyway, to promote good coding practices, let's look at how you can use a <ApiLi
In the following code we've made several changes:
-- Split the code into multiple files.
-- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
-- Added a `Router` to make our routing cleaner, without `if` clauses.
+- Split the code into multiple files.
+- Replaced `console.log` with the Crawlee logger for nicer, colourful logs.
+- Added a `Router` to make our routing cleaner, without `if` clauses.
In our `main.mjs` file, we place the general structure of the crawler:
@@ -44,8 +44,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
```
Then in a separate `routes.mjs` file:
@@ -59,41 +58,81 @@ export const router = createPlaywrightRouter();
// This replaces the request.label === DETAIL branch of the if clause.
router.addHandler('DETAIL', async ({ request, page, log }) => {
- log.debug(`Extracting data: ${request.url}`)
- const urlParts = request.url.split('/').slice(-2);
- const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');
- const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });
- const runCountString = await runsRow.textContent();
+ log.debug(`Extracting data: ${request.url}`);
+ const urlPart = request.url.split('/').slice(-1); // ['sennheiser-mke-440-professional-stereo-shotgun-microphone-mke-440']
+ const manufacturer = urlPart[0].split('-')[0]; // 'sennheiser'
+
+ const title = await page.locator('.product-meta h1').textContent();
+ const sku = await page
+ .locator('span.product-meta__sku-number')
+ .textContent();
+
+ const priceElement = page
+ .locator('span.price')
+ .filter({
+ hasText: '$',
+ })
+ .first();
+
+ const currentPriceString = await priceElement.textContent();
+ const rawPrice = currentPriceString.split('$')[1];
+ const price = Number(rawPrice.replaceAll(',', ''));
+
+ const inStockElement = page
+ .locator('span.product-form__inventory')
+ .filter({
+ hasText: 'In stock',
+ })
+ .first();
+
+ const inStock = (await inStockElement.count()) > 0;
const results = {
url: request.url,
- uniqueIdentifier: urlParts.join('/'),
- owner: urlParts[0],
- title: await page.locator('.ActorHeader-identificator h1').textContent(),
- description: await page.locator('p.ActorHeader-description').textContent(),
- modifiedDate: new Date(Number(modifiedTimestamp)),
- runCount: runCountString.replace('Runs ', ''),
- }
-
- log.debug(`Saving data: ${request.url}`)
+ manufacturer,
+ title,
+ sku,
+ currentPrice: price,
+ availableInStock: inStock,
+ };
+
+ log.debug(`Saving data: ${request.url}`);
await Dataset.pushData(results);
});
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, request, log }) => {
+ log.debug(`Enqueueing pagination for: ${request.url}`);
+ // We are now on a category page. We can use this to paginate through and enqueue all products,
+ // as well as any subsequent pages we find
+
+ await page.waitForSelector('.product-item > a');
+ await enqueueLinks({
+ selector: '.product-item > a',
+ label: 'DETAIL', // <= note the different label
+ });
+
+ // Now we need to find the "Next" button and enqueue the next page of results (if it exists)
+ const nextButton = await page.$('a.pagination__next');
+ if (nextButton) {
+ await enqueueLinks({
+ selector: 'a.pagination__next',
+ label: 'CATEGORY', // <= note the same label
+ });
+ }
+});
+
// This is a fallback route which will handle the start URL
// as well as the LIST labeled URLs.
router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {
- log.debug(`Enqueueing pagination: ${request.url}`)
- await page.waitForSelector('.ActorStorePagination-buttons a');
- await enqueueLinks({
- selector: '.ActorStorePagination-buttons a',
- label: 'LIST',
- })
- log.debug(`Enqueueing actor details: ${request.url}`)
- await page.waitForSelector('div[data-test="actorCard"] a');
+ log.debug(`Enqueueing categories from page: ${request.url}`);
+ // This means we're on the start page, with no label.
+ // On this page, we just want to enqueue all the category pages.
+
+ await page.waitForSelector('.collection-block-item');
await enqueueLinks({
- selector: 'div[data-test="actorCard"] a',
- label: 'DETAIL', // <= note the different label
- })
+ selector: '.collection-block-item',
+ label: 'CATEGORY',
+ });
});
```
diff --git a/website/versioned_docs/version-3.5/introduction/09-deployment.mdx b/website/versioned_docs/version-3.5/introduction/09-deployment.mdx
index 255056898..5273181be 100644
--- a/website/versioned_docs/version-3.5/introduction/09-deployment.mdx
+++ b/website/versioned_docs/version-3.5/introduction/09-deployment.mdx
@@ -1,7 +1,7 @@
---
id: deployment
-title: "Running your crawler in the Cloud"
-sidebar_label: "Running in the Cloud"
+title: 'Running your crawler in the Cloud'
+sidebar_label: 'Running in the Cloud'
description: Deploying Crawlee projects to the Apify Platform
---
@@ -23,12 +23,13 @@ We started this guide by using the Crawlee CLI to bootstrap the project - it off
The first step will be installing two new dependencies:
-- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+- Apify SDK, a toolkit for working with the Apify Platform. This will allow us to wire the storages (e.g. `RequestQueue` and `Dataset`) to the Apify cloud products. This will be a dependency of our Node.js project.
+
```bash
npm install apify
```
-- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
+- Apify CLI, a command-line tool that will help us with authentication and deployment. This will be a globally installed tool, you will install it only once and use it in all your Crawlee/Apify projects.
```bash
npm install -g apify-cli
```
@@ -67,7 +68,7 @@ const crawler = new PlaywrightCrawler({
requestHandler: router,
});
-await crawler.run(['https://apify.com/store']);
+await crawler.run(['https://warehouse-theme-metal.myshopify.com/collections']);
// highlight-next-line
await Actor.exit();
diff --git a/website/versioned_sidebars/version-3.0-sidebars.json b/website/versioned_sidebars/version-3.0-sidebars.json
index b1b54cc27..ad15a92be 100644
--- a/website/versioned_sidebars/version-3.0-sidebars.json
+++ b/website/versioned_sidebars/version-3.0-sidebars.json
@@ -17,7 +17,8 @@
"introduction/crawling",
"introduction/scraping",
"introduction/saving-data",
- "introduction/refactoring"
+ "introduction/refactoring",
+ "introduction/deployment"
]
},
{
@@ -48,6 +49,39 @@
"guides/apify-platform"
]
},
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "slug": "/deployment",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers."
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
{
"type": "category",
"label": "Examples",
diff --git a/website/versioned_sidebars/version-3.1-sidebars.json b/website/versioned_sidebars/version-3.1-sidebars.json
index 877b43d16..c15f71f4e 100644
--- a/website/versioned_sidebars/version-3.1-sidebars.json
+++ b/website/versioned_sidebars/version-3.1-sidebars.json
@@ -17,7 +17,8 @@
"introduction/crawling",
"introduction/scraping",
"introduction/saving-data",
- "introduction/refactoring"
+ "introduction/refactoring",
+ "introduction/deployment"
]
},
{
@@ -48,6 +49,39 @@
"guides/apify-platform"
]
},
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "slug": "/deployment",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers."
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
{
"type": "category",
"label": "Examples",
diff --git a/website/versioned_sidebars/version-3.2-sidebars.json b/website/versioned_sidebars/version-3.2-sidebars.json
index 877b43d16..c15f71f4e 100644
--- a/website/versioned_sidebars/version-3.2-sidebars.json
+++ b/website/versioned_sidebars/version-3.2-sidebars.json
@@ -17,7 +17,8 @@
"introduction/crawling",
"introduction/scraping",
"introduction/saving-data",
- "introduction/refactoring"
+ "introduction/refactoring",
+ "introduction/deployment"
]
},
{
@@ -48,6 +49,39 @@
"guides/apify-platform"
]
},
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "slug": "/deployment",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers."
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
{
"type": "category",
"label": "Examples",
diff --git a/website/versioned_sidebars/version-3.3-sidebars.json b/website/versioned_sidebars/version-3.3-sidebars.json
index 877b43d16..c15f71f4e 100644
--- a/website/versioned_sidebars/version-3.3-sidebars.json
+++ b/website/versioned_sidebars/version-3.3-sidebars.json
@@ -17,7 +17,8 @@
"introduction/crawling",
"introduction/scraping",
"introduction/saving-data",
- "introduction/refactoring"
+ "introduction/refactoring",
+ "introduction/deployment"
]
},
{
@@ -48,6 +49,39 @@
"guides/apify-platform"
]
},
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "slug": "/deployment",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers."
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
{
"type": "category",
"label": "Examples",
diff --git a/website/versioned_sidebars/version-3.4-sidebars.json b/website/versioned_sidebars/version-3.4-sidebars.json
index 877b43d16..c15f71f4e 100644
--- a/website/versioned_sidebars/version-3.4-sidebars.json
+++ b/website/versioned_sidebars/version-3.4-sidebars.json
@@ -17,7 +17,8 @@
"introduction/crawling",
"introduction/scraping",
"introduction/saving-data",
- "introduction/refactoring"
+ "introduction/refactoring",
+ "introduction/deployment"
]
},
{
@@ -48,6 +49,39 @@
"guides/apify-platform"
]
},
+ {
+ "type": "category",
+ "label": "Deployment",
+ "link": {
+ "type": "generated-index",
+ "slug": "/deployment",
+ "title": "Deployment guides",
+ "description": "Here you can find guides on how to deploy your crawlers to various cloud providers."
+ },
+ "items": [
+ {
+ "type": "doc",
+ "id": "deployment/apify-platform",
+ "label": "Deploy on Apify"
+ },
+ {
+ "type": "category",
+ "label": "Deploy on AWS",
+ "items": [
+ "deployment/aws-cheerio",
+ "deployment/aws-browsers"
+ ]
+ },
+ {
+ "type": "category",
+ "label": "Deploy to Google Cloud",
+ "items": [
+ "deployment/gcp-cheerio",
+ "deployment/gcp-browsers"
+ ]
+ }
+ ]
+ },
{
"type": "category",
"label": "Examples",