Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
TypeScript apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
docs: fix and update introduction tutorial (#1915)
Ondra Urbanc557f5fba2de
diff --git a/docs/introduction/04-pw-w-cheerio.ts b/docs/introduction/04-pw-w-cheerio.tsindex 2873909e8..b698f9fc4 100644--- a/docs/introduction/04-pw-w-cheerio.ts+++ b/docs/introduction/04-pw-w-cheerio.ts@@ -5,13 +5,13 @@ import { PlaywrightCrawler } from 'crawlee';const crawler = new PlaywrightCrawler({requestHandler: async ({ page, parseWithCheerio }) => {// Wait for the actor cards to render.- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"]');// Extract the page's HTML from browser// and parse it with Cheerio.const $ = await parseWithCheerio();// Use familiar Cheerio syntax to// select all the actor cards.- $('.ActorStoreItem').each((i, el) => {+ $('div[data-test="actorCard"]').each((i, el) => {const text = $(el).text();console.log(`ACTOR_${i + 1}: ${text}\n`);});diff --git a/docs/introduction/04-pw.ts b/docs/introduction/04-pw.tsindex 8260549a3..37becf161 100644--- a/docs/introduction/04-pw.ts+++ b/docs/introduction/04-pw.ts@@ -5,10 +5,10 @@ import { PlaywrightCrawler } from 'crawlee';const crawler = new PlaywrightCrawler({requestHandler: async ({ page }) => {// Wait for the actor cards to render.- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"]');// Execute a function in the browser which targets// the actor card elements and allows their manipulation.- const actorTexts = await page.$$eval('.ActorStoreItem', (els) => {+ const actorTexts = await page.$$eval('div[data-test="actorCard"]', (els) => {// Extract text content from the actor cardsreturn els.map((el) => el.textContent);});diff --git a/docs/introduction/04-real-world-project.mdx b/docs/introduction/04-real-world-project.mdxindex a54e29bea..775b34fd6 100644--- a/docs/introduction/04-real-world-project.mdx+++ b/docs/introduction/04-real-world-project.mdx@@ -19,7 +19,7 @@ We hear you, young padawan! First, learn how to crawl, you must. Only then, walk## Making a production-grade crawler-Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **actors** that anyone can grab and use for free.+Making a production-grade crawler is not difficult, but there are many pitfalls of scraping that can catch you off guard. So for the real world project you'll learn how to scrape [Apify Store](https://apify.com/store) instead of the Crawlee website. It is a library of scrapers and automations called **Actors** that anyone can grab and use for free.The website requires JavaScript rendering, which allows us to showcase more features of Crawlee. We've also added some helpful tips that prepare you for the real-world issues that you will surely encounter when scraping at scale.@@ -42,7 +42,7 @@ For the purposes of this tutorial, let's assume that the website cannot be scrap## Choosing the data you need-A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all actors from [Apify Store](https://apify.com/store) and for each actor we want to get its:+A good first step is to figure out what data you want to scrape and where to find it. For the time being, let's just agree that we want to scrape all Actors from [Apify Store](https://apify.com/store) and for each Actor we want to get its:- URL- Owner@@ -52,7 +52,7 @@ A good first step is to figure out what data you want to scrape and where to fin- Last modification date- Number of runs-You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the actor detail pages.+You will notice that some information is available directly on the list page, but for details such as "Last modification date" or "Number of runs" we'll also need to open the Actor detail pages.@@ -66,7 +66,7 @@ Let's take a look at the `apify.com/store` page more carefully. There are some *### Categories and sorting-When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.+When you click the categories, you'll see that they filter the results. If you remove the category, you're back to the original number of results. By going through a few categories and observing the behavior, we can quite safely assume that the default setting - with **no category selected** - shows us **all the Actors available** in the store and that's the setting we'll use to scrape. The same applies to sorting. We don't need that now.:::caution@@ -90,13 +90,13 @@ Similarly to the issue with filters explained above, the existence of pagination:::-At the time of writing the Store results counter showed 1047 results - actors. Quick count of actors on one page of results makes 24. 8 rows times 3 actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).+At the time of writing the Store results counter showed 1047 results - Actors. Quick count of Actors on one page of results makes 24. 8 rows times 3 Actors. This means that there should be 44 pages of results. 1047 divided by 24. Try going to page number 44 (or the number your own calculation produced).```https://apify.com/store?page=44```-It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain actors were hidden for some reason, but the count does not reflect it.+It's empty. 🤯 Wrong calculation? Not really. This is an example of another common issue in web scraping. The result count presented by websites very rarely matches the actual number of available results. In our case, it's simply because certain Actors were hidden for some reason, but the count does not reflect it.At the time of writing, the last results were actually on page 42. But that's ok. What's important is that on this page 42, the pagination links at the bottom are still the same as on page one, two or six. This makes it fairly certain that you can keep following those links until you scrape all the results. Good 👍@@ -104,15 +104,15 @@ If you're not convinced, you can visit a page somewhere in the middle, like `htt## The crawling strategy-Now that you know where to start and how to find all the actor details, let's look at the crawling process.+Now that you know where to start and how to find all the Actor details, let's look at the crawling process.-1. Visit the store page containing the list of actors (our start URL).-2. Enqueue all links to actor details.+1. Visit the store page containing the list of Actors (our start URL).+2. Enqueue all links to Actor details.3. Enqueue links to next pages of results.4. Open the next page in queue.- When it's a results list page, go to 2.- - When it's an actor detail page, scrape the data.-5. Repeat until all results pages and all actor details have been processed.+ - When it's an Actor detail page, scrape the data.+5. Repeat until all results pages and all Actor details have been processed.`PlaywrightCrawler` will make sure to visit the pages for you, if you provide the correct requests, and you already know how to enqueue pages, so this should be fairly easy. Nevertheless, there are few more tricks that we'd like to showcase.@@ -120,7 +120,7 @@ Now that you know where to start and how to find all the actor details, let's loLet's check that everything is set up correctly before writing the scraping logic itself. You might realize that something in your previous analysis doesn't quite add up, or the website might not behave exactly as you expected.-The example below creates a new crawler that visits the start URL and prints the text content of all the actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual actor cards.+The example below creates a new crawler that visits the start URL and prints the text content of all the Actor cards on that page. When you run the code, you will see the _very badly formatted_ content of the individual Actor cards.<Tabs groupId="sanity-check"><TabItem value="playwright" label="Playwright" default>@@ -131,9 +131,9 @@ The example below creates a new crawler that visits the start URL and prints the</TabItem></Tabs>-If you're wondering how to get that `.ActorStoreItem` selector. We'll explain it in the next chapter on DevTools.+If you're wondering how to get that `div[data-test="actorCard"]` selector. We'll explain it in the next chapter on DevTools.-## DevTools - the scrapers toolbox+## DevTools - the scraper's toolbox:::info@@ -145,7 +145,7 @@ Let's open DevTools by going to https://apify.com/store in Chrome and then right## Selecting elements-In the DevTools, choose the **Select an element** tool and try hovering over one of the actor cards.+In the DevTools, choose the **Select an element** tool and try hovering over one of the Actor cards.@@ -155,15 +155,15 @@ You'll see that you can select different elements inside the card. Instead, seleSelecting an element will highlight it in the DevTools HTML inspector. When carefully look at the elements, you'll see that there are some **classes** attached to the different HTML elements. Those are called **CSS classes**, and we can make a use of them in scraping.-Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the actor cards. You'll see that all the card's data is displayed in an `<a>` element with two classes, one of which is **ActorStoreItem**. It should now make sense how we got that `.ActorStoreItem` selector. It's just a way to find all elements that are annotated with the `ActorStoreItem`.+Conversely, by hovering over elements in the HTML inspector, you will see them highlight on the page. Inspect the page's structure around the Actor cards. You'll see that all the card's data is displayed in an `<a>` element with a `data-test` attribute **actorCard**. It should now make sense how we got that `div[data-test="actorCard"]` selector. It's just a way to find all elements that are annotated with the `actorCard`.It's always a good idea to double-check that you're not getting any unwanted elements with this class. To do that, go into the **Console** tab of DevTools and run:```ts-document.querySelectorAll('.ActorStoreItem');+document.querySelectorAll('div[data-test="actorCard"]');```-You will see that only the 24 actor cards will be returned, and nothing else.+You will see that only the 24 Actor cards will be returned, and nothing else.:::tip@@ -173,4 +173,4 @@ CSS selectors and DevTools are quite a big topic. If you want to learn more, vis## Next lesson-In the next lesson we will crawl the whole store, including all the listing pages and all the actor detail pages.+In the next lesson, we will crawl the whole store, including all the listing pages and all the Actor detail pages.diff --git a/docs/introduction/05-crawling.mdx b/docs/introduction/05-crawling.mdxindex 979779f7e..848007e66 100644--- a/docs/introduction/05-crawling.mdx+++ b/docs/introduction/05-crawling.mdx@@ -5,7 +5,7 @@ sidebar_label: "Crawling"description: Your first steps into the world of scraping with Crawlee----To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with actors - listing pages and also all the actor detail pages.+To crawl the whole [Apify Store](https://apify.com/store) and find all the data, you first need to visit all the pages with Actors - listing pages and also all the Actor detail pages.## Crawling the listing pages@@ -23,14 +23,14 @@ import { PlaywrightCrawler } from 'crawlee';const crawler = new PlaywrightCrawler({requestHandler: async ({ page, request, enqueueLinks }) => {console.log(`Processing: ${request.url}`)- // Wait for the actor cards to render,+ // Wait for the Actor cards to render,// otherwise enqueueLinks wouldn't enqueue anything.- await page.waitForSelector('.ActorStorePagination-pages a');+ await page.waitForSelector('.ActorStorePagination-buttons a');// Add links to the queue, but only from// elements matching the provided selector.await enqueueLinks({- selector: '.ActorStorePagination-pages > a',+ selector: '.ActorStorePagination-buttons a',label: 'LIST',})},@@ -43,7 +43,7 @@ The code should look pretty familiar to you. It's a very simple `requestHandler`### The `selector` parameter of `enqueueLinks()`-When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-pages a` selector, which selects all the `<a>` elements that are direct children of an element with `class=ActorStorePagination-pages`.+When you previously used `enqueueLinks()`, you were not providing any `selector` parameter, and it was fine, because you wanted to use the default value, which is `a` - finds all `<a>` elements. But now, you need to be more specific. There are multiple `<a>` links on the Store results page, and you're only interested in those that will take your crawler to the next page of results. Using the DevTools, you'll find that you can select the links you need using the `.ActorStorePagination-buttons a` selector, which selects all the `<a>` elements that are children of an element with `class=ActorStorePagination-buttons`.### The `label` of `enqueueLinks()`@@ -51,7 +51,7 @@ You will see `label` used often throughout Crawlee, as it's a convenient way of## Crawling the detail pages-In a similar fashion, you need to collect all the URLs to the actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.+In a similar fashion, you need to collect all the URLs to the Actor detail pages, because only from there you can scrape all the data you need. The following code only repeats the concepts you already know for another set of links.```tsimport { PlaywrightCrawler } from 'crawlee';@@ -65,17 +65,17 @@ const crawler = new PlaywrightCrawler({// This means we're either on the start page, with no label,// or on a list page, with LIST label.- await page.waitForSelector('.ActorStorePagination-pages a');+ await page.waitForSelector('.ActorStorePagination-buttons a');await enqueueLinks({- selector: '.ActorStorePagination-pages > a',+ selector: '.ActorStorePagination-buttons a',label: 'LIST',})// In addition to adding the listing URLs, we now also// add the detail URLs from all the listing pages.- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"] a');await enqueueLinks({- selector: '.ActorStoreItem',+ selector: 'div[data-test="actorCard"] a',label: 'DETAIL', // <= note the different label})}diff --git a/docs/introduction/06-scraping.mdx b/docs/introduction/06-scraping.mdxindex 38b2289c2..4a81741b9 100644--- a/docs/introduction/06-scraping.mdx+++ b/docs/introduction/06-scraping.mdx@@ -7,7 +7,7 @@ description: Your first steps into the world of scraping with Crawleeimport ApiLink from '@site/src/components/ApiLink';-In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the actors in the Apify Store. Let's review that and figure out ways to access the data.+In the [Real-world project chapter](./real-world-project#choosing-the-data-you-need), we created a list of the information we wanted to collect about the Actors in the Apify Store. Let's review that and figure out ways to access the data.- URL- Owner@@ -21,7 +21,7 @@ In the [Real-world project chapter](./real-world-project#choosing-the-data-you-n### Scraping the URL, Owner and Unique identifier-Some information is lying right there in front of us without even having to touch the actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!+Some information is lying right there in front of us without even having to touch the Actor detail pages. The `URL` we already have - the `request.url`. And by looking at it carefully, we realize that it already includes the `owner` and the `unique identifier` too. We can just split the `string` and be on our way then!:::info@@ -43,19 +43,13 @@ It's a matter of preference, whether to store this information separately in the:::-Now it's time to add more data to the results. Let's open one of the actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the actor.+Now it's time to add more data to the results. Let's open one of the Actor detail pages, for example the [`apify/web-scraper`](https://apify.com/apify/web-scraper) page and use our DevTools-Fu 🥋 to figure out how to get the title of the Actor.### Title-By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be.--:::caution--In some cases there could actually be more `<h1>` tags on the same page. Yes, it's wrong, but it is quite common in the wild. Be careful when scraping and don't assume that websites are programmed correctly. They aren't.--:::+By using the element selector tool, you can see that the title is there under an `<h1>` tag, as titles should be. Web pages usually have only a single `<h1>` tag, but this one has two. Because of that, we have to extend the selector with more information. The `<h1>` tag is enclosed in a `<div>` with class `ActorHeader-identificator`. We can leverage this to create a combined selector `.ActorHeader-identificator h1`. It selects any `<h1>` element that is a child of a different element with the class `ActorHeader-identificator`.:::tip@@ -63,20 +57,20 @@ Remember that you can press CTRL+F (or CMD+F on Mac) in the **Elements** tab of:::-To get the title we just need to find it using `Playwright` and a `h1` locator, which selects the first `<h1>` element it finds and throws if it finds more. That's good. It's usually better to crash the crawler than silently return bad data.+To get the title, we need to find it using `Playwright` and a `.ActorHeader-identificator h1` locator, which selects the `<h1>` element we're looking for, or throws, if it finds more than one. That's good. It's usually better to crash the crawler than silently return bad data.```js-const title = await page.locator('h1').textContent();+const title = await page.locator('.ActorHeader-identificator h1').textContent();```### Description-Getting the actor's description is a little more involved, but still pretty straightforward. You can't just simply search for a `<p>` tag, because there's a lot of them in the page. We need to narrow our search down a little. Using the DevTools we find that the actor description is actually nested inside a `<span>` tag with a class `actor-description`. And since there's no other `<span>` with that class on the page, we can safely use it.+Using the DevTools, we find that the Actor description is inside a `<p>` tag with a class `ActorHeader-description`. And since there's no other `<p>` with that class on the page, we can safely use it.```js-const description = await page.locator('span.actor-description').textContent();+const description = await page.locator('p.ActorHeader-description').textContent();```### Last modification date@@ -99,12 +93,12 @@ But we would much rather see a readable date in our results, not a unix timestamWe're finishing up with the `runCount`. There's no specific element like `<time>`, so we need to create a complex selector and then do a transformation on the result.```js-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });-const runCountString = await runsRow.locator('span').last().textContent();-const runCount = Number(runCountString.replaceAll(',', ''));+const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });+const runCountString = await runsRow.textContent();+const runCount = runCountString.replace('Runs ', '');```-The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-stats ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the number from the last (second) `<span>` element inside that `<li>`. But its type is still a `string`, so we remove the commas by replacing them with empty strings, and finally convert the result to a `number` by wrapping it with a `Number()` call.+The code looks complicated, but it only reads that we're looking for an `<ul class="ActorHeader-userMedallion ...">` element and within that element we're looking for the `<li>` element that includes the text **Runs**. We're only interested in the number of runs. So we extract the count from the last (second) `<li>` element. But it contains the "Runs " string as well, so we remove the "Runs " by replacing the word with empty string.And there we have it! All the data we needed. For the sake of completeness, let's add all the properties together, and we're good to go.@@ -113,16 +107,16 @@ const urlParts = request.url.split('/').slice(-2);const uniqueIdentifier = urlParts.join('/');const owner = urlParts[0];-const title = await page.locator('h1').textContent();+const title = await page.locator('.ActorHeader-identificator h1').textContent();-const description = await page.locator('span.actor-description').textContent();+const description = await page.locator('p.ActorHeader-description').textContent();const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');const modifiedDate = new Date(Number(modifiedTimestamp));-const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });-const runCountString = await runsRow.locator('span').last().textContent();-const runCount = Number(runCountString.replaceAll(',', ''));+const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });+const runCountString = await runsRow.textContent();+const runCount = runCountString.replace('Runs ', '');```## Trying it out@@ -138,29 +132,28 @@ const crawler = new PlaywrightCrawler({if (request.label === 'DETAIL') {const urlParts = request.url.split('/').slice(-2);const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });- const runCountString = await runsRow.locator('span').last().textContent();-+ const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });+ const runCountString = await runsRow.textContent();const results = {url: request.url,uniqueIdentifier: urlParts.join('/'),owner: urlParts[0],- title: await page.locator('h1').textContent(),- description: await page.locator('span.actor-description').textContent(),+ title: await page.locator('.ActorHeader-identificator h1').textContent(),+ description: await page.locator('p.ActorHeader-description').textContent(),modifiedDate: new Date(Number(modifiedTimestamp)),- runCount: Number(runCountString.replaceAll(',', '')),+ runCount: runCountString.replace('Runs ', ''),}- console.log(results)+ console.log(results);} else {- await page.waitForSelector('.ActorStorePagination-pages a');+ await page.waitForSelector('.ActorStorePagination-buttons a');await enqueueLinks({- selector: '.ActorStorePagination-pages > a',+ selector: '.ActorStorePagination-buttons a',label: 'LIST',})- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"] a');await enqueueLinks({- selector: '.ActorStoreItem',+ selector: 'div[data-test="actorCard"] a',label: 'DETAIL', // <= note the different label})}@@ -172,7 +165,19 @@ await crawler.run(['https://apify.com/store']);-When you run the actor, you should see the crawled URLs and their scraped data printed to the console.+When you run the crawler, you will see the crawled URLs and their scraped data printed to the console. The output will look something like this:++```json+{+ "url": "https://apify.com/clockworks/tiktok-scraper",+ "uniqueIdentifier": "clockworks/tiktok-scraper",+ "owner": "clockworks",+ "title": "TikTok Scraper",+ "description": "Powerful TikTok scraper to extract data from TikTok videos, hashtags, and users. Use it to scrape TikTok profiles, hashtags, posts, URLs, numbers of shares, followers, hearts, names, video, and music-related data. Download TikTok data as a HTML, JSON, CSV, Excel, or XML doc.",+ "modifiedDate": "2023-04-26T12:13:19.387Z",+ "runCount": "1.6M"+}+```## Next lessondiff --git a/docs/introduction/07-saving-data.mdx b/docs/introduction/07-saving-data.mdxindex 717b9f342..3d7ea9202 100644--- a/docs/introduction/07-saving-data.mdx+++ b/docs/introduction/07-saving-data.mdx@@ -31,29 +31,29 @@ const crawler = new PlaywrightCrawler({if (request.label === 'DETAIL') {const urlParts = request.url.split('/').slice(-2);const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });- const runCountString = await runsRow.locator('span').last().textContent();+ const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });+ const runCountString = await runsRow.textContent();const results = {url: request.url,uniqueIdentifier: urlParts.join('/'),owner: urlParts[0],- title: await page.locator('h1').textContent(),- description: await page.locator('span.actor-description').textContent(),+ title: await page.locator('.ActorHeader-identificator h1').textContent(),+ description: await page.locator('p.ActorHeader-description').textContent(),modifiedDate: new Date(Number(modifiedTimestamp)),- runCount: Number(runCountString.replaceAll(',', '')),+ runCount: (runCountString.replace('Runs ', '')),}await Dataset.pushData(results);} else {- await page.waitForSelector('.ActorStorePagination-pages a');+ await page.waitForSelector('.ActorStorePagination-buttons a');await enqueueLinks({- selector: '.ActorStorePagination-pages > a',+ selector: '.ActorStorePagination-buttons a',label: 'LIST',})- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"] a');await enqueueLinks({- selector: '.ActorStoreItem',+ selector: 'div[data-test="actorCard"] a',label: 'DETAIL', // <= note the different label})}diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdxindex 0a23142d5..99a55f424 100644--- a/docs/introduction/08-refactoring.mdx+++ b/docs/introduction/08-refactoring.mdx@@ -31,7 +31,7 @@ In our `main.mjs` file, we place the general structure of the crawler:```js title="src/main.mjs"import { PlaywrightCrawler, log } from 'crawlee';-import { router } from './routes.js';+import { router } from './routes.mjs';// This is better set with CRAWLEE_LOG_LEVEL env var// or a configuration option. This is just for show 😈@@ -44,11 +44,8 @@ const crawler = new PlaywrightCrawler({requestHandler: router,});-log.debug('Adding requests to the queue.');-await crawler.addRequests(['https://apify.com/store']);-// crawler.run has its own logs 🙂-await crawler.run();+await crawler.run(['https://apify.com/store']);```Then in a separate `routes.mjs` file:@@ -65,17 +62,17 @@ router.addHandler('DETAIL', async ({ request, page, log }) => {log.debug(`Extracting data: ${request.url}`)const urlParts = request.url.split('/').slice(-2);const modifiedTimestamp = await page.locator('time[datetime]').getAttribute('datetime');- const runsRow = page.locator('ul.ActorHeader-stats > li').filter({ hasText: 'Runs' });- const runCountString = await runsRow.locator('span').last().textContent();+ const runsRow = page.locator('ul.ActorHeader-userMedallion > li').filter({ hasText: 'Runs' });+ const runCountString = await runsRow.textContent();const results = {url: request.url,uniqueIdentifier: urlParts.join('/'),owner: urlParts[0],- title: await page.locator('h1').textContent(),- description: await page.locator('span.actor-description').textContent(),+ title: await page.locator('.ActorHeader-identificator h1').textContent(),+ description: await page.locator('p.ActorHeader-description').textContent(),modifiedDate: new Date(Number(modifiedTimestamp)),- runCount: Number(runCountString.replaceAll(',', '')),+ runCount: runCountString.replace('Runs ', ''),}log.debug(`Saving data: ${request.url}`)@@ -83,24 +80,24 @@ router.addHandler('DETAIL', async ({ request, page, log }) => {});// This is a fallback route which will handle the start URL-// as well as the LIST labelled URLs.+// as well as the LIST labeled URLs.router.addDefaultHandler(async ({ request, page, enqueueLinks, log }) => {log.debug(`Enqueueing pagination: ${request.url}`)- await page.waitForSelector('.ActorStorePagination-pages a');+ await page.waitForSelector('.ActorStorePagination-buttons a');await enqueueLinks({- selector: '.ActorStorePagination-pages > a',+ selector: '.ActorStorePagination-buttons a',label: 'LIST',})log.debug(`Enqueueing actor details: ${request.url}`)- await page.waitForSelector('.ActorStoreItem');+ await page.waitForSelector('div[data-test="actorCard"] a');await enqueueLinks({- selector: '.ActorStoreItem',+ selector: 'div[data-test="actorCard"] a',label: 'DETAIL', // <= note the different label})});```-Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the actor more readable and manageable.+Let's describe the changes a bit more in detail. We hope that in the end, you will agree that this structure makes the crawler more readable and manageable.## Splitting your code into multiple files@@ -110,9 +107,7 @@ There's no reason not to split your code into multiple files and keep your logicWe won't go to great lengths here to talk about `log` object from Crawlee, because you can read it all in the <ApiLink to="core/class/Log">documentation</ApiLink>, but there's just one thing that we need to stress: **log levels**.-Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. This is huge! Because you can now add a lot of debug logs in your actor, which will help you when something goes wrong and turn them on or off with a simple configuration change.--The punch line? Use `log` from Crawlee instead of `console.log` now and thank us later when something goes wrong!+Crawlee `log` has multiple log levels, such as `log.debug`, `log.info` or `log.warning`. It not only makes your log more readable, but it also allows selective turning off of some levels by either calling the `log.setLevel()` function or by setting the `CRAWLEE_LOG_LEVEL` environment variable. Thanks to this you can add a lot of debug logs to your crawler without polluting your log when they're not needed, but ready to help when you encounter issues.## Using a router to structure your crawlingdiff --git a/website/static/img/getting-started/description.jpg b/website/static/img/getting-started/description.jpgindex 85d70d201..a6ce595be 100644Binary files a/website/static/img/getting-started/description.jpg and b/website/static/img/getting-started/description.jpg differdiff --git a/website/static/img/getting-started/modified-date.jpg b/website/static/img/getting-started/modified-date.jpgindex d476ecad4..631efda5b 100644Binary files a/website/static/img/getting-started/modified-date.jpg and b/website/static/img/getting-started/modified-date.jpg differdiff --git a/website/static/img/getting-started/scraping-practice.jpg b/website/static/img/getting-started/scraping-practice.jpgindex 517195ae3..3b358015a 100644Binary files a/website/static/img/getting-started/scraping-practice.jpg and b/website/static/img/getting-started/scraping-practice.jpg differdiff --git a/website/static/img/getting-started/selected-element.png b/website/static/img/getting-started/selected-element.pngindex d272f1492..64de73a7a 100644Binary files a/website/static/img/getting-started/selected-element.png and b/website/static/img/getting-started/selected-element.png differdiff --git a/website/static/img/getting-started/title.jpg b/website/static/img/getting-started/title.jpgindex d5b059844..9c6d09d31 100644Binary files a/website/static/img/getting-started/title.jpg and b/website/static/img/getting-started/title.jpg differ