Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

1.7 KB
Raw Blame
---
id: crawl-sitemap
title: Crawl a sitemap
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';

import CheerioSource from '!!raw-loader!roa-loader!./crawl-sitemap-cheerio.ts';
import PuppeteerSource from '!!raw-loader!roa-loader!./crawl-sitemap-puppeteer.ts';
import PlaywrightSource from '!!raw-loader!roa-loader!./crawl-sitemap-playwright.ts';

We will crawl sitemap which tells search engines which pages and file are important in the website, it also provides valuable information about these files. This example builds a sitemap crawler which downloads and crawls the URLs from a sitemap, by using the <ApiLink to="utils/class/Sitemap">`Sitemap`</ApiLink> utility class provided by the <ApiLink to="utils">`@crawlee/utils`</ApiLink> module.

<Tabs groupId="crawler-type">

<TabItem value="cheerio_crawler" label="Cheerio Crawler" default>

<RunnableCodeBlock className="language-js" type="cheerio">
	{CheerioSource}
</RunnableCodeBlock>

</TabItem>

<TabItem value="puppeteer_crawler" label="Puppeteer Crawler">

:::tip

To run this example on the Apify Platform, select the `apify/actor-node-puppeteer-chrome` image for your Dockerfile.

:::

<RunnableCodeBlock className="language-js" type="puppeteer">
	{PuppeteerSource}
</RunnableCodeBlock>

</TabItem>

<TabItem value="playwright_crawler" label="Playwright Crawler">

:::tip

To run this example on the Apify Platform, select the `apify/actor-node-playwright-chrome` image for your Dockerfile.

:::

<RunnableCodeBlock className="language-js" type="playwright">
	{PlaywrightSource}
</RunnableCodeBlock>

</TabItem>

</Tabs>