Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

3.3 KB
Raw Blame
---
id: setting-up
title: "Setting up"
description: Your first steps into the world of scraping with Crawlee
---

import ThemedImage from '@theme/ThemedImage';

import ApiLink from '@site/src/components/ApiLink';

To run Crawlee on your own computer, you need to meet the following pre-requisites first:

1. Have **Node.js version 16.0** (Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)) or higher installed.
2. Have **NPM** installed, or use other package manager of your choice.

If not certain, confirm the prerequisites by running:

```bash
node -v
```

```bash
npm -v
```

## Creating a new project

The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is embedded in the `crawlee` package:

```bash
npx crawlee create my-crawler
```

A prompt will be shown, asking you to select a template. Crawlee is written in TypeScript so if you're familiar with it, choosing a TypeScript template will give you better code completion and static type checking, but feel free to use JavaScript as well. Functionally they're identical.

Let's choose the first template called **Getting started example**. The command will create a new directory in your current working directory, called **my-crawler**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.

Let's try that!

```bash
cd my-crawler
npm start
```

You will see log messages in the terminal as Crawlee boots up and starts scraping the Crawlee website.

```log
INFO  PlaywrightCrawler: Starting the crawl
INFO  PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
INFO  PlaywrightCrawler: Title of https://crawlee.dev/js/docs/examples is 'Examples | Crawlee'
INFO  PlaywrightCrawler: Title of https://crawlee.dev/js/api/core is '@crawlee/core | API | Crawlee'
INFO  PlaywrightCrawler: Title of https://crawlee.dev/js/api/core/changelog is 'Changelog | API | Crawlee'
INFO  PlaywrightCrawler: Title of https://crawlee.dev/js/docs/quick-start is 'Quick Start | Crawlee'
```

You can always terminate the crawl with a keypress in the terminal:

```text
CTRL+C
```

### Running headful browsers

Browsers controlled by Playwright run headless (without a visible window). You can switch to headful by uncommenting the `headless: false` option in the crawler's constructor. This is useful in the development phase when you want to see what's going on in the browser.

```typescript
// Uncomment this option to see the browser window.
headless: false
```

When you run the example again, after a second a Chromium browser window will open. In the window, you'll see quickly changing pages as the crawler does its job.

:::note

For the sake of this show off, we've slowed down the crawler, but rest assured, it's blazing fast in real world usage.

:::

<ThemedImage
  alt="An image showing off Crawlee scraping the Crawlee website using Puppeteer/Playwright and Chromium"
  sources={{
    light: '/img/chrome-scrape-light.gif',
    dark: '/img/chrome-scrape-dark.gif',
  }}
/>

## Next steps

Next, you will see how to create a very simple crawler and explain Crawlee components while building it.