Skip to content

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

docs: improve onboarding (#1464)

Ondra Urbanb2270a0af015
diff --git a/docs/introduction/01-setting-up.mdx b/docs/introduction/01-setting-up.mdx
index 7cc45467d..3a6e1b51b 100644
--- a/docs/introduction/01-setting-up.mdx
+++ b/docs/introduction/01-setting-up.mdx
@@ -25,39 +25,54 @@ npm -v
## Creating a new project
-The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is also embedded in the `crawlee` package:
+The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is embedded in the `crawlee` package:
```bash
-npx crawlee create my-new-project
+npx crawlee create my-crawler
```
A prompt will be shown, asking you to choose a template. Crawlee is written in TypeScript so if you're familiar with it, choosing a TypeScript template will give you better code completion and static type checking, but feel free to use JavaScript as well. Functionally they're identical.
-Let's choose the first template called **Crawlee playwright template**. The command will create a new directory in your current working directory, called **my-new-project**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
+Let's choose the first template called **Getting started example**. The command will create a new directory in your current working directory, called **my-crawler**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
Let's try that!
```bash
-cd my-new-project
+cd my-crawler
npm start
```
-You will see log messages in the terminal as Crawlee boots up and after a second a Chromium browser window will open. In the window, you'll see quickly changing pages and back in the terminal, you will see the printed titles (contents of the `<title>` HTML tags) of the pages.
+You will see log messages in the terminal as Crawlee boots up and starts scraping the Crawlee website.
-![Chrome Scrape](/img/chrome_scrape.gif)
+```log
+INFO PlaywrightCrawler: Starting the crawl
+INFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+```
-:::info
+You can always terminate the crawl with a keypress in the terminal:
-We picked the Playwright template, which uses Chromium to open pages. If you pick the Cheerio template instead, there won't be any browser window, as the requests to the target site will be done via a specialized HTTP client: [`got-scraping`](https://github.com/apify/got-scraping), instead of a browser.
+```text
+CTRL+C
+```
-:::
+### Running headful browsers
-You can always terminate the crawl with a keypress in the terminal:
+Browsers controlled by Playwright run headless (without a visible window). You can switch to headful by uncommenting the `headless: false` option in the crawler's constructor. This is useful in the development phase when you want to see what's going on in the browser.
-```bash
-CTRL+C
+
+```typescript
+// Uncomment this option to see the browser window.
+headless: false
```
+When you run the example again, after a second a Chromium browser window will open. In the window, you'll see quickly changing pages as the crawler does its job.
+
+![Chrome Scrape](/img/chrome_scrape.gif)
+
## Next lesson
The next lesson will teach you how to create a very simple crawler and explain Crawlee components while building it.
diff --git a/docs/introduction/index.mdx b/docs/introduction/index.mdx
index a6895d500..b64688861 100644
--- a/docs/introduction/index.mdx
+++ b/docs/introduction/index.mdx
@@ -12,7 +12,7 @@ Your crawlers will appear human-like and fly under the radar of modern bot prote
## What you will learn
-The goal of the introduction is to provide a step-by-step guide to all the features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
+The goal of the introduction is to provide a step-by-step guide to the most important features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
## 🛠 Features
@@ -32,7 +32,7 @@ The goal of the introduction is to provide a step-by-step guide to all the featu
- Zero config **HTTP2 support**, even for proxies
- Automatic generation of **browser-like headers**
- Replication of browser **TLS fingerprints**
-- Integrated fast **HTML parser**
+- Integrated fast **HTML parsers**. Cheerio and JSDOM
- Yes, you can scrape **JSON APIs** as well
### 💻 Real browser crawling
diff --git a/docs/quick-start/headful_playwright.ts b/docs/quick-start/headful_playwright.ts
index 149482390..378b8d4f5 100644
--- a/docs/quick-start/headful_playwright.ts
+++ b/docs/quick-start/headful_playwright.ts
@@ -1,16 +1,16 @@
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
- // When you turn off headless mode, the crawler
- // will run with a visible browser window.
- headless: false,
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
- await enqueueLinks({ strategy: 'same-domain' });
- await Dataset.pushData({ url, title });
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
},
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
});
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/quick-start/headful_puppeteer.ts b/docs/quick-start/headful_puppeteer.ts
index c8e44fcbd..37c0f2b00 100644
--- a/docs/quick-start/headful_puppeteer.ts
+++ b/docs/quick-start/headful_puppeteer.ts
@@ -1,16 +1,16 @@
import { PuppeteerCrawler, Dataset } from 'crawlee';
const crawler = new PuppeteerCrawler({
- // When you turn off headless mode, the crawler
- // will run with a visible browser window.
- headless: false,
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
- await enqueueLinks({ strategy: 'same-domain' });
- await Dataset.pushData({ url, title });
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
},
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
});
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/quick-start/index.mdx b/docs/quick-start/index.mdx
index 59e1e6f4f..1acc52bca 100644
--- a/docs/quick-start/index.mdx
+++ b/docs/quick-start/index.mdx
@@ -1,11 +1,12 @@
---
id: quick-start
title: Quick Start
-description: Quick and easy steps to get started using Crawlee today!
+description: With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.
---
import ApiLink from '@site/src/components/ApiLink';
+import Admonition from '@theme/Admonition';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
@@ -33,10 +34,30 @@ This crawler uses a headless browser to crawl, controlled by the [Puppeteer](htt
### PlaywrightCrawler
[Playwright](https://github.com/microsoft/playwright) is a more powerful and full-featured successor to Puppeteer. It can control Chromium, Chrome, Firefox, Webkit and many other browsers. If you're not familiar with Puppeteer already, and you need a headless browser, go with Playwright.
-## Installation
+:::caution before you start
-Crawlee requires [Node.js](https://nodejs.org/en/) 16 or later.
-It can be added to any Node.js project by running:
+Crawlee requires [Node.js 16 or later](https://nodejs.org/en/).
+
+:::
+
+## Installation with Crawlee CLI
+
+The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**.
+The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
+
+```bash
+npx crawlee create my-crawler
+```
+
+After the installation is complete you can start the crawler like this:
+
+```bash
+cd my-crawler && npm start
+```
+
+## Manual installation
+
+You can add Crawlee to any Node.js project by running:
<Tabs groupId="quick_start">
<TabItem value="cheerio" label="CheerioCrawler" default>
@@ -66,17 +87,24 @@ It can be added to any Node.js project by running:
## Crawling
-Run the following example to perform a recursive crawl of the Crawlee website using:
+Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler.
+
+<Admonition type="caution" title="Don't forget about module imports">
+ To run the example, add a <code>"type": "module"</code> clause into your <code>package.json</code> or
+ copy it into a file with an <code>.mjs</code> suffix. This enables <code>import</code> statements in Node.js.
+ See <a href="https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#enabling" target="_blank" rel="noreferrer">Node.js docs</a> for
+ more information.
+</Admonition>
<Tabs groupId="quick_start">
<TabItem value="cheerio" label="CheerioCrawler" default>
- <CodeBlock language="js" title="src/main.mjs">{CheerioSource}</CodeBlock>
+ <CodeBlock language="js">{CheerioSource}</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PlaywrightSource}</CodeBlock>
+ <CodeBlock language="js">{PlaywrightSource}</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PuppeteerSource}</CodeBlock>
+ <CodeBlock language="js">{PuppeteerSource}</CodeBlock>
</TabItem>
</Tabs>
@@ -91,10 +119,10 @@ Browsers controlled by Puppeteer and Playwright run headless (without a visible
<Tabs groupId="quick_start">
<TabItem value="playwright" label="PlaywrightCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PlaywrightHeadful}</CodeBlock>
+ <CodeBlock language="js">{PlaywrightHeadful}</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PuppeteerHeadful}</CodeBlock>
+ <CodeBlock language="js">{PuppeteerHeadful}</CodeBlock>
</TabItem>
</Tabs>
@@ -127,18 +155,3 @@ You can find more examples showcasing various features of Crawlee in the [Exampl
- [Configuration](./guides/configuration)
- [Request storage](./guides/request-storage)
- [Result storage](./guides/result-storage)
-
-## Local usage with Crawlee command-line interface (CLI)
-
-To create a boilerplate of your project, you can use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli) tool by running:
-
-```bash
-npx crawlee create my-cheerio-crawler
-```
-
-The CLI will prompt you to select a project boilerplate template - let's pick the **Crawlee cheerio template [TypeScript]**. The tool will create a directory called `my-cheerio-crawler` with Node.js project files. You can run the project as follows:
-
-```bash
-cd my-cheerio-crawler
-npm start
-```
diff --git a/docs/quick-start/quick_start_cheerio.ts b/docs/quick-start/quick_start_cheerio.ts
index 038aa3b03..0dff35063 100644
--- a/docs/quick-start/quick_start_cheerio.ts
+++ b/docs/quick-start/quick_start_cheerio.ts
@@ -1,21 +1,21 @@
import { CheerioCrawler, Dataset } from 'crawlee';
+// CheerioCrawler crawls the web using HTTP requests
+// and parses HTML using the Cheerio library.
const crawler = new CheerioCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, $, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = $('title').text();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/quick-start/quick_start_cheerio.txt b/docs/quick-start/quick_start_cheerio.txt
index fb7c410e3..e68fa4133 100644
--- a/docs/quick-start/quick_start_cheerio.txt
+++ b/docs/quick-start/quick_start_cheerio.txt
@@ -1,6 +1,5 @@
-INFO CheerioCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.7,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
-INFO CheerioCrawler: Title of https://crawlee.dev/: Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples: Examples | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start: Quick Start | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/api/core: @crawlee/core | API | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/api/core/changelog: Changelog | API | Crawlee
+INFO CheerioCrawler: Starting the crawl
+INFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'
diff --git a/docs/quick-start/quick_start_playwright.ts b/docs/quick-start/quick_start_playwright.ts
index d7d12c27b..9d0f0e738 100644
--- a/docs/quick-start/quick_start_playwright.ts
+++ b/docs/quick-start/quick_start_playwright.ts
@@ -1,21 +1,23 @@
import { PlaywrightCrawler, Dataset } from 'crawlee';
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
+ // Uncomment this option to see the browser window.
+ // headless: false,
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/quick-start/quick_start_puppeteer.ts b/docs/quick-start/quick_start_puppeteer.ts
index 001254010..6cb7dd6d5 100644
--- a/docs/quick-start/quick_start_puppeteer.ts
+++ b/docs/quick-start/quick_start_puppeteer.ts
@@ -1,21 +1,23 @@
import { PuppeteerCrawler, Dataset } from 'crawlee';
+// PuppeteerCrawler crawls the web using a headless
+// browser controlled by the Puppeteer library.
const crawler = new PuppeteerCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
+ // Uncomment this option to see the browser window.
+ // headless: false,
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/packages/templates/manifest.json b/packages/templates/manifest.json
index 495888040..13c9fe987 100644
--- a/packages/templates/manifest.json
+++ b/packages/templates/manifest.json
@@ -1,28 +1,36 @@
{
"templates": [
+ {
+ "name": "getting-started-ts",
+ "description": "Getting started example [TypeScript]"
+ },
+ {
+ "name": "getting-started-js",
+ "description": "Getting started example [JavaScript]"
+ },
{
"name": "cheerio-ts",
- "description": "Crawlee cheerio template [TypeScript]"
+ "description": "CheerioCrawler template project [TypeScript]"
},
{
"name": "playwright-ts",
- "description": "Crawlee playwright template [TypeScript]"
+ "description": "PlaywrightCrawler template project [TypeScript]"
},
{
"name": "puppeteer-ts",
- "description": "Crawlee puppeteer template [TypeScript]"
+ "description": "PuppeteerCrawler template project [TypeScript]"
},
{
"name": "cheerio-js",
- "description": "Crawlee cheerio template [JavaScript]"
+ "description": "CheerioCrawler template project [JavaScript]"
},
{
"name": "playwright-js",
- "description": "Crawlee playwright template [JavaScript]"
+ "description": "PlaywrightCrawler template project [JavaScript]"
},
{
"name": "puppeteer-js",
- "description": "Crawlee puppeteer template [JavaScript]"
+ "description": "PlaywrightCrawler template project [JavaScript]"
}
]
}
diff --git a/packages/templates/templates/cheerio-js/.dockerignore b/packages/templates/templates/cheerio-js/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/cheerio-js/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/cheerio-js/Dockerfile b/packages/templates/templates/cheerio-js/Dockerfile
index 438d36e17..4018d6b99 100644
--- a/packages/templates/templates/cheerio-js/Dockerfile
+++ b/packages/templates/templates/cheerio-js/Dockerfile
@@ -1,15 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
FROM apify/actor-node:16
-# copy package.json, lock file and possibly crawlee.json
-COPY package*.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
-# install only prod deps
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/packages/templates/templates/cheerio-js/package.json b/packages/templates/templates/cheerio-js/package.json
index 80a48c5d7..51fb96156 100644
--- a/packages/templates/templates/cheerio-js/package.json
+++ b/packages/templates/templates/cheerio-js/package.json
@@ -8,7 +8,6 @@
},
"scripts": {
"start": "node src/main.js",
- "build": "tsc",
"test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
},
"author": "It's not you it's me",
diff --git a/packages/templates/templates/cheerio-js/src/routes.js b/packages/templates/templates/cheerio-js/src/routes.js
index 971577b0f..299ce7913 100644
--- a/packages/templates/templates/cheerio-js/src/routes.js
+++ b/packages/templates/templates/cheerio-js/src/routes.js
@@ -3,19 +3,19 @@ import { Dataset, createCheerioRouter } from 'crawlee';
export const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('LIST', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('DETAIL', async ({ request, $, log }) => {
+router.addHandler('detail', async ({ request, $, log }) => {
const title = $('title').text();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/packages/templates/templates/cheerio-ts/.dockerignore b/packages/templates/templates/cheerio-ts/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/cheerio-ts/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/cheerio-ts/Dockerfile b/packages/templates/templates/cheerio-ts/Dockerfile
index a2d94e6cc..43815f97d 100644
--- a/packages/templates/templates/cheerio-ts/Dockerfile
+++ b/packages/templates/templates/cheerio-ts/Dockerfile
@@ -1,32 +1,50 @@
-# First, specify the base Docker image. You can read more about
+# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
FROM apify/actor-node:16 AS builder
-# Second, copy the TS source files, package.json and package-lock.json since it should be
-# the only files that affects the installation and build in the next step
-COPY package*.json tsconfig.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
-# Install all dependencies (including dev deps) and build the project
-RUN npm install --include=dev && npm run build
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
# Create final image
FROM apify/actor-node:16
-# Copy only necessary files (crawlee.json might not exist, so we need the regexp trick with `crawlee?json`)
-COPY --from=builder /usr/src/app/package.json /usr/src/app/crawlee?json ./
+# Copy only built JS files from builder image
COPY --from=builder /usr/src/app/dist ./dist
-# Install only prod deps
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
-# Run compiled code
-CMD npm run start:prod
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY . ./
+
+
+# Run the image.
+CMD npm run start:prod --silent
diff --git a/packages/templates/templates/cheerio-ts/src/routes.ts b/packages/templates/templates/cheerio-ts/src/routes.ts
index a7b0f2e67..299ce7913 100644
--- a/packages/templates/templates/cheerio-ts/src/routes.ts
+++ b/packages/templates/templates/cheerio-ts/src/routes.ts
@@ -1,21 +1,21 @@
-import { createCheerioRouter, Dataset } from 'crawlee';
+import { Dataset, createCheerioRouter } from 'crawlee';
export const router = createCheerioRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('LIST', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('DETAIL', async ({ request, $, log }) => {
+router.addHandler('detail', async ({ request, $, log }) => {
const title = $('title').text();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/packages/templates/templates/getting-started-js/.dockerignore b/packages/templates/templates/getting-started-js/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/getting-started-js/.gitignore b/packages/templates/templates/getting-started-js/.gitignore
new file mode 100644
index 000000000..02a7e1020
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/.gitignore
@@ -0,0 +1,8 @@
+# This file tells Git which files shouldn't be added to source control
+
+.idea
+dist
+node_modules
+apify_storage
+crawlee_storage
+storage
diff --git a/packages/templates/templates/getting-started-js/Dockerfile b/packages/templates/templates/getting-started-js/Dockerfile
new file mode 100644
index 000000000..afe996891
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/Dockerfile
@@ -0,0 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node-playwright-chrome:16
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/packages/templates/templates/getting-started-js/README.md b/packages/templates/templates/getting-started-js/README.md
new file mode 100644
index 000000000..6a404f0e3
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/README.md
@@ -0,0 +1,9 @@
+# Getting started with Crawlee
+
+This example uses `PlaywrightCrawler` to recursively crawl https://crawlee.dev using the browser automation library [Playwright](https://playwright.dev).
+
+You can find more examples and documentation at the following links:
+
+- [Step-by-step tutorial](https://crawlee.dev/docs/introduction) for Crawlee
+- `PlaywrightCrawler` [API documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)
+- Other [examples](https://crawlee.dev/docs/examples/playwright-crawler)
diff --git a/packages/templates/templates/getting-started-js/package.json b/packages/templates/templates/getting-started-js/package.json
new file mode 100644
index 000000000..2ab4897db
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/package.json
@@ -0,0 +1,16 @@
+{
+ "name": "crawlee-getting-started-js",
+ "version": "0.0.1",
+ "type": "module",
+ "description": "This is an example of a Crawlee project.",
+ "dependencies": {
+ "crawlee": "^3.0.0",
+ "playwright": "*"
+ },
+ "scripts": {
+ "start": "node src/main.js",
+ "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
+ },
+ "author": "It's not you it's me",
+ "license": "ISC"
+}
diff --git a/packages/templates/templates/getting-started-js/src/main.js b/packages/templates/templates/getting-started-js/src/main.js
new file mode 100644
index 000000000..256fc3a57
--- /dev/null
+++ b/packages/templates/templates/getting-started-js/src/main.js
@@ -0,0 +1,24 @@
+// For more information, see https://crawlee.dev/
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
+const crawler = new PlaywrightCrawler({
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
+
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
+ },
+ // Uncomment this option to see the browser window.
+ // headless: false,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/packages/templates/templates/getting-started-ts/.dockerignore b/packages/templates/templates/getting-started-ts/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/getting-started-ts/.gitignore b/packages/templates/templates/getting-started-ts/.gitignore
new file mode 100644
index 000000000..02a7e1020
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/.gitignore
@@ -0,0 +1,8 @@
+# This file tells Git which files shouldn't be added to source control
+
+.idea
+dist
+node_modules
+apify_storage
+crawlee_storage
+storage
diff --git a/packages/templates/templates/getting-started-ts/Dockerfile b/packages/templates/templates/getting-started-ts/Dockerfile
new file mode 100644
index 000000000..e6477ffc3
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/Dockerfile
@@ -0,0 +1,51 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
+FROM apify/actor-node-playwright-chrome:16 AS builder
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY --chown=myuser . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
+
+# Create final image
+FROM apify/actor-node-playwright-chrome:16
+
+# Copy only built JS files from builder image
+COPY --from=builder --chown=myuser /home/myuser/dist ./dist
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
+
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
+RUN npm --quiet set progress=false \
+ && npm install --omit=dev --omit=optional \
+ && echo "Installed NPM packages:" \
+ && (npm list --omit=dev --all || true) \
+ && echo "Node.js version:" \
+ && node --version \
+ && echo "NPM version:" \
+ && npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image. If you know you won't need headful browsers,
+# you can remove the XVFB start script for a micro perf gain.
+CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod --silent
diff --git a/packages/templates/templates/getting-started-ts/README.md b/packages/templates/templates/getting-started-ts/README.md
new file mode 100644
index 000000000..6a404f0e3
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/README.md
@@ -0,0 +1,9 @@
+# Getting started with Crawlee
+
+This example uses `PlaywrightCrawler` to recursively crawl https://crawlee.dev using the browser automation library [Playwright](https://playwright.dev).
+
+You can find more examples and documentation at the following links:
+
+- [Step-by-step tutorial](https://crawlee.dev/docs/introduction) for Crawlee
+- `PlaywrightCrawler` [API documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)
+- Other [examples](https://crawlee.dev/docs/examples/playwright-crawler)
diff --git a/packages/templates/templates/getting-started-ts/package.json b/packages/templates/templates/getting-started-ts/package.json
new file mode 100644
index 000000000..32aaa1bec
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/package.json
@@ -0,0 +1,24 @@
+{
+ "name": "crawlee-getting-started-ts",
+ "version": "0.0.1",
+ "type": "module",
+ "description": "This is an example of a Crawlee project.",
+ "dependencies": {
+ "crawlee": "^3.0.0",
+ "playwright": "*"
+ },
+ "devDependencies": {
+ "@apify/tsconfig": "^0.1.0",
+ "ts-node": "^10.8.0",
+ "typescript": "^4.7.4"
+ },
+ "scripts": {
+ "start": "npm run start:dev",
+ "start:prod": "node dist/main.js",
+ "start:dev": "ts-node-esm -T src/main.ts",
+ "build": "tsc",
+ "test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
+ },
+ "author": "It's not you it's me",
+ "license": "ISC"
+}
diff --git a/packages/templates/templates/getting-started-ts/src/main.ts b/packages/templates/templates/getting-started-ts/src/main.ts
new file mode 100644
index 000000000..256fc3a57
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/src/main.ts
@@ -0,0 +1,24 @@
+// For more information, see https://crawlee.dev/
+import { PlaywrightCrawler, Dataset } from 'crawlee';
+
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
+const crawler = new PlaywrightCrawler({
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, page, enqueueLinks, log }) {
+ const title = await page.title();
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
+
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
+ },
+ // Uncomment this option to see the browser window.
+ // headless: false,
+});
+
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/packages/templates/templates/getting-started-ts/tsconfig.json b/packages/templates/templates/getting-started-ts/tsconfig.json
new file mode 100644
index 000000000..d7bd5a048
--- /dev/null
+++ b/packages/templates/templates/getting-started-ts/tsconfig.json
@@ -0,0 +1,13 @@
+{
+ "extends": "@apify/tsconfig",
+ "compilerOptions": {
+ "module": "ES2022",
+ "target": "ES2022",
+ "outDir": "dist",
+ "noUnusedLocals": false,
+ "lib": ["DOM"]
+ },
+ "include": [
+ "./src/**/*"
+ ]
+}
diff --git a/packages/templates/templates/playwright-js/.dockerignore b/packages/templates/templates/playwright-js/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/playwright-js/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/playwright-js/Dockerfile b/packages/templates/templates/playwright-js/Dockerfile
index 13aa67daa..afe996891 100644
--- a/packages/templates/templates/playwright-js/Dockerfile
+++ b/packages/templates/templates/playwright-js/Dockerfile
@@ -1,15 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
FROM apify/actor-node-playwright-chrome:16
-# copy package.json, lock file and possibly crawlee.json
-COPY package*.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# install only prod deps
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/packages/templates/templates/playwright-js/README.md b/packages/templates/templates/playwright-js/README.md
index 3cf608da8..5dfa84acf 100644
--- a/packages/templates/templates/playwright-js/README.md
+++ b/packages/templates/templates/playwright-js/README.md
@@ -6,21 +6,3 @@ If you're looking for examples or want to learn more visit:
- [Documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)
- [Examples](https://crawlee.dev/docs/examples/playwright-crawler)
-
-## Writing a README
-
-See our tutorial on [writing READMEs for your actors](https://help.apify.com/en/articles/2912548-how-to-write-great-readme-for-your-actors) if you need more inspiration.
-
-### Table of contents
-
-If your README requires a table of contents, use the template below and make sure to keep the `<!-- toc start -->` and `<!-- toc end -->` markers.
-
-<!-- toc start -->
-- Introduction
-- Use Cases
- - Case 1
- - Case 2
-- Input
-- Output
-- Miscellaneous
- <!-- toc end -->
diff --git a/packages/templates/templates/playwright-js/package.json b/packages/templates/templates/playwright-js/package.json
index e087d4fcf..d331e1190 100644
--- a/packages/templates/templates/playwright-js/package.json
+++ b/packages/templates/templates/playwright-js/package.json
@@ -9,7 +9,6 @@
},
"scripts": {
"start": "node src/main.js",
- "build": "tsc",
"test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
},
"author": "It's not you it's me",
diff --git a/packages/templates/templates/playwright-js/src/routes.js b/packages/templates/templates/playwright-js/src/routes.js
index f26623a67..1d758a8f7 100644
--- a/packages/templates/templates/playwright-js/src/routes.js
+++ b/packages/templates/templates/playwright-js/src/routes.js
@@ -3,19 +3,19 @@ import { Dataset, createPlaywrightRouter } from 'crawlee';
export const router = createPlaywrightRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('handleList', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('handleDetail', async ({ request, page, log }) => {
+router.addHandler('detail', async ({ request, page, log }) => {
const title = await page.title();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/packages/templates/templates/playwright-ts/.dockerignore b/packages/templates/templates/playwright-ts/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/playwright-ts/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/playwright-ts/Dockerfile b/packages/templates/templates/playwright-ts/Dockerfile
index 8ffc524b0..e6477ffc3 100644
--- a/packages/templates/templates/playwright-ts/Dockerfile
+++ b/packages/templates/templates/playwright-ts/Dockerfile
@@ -1,32 +1,51 @@
-# First, specify the base Docker image. You can read more about
+# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
-FROM apify/actor-node:16 AS builder
+FROM apify/actor-node-playwright-chrome:16 AS builder
-# Second, copy the TS source files, package.json and package-lock.json since it should be
-# the only files that affects the installation and build in the next step
-COPY package*.json tsconfig.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# Install all dependencies (including dev deps) and build the project
-RUN npm install --include=dev && npm run build
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY --chown=myuser . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
# Create final image
FROM apify/actor-node-playwright-chrome:16
-# Copy only necessary files (crawlee.json might not exist, so we need the regexp trick with `crawlee?json`)
-COPY --from=builder /usr/src/app/package.json /usr/src/app/crawlee?json ./
-COPY --from=builder /usr/src/app/dist ./dist
+# Copy only built JS files from builder image
+COPY --from=builder --chown=myuser /home/myuser/dist ./dist
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# Install only prod deps
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
-# Run compiled code
-CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image. If you know you won't need headful browsers,
+# you can remove the XVFB start script for a micro perf gain.
+CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod --silent
diff --git a/packages/templates/templates/playwright-ts/README.md b/packages/templates/templates/playwright-ts/README.md
index 2e8e6970e..9f3345c75 100644
--- a/packages/templates/templates/playwright-ts/README.md
+++ b/packages/templates/templates/playwright-ts/README.md
@@ -6,21 +6,3 @@ If you're looking for examples or want to learn more visit:
- [Documentation](https://crawlee.dev/api/playwright-crawler/class/PlaywrightCrawler)
- [Examples](https://crawlee.dev/docs/examples/playwright-crawler)
-
-## Writing a README
-
-See our tutorial on [writing READMEs for your actors](https://help.apify.com/en/articles/2912548-how-to-write-great-readme-for-your-actors) if you need more inspiration.
-
-### Table of contents
-
-If your README requires a table of contents, use the template below and make sure to keep the `<!-- toc start -->` and `<!-- toc end -->` markers.
-
-<!-- toc start -->
-- Introduction
-- Use Cases
- - Case 1
- - Case 2
-- Input
-- Output
-- Miscellaneous
- <!-- toc end -->
diff --git a/packages/templates/templates/playwright-ts/src/routes.ts b/packages/templates/templates/playwright-ts/src/routes.ts
index f26623a67..1d758a8f7 100644
--- a/packages/templates/templates/playwright-ts/src/routes.ts
+++ b/packages/templates/templates/playwright-ts/src/routes.ts
@@ -3,19 +3,19 @@ import { Dataset, createPlaywrightRouter } from 'crawlee';
export const router = createPlaywrightRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('handleList', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('handleDetail', async ({ request, page, log }) => {
+router.addHandler('detail', async ({ request, page, log }) => {
const title = await page.title();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/packages/templates/templates/puppeteer-js/.dockerignore b/packages/templates/templates/puppeteer-js/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/puppeteer-js/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/puppeteer-js/Dockerfile b/packages/templates/templates/puppeteer-js/Dockerfile
index 9bdbbf8a6..e7e8a8a8c 100644
--- a/packages/templates/templates/puppeteer-js/Dockerfile
+++ b/packages/templates/templates/puppeteer-js/Dockerfile
@@ -1,15 +1,29 @@
+# Specify the base Docker image. You can read more about
+# the available images at https://crawlee.dev/docs/guides/docker-images
+# You can also use any other image from Docker Hub.
FROM apify/actor-node-puppeteer-chrome:16
-# copy package.json, lock file and possibly crawlee.json
-COPY package*.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# install only prod deps
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
+
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image.
+CMD npm start --silent
diff --git a/packages/templates/templates/puppeteer-js/README.md b/packages/templates/templates/puppeteer-js/README.md
index 07545daa1..01c3e360a 100644
--- a/packages/templates/templates/puppeteer-js/README.md
+++ b/packages/templates/templates/puppeteer-js/README.md
@@ -6,21 +6,3 @@ If you're looking for examples or want to learn more visit:
- [Documentation](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler)
- [Examples](https://crawlee.dev/docs/examples/puppeteer-crawler)
-
-## Writing a README
-
-See our tutorial on [writing READMEs for your actors](https://help.apify.com/en/articles/2912548-how-to-write-great-readme-for-your-actors) if you need more inspiration.
-
-### Table of contents
-
-If your README requires a table of contents, use the template below and make sure to keep the `<!-- toc start -->` and `<!-- toc end -->` markers.
-
-<!-- toc start -->
-- Introduction
-- Use Cases
- - Case 1
- - Case 2
-- Input
-- Output
-- Miscellaneous
- <!-- toc end -->
diff --git a/packages/templates/templates/puppeteer-js/package.json b/packages/templates/templates/puppeteer-js/package.json
index ce504b614..e34913979 100644
--- a/packages/templates/templates/puppeteer-js/package.json
+++ b/packages/templates/templates/puppeteer-js/package.json
@@ -9,7 +9,6 @@
},
"scripts": {
"start": "node src/main.js",
- "build": "tsc",
"test": "echo \"Error: oops, the actor has no tests yet, sad!\" && exit 1"
},
"author": "It's not you it's me",
diff --git a/packages/templates/templates/puppeteer-js/src/routes.js b/packages/templates/templates/puppeteer-js/src/routes.js
index 7f2b1d4bc..52f55358c 100644
--- a/packages/templates/templates/puppeteer-js/src/routes.js
+++ b/packages/templates/templates/puppeteer-js/src/routes.js
@@ -3,19 +3,19 @@ import { Dataset, createPuppeteerRouter } from 'crawlee';
export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('handleList', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('handleDetail', async ({ request, page, log }) => {
+router.addHandler('detail', async ({ request, page, log }) => {
const title = await page.title();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/packages/templates/templates/puppeteer-ts/.dockerignore b/packages/templates/templates/puppeteer-ts/.dockerignore
new file mode 100644
index 000000000..ceb85b1ce
--- /dev/null
+++ b/packages/templates/templates/puppeteer-ts/.dockerignore
@@ -0,0 +1,10 @@
+# configurations
+.idea
+
+# crawlee and apify storage folders
+apify_storage
+crawlee_storage
+storage
+
+# installed files
+node_modules
diff --git a/packages/templates/templates/puppeteer-ts/Dockerfile b/packages/templates/templates/puppeteer-ts/Dockerfile
index 18b9acee4..19f6fa626 100644
--- a/packages/templates/templates/puppeteer-ts/Dockerfile
+++ b/packages/templates/templates/puppeteer-ts/Dockerfile
@@ -1,32 +1,51 @@
-# First, specify the base Docker image. You can read more about
+# Specify the base Docker image. You can read more about
# the available images at https://crawlee.dev/docs/guides/docker-images
# You can also use any other image from Docker Hub.
-FROM apify/actor-node:16 AS builder
+FROM apify/actor-node-puppeteer-chrome:16 AS builder
-# Second, copy the TS source files, package.json and package-lock.json since it should be
-# the only files that affects the installation and build in the next step
-COPY package*.json tsconfig.json crawlee?json ./
-COPY src ./src
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# Install all dependencies (including dev deps) and build the project
-RUN npm install --include=dev && npm run build
+# Install all dependencies. Don't audit to speed up the installation.
+RUN npm install --include=dev --audit=false
+
+# Next, copy the source files using the user set
+# in the base image.
+COPY --chown=myuser . ./
+
+# Install all dependencies and build the project.
+# Don't audit to speed up the installation.
+RUN npm run build
# Create final image
FROM apify/actor-node-puppeteer-chrome:16
-# Copy only necessary files (crawlee.json might not exist, so we need the regexp trick with `crawlee?json`)
-COPY --from=builder /usr/src/app/package.json /usr/src/app/crawlee?json ./
-COPY --from=builder /usr/src/app/dist ./dist
+# Copy only built JS files from builder image
+COPY --from=builder --chown=myuser /home/myuser/dist ./dist
+
+# Copy just package.json and package-lock.json
+# to speed up the build using Docker layer cache.
+COPY --chown=myuser package*.json ./
-# Install only prod deps
+# Install NPM packages, skip optional and development dependencies to
+# keep the image small. Avoid logging too much and print the dependency
+# tree for debugging
RUN npm --quiet set progress=false \
- && npm install --only=prod --no-optional \
+ && npm install --omit=dev --omit=optional \
&& echo "Installed NPM packages:" \
- && (npm list --only=prod --no-optional --all || true) \
+ && (npm list --omit=dev --all || true) \
&& echo "Node.js version:" \
&& node --version \
&& echo "NPM version:" \
&& npm --version
-# Run compiled code
-CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod
+# Next, copy the remaining files and directories with the source code.
+# Since we do this after NPM install, quick build will be really fast
+# for most source file changes.
+COPY --chown=myuser . ./
+
+
+# Run the image. If you know you won't need headful browsers,
+# you can remove the XVFB start script for a micro perf gain.
+CMD ./start_xvfb_and_run_cmd.sh && npm run start:prod --silent
diff --git a/packages/templates/templates/puppeteer-ts/README.md b/packages/templates/templates/puppeteer-ts/README.md
index 03d07ad62..68e0895ea 100644
--- a/packages/templates/templates/puppeteer-ts/README.md
+++ b/packages/templates/templates/puppeteer-ts/README.md
@@ -6,21 +6,3 @@ If you're looking for examples or want to learn more visit:
- [Documentation](https://crawlee.dev/api/puppeteer-crawler/class/PuppeteerCrawler)
- [Examples](https://crawlee.dev/docs/examples/puppeteer-crawler)
-
-## Writing a README
-
-See our tutorial on [writing READMEs for your actors](https://help.apify.com/en/articles/2912548-how-to-write-great-readme-for-your-actors) if you need more inspiration.
-
-### Table of contents
-
-If your README requires a table of contents, use the template below and make sure to keep the `<!-- toc start -->` and `<!-- toc end -->` markers.
-
-<!-- toc start -->
-- Introduction
-- Use Cases
- - Case 1
- - Case 2
-- Input
-- Output
-- Miscellaneous
- <!-- toc end -->
diff --git a/packages/templates/templates/puppeteer-ts/src/routes.ts b/packages/templates/templates/puppeteer-ts/src/routes.ts
index 7f2b1d4bc..52f55358c 100644
--- a/packages/templates/templates/puppeteer-ts/src/routes.ts
+++ b/packages/templates/templates/puppeteer-ts/src/routes.ts
@@ -3,19 +3,19 @@ import { Dataset, createPuppeteerRouter } from 'crawlee';
export const router = createPuppeteerRouter();
router.addDefaultHandler(async ({ enqueueLinks, log }) => {
- log.info(`Handle Start URLs`);
+ log.info(`enqueueing new URLs`);
await enqueueLinks({
- globs: ['https://crawlee.dev/**'],
- label: 'DETAIL',
+ globs: ['https://crawlee.dev/*'],
+ label: 'detail',
});
});
-router.addHandler('handleList', async ({ log }) => {
- log.info(`Handle pagination`);
-});
-
-router.addHandler('handleDetail', async ({ request, page, log }) => {
+router.addHandler('detail', async ({ request, page, log }) => {
const title = await page.title();
- log.info(`Handle details: ${title} [${request.loadedUrl}]`);
- await Dataset.pushData({ url: request.loadedUrl });
+ log.info(`${title}`, { url: request.loadedUrl });
+
+ await Dataset.pushData({
+ url: request.loadedUrl,
+ title,
+ });
});
diff --git a/website/docusaurus.config.js b/website/docusaurus.config.js
index 51fee3ad3..5b0c51fd6 100644
--- a/website/docusaurus.config.js
+++ b/website/docusaurus.config.js
@@ -33,7 +33,7 @@ const packagesOrder = [
/** @type {Partial<import('@docusaurus/types').DocusaurusConfig>} */
module.exports = {
title: 'Crawlee',
- tagline: 'The scalable web crawling, scraping and automation library for JavaScript/Node.js',
+ tagline: 'Build reliable crawlers. Fast.',
url: 'https://crawlee.dev',
baseUrl: '/',
trailingSlash: false,
diff --git a/website/src/components/Highlights.jsx b/website/src/components/Highlights.jsx
index 937c2d6e5..7db438f1d 100644
--- a/website/src/components/Highlights.jsx
+++ b/website/src/components/Highlights.jsx
@@ -4,61 +4,62 @@ import styles from './Highlights.module.css';
const FeatureList = [
{
- title: 'Runs on JavaScript',
+ title: 'JavaScript & TypeScript',
Svg: require('../../static/img/features/runs-on-js.svg').default,
description: (
<>
- JavaScript is the language of the web. Crawlee builds on popular tools like <a href="https://www.npmjs.com/package/playwright">Playwright</a>, {' '}
- <a href="https://www.npmjs.com/package/puppeteer">Puppeteer</a> and <a href='https://www.npmjs.com/package/cheerio'>cheerio</a>,
- to deliver large-scale high-performance web scraping and crawling of any website. Works best with <b>TypeScript</b>!
+ We believe websites are best scraped in the language they're written in. Crawlee <b>runs on Node.js
+ and it's built in TypeScript</b> to improve code completion in your IDE, even if you don't use TypeScript yourself.
</>
),
},
{
- title: 'Automates any web workflow',
- Svg: require('../../static/img/features/automate-everything.svg').default,
+ title: 'HTTP scraping',
+ Svg: require('../../static/img/features/fingerprints.svg').default,
description: (
<>
- Run headless Chrome, Firefox, WebKit or other browsers, manage lists and queues of URLs to crawl, run crawlers in parallel at maximum
- system capacity. Handle storage and export of results and rotate proxies.
+ Crawlee makes HTTP requests that <b>mimic browser headers and TLS fingerprints</b>. It also rotates
+ them automatically based on data about real-world traffic. Popular HTML parsers <b>Cheerio and JSDOM</b> are included.
</>
),
},
{
- title: 'Works on any system',
+ title: 'Headless browsers',
Svg: require('../../static/img/features/works-everywhere.svg').default,
description: (
<>
- Crawlee can be used stand-alone on your own systems or it can run as a serverless microservice on the {' '}
- <a href="https://console.apify.com/actors">Apify Platform</a>.
+ Switch your crawlers from HTTP to headless browsers in 3 lines of code. Crawlee builds on top of <b>Puppeteer
+ and Playwright</b> and adds its own <b>anti-blocking features and human-like fingerprints</b>. Chrome, Firefox and more.
</>
),
},
{
- title: 'Automatic scaling',
+ title: 'Automatic scaling and proxy management',
Svg: require('../../static/img/features/auto-scaling.svg').default,
description: (
<>
- All the crawlers are automatically scaled based on available system resources using the <code>AutoscaledPool</code> class.
- Advanced options are available to fine-tune scaling behaviour.
+ Crawlee automatically manages concurrency based on available system resources and smartly rotates proxies.
+ Proxies that often time-out, return network errors or bad HTTP codes like 401 or 403 are discarded.
</>
),
},
{
- title: 'Generated fingerprints',
+ title: 'Queue and Storage',
Svg: require('../../static/img/features/fingerprints.svg').default,
description: (
<>
- Never get blocked with unique fingerprints for browsers generated based on real world data.
+ You can save files, screenshots and JSON results to disk with one line of code or plug an adapter
+ for your DB. Your URLs are kept in a queue that ensures their uniqueness and that you don't lose progress when something fails.
</>
),
},
{
- title: 'Browser like requests from Node.js',
+ title: 'Helpful utils and configurability',
Svg: require('../../static/img/features/node-requests.svg').default,
description: (
<>
- Crawl using HTTP requests as if they were from browsers, using auto-generated headers based on real browsers and their TLS fingerprints.
+ Crawlee includes tools for extracting social handles or phone numbers, infinite scrolling, blocking
+ unwanted assets and many more. It works great out of the box, but also provides rich configuration options.
</>
),
},
diff --git a/website/src/pages/index.js b/website/src/pages/index.js
index f836d9a53..f679d66f5 100644
--- a/website/src/pages/index.js
+++ b/website/src/pages/index.js
@@ -1,11 +1,12 @@
import React from 'react';
import clsx from 'clsx';
+import Admonition from '@theme/Admonition';
import Layout from '@theme/Layout';
import Link from '@docusaurus/Link';
import CodeBlock from '@theme/CodeBlock';
import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
import styles from './index.module.css';
-import Hightlights from '../components/Highlights';
+import Highlights from '../components/Highlights';
function Hero() {
return (
@@ -55,20 +56,27 @@ function Features() {
return (
<section className={clsx('container', styles.features)}>
<div className="row">
- <div className="col col--4">
- <h2>Easy crawling</h2>
- </div>
- </div>
- <div className="row">
- <div className="col col--4">
+ <div className="col col--6">
+ <h2>Reliable crawling 🏗</h2>
+ <p>
+ Crawlee won't fix broken selectors for you (yet), but it helps you <b>build and maintain your crawlers faster</b>.
+ </p>
+ <p>
+ When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to
+ one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back.
+ </p>
<p>
- There are three main classes that you can use to start crawling the web in no time. Need to crawl plain HTML?
- Use the blazing fast CheerioCrawler. For complex websites that use React, Vue or other front-end javascript libraries and require
- JavaScript execution, spawn a headless browser with PlaywrightCrawler or PuppeteerCrawler.
+ It keeps your proxies healthy by rotating them smartly with good fingerprints that make your crawlers
+ look human-like. It's not unblockable,
+ but <a href="https://blog.apify.com/daltix-python-vs-apify-sdk/" target="_blank" rel="noreferrer"><b>it will save you money in the long run</b></a>.
+ </p>
+ <p>
+ Crawlee is built by people who scrape for a living and use it every day to scrape millions of pages.
+ <a href="https://discord.com/invite/jyEM2PRvMU" target="_blank" rel="noreferrer"><b> Meet our community on Discord</b></a>.
</p>
</div>
- <div className="col col--8 text--center">
- <img src={require('../../static/img/chrome_scrape.gif').default} className={clsx(styles.hideSmall)} />
+ <div className="col col--5 col--offset-1">
+ <img src={require('../../static/img/logo-blur.png').default} className={clsx(styles.hideSmall)} />
</div>
</div>
</section>
@@ -77,19 +85,26 @@ function Features() {
const example = `import { PlaywrightCrawler, Dataset } from 'crawlee';
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
- async requestHandler({ request, page, enqueueLinks }) {
+ // Use the requestHandler to process each of the crawled pages.
+ async requestHandler({ request, page, enqueueLinks, log }) {
const title = await page.title();
- console.log(\`Title of $\{request.loadedUrl} is '$\{title}'\`);
+ log.info(\`Title of \${request.loadedUrl} is '\${title}'\`);
- // save some results
+ // Save results as JSON to ./storage/datasets/default
await Dataset.pushData({ title, url: request.loadedUrl });
- // enqueue all links targeting the same hostname
+ // Extract links from the current page
+ // and add them to the crawling queue.
await enqueueLinks();
- }
+ },
+ // Uncomment this option to see the browser window.
+ // headless: false,
});
+// Add first URL to the queue and start the crawl.
await crawler.run(['https://crawlee.dev']);
`;
@@ -97,19 +112,28 @@ function ActorExample() {
return (
<section className={clsx(styles.try, 'container')}>
<div className="col">
- <h2>Try it out</h2>
- <p>Install Crawlee into a Node.js project. You must have Node.js 16 or higher installed.</p>
+ <h2>Try Crawlee out 👾</h2>
+ <Admonition type="caution" title="before you start">
+ Crawlee requires <a href="https://nodejs.org/en/" target="_blank" rel="noreferrer"><b>Node.js 16 or higher</b></a>.
+ </Admonition>
+ <p>
+ The fastest way to try Crawlee out is to use the <b>Crawlee CLI</b> and choose the <b>Getting started example</b>.
+ The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
+ </p>
+ <CodeBlock className="language-bash">
+ npx crawlee create my-crawler
+ </CodeBlock>
+ <p>
+ If you prefer adding Crawlee <b>into your own project</b>, try the example below.
+ Because it uses <code>PlaywrightCrawler</code> we also need to install Playwright.
+ It's not bundled with Crawlee to reduce install size.
+ </p>
<CodeBlock className="language-bash">
npm install crawlee playwright
</CodeBlock>
- <p>Copy the following code into a file in the project, for example <code>main.mjs</code>:</p>
<CodeBlock className="language-typescript">
{example}
</CodeBlock>
- <p>Execute the following command in the project's folder and watch it recursively crawl the Crawlee website with Playwright and Chromium.</p>
- <CodeBlock className="language-bash">
- node main.mjs
- </CodeBlock>
</div>
</section>
);
@@ -123,7 +147,7 @@ export default function Home() {
title={`${siteConfig.title} · ${siteConfig.tagline}`}
description={siteConfig.description}>
<Hero />
- <Hightlights />
+ <Highlights />
<Features />
<ActorExample />
<div className="container">
diff --git a/website/static/img/chrome_scrape.gif b/website/static/img/chrome_scrape.gif
index 83de50c8f..42a8c272f 100644
Binary files a/website/static/img/chrome_scrape.gif and b/website/static/img/chrome_scrape.gif differ
diff --git a/website/static/img/chrome_scrape_old.gif b/website/static/img/chrome_scrape_old.gif
deleted file mode 100644
index 08a455e70..000000000
Binary files a/website/static/img/chrome_scrape_old.gif and /dev/null differ
diff --git a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
index 7cc45467d..9aee7e42e 100644
--- a/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
+++ b/website/versioned_docs/version-3.0/introduction/01-setting-up.mdx
@@ -9,9 +9,9 @@ import ApiLink from '@site/src/components/ApiLink';
To run Crawlee on your own computer, you need to meet the following pre-requisites first:
1. Have **Node.js version 16.0** or higher installed.
- - Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
+- Visit [Node.js website](https://nodejs.org/en/download/) to download or use [fnm](https://github.com/Schniz/fnm)
2. Have **NPM** installed, or use other package manager of your choice.
- - NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
+- NPM comes bundled with Node.js, so you should already have it. If not, reinstall Node.js.
If not certain, confirm the prerequisites by running:
@@ -25,39 +25,54 @@ npm -v
## Creating a new project
-The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is also embedded in the `crawlee` package:
+The fastest and best way to create new projects with Crawlee is to use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli). You can use the `npx` utility to download and run the CLI - it is embedded in the `crawlee` package:
```bash
-npx crawlee create my-new-project
+npx crawlee create my-crawler
```
A prompt will be shown, asking you to choose a template. Crawlee is written in TypeScript so if you're familiar with it, choosing a TypeScript template will give you better code completion and static type checking, but feel free to use JavaScript as well. Functionally they're identical.
-Let's choose the first template called **Crawlee playwright template**. The command will create a new directory in your current working directory, called **my-new-project**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
+Let's choose the first template called **Getting started example**. The command will create a new directory in your current working directory, called **my-crawler**, add a **package.json** to this folder and install all the necessary dependencies. It will also add example source code that you can immediately run.
Let's try that!
```bash
-cd my-new-project
+cd my-crawler
npm start
```
-You will see log messages in the terminal as Crawlee boots up and after a second a Chromium browser window will open. In the window, you'll see quickly changing pages and back in the terminal, you will see the printed titles (contents of the `<title>` HTML tags) of the pages.
+You will see log messages in the terminal as Crawlee boots up and starts scraping the Crawlee website.
-![Chrome Scrape](/img/chrome_scrape.gif)
+```log
+INFO PlaywrightCrawler: Starting the crawl
+INFO PlaywrightCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core is '@crawlee/core | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/api/core/changelog is 'Changelog | API | Crawlee'
+INFO PlaywrightCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+```
-:::info
+You can always terminate the crawl with a keypress in the terminal:
-We picked the Playwright template, which uses Chromium to open pages. If you pick the Cheerio template instead, there won't be any browser window, as the requests to the target site will be done via a specialized HTTP client: [`got-scraping`](https://github.com/apify/got-scraping), instead of a browser.
+```text
+CTRL+C
+```
-:::
+### Running headful browsers
-You can always terminate the crawl with a keypress in the terminal:
+Browsers controlled by Playwright run headless (without a visible window). You can switch to headful by uncommenting the `headless: false` option in the crawler's constructor. This is useful in the development phase when you want to see what's going on in the browser.
-```bash
-CTRL+C
+
+```typescript
+// Uncomment this option to see the browser window.
+headless: false
```
+When you run the example again, after a second a Chromium browser window will open. In the window, you'll see quickly changing pages as the crawler does its job.
+
+![Chrome Scrape](/img/chrome_scrape.gif)
+
## Next lesson
The next lesson will teach you how to create a very simple crawler and explain Crawlee components while building it.
diff --git a/website/versioned_docs/version-3.0/introduction/index.mdx b/website/versioned_docs/version-3.0/introduction/index.mdx
index a6895d500..b64688861 100644
--- a/website/versioned_docs/version-3.0/introduction/index.mdx
+++ b/website/versioned_docs/version-3.0/introduction/index.mdx
@@ -12,7 +12,7 @@ Your crawlers will appear human-like and fly under the radar of modern bot prote
## What you will learn
-The goal of the introduction is to provide a step-by-step guide to all the features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
+The goal of the introduction is to provide a step-by-step guide to the most important features of Crawlee. It will walk you through creating the simplest of crawlers that only prints text to console, all the way up to a full-featured scraper that collects links from a website and extracts data.
## 🛠 Features
@@ -32,7 +32,7 @@ The goal of the introduction is to provide a step-by-step guide to all the featu
- Zero config **HTTP2 support**, even for proxies
- Automatic generation of **browser-like headers**
- Replication of browser **TLS fingerprints**
-- Integrated fast **HTML parser**
+- Integrated fast **HTML parsers**. Cheerio and JSDOM
- Yes, you can scrape **JSON APIs** as well
### 💻 Real browser crawling
diff --git a/website/versioned_docs/version-3.0/quick-start/headful_playwright.ts b/website/versioned_docs/version-3.0/quick-start/headful_playwright.ts
index 149482390..378b8d4f5 100644
--- a/website/versioned_docs/version-3.0/quick-start/headful_playwright.ts
+++ b/website/versioned_docs/version-3.0/quick-start/headful_playwright.ts
@@ -1,16 +1,16 @@
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
- // When you turn off headless mode, the crawler
- // will run with a visible browser window.
- headless: false,
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
- await enqueueLinks({ strategy: 'same-domain' });
- await Dataset.pushData({ url, title });
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
},
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
});
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.0/quick-start/headful_puppeteer.ts b/website/versioned_docs/version-3.0/quick-start/headful_puppeteer.ts
index c8e44fcbd..37c0f2b00 100644
--- a/website/versioned_docs/version-3.0/quick-start/headful_puppeteer.ts
+++ b/website/versioned_docs/version-3.0/quick-start/headful_puppeteer.ts
@@ -1,16 +1,16 @@
import { PuppeteerCrawler, Dataset } from 'crawlee';
const crawler = new PuppeteerCrawler({
- // When you turn off headless mode, the crawler
- // will run with a visible browser window.
- headless: false,
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
- await enqueueLinks({ strategy: 'same-domain' });
- await Dataset.pushData({ url, title });
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
+ await Dataset.pushData({ title, url: request.loadedUrl });
+ await enqueueLinks();
},
+ // When you turn off headless mode, the crawler
+ // will run with a visible browser window.
+ headless: false,
});
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.0/quick-start/index.mdx b/website/versioned_docs/version-3.0/quick-start/index.mdx
index 59e1e6f4f..1acc52bca 100644
--- a/website/versioned_docs/version-3.0/quick-start/index.mdx
+++ b/website/versioned_docs/version-3.0/quick-start/index.mdx
@@ -1,11 +1,12 @@
---
id: quick-start
title: Quick Start
-description: Quick and easy steps to get started using Crawlee today!
+description: With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.
---
import ApiLink from '@site/src/components/ApiLink';
+import Admonition from '@theme/Admonition';
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
@@ -33,10 +34,30 @@ This crawler uses a headless browser to crawl, controlled by the [Puppeteer](htt
### PlaywrightCrawler
[Playwright](https://github.com/microsoft/playwright) is a more powerful and full-featured successor to Puppeteer. It can control Chromium, Chrome, Firefox, Webkit and many other browsers. If you're not familiar with Puppeteer already, and you need a headless browser, go with Playwright.
-## Installation
+:::caution before you start
-Crawlee requires [Node.js](https://nodejs.org/en/) 16 or later.
-It can be added to any Node.js project by running:
+Crawlee requires [Node.js 16 or later](https://nodejs.org/en/).
+
+:::
+
+## Installation with Crawlee CLI
+
+The fastest way to try Crawlee out is to use the **Crawlee CLI** and choose the **Getting started example**.
+The CLI will install all the necessary dependencies and add boilerplate code for you to play with.
+
+```bash
+npx crawlee create my-crawler
+```
+
+After the installation is complete you can start the crawler like this:
+
+```bash
+cd my-crawler && npm start
+```
+
+## Manual installation
+
+You can add Crawlee to any Node.js project by running:
<Tabs groupId="quick_start">
<TabItem value="cheerio" label="CheerioCrawler" default>
@@ -66,17 +87,24 @@ It can be added to any Node.js project by running:
## Crawling
-Run the following example to perform a recursive crawl of the Crawlee website using:
+Run the following example to perform a recursive crawl of the Crawlee website using the selected crawler.
+
+<Admonition type="caution" title="Don't forget about module imports">
+ To run the example, add a <code>"type": "module"</code> clause into your <code>package.json</code> or
+ copy it into a file with an <code>.mjs</code> suffix. This enables <code>import</code> statements in Node.js.
+ See <a href="https://nodejs.org/dist/latest-v16.x/docs/api/esm.html#enabling" target="_blank" rel="noreferrer">Node.js docs</a> for
+ more information.
+</Admonition>
<Tabs groupId="quick_start">
<TabItem value="cheerio" label="CheerioCrawler" default>
- <CodeBlock language="js" title="src/main.mjs">{CheerioSource}</CodeBlock>
+ <CodeBlock language="js">{CheerioSource}</CodeBlock>
</TabItem>
<TabItem value="playwright" label="PlaywrightCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PlaywrightSource}</CodeBlock>
+ <CodeBlock language="js">{PlaywrightSource}</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PuppeteerSource}</CodeBlock>
+ <CodeBlock language="js">{PuppeteerSource}</CodeBlock>
</TabItem>
</Tabs>
@@ -91,10 +119,10 @@ Browsers controlled by Puppeteer and Playwright run headless (without a visible
<Tabs groupId="quick_start">
<TabItem value="playwright" label="PlaywrightCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PlaywrightHeadful}</CodeBlock>
+ <CodeBlock language="js">{PlaywrightHeadful}</CodeBlock>
</TabItem>
<TabItem value="puppeteer" label="PuppeteerCrawler">
- <CodeBlock language="js" title="src/main.mjs">{PuppeteerHeadful}</CodeBlock>
+ <CodeBlock language="js">{PuppeteerHeadful}</CodeBlock>
</TabItem>
</Tabs>
@@ -127,18 +155,3 @@ You can find more examples showcasing various features of Crawlee in the [Exampl
- [Configuration](./guides/configuration)
- [Request storage](./guides/request-storage)
- [Result storage](./guides/result-storage)
-
-## Local usage with Crawlee command-line interface (CLI)
-
-To create a boilerplate of your project, you can use the [Crawlee CLI](https://www.npmjs.com/package/@crawlee/cli) tool by running:
-
-```bash
-npx crawlee create my-cheerio-crawler
-```
-
-The CLI will prompt you to select a project boilerplate template - let's pick the **Crawlee cheerio template [TypeScript]**. The tool will create a directory called `my-cheerio-crawler` with Node.js project files. You can run the project as follows:
-
-```bash
-cd my-cheerio-crawler
-npm start
-```
diff --git a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.ts b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.ts
index 038aa3b03..0dff35063 100644
--- a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.ts
+++ b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.ts
@@ -1,21 +1,21 @@
import { CheerioCrawler, Dataset } from 'crawlee';
+// CheerioCrawler crawls the web using HTTP requests
+// and parses HTML using the Cheerio library.
const crawler = new CheerioCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, $, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = $('title').text();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt
index fb7c410e3..e68fa4133 100644
--- a/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt
+++ b/website/versioned_docs/version-3.0/quick-start/quick_start_cheerio.txt
@@ -1,6 +1,5 @@
-INFO CheerioCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.7,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":null}}}
-INFO CheerioCrawler: Title of https://crawlee.dev/: Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples: Examples | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start: Quick Start | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/api/core: @crawlee/core | API | Crawlee
-INFO CheerioCrawler: Title of https://crawlee.dev/api/core/changelog: Changelog | API | Crawlee
+INFO CheerioCrawler: Starting the crawl
+INFO CheerioCrawler: Title of https://crawlee.dev/ is 'Crawlee · Build reliable crawlers. Fast. | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/examples is 'Examples | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/quick-start is 'Quick Start | Crawlee'
+INFO CheerioCrawler: Title of https://crawlee.dev/docs/guides is 'Guides | Crawlee'
diff --git a/website/versioned_docs/version-3.0/quick-start/quick_start_playwright.ts b/website/versioned_docs/version-3.0/quick-start/quick_start_playwright.ts
index d7d12c27b..9d0f0e738 100644
--- a/website/versioned_docs/version-3.0/quick-start/quick_start_playwright.ts
+++ b/website/versioned_docs/version-3.0/quick-start/quick_start_playwright.ts
@@ -1,21 +1,23 @@
import { PlaywrightCrawler, Dataset } from 'crawlee';
+// PlaywrightCrawler crawls the web using a headless
+// browser controlled by the Playwright library.
const crawler = new PlaywrightCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
+ // Uncomment this option to see the browser window.
+ // headless: false,
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);
diff --git a/website/versioned_docs/version-3.0/quick-start/quick_start_puppeteer.ts b/website/versioned_docs/version-3.0/quick-start/quick_start_puppeteer.ts
index 001254010..6cb7dd6d5 100644
--- a/website/versioned_docs/version-3.0/quick-start/quick_start_puppeteer.ts
+++ b/website/versioned_docs/version-3.0/quick-start/quick_start_puppeteer.ts
@@ -1,21 +1,23 @@
import { PuppeteerCrawler, Dataset } from 'crawlee';
+// PuppeteerCrawler crawls the web using a headless
+// browser controlled by the Puppeteer library.
const crawler = new PuppeteerCrawler({
+ // Use the requestHandler to process each of the crawled pages.
async requestHandler({ request, page, enqueueLinks, log }) {
- const { url } = request;
-
- // Extract HTML title of the page.
const title = await page.title();
- log.info(`Title of ${url}: ${title}`);
+ log.info(`Title of ${request.loadedUrl} is '${title}'`);
- // Add links from the page that point
- // to the same domain as the original request.
- await enqueueLinks({ strategy: 'same-domain' });
+ // Save results as JSON to ./storage/datasets/default
+ await Dataset.pushData({ title, url: request.loadedUrl });
- // Save extracted data to storage.
- await Dataset.pushData({ url, title });
+ // Extract links from the current page
+ // and add them to the crawling queue.
+ await enqueueLinks();
},
+ // Uncomment this option to see the browser window.
+ // headless: false,
});
-// Add a start URL to the queue and run the crawler.
-await crawler.run(['https://crawlee.dev/']);
+// Add first URL to the queue and start the crawl.
+await crawler.run(['https://crawlee.dev']);